Our last team meeting for 2016 from the 24th to the 25th November took place at Cologne. Here in holy city of Cologne is the North Rhine-Westphalian Library Centre (hbz) where four people of the OER World Map team are located. So the other members of our geographically dispersed team had to travel this time. The two days were filled with constructive discussions and effective teamwork. Continue reading
It has been a while since the last technical progress report has been published. Luckily, this is not because nothing has happened since, but rather because we were busy building things. This blog post briefly summarizes the most important of those things.
The most prominent changes are naturally related to the user interface. The layout is now based on three interrelated columns; one for the map, one for search / filter result listings and one for individual entries. On top of that, additional information such as a feed of recent additions and statistics are available in a popup window. Less visible, but an enhancement nevertheless, is the fact that navigating the map no longer requires full page loads and is thus much smoother.
The templates that editors of the OER World Map use to input data have also been slighlty reworked. In order to reduce the number of fields that are presented to the users, the inverses of fields for links that have a more or less natural direction have been hidden. In order to clarify the semantics of data elements, descriptions are available via tooltips. The most significant simplification probably is that Markdown is now supported for fields that hold running text.
Finally, first elements of an administrative interface have been implemented. Among these are the administration of roles for registered users, a UI for data migrations and a precise log of all changes that have been made to the database.
During phase II, the emphasis was to win editors for the OER World Map by individually inviting them to collaborate. An important step towards growing a bigger community of OER World Map users now is the possibility for anyone to register a user account. Once registered, it is possible to create a personal profile and thus represent oneself on the map. Also, the possibility to comment on entries is only available to registered users. You are very welcome to participate in editing data beyond that; get in touch if you are interested!
From a technical point of view, we have switched to what can be described as a perimeter security model. User authentification and authorization is now done by an Apache reverse proxy before a request even hits the OER World Map web application. On the one hand, this separation of concerns brings a performance gain. On the other hand, it would be hard to compete with Apache’s battle proven security anyways.
While invisible to most users, there have been very important improvements in the archticture of the back end of the system. While using an Elasticsearch index as our main data sink allowed us to quickly grow the system during phase II, some of the limitations of that approach became evident once more editorial activity was recorded.
On the one hand, data needs to be denormalized quite heavily to fully embrace the features of a document oriented system such as Elasticsearch, especially when it comes to aggregations. On the other hand, the data in the OER Data Hub is highly interlinked. This combination makes write-operations quite expensive because a single update operation often modifies multiple JSON documents in the index. A successful write operation could only be assumed once the data trickled into all places it was supposed to be, which made waiting times unacceptable.
To complement the extremely fast read operations that Elasticsearch provides with equally fast write operations, a special type of relational database, a triple store, was added to the technology stack. It is now our primary data store and the single source of truth in the system which asynchronously feeds the Elasticsearch index after write operations.
Another precondition to gradually open the platform to a bigger circle of editors has been data versioning. In order to ensure data quality, it must be possible to retrace the evolution of the dataset. In other words, it is necessary to completely understand who changed which parts of the data, and when the changes happened. Naturally being familiar with the way source code is versioned, we adopted the structure of Git commits to the RDF data in our triple store:
Author: felix.ostrowski@XYZ.com Date: 2016-07-01T15:55:51.012+02:00 + <urn:uuid:123> <http://schema.org/name> "Felix Ostrowski" . - <urn:uuid:123> <http://schema.org/name> "Felix Ostrowsko" . Author: felix.ostrowski@XYZ.com Date: 2016-06-29T18:01:40.587+02:00 + <urn:uuid:123> <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://schema.org/Person> . + <urn:uuid:123> <http://schema.org/name> "Felix Ostrowsko" .
A nice side effect of all our data being a series of commits such as the ones above, is that the back-up strategy is simply a matter of saving plain text commit files. These are the only precondition to completely recover our data set after potential failures.
On the data model side of things, a tag field has been added to all resources. This, along with the corresponding filter, allows editors to create arbitrary custom subsets of the data. With regards to the
Service type, a
license field is now exposed, along with a controlled vocabulary of licenses and the corresponding filter. Finally, the
funder property is now available for Projects.
We now have our first video content available through OER World Map! You can review the entries at the following URLs:
Open Educational Resources in Africa
NOVA’s OER-Based Associate Degree Project
We hope to add more stories in this way over the coming months. If you have a video relating to OER that you would like to share with the community then get in touch.
Content published on YouTube with a licence that permits sharing can easily be repurposed for the map in this way using the embed code provided in the sharing menu.
Just before the end of a busy year, this post briefly summarizes the last steps takes towards the release of v1.0 of the OER World Map, the final one for phase II of the project.
The basic input templates have been refined. Most importantly, look-up functionalities for different value types are in place, which significantly enhances the usability. Controlled vocabularies such as the subject classification are presented in their hierarchies. The target audience(s) of a service can be entered in a similar way.
Linking entries with one another is another major use-case, consider e.g. adding a contact person to an organization. In the same way, services can be linked to their providers, projects with their participants etc pp. This lets the OER World Map grow into a highly interlinked database that allows to reveal many explicit and implicit connections.
The language of text values can easily be set and currently defaults to English. While not it is not exposed yet, we do support gathering multilingual data.
Events have been added to the data model and the input templates. An enhanced location-lookup widget is still in the making.
Query results can now also be downloaded as CSV to be further processed as a spreadsheet. The UI is currently not exposing this functionality yet, nevertheless it is available using content negotiation.
$ curl -H "Accept: text/csv" "https://beta.oerworldmap.org/resource/?filter.about.location.address.addressCountry=DE&filter.about.@type=Organization"
JSON can be be obtained in the same way, using the corresponding header “Accept: application/json”. Expect the different representations to be available from the UI early 2016.
The map of individual countries can now be embedded into external website with ease. A precondition for this is that iframe-tags are supported by the target platform:
This blog for example is hosted on wordpress.com which does not allow iframes. Being hosted, we cannot add the plugin that would allow us to do so. See this post on blogger.com for an example:
We look forward to working on phase III of the OER World Map project. Refinements of all major functionalities are planned for 2016, see the proposal for more information. Happy Holidays!
This post summarizes the progress made since v0.2 and provides an outlook to version 1.0 which will be launched by the end of the year. Feedback is greatly appreciated either in the comments or by mail.
The front page now features statistics on the distribution of subjects, languages and grade levels of services on the OER World Map. They are backed by Elasticsearch aggregations which have proven to be an excellent basis. For the visual representations of the statistics, currently only tables and pie charts are used.
More statistics and alternate visualizations are planned for the next release. Once the openness indicator is in place, statistics on the openness of services will be implemented. The vocabulary for OER activity fields will be the basis to summarize projects based on those categories. Where applicable, the visualizations will include bar charts and scattergrams.
Complementing the statistics, a filter section has been implemented in the resource listing. Currently filters for type, language, country, subject and intended audience (i.e. grade level) of entries are available. These are again based on Elasticsearch aggregations.
The exact behavior, i.e. possible combinations and operators, of the filters is yet to be defined. An advanced user interface is planned for the next release.
Authentication & authorization
The components for authentication are in place. The OER World Map is using the users’ email accounts as identity provider for authentication. Upon request, a one time password is created and sent to the user. Using his email address and this password, users can log in until a timeout occurs or they explicitly log out. This ensures that no weak passwords can be used and frees us of the burdens of password reminders and the like. We consider email accounts as safe; if a users email is hacked, the trouble is big anyways as email is usually used for password-resets of online accounts.
While authentication verifies who a user is, authorization verifies what she is allowed to do in the system. Components necessary for advanced, activity based authorization are ready to be deployed. Thus in theory, general users could be granted write access. To ensure the well functioning of the system and more importantly data quality, the decision was made to open the platform incrementally, with write access for country champions only for now. This means that for the time being, there are three roles users can take in the OER World Map: visitors, editors (country champions) and administrators.
The most important feature that makes authentication and authorization necessary is data manipulation. While this has been possible using the JSON API for quite a while now, work on the user interface to do so from within a browser has progressed. We still following the idea of a progressively enhanced hypermedia application.
This is why the first iteration of the UI is still rather cumbersome: pure HTML forms act as hypermedia controls. They are in place for all resource types. Before opening write access to the public, the current forms will be progressively enhanced to increase usability. The design for all necessary widgets is done and implementation of those designs is currently in progress.
Last but not least, new data has been imported into the OER World Map. To support the use cases of filtering services by subject and grade level, two controlled vocabularies have been integrated. The educational subjects classification extends the ISCED-2013 classification, so far mostly by adding concrete languages to the concept of language acquisition. ISCED-1997 is used to classify services by their intended audiences, i.e. targeted grade levels. The integration of these classifications allowed us to import the list of OER repositories, created by Javiera Atenas and Leo Havemann, and index it by subject and grade level.
Additionally, the OEC member list was added, resulting in many new entries and thus the need for clustering pins on the map. Finally, the list of country champions is constantly updated as our network grows.
As we move toward the release of version 1.0 of the OER World Map, we will first focus on the usability of the editing templates. In this context there will also be a general revision of UI, including context-sensitive help. After that, the map will be extended with a calendar view for event data, a feature to export search results as CSV and finally the possibility to embed custom views of the OER World Map into external platforms such as blogs.
Shortly before the launch of version 0.2 of The OER World Map, this report gives a brief overview of the development that has happened since the release of version 0.1 as well as an outlook on the next steps.
Gathering and ingesting data
Following an unconference session at The William and Flora Hewlett Foundation’s OER Grantees Meeting 2015, the use case of collecting and presenting stories about how OER are created and used was put at the center of our attention. In order to not divert too far from our core project goal (gathering and exposing structured data on the global OER movement), the stories that were sent in after our initial Call for OER Stories were postprocessed by an editorial team. This means that those entities that are mentioned in a story have been extracted and added as self-contained entries. An example for the result of the postprocessing is the story “Darakht-e Danesh Library Brings Knowledge to Afghan Women”, from which the description of The Darakht-e Danesh Library is derived. The collection and processing of OER stories is an ongoing effort, the list of stories will grow over time.
A consequence of this approach was to implement basic support for all main types of our data model (organizations, people, services, projects and events) in the back end. It should be noted that this enables us to also ingest data independently of stories, the OEC membership list for example is ready to be put on the map. We believe that the approach of contextualizing data through editorial stories leads to higher quality, though.
In order to present stories and related entities on the geographical world map, the rough multi-column sketch has been further developed into a wireframe. While not all details have been implemented yet, the list of stories is available on the entry page and there are views not only for individual stories but also for single entities such as Livro Didático Público. Additionally, a view of a country with its country champions, basic statistics and associated entities is available.
In order to find relevant content on The OER World Map, a basic search is available. This allows for simple full text search as well as field-based filtering, e.g. by type. The latter will be exposed separately in the user infterface in the next release.
For the tech-savvy, data can not only be ingested but is now also available as JSON using Content negotiation by providing the correct HTTP Accept Header:
$ curl -H "Accept: application/json" https://beta.oerworldmap.org/country/gb
Speaking of Content negotiation and given the international character of the platform, multilinguality is also prepared for on the data level. While all currently available stories are provided in English only, our JSON schema allows to store multilingual values where it makes sense.
Besides gathering, processing and adding more stories, we plan to ingest an initial dataset of OER services (in this case repositories) next. Once this is done, we would like to qualify that dataset, most importantly by adding the targeted levels and fields of education as well as languages and licences of the supplied OERs. In order to do so, the necessary front end components to edit such a dataset will be developed. Since we believe that it is more sensible to locate a repository by its content rather than its physical location, filters to browse repositories by grade level, field of education and language will be added to the UI. Exposing those traits as reports in form of charts rounds of the planning.
General software development process
All prerequisits for the development process and quality control have been taken. The code is available openly on github. User Stories and other issues are handled as issues, which are managed through a Kanban board. The inital backlog was filled in a two-day user story workshop. To ensure code quality, Travis CI is used for continous integration.
Development of the landing page
Initially, it was planned to very quickly release a landing page with basic project information and the possibility to register for the project newsletter. Instead of only describing what is planned though, the idea to actually build a first minimal yet functional version demonstrating the way that the OER World Map can provide an overview of the global OER ecosystem arose. The idea was to implement a heat map displaying an aggregation of users by country. Thus,
- getting basic project information,
- signing up for a newsletter,
- and a first implementation of an actual OER World Map
were the core features to realize.
Given the international character of the platform, multilinguality has been prepared for from the beginning. While only an English interface is provided for the time being, more languages can easily be added by creating text files in a very simple format.
Given these features, the designer and the front-end development team sat together to draft a first concept for the user interface. This process resulted in a basic page layout and color scheme as can be seen below.
In order to implement the functionalities and the design, technological choices had to be made. These choices were guided by the standpoint that at least a very basic version of the platform should be accessible by low-end clients such as old browsers, screen readers and search engine spiders. Instead of building a hi-fi version first and then making big efforts to make things work under less than ideal circumstances, it was decided to embrace the philosophy of progressive enhancement:
[G]raceful degradation starts from the status quo of complexity and tries to fix for the lesser experience whereas progressive enhancement starts from a very basic, working example and allows for constant extension for future environments. Degrading gracefully means looking back whereas enhancing progressively means looking forward whilst keeping your feet on firm ground. http://www.w3.org/wiki/Graceful_degradation_versus_progressive_enhancement
This meant to divert partly from the technology stack that was originally envisioned; most notably it was decided to not use AngularJS but relying on classic server-side rendering. Bootstrap is being used as a CSS framework as it eases the implementation of responsive design.
The geographical map itself is based on jVectorMap, which allowed very quick implementation of a global scale map. Several shortcomings – most significantly the fact that it is based on a custom data model instead of wide spread standards such as GeoJSON – make it very likely to be replaced by a D3 based solution in the course of the project, as was originally planned.
On the back-end side, the original technology stack that was proposed remains mostly unchanged. Data is indexed in Elasticsearch so that it can easily be exposed in two ways that are most important: as a list of search hits and as aggregations that can be displayed in various forms of visualizations. The data model is expressed as a JSON-Schema that is also used to validate instance data. Currently all elements of the data model stem from the widespread and well adopted schema.org vocabulary. Countries are referenced by their ISO 3166 alpha-2 country-codes which is also well accepted, widespread and integrates well with schema.org.
The actual newsletter is handled outside of the OER World Map platform by GNU Mailman, a widespread and well tested system for handling mailing lists. Comitted to privacy by design, this makes it possibly that the database driving the OER World Map itself currently does not hold clear text email adresses.
The main next feature to be implemented is the possibility to describe organizations producing or using OER or otherwise engaging in the community. This has several implications. Most importantly,
- authorization is needed to prevent spam,
- an editing interface has to be implemented to easily edit data,
- and an advanced layout is necessary that provides the means for more complex user interaction.
Work on all three of these has already begun. At this stage, authorization will be provided by making it possible for people that have signed up for the newsletter to request one-time-passwords that are delivered to their email address and are valid until logging out. This allows for Basic access authentication (over SSL, which the entire OER World Map uses) without the need to set up actual user accounts.
For the editing interface, several solutions to provide user-friendly geocoding are being evaluated. Among the most promising are solutions that are based on OpenStreetMap data and Elasticsearch, such as Photon and Pelias.
The layout of the OER World Map beyond the current landing page is based on responsive web design. Depending on the device being used, one to three columns are displayed at once. The first one presents the actual world map. The second one contains aggregations resp search results while the third is reserved for views of individual items. This approach keeps it open to add additional columns or a vertical dimension to individual columns so that future extensions can easily be incorporated.
If you would like to know more details or have additional questions, you are very welcome to post a comment.