Easy use and creation of vocabularies: the LoCloud vocabulary microservice

Gerda and Walter Koch of AIT Applied Information Technology Graz, Austria write:

LoCloud provides support services and tools to regional and local heritage institutions. These tools and services simplify ingestion of data into Europeana and help to improve metadata quality with the use of cloud-based services. One of the LoCloud services developed during the second project year is the vocabulary microservice.

Microservices? An introduction

The past years have seen a move from monolithic applications towards microservice architectures. Microservice architectures are generally described as suites of small and independent services compiled into single applications, in opposite to monolithic applications that are built as single units, with a client-side user interface, a database, and a server-side application. While any changes to monolithic applications require the re-building and deployment of the entire applications, in microservice architectures it is quite easy to introduce new versions of the individual services or integrate new services. This will become even more important in the future, as applications will increasingly be deployed to the cloud.

Monolithic versus microservices architecture

Figure 1 – Monolithic versus microservices architecture

Microservices are independently deployable and scalable and they could even be written in different programming languages by different software teams. Usually. the services are built upon business capabilities and communicate via web service requests or remote procedure calls.

The LoCloud microservices have been developed by different project partner teams and are being deployed in the LoCloud MORe [1] aggregator. Additionally, some of the services provide front-end applications. The LoCloud vocabulary microservice provides the business capability for vocabulary management and enrichment of metadata.

Vocabulary Microservice

Vocabulary services support the enrichment of metadata (catalogue data) by adding vocabulary terms to metadata records. This can be done right “at the beginning” when the object is registered in the local cataloguing system. Alternatively, the data can be enriched by an automated service “after” cataloguing has taken place. The second possibility often prevails when data is ingested into joint virtual catalogues (like Europeana) where common vocabularies provide a means to semantically link data and to support easy browsing through the entire repository.

The requirement for the LoCloud vocabulary service team was to implement a cloud-based vocabulary web service and vocabulary application for the LoCloud network. The vocabulary application should support the development of multilingual, semantic thesauri for local heritage content and the vocabulary webservice had to be based on international standards such as SKOS [2] and the ISO thesaurus norms [3].
Four main use cases for the vocabulary services exist:
Three uses cases for vocabulary provision: Use the vocabulary microservice…
a)            …in the various enrichment workflows automatically through the generic enrichment service
b)           …through the Aggregator User interface by choice
c)            …in local cataloguing systems via web services.
One use case for vocabulary creation including import of existing vocabularies: Use the service
d)           …with a cloud based online tool.
There was a six-month implementation period and a three-month testing period for the service scheduled in the LoCloud workplan. A systematic research of already existing applications revealed that the open source tool TemaTres (http://sourceforge.net/projects/tematres/) was the best starting point for a rapid development and implementation of such a service.
TemaTres [4] supports the handling of vocabularies in accordance with the ISO standard thesaurus norms. It allows for import and export of data as simple text files or in SKOS format. The tool was installed in the cloud testlab of LoCloud and then adapted to the project’s needs. These needs concerned mainly the implementation of a simplified administration and import facility. Additionally, we implemented a note extension that allows storing coordinates for place names. Two new web service [5] calls “import” and “linkTerm” were added to the extensive list of already available TemaTres web services. For the import process of multilingual vocabularies a new identifier for concepts was introduced, connecting automatically the terms in the various languages. This identifier also carries the link information of the publicly published vocabulary and is later added to the metadata during the enrichment process.
How the LoCloud vocabulary services may be used.
Use Case 1 – Automatic usage through the generic enrichment service.

The LoCloud microservice “Generic enrichment” automatically receives the vocabularies available in the vocabulary tool and uses them during the automated enrichment process conducted in the MORe aggregator.

To date around 30 vocabularies are imported to the vocabulary tool. A list of these vocabularies can be found here  > LoCloud Vocabularies.

Figures 2 and 3 show how the microservices Vocabulary and Vocabulary Matching (available currently for  English and Spanish language) are added to a data source enrichment plan in the the MORe aggregator.



Figure 4 – Use Case 1 – Automatic usage through the generic enrichment service

Consequently, the EDM metadata is checked against the LoCloud vocabularies and where appropriate the vocabulary links are added.

Use Case 2 – Match vocabulary terms in the aggregator.

Figure 5 – Selection of individual vocabulary terms for a Subject Collection in MORe

Instead of matching the vocabularies automatically to the metadata it is possible to selected individual terms from the LoCloud vocabularies that should be added to the metadata. These selections are called Subject Collections in MORe.

Use case 3 – Use the vocabularies in your local cataloguing systems via web services

Figure 6 – Integration of a vocabulary to a local cataloguing tool via webservice

One of the benefits of microservices is that these small and independent services may be plugged in into other application and can be used via the APIs [6] they provide. Figure 6 depicts an example of a local cataloguing system for music archiving that calls the vocabulary web service in the metadata field “Genre(s)”. The technical information on how the services can be integrated is published on the LoCloud online support centre. There are currently fifteen different web service calls for vocabulary integration available.

Use case 4 – The vocabulary experimental application.

The LoCloud vocabularies have been imported into the Tematres tool in order to use them via web services in the aggregation process. But

Figure 7 – LoCloud Vocabularies, Website

Tematres can also be used to create vocabularies from the scratch or to import already existing vocabularies. The advantage of importing vocabularies to the tool is clear: The vocabularies become available in SKOS format with an own web presence and can be used for semantic linking afterwards. With the consent of the vocabulary owner, the vocabulary might then become available for other LoCloud partners in the MORe aggregator. Moreover, the LoCloud Tematres installation allows collaborating online in creating and extending vocabularies (for example in order to create new translations of existing vocabularies).

The vocabulary application is accessible through this link (see figure 7). Please check the LoCloud support portal for information on a test user account which is necessary to access and create a new vocabulary.



[1]https://support.locloud.eu/MORE 26 January, 2015

[2]http://www.w3.org/2004/02/skos/ 26 January, 2015

[3]http://www.niso.org/schemas/iso25964/ 26 January, 2015

[4]http://www.vocabularyserver.com/ 26 January, 2015

[5]http://en.wikipedia.org/wiki/Web_service 26 January, 2015

[6]http://en.wikipedia.org/wiki/Application_programming_interface 26 January, 2015

Hackathon winner!

MoRe Quality by Vangelis Banos (Future LIbrary – Greece) is the winning prototype application of the LoCloud hackathon which took place on the 11th of February 2015 at the premises of the Google Cultural Institute in Paris, France.The hackathon was organised by LoCloud and Europeana in the context of EuropeanaTech 2015.

Image of Vangelis Banos at the Google culture institute
Vangelis Banos winner of the Locloud hackathon
Vangelis Banos, winner of the LoCloud Hackathon


The Metadata & Object Repository (MoRe) is an easy and powerful tool to aggregate information and harvest metadata from multiple sources in multiple schemas. Such aggregation schemas usually create problematic situations regarding the quality of the harvested metadata.

Metadata may pass the standard Europeana XML validity tests but they may include problematic metadata values. For instance:

* a dc:date value could be formatted in the wrong way:

<dc:date>approximately 18th century</dc:date>

This format is not correct according to established date formats.

*  an author name could be incomplete according to bibliographic standards.

Example: <dc:creator>Mike</dc:creator>.

* a URL may be invalid. E.g.: <ese:isShownAt>http://invalidurl.com/error-url</ese:isShownAt>

The aim of the MoRe Quality tool is to implement a validation system which could be able to catch these errors and produce useful reports to the collection administrators.

MoRe Quality application –The prototype functionality

MoRe Quality communicates with the LoCloud MoRe instance using a specific user API Key, retrieves the ingested metadata and performs evaluations to identify common errors such as:

* Invalid date formats (ISO 8601 standard)

* Invalid hyperlinks

* Invalid language codes (ISO 639 standard)

* Invalid author names

The results are presented to the user in a simple report.

The application is implemented in such a way that enables developers to add extra evaluation rules in an easy and intuitive way by implementing simple functions – plugins.

Technical information

MoRe Quality is implemented using linux and python 2.7.

Some common python modules are utilised:

* Virtual environments

* Flask

* Python Requests

* BeautifulSoup4

* pycountry

* iso8601

The prototype is not currently running on a production server but the full source code freely available at: https://bitbucket.org/vbanos/more-quality/

Anyone interested in MoRe Quality should feel free to contact the author for more information.

LoCloud and MINT

The MINT service is  a web based platform  designed and developed by  NTUA (the National Technical University of Athens – partner in LoCloud) to facilitate the aggregation of  digital cultural heritage content and metadata in Europe.

The service includes all the various  steps of  workflows, from the ingestion, mapping and aggregation of metadata records to  the implementation of a variety of remediation approaches for the resulting repository. The platform offers  users an organisation management system enabling the deployment and operation of different aggregation schemes (thematic or cross-domain, international, national or regional) and corresponding access rights. Registered organisations can upload (http, ftp, oai-pmh) their metadata records in xml or csv serialisation in order to manage, aggregate and publish their collections.

A reference metadata model serves as the aggregation schema to which the ingested (standard or proprietary) schemata are aligned to. Users can define their metadata crosswalks with the help of a visual mappings editor for the XSL language. The mapping is performed with simple drag-and-drop or input operations, which are then translated into the corresponding code. The mapping editor visualises both the input and target XSD, in an intuitive interface that provides access and navigation of the structure and data of the input schema, as well as the structure, documentation and restrictions of the target one. It supports string manipulation functions for input elements in order to perform 1-n and m-1 (with the option between concatenation and element repetition) mappings between the two models. Additionally, structural element mappings are allowed, as well as constant or controlled value (target schema enumerations) assignment, conditional mappings (with a complex condition editor) and value mappings between input and target value lists. Mappings can be applied to ingested records, edited, downloaded and shared as templates between users of the platform.

Preview interfaces present to users the steps of the aggregation including the current input xml record, the XSLT of their mappings, the transformed record in the target schema, subsequent transformations from the target schema to other models of interest (e.g. Europeana’s metadata schema), and available html renderings of each xml record. Users can transform their selected collections using complete and validated mappings in order to publish them in available target schemas for the required aggregation and remediation steps.

The MINT platform has been deployed for a variety of aggregation workflows corresponding to the whole or parts of the backend services. Specifically, it has served the aggregation of a significant amount of museum content for Europeana through the ATHENA project, that has ingested and aligned to the LIDO format over 4 million items from 135 organisations. The resulting repository offers an OAI-PMH interface presenting the records in the Europeana Semantic Elements schema (ESE). The use of a reference model allowed the rapid support of updated ESE versions that were introduced during  the project (2008-2011), with minimal input from providers. The users’ effort to align their data to an adopted domain model also motivated them to update their collection management systems and improve the quality of their annotations in order to take advantage of a well defined, machine understandable model and, subsequently, control and enrich their organisation’s contribution and visibility through the aggregator and Europeana.

Τhe MINT ingestion platform that is used in the LoCloud project is meant for large-scale ingestion of metadata with the final aim of  delivering  to Europeana a significant content from small and medium cultural institutions. Τhe development of MINT started within the ATHENA project when the NTUA team integrated all the necessary components for ingesting, mapping and publishing metadata to Europeana into a common technology platform, while it evolved through its use in other Europeana-feeder projects such as Linked Heritage, EuScreen, ECLAP, Carare, Europeana Fashion, Europeana Photography and others. The MINT platform provides content holders with the ability to perform the required mapping of their own metadata schemas into LIDO, Carare2.0 and EDM. It enables the ingestion of metadata from multiple sources, the mapping of the imported records to a target metadata schema and the transformation and storage of the metadata in a repository. Although its deployment is also guided by expediency, the system has been developed using established tools and standards and embodying best practices in order to animate familiar content provider procedures in an intuitive and transparent way also for newcomers.

Vassilis Tzouvaras
Senior Researcher
National Technical University of Athens

Cloud computing take-up in the public sector workshop

On 20 May 2014 Walter Koch, AIT, Austria, has presented LoCloud at the Cloud computing uptake in the public sector workshop, held at the European Commission’s premises in Brussels.

The objective of the  Cloud computing uptake in the public sector workshop was to bring together organisations that have expressed clear interest in cloud computing and which could benefit from interaction. The workshop also aimed to provide input for identifying possible future actions in this area.

Below is a short report by Walter Koch (AIT, Austria). All presentations made at the event are available for download from the workshop website.


Report of the workshop Cloud Computing Uptake In The Public Sector, Brussels, 20 May 2014

The workshop was introduced by Ken Ducatel, European Commission, Head of Unit: Software and Services.

Mr. Ducatel presentation focused on cloud computing within the framework of the European Commission programme Horizon 2020. He addressed issues such as like Service Level Agreements, Security Services, Standard Terms and pointed out to the final report Establishing a Trusted Cloud Europe. This document presents the results of a collaborative work by public administrations, cloud businesses, and data protection advocates who have joined forces through the European Cloud Partnership in order to establish a roadmap for European leadership in the cloud.

The first presentation at the workshop was made by Dirk van Rooy, Head of Sector European Cloud Partnership, European Commission DG Connect E2, who provided an overview of ICT8-2015: Boosting public sector productivity and innovation through cloud computing services, one of the challenges of the Horizon 2020 Work Programme. This specific challenge is divided into two areas:

a. Pre-commercial procurement for public sector cloud computing services (PCP)

b. Public procurement of innovative cloud computing solutions (PPI)

The PCP and PPI activities have been presented in detail by Lieve Bos, DG Connect Unit F2: Innovation. An overview of PCP and PPI related calls for proposals is included in the presentation (available from the event web site). The EC funding for the next call is € 9 million.

A session on Practical Examples Of Public Sector Cloud Use/Procurement followed. The final session of the workshop included presentations about cloud initiatives in various societal domains. This session was devoted to a series of themes including science, education, European cities, culture and transport.  Europeana was presented by Pavel Kats while Walter Koch outlined some technical aspects of the LoCloud services.  The workshop was mainly focused on procurement of cloud computing services and cloud computing solutions. The system presented by Maryline Lengert and Bob Jones, CERN may be of interest to Europeana. They presented the Helix Nebula System which could provide an infrastructure to Europeana developments and services.


Univ.-Prof.em.Dr.Walter Koch
Angewandte Informationstechnik Forschungsgesellschaft mb, Austria

Micro-services and a lightweight digital library

LoCloud will implement a series of micro-services aimed to benefit Europeana’s user’s experience as well as the content providers institutions.

Currently, work is in progress on the development of an experimental application to enable local cultural institutions to collaborate in the development of multilingual vocabularies for local history and archaeology. Moreover, some of the technical partners are implementing Natural Language Processing (NLP) tools to analyse and enrich the metadata being provided to Europeana as well as to develop and test an application on contents uploaded to Wikimedia and capture and enrich the available metadata. The idea behind this effort is to test the potential of exploiting Wikimedia for crowd sourcing projects, as well as for small institutions and individuals for capturing the local heritage.

A prototype geo-location tool is being developed and it will be made available to partners for testing in the coming months. This service will offer content providers the opportunity  to use a combination of local place names data and spatial filtering mechanisms to assign geo-location coordinates to a given digital object as well as to improve the accuracy of the existing geo-coding. Moreover, the spatial metadata will become useful for detailed navigation such as mobile devices, offering mobile users the chance to explore cultural content relevant to the place they are in.

Other services under development are an historic place names service, metadata enrichment services allowing a wide range of connections between items in a collection, and a Wikimedia application which will enable the integration of Wikimedia services with the LoCloud infrastructure.

Finally, work has progressed with regard to the development of a Lightweight Digital Library System. A prototype will hopefully be ready before the end of 2014 when it will undergo a testing phase by the LoCloud content provider partners. This tool will be useful for those small institutions that currently do not have a digital archive and therefore need a user-friendly system for cataloguing their digital content and metadata.