New workflow functions

New plugin types for OPACs, servlets and export

Version 2.0 of Goobi has been extended to support additional plugin types. Alongside the existing types (‘step’, ‘import’, ‘command’ and ‘validation’), the extended infrastructure now supports ‘opac’, ‘servlet’ and ‘export’ plugin types.

/opt/digiverso/goobi/plugins/command/
/opt/digiverso/goobi/plugins/export/
/opt/digiverso/goobi/plugins/import/
/opt/digiverso/goobi/plugins/opac/
/opt/digiverso/goobi/plugins/servlet/
/opt/digiverso/goobi/plugins/step/
/opt/digiverso/goobi/plugins/validation/

To ensure that Goobi remains flexible with regard to the selection of presentation software, the previous export options have been replaced by a plugin-based export. Compatibility has been maintained by switching both manual and automatic exports to a plugin basis.

Integration of waiting times into the workflow

For some time, our aim has been to integrate waiting times into the workflow in order to block any further processing for a specified period. This functionality is particularly useful as a way of ensuring compliance with copyright protection and publication terms. It can now be firmly integrated into the workflow to establish a waiting period before the definitive archiving or amendment of data. The waiting period can be implemented using one or more ‘delay’ plugins.

Exporting processes without images

To ensure that it can interact fully with external repositories, Goobi has now been extended to allow users to configure the export of data without adding contents from the ‘media’ directory. This means it is now possible to export just metadata, for example as a METS file.

Fully automated workflows without manual intervention

The efficiency of large-scale processing with Goobi has been increased through the option to automate even more workflow steps. As well as the existing functions that allows users to initiate workflows with a series of automated steps, those steps can now also be performed automatically from the administrator’s interface following a change of status or after sending off solutions to specific problems. This further reduces the need for manual intervention by users.

Additional properties for new ‘link’ and ‘readonly’ types

Up to now, users have been offered various property types on an individual basis when they are processing workflow steps in order to receive data input from users. Two new types (‘link’ and ‘readonly’) have now been added, thus giving users the option to be shown a clickable link or to be presented with values that are not to be changed.

New functions for handling metadata

Metadata grouping

One of the biggest changes made during the development of Goobi Version 2.0 was the integration of new metadata processing options. These include a facility to group metadata that allows users to display and save various metadata fields together.

This means that different items of information can be captured, validated and then exported, e.g. in MODS data format, for individual metadata.

Support for additional personal data (biographical dates and epithets)

Thanks to the introduction of a metadata grouping function in Goobi version 2.0, the exporting of data has been extended to allow the use of additional personal data.

Depending on the individual configuration in Goobi’s rulesets, you can now specify whether to include the information ‘termsOfAddress’ and ‘date’ in the Edit metadata screen alongside the first name, surname and role.

Connection to authority databases

As well as the option to record personal data, Goobi 2.0 now allows users to enter authority data. This means you can export authority data to the intranda viewer to facilitate user-friendly research and presentation. Users can then work directly with the authority data of Germany’s National Library (Deutsche Nationalbibliothek), for example, and integrate that data live into the digital presentation. Users can also interact with other authority databases, thus providing a connection between your data and a range of external and in-house data sources.

You can use the rulesets to configure the way Goobi interacts with external authority databases. This configuration has a direct impact on the editing of information within the Goobi METS Editor.

Extension of METS Editor to allow handling of process images

We have added a new section to the METS Editor that gives users direct access to digitised material on the Goobi server. Staff tasked with the processing of metadata can use this facility to easily delete duplicate images without having to work their way through laborious correction loops involving additional workflow steps.

In all cases, the handling of images within the digitised material affects each directory for the related process. If master images are renamed or deleted, there will be an immediate change in the corresponding derivatives.

Simpler file renaming for digitised material using the METS Editor

Goobi 2.0 allows you to rename digitised material within the file system. This can be done directly from the METS Editor. Depending on the number of files being renamed, the save process can be lengthy. The METS Editor now includes a progress bar that provides active feedback on the progress of renaming all the files in a process.

Extension of METS Editor to allow copying of structure elements

The METS Editor in Goobi 2.0 now includes an option to copy existing structure elements instead of having to create them manually each time.

Extension of METS Editor to allow copying of structure elements from other processes

The METS Editor in Goobi 2.0 now includes an option that allows users to copy structure elements from the METS files of one process into another process. This involves specifying the source process containing the data, selecting the structural element you wish to copy from the tree view and specifying the destination process within the current METS file.

Extension of METS Editor to simplify processing of digitised newspaper material

Within the METS Editor, the tree view for structural data now displays newspaper issue numbers and publication dates directly, i.e. users no longer have to open the structure elements individually to view the information. This makes it possible to carry out rapid checks even in the case of newspaper volumes containing hundreds of issues.

Importing, processing and exporting newspapers

One of the requirements that emerged from our newspaper digitisation projects was a function that would allow users to import digitised material for newspaper volumes on an annual basis. Using the interface for import plugins, it is now possible to establish import processes that automatically generate the corresponding issues and any enclosures including the associated metadata for the issue number and the publication data within the METS files.

Integration of additional catalogue types for data import

Version 2.0 of Goobi now supports native import processes from MARC-XML and other data formats. As well as catalogue imports for aleph catalogues, users can now import additional data from other sources such as Faust databases and from other systems.

Support for additional metadata formats METS 1.10, MODS 3.5 and LIDO

Goobi 2.0 contains a series of improvements with regard to the internal processing of metadata and now supports the latest version of the METS schema (version 1.10). For more details of the current METS schema please see the following site:

http://www.loc.gov/standards/mets/version110/mets.xsd

The storage of metadata within the MODS area has also been expanded, and the current MODS schema has now been implemented (version 3.5). Details of the changes made in MODS version 3.5 can be found at:

http://www.loc.gov/standards/mods/changes-3-5.html

As part of the update, we migrated from version 3.3 to 3.5 of MODS. The following link may help with further research as the site also describes the features of the schema update to version 3.4:

http://www.loc.gov/standards/mods/changes-3-4.html

As well as support for METS and MODS within the UGH metadata library, which is used by Goobi, the metadata format LIDO has also been extended. This means that Goobi can now be used to describe museum items directly in the central standardised international museum format. Further information about LIDO can be found at:

http://network.icom.museum/cidoc/working-groups/data-harvesting-and-interchange/what-is-lido/

METS export allows additional data export to DFG Viewer (user profile 2.1)

New information fields (e.g. licence, sponsors) have been created to display digitised material using the DFG Viewer.

Depending on the specification, these new fields will also be exported to the DFG Viewer.

http://dfg-viewer.de/fileadmin/groups/dfgviewer/METS-Anwendungsprofil_2.1.pdf

Improved search function

Goobi’s search functionality has been massively improved during the development of version 2.0 with the result that it is now easier to start working with large volumes of data. This involved making improvements to existing mechanisms and implementing new search options.

Extended searching

The ‘Search process’ page has been adapted to allow newly developed search functions to be shown in the input box.

Each line represents an individual search parameter. The first column allows the user to select one or more search fields. The input options vary depending on the selection. In the case of workflow steps and projects, for example, users can select a value from a list. In the case of titles and IDs, there is a simple text box, while for properties and metadata the two forms can be combined.

The plus and minus buttons at the end of each line allow users to add further search parameters or remove existing ones.

Additional search functionality for batch IDs

Goobi 2.0 now allows users to conduct a search based on batch ID. This makes the task of searching for data within the entire system even more flexible.

Metadata searches

Goobi 2.0 now allows users to search for previously recorded metadata, thus complementing the existing search parameters based on production data. To make this possible, a list of the available metadata in the database is stored in the METS Editor in addition to the METS file. This means that users can subsequently perform a search of all database content and of the METS file.

Complex searches

Goobi’s extended search function has been improved to allow the use of OR in addition to the standard Boolean operators AND and NOT. Brackets can now also be used to submit even more complex search requests. This means that individual search parameters can now be used more than once in the same request.

General modifications

A number of other general improvements have been made in Goobi 2.0 to improve the application’s functionality and make it more user-friendly.

Change to method of adding users for automatic entry in the LDAP

It is now much simpler to add new users in Goobi. Up to now this has always been slightly awkward, as users had to be processed a second time in Goobi in order to enter them in the corresponding LDAP server. New users can now be entered in the LDAP server directly through the ‘Add new user’ dialogue box. This involves entering the person’s details and then adding the new user in the LDAP before assigning specific projects and activities. As well as being more intuitive and straightforward, this method is also much less likely to produce an error.

Storing search results

Process hit lists can be stored in either Excel or pdf format. With Goobi 2.0, you can now specify which values or columns should be added to the file for storage purposes.

Improvements to GoobiScripts

Goobi 2.0 provides an option to update the metadata from a search hit list in the actual database. This involves scanning all Goobi processes, loading the metadata from each associated METS file and then saving the metadata to the database.

action:updateMetadata

Any old dataset entries found will be deleted before saving.

New option to execute several GoobiScripts in a single call

Goobi 2.0 provides a new and convenient option that allows you to modify numerous processes all together by simultaneously executing several GoobiScripts. To do so, simply enter semi-colons as separators between the individual scripts.

Importing authority data from the OPAC

If authority data (e.g. from the GND) has already been added to a dataset in the library catalogue, this data can now be imported directly into Goobi when creating new processes. As well as links to authority data on individuals, you can now include all technical terms, geographical data and organisations.

Technical infrastructure

Goobi 2.0 is the result of a comprehensive internal revision. The biggest changes concern the database connection.

Changes to database connection

Whereas in previous versions we used the database framework Hibernate to make communication with the database as simple as possible, this connection has now been completely restructured. From now on, communication with the database will take place using Apache DBCP in the form of SQL statements. This migration will give programmers much greater scope to influence actual database communications and will also make it easier to trace errors and carry out improvements.

Updated web framework JavaServer Faces 2.2

The web framework JSF 2.2 will now run alongside version 2.0 of Goobi. In conjunction with the templating mechanism Facelets, this makes it possible to adapt and expand the HTML pages in the web interface on a more individual basis. The main advantage of this change is that plugins can now be integrated into the workflow from within the user interface.

Expansion of existing folder infrastructure for additional data

The conventions for Goobi’s internal folder structures have been modified to provide more flexible support for other data formats and workflow actions. In addition to the ‘images’ and ‘ocr’ directories that could be found in the first version of Goobi and were used to store output from a text recognition task, several new folders have now been introduced and have specific functions within the workflow:

source
import
export
taskmanager
validation

Refactoring of Java code for managed beans

While modifying the database connection, we also carried out substantial refactoring work within ‘managed beans’. As well as changing the package name to org.goobi.managedbeans.* we completely re-implemented several tasks, especially the submission of data requests and the loading of data from the database.

Special characters in project and docket variables

The names and content of variables can now include special characters and spaces. This additional functionality can be used for project configuration purposes when generating METS files, calling scripts and creating dockets.

New variables for use with Goobi

In addition to the variable $(meta.NAME), which is replaced by the value of the first metadata item to match the selected name, it is now possible to use the variable $(metas.NAME) to export all the available metadata items with that name using comma separators. This can be useful if you wish to send all the languages used in a process to the OCR engine.

Local configuration files for easier maintenance

The concept of local configuration files that proved very successful in the development of the Goobi viewer has now been adopted for Goobi 2.0. As a result, it is now possible to save the configuration file goobi_config.properties within the central configuration directory /opt/digiverso/goobi/config/ and in doing so to consistently overwrite any values from the web application that differ from the default configuration. This makes it easier to update Goobi without having to deal with individual configuration files in the web application.

/opt/digiverso/goobi/config/goobi_digitalCollections.xml
/opt/digiverso/goobi/config/goobi_metadataDisplayRules.xml
/opt/digiverso/goobi/config/goobi_opac.xml
/opt/digiverso/goobi/config/goobi_processProperties.xml
/opt/digiverso/goobi/config/goobi_projects.xml
/opt/digiverso/goobi/config/goobi_webapi.xml
/opt/digiverso/goobi/config/messages_de.properties
/opt/digiverso/goobi/config/messages_en.properties
/opt/digiverso/goobi/config/messages_es.properties
/opt/digiverso/goobi/config/goobi_config.properties

As well as overwriting the central configuration file, you can now overwrite translations of individual texts if you do not wish to use the default translation for them. To do this, simply create an appropriate message file (e.g. messages_de.properties) within the central configuration directory /opt/digiverso/goobi/config/. This file should contain the texts you wish to display in the user interface in place of the default translations.

Error correction

Smaller corrections of existing bugs

It is no longer possible to allocate a project to users who are already members of that project or to allocate a group to users who are already members of that group.