Harvesting¶

The harvesting of other registries can be both compute and network intensive, both for the harvesting registry and the harvested registries. This registry implementation has several tools that allow the operator bootstrap and manage the process in a flexible way that

minimises the network load on the harvested registries - especially in the case of failures.
allows the operator to control the process and its schedule, and
allows the operator to inspect the results and diagnose problems.

The Harvest cache¶

The harvest cache is a local storage area where harvested records are temporarily stored before being processed and loaded into the main database. This helps to reduce the load on the harvested registries and provides a way to recover from failures.

The cache consists of a series of directories (one for each remote repository) containing a set of files of raw OAI-PMH ListRecord responses. The cache root directory is configured with the ivoa.registry.harvesting.cache-dir property (see Configuration). It is populated for a particular registry ivoa identifier by the POST /harvesting/harvest/source/{key}/cache endpoint, and can be loaded into the database with POST /harvesting/harvest/source/{key}/cache/load. The cache is not used for incremental harvesting, but it can be used to bootstrap a harvest or to recover from a failure.

Bootstrapping the harvest¶

It is possible to bootstrap the harvest from a local copy of the Registry of Registries (RofR) instead of fetching it from the IVOA. This is useful for testing and for reducing the load on the RofR. The local copy is bundled with the application and can be used by setting ivoa.registry.harvesting.rofr-use-local=true in the configuration.

In addition, it is often a good idea to use an existing cache for the larger registries such as Vizier (ivo://cds.vizier/registry) and the EuroVO registry (ivo://esavo/registry).

Harvest record¶

The registry keeps a catalog of OAI-PMH sources, persisted in the database at harvest/sources.xml. The seed is the RofR (or the bundled catalog when rofr-use-local=true). Each source has a status: ACTIVE, QUEUED, RUNNING, FAILED, DISABLED, REJECTED (and CACHED for a source that has a local cache waiting to be loaded). Harvests are incremental from each source’s last successful harvest. Harvested records are stored under harvested/<source-key>/.

Harvesting runs on the cron schedule, or on request through the admin endpoints (all need the update role):

Endpoint	Purpose
`POST /harvesting/harvest`	Queue a harvest (`sourceKey`, `immediate` query parameters)
`GET /harvesting/harvest/status`	Current harvest status
`GET /harvesting/harvest/sources` / `POST /harvesting/harvest/sources`	List / add sources
`GET /harvesting/harvest/runs`	Recent run records
`POST /harvesting/harvest/source/{key}/disable`	Stop harvesting a source
`POST /harvesting/harvest/source/{key}/reset`	Reset a source so that the next harvest starts from the beginning
`GET`/`POST /harvesting/harvest/source/{key}/cache`	Inspect / populate the file cache for a source
`POST /harvesting/harvest/source/{key}/cache/load`	Load a source’s cached files into the database

Run log¶

Each source keeps its most recent run records: start/end time, records stored, new sources discovered, outcome (SUCCESS/FAILED) and details. With RegTAP enabled the record also contains relationalAdded, relationalDeleted, relationalFailed and up to 20 relationalFailure entries (identifier: message). A resource that cannot be loaded relationally does not fail the harvest.

Deleted records¶

Records flagged deleted by a source are removed from the XML database and, if enabled, from the RegTAP tables.

Discovery¶

With discovery enabled, registries described in harvested records are added as new sources, bounded by the discovery.* limits in Configuration.