Package org.javastro.ivoa.registry.harvesting

Class HarvestOrchestrator

java.lang.Object
org.javastro.ivoa.registry.harvesting.HarvestOrchestrator
@ApplicationScoped public class HarvestOrchestrator extends Object
Queue-based orchestrator for asynchronous incremental harvesting of IVOA registries.

Architecture

  • The RofR seed source is enrolled at first scheduler invocation.
  • Each scheduler tick dequeues one pending source, harvests it incrementally (using the per-source lastSuccessfulUntil cursor), stores the records, and parses them to find new sources.
  • Newly discovered sources are enqueued through the same queue.
  • The queue is deduplicated: a source that is already QUEUED or RUNNING is never enqueued twice.
  • All source state is persisted in BaseX via HarvestSourceCatalog and survives service restart.

Scheduling is controlled by the ivoa.harvesting.rofr.cron property (default "off" = disabled). Set it to a cron expression such as "0 0/10 * * * ?" to enable periodic harvesting.

  • Constructor Details

    • HarvestOrchestrator

      public HarvestOrchestrator()

  • Method Details

    • enqueue

      public boolean enqueue(String sourceKey)
      Enqueue a specific source for harvesting.
      Parameters:
      sourceKey - the key of a known source
      Returns:
      true if the source was enqueued; false if it was already queued/running, unknown, disabled, or rejected.

    • cache

      public boolean cache(String sourceKey)

    • loadFromCache

      public int loadFromCache(String sourceKey)
      Load the cached chunk files of a source into the registry store as a single document at harvested/<source-key>/rec.xml, replacing any existing one.
      Parameters:
      sourceKey - the source key
      Returns:
      number of records stored

    • triggerHarvest

      public int triggerHarvest(String sourceKey)
      Trigger an immediate, synchronous harvest of the seed (or a specific source).
      Parameters:
      sourceKey - seed is used when null
      Returns:
      number of records stored

    • getStatus

      public HarvestOrchestrator.HarvestStatus getStatus()
      Returns a snapshot of the current harvest status.

    • getSources

      public List<HarvestSource> getSources()
      Returns all sources in the catalog.

    • getRecentRuns

      public List<HarvestRunRecord> getRecentRuns()
      Returns recent run records from all sources (most-recent first, bounded).

    • setSourceStatus

      public boolean setSourceStatus(String sourceKey, SourceStatus status)
      Set the status of a source (e.g. to SourceStatus.DISABLED).
      Returns:
      true if the source was found, false otherwise.

    • addSource

      public void addSource(String key, String url)

    • discoverNewSources

      public List<String> discoverNewSources(List<org.javastro.ivoa.entities.resource.Resource> resources, String fromSourceKey, int parentDepth, HarvestSourceCatalog catalog)
      Inspect the list of harvested resources and register any newly discovered harvestable registries in catalog.
      Parameters:
      resources - records returned by a harvest run
      fromSourceKey - the source key that produced these records (for provenance)
      parentDepth - discovery depth of the producing source
      catalog - the catalog to register new sources into
      Returns:
      keys of sources that were newly accepted

    • resetSource

      public boolean resetSource(String key)