World Archive

Instructions

Making a manifest

A manifest is the fingerprint of a dataset: 256-bit perceptual hashes plus file metadata, about 120 KB per hour of footage. It holds no frames and no pixels, which is why it is the thing you upload here and the footage never moves.

  1. 01

    Install the tool

    Open source, and it runs entirely on your machine. Nothing it does touches the network.

    $ pip install git+https://github.com/World-Archive/overlap

  2. 02

    Fingerprint the footage you were offered

    Videos, MCAP, ROS bags. The slow step, and the only one that reads pixels — roughly real-time per core.

    $ overlap --index ./offer.ovl index /path/to/offer

  3. 03

    Export a manifest

    Writes offer.ovlm — about 120 KB per hour of footage. --stride 1 keeps full density; a downsampled manifest recovers markedly less against re-cut footage.

    $ overlap --index ./offer.ovl export -o offer.ovlm --stride 1

Worth knowing

  • A very large corpus exports as a directory of parts plus parts.json. Drop the whole directory here — a partial set is refused rather than silently under-reported.
  • Only algo_id and prep_id have to match between two sides. Your sampling rate is your own choice; lowering it only weakens your own detection.
  • Quote one dataset out of a mixed inventory with --only PREFIX, which exports a subset carrying its own Merkle root, so it cannot be passed off as the parent corpus.
  • You can skip this service entirely. overlap import pulls the published catalog into your own index and overlap compare runs the same comparison locally, with nothing leaving your machine.
Upload a manifest