polycomb

polycomb automates the ingestion of public datasets — unimodal and multimodal, across many modalities — into homeobox atlases.

Turning a public dataset into a finalized atlas means downloading and organizing files, aligning their ad-hoc columns and values to a target schema, and streaming the matrices into storage. polycomb provides:

• Standardized data packages — the Collection API for assembling related datasets into a tagged, serialized collection.json package, the unit the rest of the pipeline operates on.
• Schema alignment with resolution — a resolver pipeline that maps raw values to canonical identifiers: ontology terms (cell types, tissues, diseases), registries, and external database cross-references (Ensembl, UniProt, PubChem, Cellosaurus, …), cached in a LanceDB reference database with external-API fallbacks.
• Auditable curation — every table mutation flows through a curation system that records each transformation of the original obs and var tables, with full provenance, in an audit log.
• Agent skills — a set of skills that guide an agent through the end-to-end workflow systematically, one stage at a time.

Where to start

• Workflow — the end-to-end pipeline, the skill for each stage, the scripts, and the on-disk layout. Start here for the big picture.
• Collections — the Collection API: tagging files, building datasets, coalescing, and the collection.json manifest.
• Resolvers — the resolver pipeline, the LanceDB reference cache, fallback APIs, and the registries that bind schema fields to resolvers.
• Curation — the curation operations, the applicator, and the audit log of transformations.