oncoref API Guide

oncoref keeps its historical flat top-level imports for compatibility, but new code should prefer the semantic submodules below. They make the domain boundary clear and avoid guessing whether a broad name such as coverage or peptides is general or specific to cancer-testis antigens (CTAs).

Package boundary: oncoref is the upstream home for empirical base facts and canonical identifiers that are ready to be reused across the PIRL stack. pirlygenes owns purpose-specific gene sets and panels; trufflepig owns per-sample interpretation, QC narration, and rule firing. As a rule of thumb, source-anchored measurements with denominators, confidence intervals, cohorts, PMIDs/DOIs, or shared ontology implications belong in oncoref. Opinionated gene selections and target-to-therapy registries belong in pirlygenes. One-sample rules belong in trufflepig. When a missing data field, gene universe, bundle-integrity rule, or source-QC decision affects shared reference artifacts, the durable fix should live or be exposed here rather than only in a downstream compatibility layer.

Guide Map

Read the guide from concepts to operations:

Layer Start here
Canonical cancer and gene identities Cancer Vocabulary, Gene Identity
Expression reads, artifacts, and normalization Expression And Normalization
Clinical and epidemiological reference facts ICI Response, Burden, TMB, Fusions, and Signatures
Cancer-testis antigen and panel calculations CTA Antigens, Generic Antigen Panels
Downloads, caches, and release metadata Data Management
Historical import paths Compatibility Modules

Within each section, the intended use and primary modules come first. Detailed schema, provenance, fallback, and migration contracts follow.

Cancer Vocabulary

  • oncoref.cancer_ontology — cancer-type registry, aliases, parent/child tree, lineage/family groupings, molecular subtype axes, mismatch repair (MMR) and microsatellite instability (MSI) classifier-status semantics, matched normal tissues, source-scoped evidence resolution, and display helpers.
  • oncoref.cohorts — expression/source cohort IDs, computed aggregate cohorts, source versions, and mixture-cohort flags.

Use these when asking "what cancer type or cohort does this code mean?" Prefer the DataFrame-returning query helpers when code will be passed into other oncoref domains; they keep the result type and columns stable.

Ontology and category model

The registry separates hierarchy from taxonomic level. parent_code is the tree edge, while ontology_level (grouping, type, molecular_subtype, evidence_scope) and ontology_kind (computed_union, source_scope, anatomic_type, molecular_status_subtype, etc.) say what kind of node a row is. Do not infer semantic level from mixture_cohort; that legacy flag only says the reference cohort/source is pooled or source-scoped. For example CRC_MSI is a molecular_subtype under CRC but remains a source-scope clinical evidence row, while OV is an anatomical grouping and FTC / PPC are anatomical cancer types. Pure clinical fact scopes such as NET_NONPANCREATIC and NEN_EXTRAPULMONARY_HG use ontology_level="evidence_scope" so they do not look like groupings with missing children. Differentiation and grade are orthogonal sparse axes: use differentiation="NEC" for native neuroendocrine lineage labels, or grade_tier="high" for normalized high-grade rows, without treating either one as a parentless cancer type.

Expression and classification backing

Expression/classification backing is explicit. Use reference_source, cancer_type_reference_source(), cancer_type_reference_code(), or cancer_type_records(reference_source=...) instead of inferring from mixture_cohort or from whether a code is anatomical or molecular. The enum is data-driven:

  • own_cohort — this code has its own separable expression cohort.
  • member_union — this code is backed by a union of expression-bearing member cohorts and can be reported as a coarser call.
  • parent — this code carries an annotation/slice but should be reported at its nearest reportable ancestor.
  • none — pure provenance or unsupported scope; walk up the tree if a coarser call is needed.

is_classification_target, classification_target_codes, and cancer_type_records(classification_target=True) are compatibility views over that enum: a code is returnable when reference_source is own_cohort or member_union. For example COAD_MSI, COAD_MSS, READ_MSI, and READ_MSS are own_cohort because the TCGA COAD/READ MSI partitions have separable expression; CRC_MSI is a legitimate member_union over COAD_MSI ∪ READ_MSI, not merely an annotation. A molecular slice only falls to parent when oncoref has not measured a separable cohort, as with the current STAD/UCEC molecular subtype rows.

Computed expression pools are also explicit. Use computed_union_codes() for registry rows whose expression_source="computed" and reference_source_codes("member_union") for all reportable member-union references, including source-scope unions such as CRC_MSI, NSCLC, BTC, and SGC.

Category queries

For category-aware downstream code, start with cancer_type_category_schema() and cancer_type_category_summary(). The schema is the compact public vocabulary for ontology_level, the observed ontology_kind values, and reference_source; the summary reports counts and example codes for every observed level/kind/reference-source combination. This is the intended replacement for ad hoc tests like "has children", "is mixture_cohort", or "does the code name contain MSI".

Examples

from oncoref import cancer_ontology, cohorts, expression

cancer_ontology.resolve_cancer_type("prostate")
cancer_ontology.cancer_type_tree("CRC")
cancer_ontology.cancer_type_path("COAD_MSI")

# CRC plus anatomical children and molecular leaves.
crc = cancer_ontology.cancer_type_records(under="CRC")
crc["code"].tolist()
crc[["code", "parent_code", "ontology_level", "ontology_kind"]]

# Cross-cutting molecular axes can be intersected with hierarchy or lineage.
msi_crc = cancer_ontology.cancer_type_records(subtype_group="MSI", under="CRC")
epithelial_msi = cancer_ontology.cancer_type_records(
    subtype_group="MSI", lineage_group="Epithelial"
)
source_scope_msi = cancer_ontology.cancer_type_records(
    under="CRC", ontology_level="molecular_subtype", ontology_kind="molecular_source_scope"
)
classification_targets = cancer_ontology.cancer_type_records(classification_target=True)
clinical_fact_scopes = cancer_ontology.cancer_type_records(classification_target=False)
computed_pools = cancer_ontology.computed_union_codes()
member_union_refs = cancer_ontology.reference_source_codes("member_union")
cancer_ontology.cancer_type_category_schema()
cancer_ontology.cancer_type_category_summary()
cancer_ontology.cancer_type_reference_source("CRC_MSI")
cancer_ontology.cancer_type_reference_code("STAD_MSI")

# The MMR/MSI classifier axis keeps positive, negative, and confounder classes
# explicit. STAD_MSI exists as an ontology code, but expression_only=True
# excludes it until split STAD subtype expression shards are built.
cancer_ontology.mmrd_cancer_codes()
cancer_ontology.pmmr_cancer_codes(under="CRC")
cancer_ontology.mmr_confounder_cancer_codes()
cancer_ontology.mmr_hypermutated_confounder_codes()
cancer_ontology.mmrd_cancer_codes(expression_only=True)
cancer_ontology.cancer_mismatch_repair_status("UCEC_POLE")

# COAD_MSI / READ_MSI keep anatomical expression context but resolve evidence
# rows through CRC_MSI when published sources are colorectal-level.
msi_crc[["code", "evidence_source_code", "normal_tissue_code", "hpa_tissues"]]

# Join scalar references for the returned codes.
cancer_ontology.cancer_type_reference_data(msi_crc)

# Ask whether each ontology node has a direct expression reference, a computed
# member-union reference, parent fallback, or no expression backing.
cancer_ontology.expression_reference_coverage(subtype_group="MSI", under="CRC")
cancer_ontology.coverage_for_cancer_type("ASTB")

# Use codes directly with expression accessors.
codes = cancer_ontology.cancer_type_codes(subtype_group="MSI", under="CRC")
expression.cancer_reference_expression(codes)

# Matched normal RNA expression is an explicit HPA read.
cancer_ontology.matched_normal_tissue_expression("COAD", genes=["ENSG00000141510"])

cohorts.cohort_registry_df()

Consumer readiness

expression_reference_coverage() is the ontology-wide readiness table for classifier consumers. It reports direct observed-bulk source-matrix coverage, computed member-union references for curated grouping/source-scope codes such as NET, CRC, CRC_MSI, NSCLC, BTC, and SGC, parent fallback via classification_reference_code, explicit is_classification_target eligibility, matched normal tissue availability, molecular/fusion-only definitions, canonical gene/proteoform space, data/source matrix versions, and a conservative consumer_recommendation: direct_reference, computed_reference, reference_only, parent_reference, molecular_only, or unsupported. SGC is reference_only: its histology-member union remains available for comparison, but the source/therapy grouping is not a valid final classification label. has_direct_expression_reference remains literal; computed groupings use expression_reference_kind="computed_union" and expose their pooled member codes in computed_expression_member_codes. The table intentionally does not synthesize marker-program or discriminator fallbacks; those remain consumer-layer choices in packages such as trufflepig.

Gene Identity

  • oncoref.gene_ids — canonical ENSG space, alt-haplotype / retired Ensembl ID migration, symbol/synonym resolution, and report-facing gene labels.
  • oncoref.genome — optional (pip install 'oncoref[genome]') pyensembl-backed transcript/gene lookup and transcript-to-gene aggregation for source matrices.

Resolution contract

Use the gene-id helpers before building expression artifacts or joining downstream gene sets to oncoref references. canonical_gene_id() is the primary any-identifier entry point for the shipped ENSG + symbol/synonym space: it normalizes versioned or case-varied Ensembl gene IDs, follows retired/alt ENSG aliases into the canonical space, resolves symbols and synonyms, and returns None for inputs that cannot be mapped to a canonical oncoref gene. canonical_gene_symbol(), display_gene_name(), and short_gene_name() use the same resolver so report code does not invent a separate symbol mapping. entrez_gene_mappings() and resolve_entrez_id() expose the filtered NCBI Entrez/GeneID table used by the resolver; it covers live IDs from NCBI dbXrefs or current symbols plus discontinued IDs redirected through NCBI gene_history. gene_identifier_mapping_coverage() and gene_identifier_mapping_summary() make the shipped ENSG, symbol/synonym, and Ensembl-alias coverage explicit for migration audits, including non-unique symbols and missing-symbol rows. They do not claim that RefSeq or UniProt coverage is complete.

Examples

from oncoref import canonical_gene_id, canonical_gene_symbol, display_gene_name, gene_ids

canonical_gene_id("GNB2L1")        # previous symbol -> ENSG00000204628
canonical_gene_id("7157")          # Entrez/GeneID -> ENSG00000141510
canonical_gene_symbol("GNB2L1")    # previous symbol -> RACK1
display_gene_name("ENSG00000005955")  # retired Ensembl id -> GGNBP2
gene_ids.gene_identifier_mapping_summary()

ICI Response

  • oncoref.ici_response — checkpoint-inhibitor response anchors, anti-PD-1 shortcuts, regimen-aware lookups, extracted objective response rate (ORR) estimates, and pooled response summaries.

Regimen selection

DEFAULT_ICI_REGIMEN_PRIORITY is the unpinned regimen priority (PD-1, then PD-L1, then PD-1+CTLA-4). The older REGIMEN_FALLBACK name remains available in oncoref.ici for compatibility.

Examples

from oncoref import ici_response

ici_response.apd1_response("SKCM")
ici_response.best_available_ici_response("SARC_ASPS")
ici_response.ici_response_by_regimen("SKCM")
ici_response.ici_response_estimates_df()

Evidence rows

ici_response_estimates_df() is the auditable long table behind the compact ORR anchors. Each row has a stable estimate_id; compact ici_response_record(...) / apd1_response_df() rows expose that pointer as source_estimate_id. CI provenance is explicit through ci_basis, ci_low_status, ci_high_status, and value_status, so NR/NE bounds and not-reached medians are distinguishable from fields that still need extraction. source_locator_status is currently not_extracted for legacy rows until each paper/table/supplement locator is audited row by row.

CTA Antigens

A cancer-testis antigen (CTA) is encoded by a gene that is normally restricted to reproductive tissues but can be reactivated in tumors. In oncoref, a candidate is called a CTA by an explicit Human Protein Atlas (HPA) normal-tissue expression rule; the call is not evidence that its antigen is presented by the major histocompatibility complex (MHC) or that it is a validated therapy target.

Restriction synthesis

synthesize_restriction() prefers the HPA protein restriction when protein data exist and otherwise uses the RNA restriction. Confidence increases when the modalities agree. The broad RNA call REPRODUCTIVE supports TESTIS, PLACENTAL, or REPRODUCTIVE protein calls; it never supports a SOMATIC protein call.

  • oncoref.cta — CTA definition, HPA restriction tiers, axes, aliases, and gene ID/name sets. Strict helpers such as cta_gene_names() and cta_filtered_gene_names() preserve the HPA reproductive-restriction default; cta_clinical_target_evidence() exposes a separate clinical/canonical tier for source-anchored CTA targets that may be strict-pass, HPA-excluded, or candidate-only. cta_specificity_audit() exposes machine-readable specificity demotion and candidate-only decisions for genes whose normal-tissue evidence makes strict-default inclusion unsafe or unresolved.
  • oncoref.cta_coverage — CTA patient coverage over per-sample expression matrices.
  • oncoref.cta_peptides — CTA-specific 9-mer counts and load.

cta_specific_9mer_count_map() returns a map from a join key to n_specific_9mers; those counts are used as weights when computing cta_specific_9mer_load().

Broader therapy-target curation, mass-spectrometry evidence, and downstream prioritization rules can live in consumer packages while they remain package-specific.

from oncoref import cta, cta_coverage, cta_peptides

cta.cta_gene_names()
cta.cta_clinical_target_evidence()
cta.cta_specificity_audit()
cta_coverage.cta_addressable_fraction("LUAD")
cta_peptides.cta_specific_9mer_count_map(by="proteoform_key")

Generic Antigen Panels

  • oncoref.antigen_coverage — coverage helpers for caller-supplied gene lists.

Use this when the panel is not necessarily CTA. The function names require gene_ids= so a caller cannot accidentally rely on the CTA default. This module computes coverage for a supplied list; it does not make oncoref the owner of downstream panel curation.

from oncoref import antigen_coverage

antigen_coverage.addressable_antigen_fraction("LUAD", gene_ids={"ENSG00000141510"})
antigen_coverage.greedy_antigen_coverage("LUAD", gene_ids={"ENSG00000141510"})

Expression And Normalization

Expression readers are the stable downstream surface. Builder, registry, and engine modules produce and audit those artifacts; they are separated below so read-time choices do not get mixed with source-ingestion details. Expression values use transcripts per million (TPM) unless a section states a different unit.

Reader APIs

  • oncoref.expression — read-time accessors for per-sample expression, percentile vectors, representative samples, within-sample top fractions, and pan-cancer reference tables. sample_expression_qc reports per-sample detected-gene counts, literal-zero fraction, top-gene/top-10 concentration, biological-housekeeping detection, source-scale class, and source-type caveats so sparse source-matrix artifacts can be audited before using absolute TPM floors or housekeeping normalization. per_sample_expression(..., sample_qc="pass" | "pass_or_warn" | "all") filters sample columns at read time; the raw per-sample accessor defaults to "all" for forensic access, while live summaries such as cohort_stats and pooled_cohort_stats default to QC-passing samples. source_matrix_sample_qc_manifest, expression_artifact_build_metadata, and expression_artifact_build_summary read the optional QC/build metadata emitted by regenerated expression bundles. Until a regenerated heavy bundle ships those files, they return schema-stable empty metadata by default; use on_missing="raise" when a downstream migration requires the manifests. housekeeping_cancer_expression_coverage(...) is the reusable #202 audit surface for evaluating clean-TPM biological housekeeping candidates across cancer cohorts: it reports per-gene detection, >= floor coverage, p1/p5/ median clean TPM, sample-QC mode, and source-scale metadata. Treat absolute TPM floors as hard evidence only where recommended_for_absolute_tpm_floor is true; microarray/proxy or otherwise non-linear sources stay visible as warning/rank calibration inputs, not vetoes.

Builder APIs

  • oncoref.expression_builders — build-time ingestion and artifact cores used by data-bundle generation scripts. GeoMatrixSource / build_source_matrices own the generic supplementary-matrix path from raw source file to canonical per-code per-sample TPM parquet, mapping audit, parse diagnostics, sample-QC sidecars, and SourceMatrixBuildResult.summary_rows. summarize_source_matrix is the standalone producer for those per-gene-per-cohort reference-expression rows: raw TPM stats, clean-TPM 16/9/75 stats, n_samples, n_detected, and source provenance in one schema. geo_matrix_source_from_registry and scripts/build_geo_matrix.py make source_type: geo-matrix entries in the packaged source registry directly buildable; GeoMatrixSource also preserves summary-row provenance (notes, pipeline_stem, tumor_origin, metastasis_site) so downstream shard writers do not need a parallel source registry. tumor_origin is validated against TUMOR_ORIGIN_VALUES (primary, metastasis, recurrence, cell_line, pdx, normal_tissue, mixed). GdcSource, query_gdc_star_count_manifest, build_gdc_sample_manifest, read_gdc_star_counts_tpm, build_gdc_source_matrices, and scripts/build_gdc_source.py own the common GDC STAR-counts path: open RNA-seq file discovery, deterministic sample-per-case selection, per-sample TPM matrix assembly, canonicalization, sample QC, and summary-row sidecars. Source-specific GDC lineage routing can now attach to this shared contract instead of carrying separate BL/MM/TARGET-style builders. Recount3Source, recount3_gene_sums_to_tpm, build_recount3_source_matrices, and scripts/build_recount3_source.py do the same for source_type: recount3 entries, including run-to-sample aggregation and metadata-based routing before writing the standard source-matrix artifact set. SraSalmonSource, sra_salmon_source_from_registry, build_sra_salmon_source_matrices, and scripts/build_sra_salmon_source.py own raw-read-only SRA studies: the registry pins run roles, read checksums, and the Ensembl transcriptome; all declared runs are audited while only explicitly routed tumor runs enter reference matrices. See Raw SRA Expression Sources. TreehouseSource, treehouse_source_from_registry, treehouse_cohorts_for_group, and scripts/build_treehouse_source.py own the direct Treehouse-compendium path: clinical disease-label routing, log2(TPM+1) inverse transform, symbol canonicalization, sample QC, and summary-row sidecars. Treehouse selectors are registry-native for direct clinical routing ("" and tcga) plus selected side-table-backed routes: GDC project membership, cBioPortal patient/sample clinical attributes, and cBioPortal mutation-positive case sets. scripts/rebuild_expression_artifacts.py then applies the same sample-QC policy to derived shards by default (--sample-qc pass) and emits source-matrix-sample-qc.csv plus expression-artifact-build-metadata.* in the staging directory so bundle releases record which source samples fed percentiles, representatives, proteoform summaries, and within-sample summaries. The rebuild also assigns every representative source group to the released train/validation partition and records the policy, role counts, and per-cohort validation coverage in the same metadata. Representative sample selection uses representative_sample_columns / cohort_medoids on the biological clean-TPM view, then stores the selected samples' full clean_tpm_16_9_75 vectors. Release builds retain curated cohorts that have no strict QC-pass samples only through explicit source-aware fallbacks recorded in the build metadata, and clip invalid negative source expression values to zero with per-cohort counts.

Registry and low-level APIs

  • oncoref.expression_registry — source-registry inspection helpers over the bundled expression_sources.yaml. Use expression_source_registry_entries() for the full raw YAML dictionaries, expression_source_registry_entries(source_type="geo-matrix") for generic GEO build configs, or expression_source_registry_path() only when a subprocess needs the packaged registry path. Downstream packages should use these helpers instead of shipping a second copy of the registry.
  • oncoref.expression_engine — reusable low-level builder primitives for expression tables: identity/value column detection, transcript-to-gene aggregation, source row ID-type detection, source gene-row mapping audits, missing-vs-non-parsing numeric diagnostics, and canonical ENSG aggregation in linear expression space. It is an explicit public module, so downstream builders can import oncoref.expression_engine.map_source_gene_rows, canonicalize_source_gene_matrix, and coerce_source_expression_values without reaching into scripts. Use these in builders before committing a source matrix so unresolved high-expression rows and duplicate canonical IDs are explicit artifacts rather than hidden cleanup. The source audit frames are intentionally unversioned public API objects: provenance belongs in build metadata, while the frames themselves use stable canonical columns such as sample_qc_status, sample_qc_reasons, source_expression_nonzero_samples, and source_expression_sample_with_max.
  • oncoref.source_matrices — raw per-cohort source-matrix cache/fetch helpers. Use source_matrices.sample_qc(code) for the live source-matrix QC audit and source_matrices.sample_qc_manifest(...) for the optional generated-bundle QC manifest that records which samples fed derived artifacts.

Normalization API

  • oncoref.normalization — TPM conversion, clean TPM, technical-RNA filtering, log transforms, percentile ranks, and housekeeping normalization.

The normalization helpers are intended to be reusable directly. Expression accessors and bundles are also reusable, but downstream packages may keep their own packaged expression artifacts until row-set, value, provenance, and QC contracts are parity-clean for the specific accessor they want to replace.

Pan-cancer table

expression.pan_cancer_expression() defaults to oncoref's entity-first schema: HPA normal tissue columns are <tissue>_nTPM_raw, TCGA source/provenance columns are <CODE>_FPKM_raw, deterministic TCGA TPM companions are <CODE>_TPM_raw, and analysis columns append _clean, _hk, _percentile, or _log1p. For migration code that needs pirlygenes' unsuffixed column names, use column_style="pirlygenes"; the legacy to_tpm=True keyword is accepted as a compatibility alias for that view and maps the default call to normalize="tpm". The pan-cancer view also emits raw-TPM companion columns for member-backed grouping/source-scope references (NET, CRC, NSCLC, BTC, SGC) by pooling the selected cancer-reference-expression summary rows with n-sample weights. Existing directly sourced columns, including SARC and OV, keep their current source-table behavior.

Cohort reference expression

expression.cancer_reference_expression() returns cohort-level tumor reference expression with stable long or wide output. It accepts canonical cancer codes, aliases, and aggregate cohorts, resolves gene filters by ENSG or symbol, and can return one or more normalization modes in one call:

  • normalize="tpm_clean" / "clean_tpm" — shipped biological clean-TPM percentiles.
  • normalize="tpm_clean_biological" — explicit name for that biological-only reference artifact.
  • normalize="tpm_clean_log1p" — stored log1p biological clean-TPM percentiles.
  • normalize="tpm_raw" / "tpm" — source-matrix raw TPM summaries recomputed through cohort_stats.

Long output includes source/provenance columns by default, including source cohort, source project/version, tumor origin, source type/unit, source scale class, reference method, selected source gene/sample counts, DATA_VERSION, and SOURCE_MATRIX_VERSION. This accessor is the compatibility surface for reference-expression reads; expression artifact row-set/value parity is tracked separately in the upstream parity issues.

reference_source="artifact" is the historical default: clean/log clean TPM comes from shipped percentile shards, and raw TPM is recomputed from source matrices. reference_source="summary_rows" uses the shipped cancer-reference-expression per-source sidecars when sample_qc="all" and selects one source per cancer code by a deterministic richest-source-wins rule: most genes, then most samples, then primary before mixed/metastatic, then source cohort name. For sample_qc="pass" or "pass_or_warn", the summary-row source selector intentionally recomputes via cohort_stats(..., sample_qc=...) so QC-filtered reference-expression views are shaped at read time rather than by a build-time drop. This keeps the source sidecars as all-sample evidence while allowing downstream code to ask for QC-passing summaries without maintaining a private filtered bundle.

Use reference_source="summary_rows_all", sample_qc="all" when downstream code needs the full source-union table rather than one selected source per cancer code. This long-only mode returns one row per gene, cancer code, normalization, and source cohort; even with include_provenance=False, it keeps source_cohort and sample-count columns because they are part of the source-row identity. With provenance enabled, it also preserves sidecar fields such as processing_pipeline and notes. It accepts source_kind=..., source_cohort=..., exclude_microarray_proxy=True, and pool=True for an explicit n-sample-weighted pooled view. Because these sidecars are all-sample artifacts, this source-union mode intentionally rejects format="wide", sample_qc="pass", and sample_qc="pass_or_warn"; QC-filtered all-source reference artifacts remain part of the expression-artifact rebuild work.

For compatibility with pirlygenes reference-expression consumers, the accessor also exposes the gene-to-proteoform bridge columns on every long-form row: Proteoform_ID is the cDNA/read-recovery identity that the row maps to, and Member_Ensembl_Gene_IDs is the row's member ENSG list. Without a collapse flag this is only an annotation; it does not fold rows. Use collapse_cdna_identical=True for the read-recovery space: byte-identical CDS groups plus the curated proteoform-collapse overrides. Use collapse_protein_identical=True for the genome-wide identical-protein space. Set at most one. These modes sum expression, q1, and q3 in linear TPM space inside each source context and leave wide output in the historical Ensembl_Gene_ID, Symbol, value-column shape.

Availability and missing data

Use expression.cancer_reference_expression_availability() before delegating a downstream reference-expression accessor that must distinguish unavailable oncoref artifacts from empty gene filters. It returns one row per requested code/mode with requested_code, expanded cancer_code, request_kind, available, missing_reason, provenance fields, and the reference-expression schema/data versions. expression.cancer_reference_expression(..., on_missing="empty") returns a schema-stable empty frame and stores the same missing rows in df.attrs["missing_requests"]; on_missing="raise" fails fast for required cohorts. include_request_metadata=True adds request/availability columns to long expression output, which is useful when a requested aggregate expands to child expression cohorts.

Representative evaluation partitions

Representative vectors are real source samples. A model trained on all of them must not report accuracy on those same vectors as an estimate of generalization. Use expression.representative_partition_manifest() to obtain the released training and evaluation assignment before fitting or benchmarking:

from oncoref import representative_partition_manifest

partition = representative_partition_manifest()
train_ids = partition.loc[partition["partition_role"] == "train", "representative_id"]
validation_ids = partition.loc[
    partition["partition_role"].isin(["validation", "validation_external"]),
    "representative_id",
]

The physical source_group_id, not the displayed representative ID or cancer label, is the partition unit. Parent labels, subtypes, and compatibility aliases backed by the same sample therefore always receive one shared role:

  • train — eligible for model fitting.
  • validation — deterministic within-cohort holdout.
  • validation_external — a complete independent source project held out when a cohort spans projects.
  • audit_only — retained for inspection but excluded from fitting and evaluation because its source group is not benchmark-eligible.

The ordinary within-cohort policy holds out up to two independent groups, never more than half of a cohort. A usual five-representative cohort therefore has three training and two validation groups. When a cohort spans independent source projects, the smallest viable project is instead held out whole and labeled validation_external. A cohort with only one eligible group remains train-only and reports partition_status="insufficient_independent_groups" instead of claiming validation coverage. partition_policy_version makes the exact assignment contract release-visible.

Derived artifact fields

Representative and percentile artifact readers have explicit downstream-facing contracts:

  • expression.representative_cohort_samples(..., format="long", include_provenance=True) includes the representative id, source cohort/project, source sample id and stable source-group id, source diagnosis/morphology when a sample has been reviewed, effective QC status/reasons, source scale class, linear-TPM and absolute-floor comparability flags, representative role and benchmark eligibility, partition role/status/policy, review evidence, cohort sample count, deterministic selection rank/method/basis, artifact schema version, DATA_VERSION, and SOURCE_MATRIX_VERSION. Treehouse PolyA parent, subset, and annotation-derived cohorts share one physical sample namespace, so aliases of the same source vector receive the same source_group_id even when their displayed source cohorts differ. Public representative ids default to pirlygenes-compatible CODE_rep01 columns/values. Pass representative_id_style="internal" to expose the underlying bundle/provenance ids (CODE__rep1). Representatives are selected by central-medoid plus farthest-first traversal in log1p biological clean-TPM space, with stable sample-id tie-breaking; the persisted vectors remain full clean_tpm_16_9_75.
  • expression.representative_cohort_availability() returns one row per shipped cohort with the same QC/scale qualification and a machine-readable availability reason. available_representative_cohorts(linear_tpm_comparable=True, benchmark_eligible=True) gives a fail-closed classifier-ready cohort list while retaining proxy cohorts such as MTC for rank/percentile workflows.
  • expression.cohort_gene_percentiles(..., include_provenance=True) appends the cohort code, normalization, expression unit, percentile basis, artifact schema version, DATA_VERSION, and SOURCE_MATRIX_VERSION.
  • Gene-level representative and percentile readers default to canonical oncoref ENSG IDs. For pirlygenes migration wrappers, pass gene_id_style="pirlygenes" to present known one-to-one remapped_to_oncoref rows with their legacy pirlygenes ENSG IDs. This is intentionally a presentation shim: it does not synthesize missing rows or alter expression values.
  • Gene-level reference, representative, and percentile readers default to gene_universe="artifact", which preserves the exact shipped row set. Pass gene_universe="tumor_signal" to drop rows explicitly audited as oncoref-only filterable extras for the requested artifact/cohort: strict technical extras plus biotype-resolved non-signal extras such as pseudogene, small-RNA, and immune-receptor segment rows. Protein-coding and lncRNA oncoref-only rows are retained as biological extras. Pass gene_universe="pirlygenes" only for migration parity: it starts from the tumor-signal policy, then also drops audited oncoref-only biological or unresolved extras unless the row is a documented remap target for a pirlygenes legacy ENSG ID. Combined with gene_id_style="pirlygenes", this can alias-expand a documented remap row when current pirlygenes exposes both the legacy and canonical ENSG IDs for the same measured vector. This reproduces pirlygenes row-universe expectations without inventing missing expression measurements. Pass include_gene_universe_flags=True for long reference output or any representative/percentile output to append row-level artifact_row_class, is_filterable_extra, is_technical_extra, is_missing_biological, and recommended_consumer_action columns. These options filter or label known artifact row classes; they never invent missing biological expression rows and only the explicit pirlygenes mode drops biological oncoref-only extras.
  • Representative and percentile readers default to sample_qc="pass" and validate any shipped expression-artifact-build-metadata.csv rows before returning a precomputed shard. If the metadata says a shard was built with sample_qc="all" or another policy, the reader raises rather than silently treating the shard as QC-pass. Use sample_qc="artifact" only for explicit legacy/audit reads where the caller wants exactly whatever policy the bundle used. Metadata-missing legacy bundles remain readable but expose df.attrs["artifact_sample_qc_verified"] = False.
  • Gene-level reference, representative, and percentile readers attach df.attrs["gene_universe_delta_summary"] and df.attrs["gene_universe_delta_n"] for the requested cohort/product. These attrs summarize the known pirlygenes/oncoref row-universe deltas that still apply to the returned artifact, so migration wrappers can separate remapped rows, missing upstream data, and intentional oncoref-only rows without reimplementing the audit-table matching logic.
  • Missing percentile shards still raise by default. Use on_missing="empty" to return an empty but schema-stable frame with df.attrs["missing_reason"], which is useful for compatibility adapters that need to distinguish unavailable upstream data from private downstream fallback data.

Bundle contents and gene-universe parity

The QC-policy expression bundle ships representative, percentile, within-sample, CTA-scope proteoform percentile, CTA-scope proteoform within-sample, sample-QC, and build-metadata artifacts. Non-shipped proteoform scopes can still recompute from cached source matrices. Row-set and value parity with pirlygenes is still governed by the gene-universe and expression-artifact parity issues.

expression.expression_artifact_gene_universe_deltas() exposes the known pirlygenes/oncoref row-universe deltas from the current parity audit: canonical remaps such as legacy PAXX to its oncoref ENSG, sequence-identical representative-sample remaps where the measured oncoref artifact row can be presented under the pirlygenes legacy ID, and the full current set of oncoref-only representative extras. The prior broad unresolved_oncoref_extra bucket is resolved where possible by current oncoref gene metadata into strict technical extras, non_signal_oncoref_extra rows that gene_universe="tumor_signal" filters, biological_oncoref_extra rows that stay visible in the default and tumor-signal views, sequence_identical_remapped_to_oncoref rows that gene_id_style="pirlygenes" can present or alias-expand without inventing missing expression measurements, or a small remaining unresolved set with no current biotype. In the current audit table, no rows remain flagged as missing biological; only 29 oncoref-only rows remain truly unresolved_oncoref_extra. The resolved status labels are deliberately explicit so downstream wrappers do not need to infer policy from biotypes. Use expression.expression_artifact_gene_universe_delta_summary() for counts by product/cohort/status, or expression.expression_artifact_gene_universe_delta_report(product, cancer_types) for the compact request-scoped report used by accessor attrs. These tables include gene_biotype, artifact_row_class, is_filterable_extra, is_technical_extra, is_missing_biological, and recommended_consumer_action so current-bundle row classes do not have to be inferred from prose. Use expression.expression_artifact_technical_extra_gene_ids(...) to get the oncoref-only technical-extra ENSG IDs for a product/cohort filter. This surface is intentionally provenance: it makes differences explicit for migration code, but does not synthesize missing expression rows or alter artifact values.

Clean TPM compartments

Clean TPM has one public compartment contract:

  • clean-tpm-censored-genes.csv:category == "ribosomal_protein" — 16% ribosomal compartment.
  • clean-tpm-censored-genes.csv:category == "technical" — 9% other-technical compartment.
  • genes absent from the censored table — 75% biological compartment.

The category-specific helper sets are available from oncoref.gene_families:

from oncoref import gene_families

gene_families.clean_tpm_ribosomal_gene_ids()
gene_families.clean_tpm_other_technical_gene_ids()
gene_families.clean_tpm_censored_gene_ids()

Housekeeping normalization

For clean-TPM housekeeping denominators, use the biological HPA-stable panel:

gene_families.clean_tpm_biological_housekeeping_gene_ids()
gene_families.clean_tpm_biological_housekeeping_genes()
gene_families.clean_tpm_biological_housekeeping_genes(primary_only=False)

Housekeeping normalization is defined as a median-of-ratios size factor against a fixed, versioned per-gene reference profile:

from oncoref import normalization

normalization.housekeeping_reference_profile()
normalization.tpm_to_housekeeping_normalized(matrix)

For each sample, oncoref computes:

size_factor = median(housekeeping_clean_tpm[g] / reference_tpm[g])
normalized_expression[g] = clean_tpm[g] / size_factor

The default reference is the HPA v23-derived clean-TPM biological housekeeping panel (HOUSEKEEPING_REFERENCE_PROFILE_VERSION). This is a sample-scale estimate relative to a fixed biological HK profile, not the old "divide by the panel's geometric mean" ratio. Prefer log1p clean TPM or percentile-rank clean TPM unless the analysis specifically needs an HK-derived size factor.

The old geNorm-style denominator is deliberately buried behind method="legacy_geomean" for explicit audits of historical outputs. The shorter method="geomean" spelling is not accepted.

The legacy qPCR/reference-gene panel remains available as legacy_qpcr_housekeeping_* and through the historical housekeeping_* helpers, but it is not the clean-TPM biological denominator.

Genes and Proteoforms

  • oncoref.gene_ids — bundled canonical Ensembl gene space, cross-release alias resolution, symbol synonyms, and biotype checks.
  • oncoref.genome — optional (pip install 'oncoref[genome]') pyensembl-backed gene/transcript lookup against installed Ensembl releases.
  • oncoref.proteoforms — identical-protein paralog grouping and expression collapse helpers.
  • oncoref.gene_qc / oncoref.gene_families — technical-RNA and gene-family classification used by normalization. These are normalization/QC reference families, not the general home for pirlygenes marker panels.

Burden, TMB, Fusions, and Signatures

  • oncoref.tmb — tumor mutational burden reference values. tmb.cancer_tmb_df() includes evidence-schema columns (estimate_type, source_scope, missing_reason), and tmb.cancer_tmb_record() / tmb.resolve_tmb_source() preserve requested-code metadata for source-scoped lookups such as COAD_MSI or READ_MSI resolving through CRC_MSI. Direct audited gaps use inheritance_kind="direct_missing" so callers can distinguish “known no supported site-specific estimate” from an unmapped cancer code.
  • oncoref.incidence — incidence/mortality burden and burden categories. incidence.cancer_burden_df() is the auditable burden table: percentages are the public lookup values, and raw-count, source-locator, source-site, derivation, and rounding columns are preserved as provenance fields. Existing legacy rows keep locator status values such as not_extracted until exact ACS/GLOBOCAN table/export anchors are filled in.
  • oncoref.fusions — defining fusions and partner-family lookups.
  • oncoref.response_signatures — legacy/compatibility response-signature surface used by oncoref plots. Treat it as transitional: new or extended therapy-response signature panels belong in pirlygenes unless they are recast as source-anchored empirical fact/provenance rows.

Data Management

Dataset catalog

  • oncoref.catalog — unified dataset inventory and fetch/status/path operations.

Expression bundle

  • oncoref.data_bundle — heavy expression bundle cache. Use data_bundle.bundle_contract() to inspect the downstream-stable package/data version linkage, release asset URLs, cache environment variables, completion marker policy, and expected artifact inventory for the active bundle. The inventory includes the generated sample-QC manifest, per-cohort build metadata, within-sample prevalence shards, and CTA-scope proteoform percentile/prevalence shards, not just the legacy pirlygenes expression tables. data_bundle.bundle_is_local() reports whether the entire downloadable cache is populated. For a side-effect-free check of one required artifact across both an in-repository/package data directory and a partial cache, use data_bundle.item_is_local(path) or data_bundle.find_local_item(path). These item-level probes never fetch data and reject empty files or directories. Use data_bundle.bundle_release_manifest() to fetch and validate only the small release manifest/checksum for the active DATA_VERSION, including tarball sha256 plus any artifact inventory, builder commit, source-matrix version, and sample-QC policy metadata published with the release. Use data_bundle.bundle_metadata() when a downstream package needs one no-heavy-download JSON object containing the static contract, local cache path and completeness state, local artifact inventory, and validated release manifest. CLI equivalents are available for CI/notebooks: oncoref data contract prints the static bundle contract, oncoref data metadata [oncoref|pirlygenes] prints the composed dependency state, and oncoref data release-manifest [oncoref|pirlygenes] prints only the validated release manifest/checksum metadata.

HPA data

  • oncoref.reference_data / oncoref.hpa — HPA reference-data cache and HPA tissue/cell-type accessors.

Compatibility Modules

These modules remain importable but are less discoverable than the organized facades above:

  • oncoref.apd1 — legacy anti-PD-1 response slice; prefer oncoref.ici_response.
  • oncoref.ici — core ICI implementation; prefer oncoref.ici_response for the organized public surface.
  • oncoref.coverage — original mixed CTA/generic antigen-panel coverage module; prefer oncoref.cta_coverage or oncoref.antigen_coverage.
  • oncoref.peptides — original CTA-specific 9-mer module; prefer oncoref.cta_peptides.