The reference & how it is repaired
The global task (Track 1 · Subtrack 1) scores your alignment against a gold reference. Because that reference is derived from UMLS and Mondo, it can be logically incoherent — so we repair it. This page explains the idea; the evaluation metrics page has the formal definitions.
Why a gold reference needs repairing
The reference equivalence mappings are assembled from the UMLS Metathesaurus and Mondo. Each source is reasonable on its own, but merging them can create logical incoherence: taken together with the two ontologies' own axioms, some reference mappings imply that a class must be both something and its own disjoint opposite — an unsatisfiable class. A system that reproduced such an "incoherent" reference exactly would be rewarded for producing a logically broken alignment. So before scoring, the reference is repaired into a coherent one.
The repair: a union of what three repair tools remove
Rather than trust any single repair tool, we take the union of the mappings removed by three independent, well-established systems run over the reference:
- ALCOMO (under the ELK reasoner) — a dedicated alignment-debugging tool;
- LogMap (its repair/DEBUG mode) — which can also weaken a mapping's relation rather than delete it;
- AML (AgreementMakerLight) — a structural repair.
A mapping is treated as problematic if any of the three removed it. This union is deliberately conservative: it errs toward flagging a mapping rather than silently keeping an incoherent one.
Annotation: keep, weaken, or set aside
We do not simply delete flagged mappings — that would throw away information. Instead, each reference mapping is annotated:
Standard reference
Complete, but possibly incoherent. Every mapping is present as
=. Scored with traditional precision / recall / F1.
Repaired reference headline
Coherence-aware. Survivors keep their relation; fully-removed mappings become
? and are ignored from scoring.
- Survivors keep their relation. A mapping no tool removed stays as
=; where LogMap weakened a mapping, the reference keeps that weaker</>(subsumption takes priority over deletion). - Fully-removed mappings become
?(“unknown”). They are neither correct nor incorrect — they are simply set aside during scoring. owl:deprecatedclasses are dropped from both the reference and predictions as out-of-task.
What this means for scoring
Systems are scored against both references:
- Against the standard reference — traditional P/R/F1 (every mapping counts).
- Against the repaired reference — coherence-aware and
relation-agnostic:
?mappings are ignored from both the precision and recall denominators, and a reference</>is credited by a predicted correspondence of any relation.
The repaired, coherence-aware F1 is the headline for the global leaderboard, reported with a reasoner-checked Global Coherence score (does your own alignment leave the merged ontology satisfiable?). See the evaluation metrics and the baselines for the numbers — including the coherence of the reference before and after repair.