Evaluation Metrics

This page summarises how OAEI Bio-ML submissions are scored, per track. A full description appears in the supplementary material (available at track launch).

Subtrack 1 — Global equivalence alignment

Each submission is a full alignment per pair (full OWL IRIs). It is scored with precision, recall and F1 against two references:

• Repaired reference (headline). Coherence-aware and relation-agnostic: correspondences flagged as uncertain (?) are ignored from both the predictions and the reference, and a reference subsumption (< / >) is credited by a predicted correspondence of any relation. The headline score is the repaired, coherence-aware F1.
• Standard reference (secondary). The complete, possibly-incoherent reference scored with traditional P/R/F1.

The two references are not directly comparable. They differ in both membership and scoring rules, so a repaired-vs-standard difference is not an accuracy delta — read each within its own reference.

Global Coherence. Alongside P/R/F1 we report a reasoner-checked coherence measure — the degree of logical incoherence induced by the submitted alignment on the merged ontologies (0 = coherent). Because it needs a description-logic reasoner, coherence is computed organiser-side, not in the participant scoring kit.

Note: as coherence-aware precision, recall, and F1 are headline measures, we only report those here; both the standard (i.e., unrepaired) and the repaired reference are scored via CodaBench.

Subtrack 2 — Local equivalence ranking

For each source entity, the system ranks a fixed candidate pool best-first. Writing $\mathrm{rank}(q)$ for the 1-based position of the correct target in query $q$‘s ranking, over the query set $Q$:

Reported: MRR and Hits@{1,5,10}. MRR rewards placing the correct target as early as possible (rank 1 scores 1, rank 2 scores 1/2, and so on); Hits@$k$ is the fraction of queries whose correct target lands in the top $k$.

Averaging

Every headline metric is macro-averaged over the three pairs (NCIT–DOID, SNOMED–FMA, SNOMED–NCIT) — the unweighted mean of the per-pair values, so a system must do well on all three rather than being carried by the largest.

Timeline

The evaluation window and reporting dates (all 00:00 Anywhere on Earth):