Pilot 7 Synthetic Data Update

Building benchmark and ground truth from EDC extraction to SDTM and ADaM

Yilong Zhang, Peng Zhang, Ming-chun Chen, Bingjun Wang

2026-09-25

Disclaimer

The views and opinions expressed in this presentation are those of the individual presenters. They do not represent those of their affiliated organizations or institutions.

Contributors

  • Peng Zhang CIMS Global
  • Bingjun Wang Genentech
  • Ming-chun Chen Summit Therapeutics
  • Yilong Zhang BBSW AI Committee

Goal 1: establish benchmarks

  • ODM -> SDTM -> ADaM
  • Minimum viable
    • Specification
    • Representative source data
    • Golden output
  • Human baseline
    • Experts reviewed before finalization
    • Clear review and decision making history

Goal 2: establish ground truth from full studies

  • Build ground truth once. Reuse it to evaluate every agentic workflow
  • CDISC pilots (SDTM -> ADaM)
    • cdiscpilot01 and RConsortium submission pilots 3, 5, and 6
  • OpenClinica real sample data (ODM -> SDTM -> ADaM)
    • Real sample data for testing the full data journey
  • Synthetic studies (ODM -> SDTM -> ADaM)
    • KN564 and KN189 as scalable test beds

Where we are

~150 Benchmarks
SDTM and ADaM derivation benchmarks
6 Studies
PR ready for pilot 3, 5, and 6

https://elong0527.github.io/yamaa/benchmark/adam-adsl-age-group.html

Scroll inside the frame to explore the spec, inputs, and golden output

Explore all benchmarks

Help: finalize benchmarks

  • Review a benchmark: inspect the specification and golden output
  • Add comments: challenge assumptions, clarity, and conformance
  • File issues: record anything ambiguous or incorrect
  • Open pull requests: contribute fixes and new benchmarks

Help: creating benchmarks

  • Assess privately: Run yamaa against a proprietary submission package inside your own environment.
  • Share safely: Contribute a synthetic benchmark or an issue that captures the pattern without disclosing source data.

Help: broadcasting

The next useful contribution is one reviewed benchmark.

Help publish an RConsortium blog to attract contributors to finalize benchmarks.

Introduction: yamaa

yamaa: four components

Component Purpose Links
Schema Declares the vocabulary of the specification. Repository / Introduction
Rules Defines the meaning of each specification. Repository / Rules
Engine Runs specifications in Python and R according to the rules. Python / R
Benchmark Provides runnable specification examples. Repository / Dashboards

yamaa: one principle

Determinism is a design constraint

A yamaa specification has exactly one execution. Where it would have two, yamaa fails instead of choosing.

Ambiguity is surfaced during planning, before it becomes a silent difference in the output. (e.g. rounding, missing value handling)

  • For people: the specification is readable.
  • For agents: the rules avoid ambiguity.

Inheritance

Inheritance reduces duplication while keeping every resolved study specification self-contained and reviewable.

From benchmark to deployment

Ongoing collaboration with keiji.ai on a user-friendly agentic workflow.

Reference

Finalize one benchmark the next contributor can trust.