Towards an AI-First Workflow for Clinical Development and its Analysis and Reporting

Starting from CDISC Data Standardization

Yilong Zhang
Pingye (Eric) Zhang

81st Annual Deming Conference on Applied Statistics

2026-12-07

Welcome

Outline

The course runs about three hours, with a break after Parts 1 and 2.

Topic Instructor
Part 1 AI agents and AI-first workflow Yilong
Part 1 Agentic CDISC data standardization Yilong
Break
Part 2 Giving agents the right context for domain-specific workflows Eric
Break
Part 3 Benchmarks Yilong
Part 3 Case study: reviewing benchmarks Eric and Yilong

Disclaimer

The views and opinions expressed in this presentation are those of the individual presenters. They do not represent those of their affiliated organizations or institutions.

Training objectives

This workshop aims to help you:

  • Understand the building blocks of an agentic workflow, with CDISC data standardization as one example.
  • Recognize where an AI agent’s performance may be improved.
  • Get started with creating and reviewing benchmarks that assess AI agents.

Note

Toolchains, processes, and formats differ across organizations. This workshop shows one possible approach, not the only one.

Resources

  • During the workshop, we will use three projects:
    • yamaa: tools and benchmarks for CDISC data standardization.
    • R Consortium Pilot 7 project: synthetic trials that demonstrate CDISC data standardization.
    • ai4csr: the book and slides for this short course.
  • Post your questions in the group chat.

AI agents

You will: name the parts of an AI agent and say what each one can and cannot do.

Example: arena.ai

Try it at arena.ai/agent

Use the Pilot 5 ADSL dataset to generate a baseline characteristics table using R and gtsummary.

Data: https://github.com/RConsortium/submissions-pilot7-synthetic-data/tree/main/submission-pilot5/data/adam

To finish, the agent has to:

  1. Find adsl.parquet in the folder.
  2. Read it in R.
  3. Pick the population, arms, and grouping variables.
  4. Build the table with gtsummary.
  5. Return the table and its code.

What to check: the prompt never names the population, variables, or statistics, so the agent has to guess them. Do they match your table shell?

Example: Muse

Try it at muse.ai

Muse is a personal agent from Meta. “Give Muse a goal or task and it handles the rest”: sort email, book a dinner, or fill out a form.

Muse also helps me maintain open-source projects in the background:

  • Identify project gaps from published PHUSE papers.
  • Bring new benchmark ideas for CDISC standardization.

What is an AI agent?

What is a large language model?

An LLM only predicts the next token from its context. It cannot open a file or run code: a harness does that and puts the results back into the context.

What is a harness?

The LLM is like the CPU; the harness is like the operating system.

Reference: Agent vocabulary

Term Meaning
Prompt The request, with every rule you wrote into it.
Token A word or part of a word: the unit a model reads and writes.
Context The ordered tokens the model sees in one call: prompt, files read, tool results.
LLM Predicts the next token from the context.
Harness Builds the context, calls the model, runs tools, logs each step: OpenCode.
Tool A program the harness runs for the model: read a file, run R.
Skill Task instructions the harness loads: the agent’s SOP.
Agent LLM, harness, and tools, looping until the task is done.

AI-first workflow

You will: design an AI-first workflow and the principles that keep it trustworthy.

A mental game

Shaping an AI-Native Biometrics Group from Day 0

What would you do?

You have been hired by the CMO to build a biometrics group.

Blank Slate

  • No legacy systems to migrate.
  • No “we’ve always done it this way” culture.

Rare Opportunity

  • Build an AI-native group from scratch.
  • Skip the “digital transformation” pain.

Let’s brainstorm the strategy for a small- to midsize biopharma company.

What is AI-native?

AI-native is an organization whose workflows are mostly AI-first and connected through shared, inspectable artifacts.

AI adoption AI-first AI-native
AI added to existing work AI is the default starting point AI is a core collaborator
AI helps with isolated tasks; the workflow does not change. AI starts and executes the tasks of some workflows. AI-human collaboration is designed across workflows.
Example: ChatGPT drafts an email or summarizes a document. Example: every analysis starts with a prompt to an AI agent. Example: AI agents run CDISC data standardization end to end.

AI-native is a mindset, not a tool: every team member orchestrates AI agents, handles the exceptions, and keeps improving the system.

Design an AI-first workflow

Let’s design an AI-first workflow for CDISC data standardization.

Brainstorm: where to start building

You will build an AI agent for this workflow. Where do you start, and why?

Opinionated proposal

  • As a Biometrics organization, focus on defining the workflow.
  • Leverage internal and external AI infrastructure, with proper skills and tools.

Agentic workflow design principles

Primary goal: trustworthy assistance. The agent’s work can be inspected, repeated, challenged, and reviewed.

Principles Goal
Plain text first Inspectable artifacts
Single source of truth Explicit state
Deterministic first Tools and evidence
Agent-specific identity Attributable actions

For CDISC data standardization, we recommend:

  • Do not use the annotated CRF as an input; generate it as an output.
  • Do not use a specification in Excel, Word, or PDF as an input; use a plain-text specification.

Agentic CDISC data standardization

You will: see why agents disagree on CDISC derivations and how a reviewed, plain-text specification removes the guesswork.

Contributors

  • Peng Zhang CIMS Global
  • Bingjun Wang Genentech
  • Ming-chun Chen Summit Therapeutics
  • Jingwei Xiong Merck & Co., Inc.
  • Yilong Zhang BBSW AI Committee

Motivation

  • Two AI agents derive the same analysis dataset from the same specification (example from arena.ai).
  • Both report success, but their code and outputs may differ.
  • A reviewer must resolve every difference before signing off.

From an FDA warning letter.

“you must review the AI-generated documents to ensure they were accurate and actually compliant…”

One root cause

  • The user prompt leaves business rules implicit, so agents must infer them.
  • Business rules can differ from language defaults:
    • R and Python round ties to even.
    • SAS compares missing numeric values as lower than non-missing values.

Evidence: prompt completeness decides success

Prompt tier Python R opensas
Full 99.3% (133/134) 98.5% (132/134) 86.6% (116/134)
Conventions 35.8% (48/134) 33.6% (45/134) 34.3% (46/134)
Brief 18.7% (25/134) 17.9% (24/134) 15.7% (21/134)

Same 134 tasks, same agent; only the prompt changed. Writing the business rules into the prompt lifts the pass rate from under 20% to over 85%.

opensas: https://github.com/kirha-ai/opensas
Agent: opencode 1.18.33. Model: opencode-go/muse-spark-1.3-contributor, 1 attempt, Oct 12 2026

Mean arterial pressure: three prompts

Shared Prompt:

  • create an output ADVS from the provided input ADVS, following CDISC ADaM standards; one record per subject, visit, and parameter.
  • STUDYID, USUBJID, PARAMCD, PARAM, AVAL, VISIT, DTYPE
  • read from /app/input; write /app/output/advs.csv.
Rule in the prompt Full - passes Conventions - passes Brief - fails
MAP identifiers PARAMCD = "MAP"
PARAM = "Mean Arterial Pressure (mmHg)"
Same MAP identifiers Not stated
Records to add One per subject and visit with both SYSBP and DIABP; none when either is absent or missing. Not stated Not stated
AVAL Two-thirds DIABP plus one-third SYSBP;
not rounded.
Not rounded;
formula not stated.
Not stated
DTYPE
(sponsor convention)
Added record: CALCULATION.
Collected records: no value.
CALCULATION or no value;
assignment not stated.
Not stated

Brief omits both the MAP formula and the rule for adding a record.
The ADaM IG uses DTYPE for records derived differently within a parameter; setting it on a whole derived parameter is a sponsor rule the agent cannot infer.
SYSBP = systolic blood pressure; DIABP = diastolic blood pressure.
https://elong0527.github.io/yamaa/benchmark/adam-advs-map.html

Goal

  • Reduce human review effort.
  • Leverage organization templates.
  • Capture business rules that AI agents can apply.
  • Move human-AI collaboration to the planning stage.

yamaa: collaboration at the planning stage

Human and AI review the specification during planning; the engine executes the agreed rules.

yamaa: YAML specification example

yamaa: "1.0"
type: spec
model: ADaM
domain: ADSL
keys: [STUDYID, USUBJID]
input:
  SOURCE:
    path: input/adsl.csv
    types: {HEIGHTCM: float, WEIGHTKG: float}

output:
  path: adsl.csv
  columns: [STUDYID, USUBJID, BMI]

columns:
  - {name: STUDYID, type: str, label: Study Identifier, derivation: SOURCE.STUDYID}
  - {name: USUBJID, type: str, label: Unique Subject Identifier, derivation: SOURCE.USUBJID}
  - {name: HEIGHTCM, type: float, label: Height (cm), derivation: SOURCE.HEIGHTCM}
  - {name: WEIGHTKG, type: float, label: Weight (kg), derivation: SOURCE.WEIGHTKG}
  - name: BMI
    type: float
    label: Body Mass Index (kg/m2)
    derivation: "WEIGHTKG / POWER(NULLIF(HEIGHTCM, 0) / 100, 2)"

yamaa: core principle

A yamaa specification has exactly one execution. Where it would have two, yamaa fails instead of choosing.

output_dataset = derive(input_datasets, spec)

The specification is a contract between people and AI agents:

  • spec: written in the closed vocabulary of the schema.
  • derive: applies each business rule with one defined meaning.

yamaa: where the uncertainty lives

The AI agent is probabilistic, so people review the specification. The engine that executes it is deterministic: the same specification and inputs give the same datasets.

yamaa: inheritance

Inheritance reduces duplication while keeping every resolved study specification self-contained and reviewable. (example)

Summary of the workflow

People and AI agents collaborate on one version-controlled specification at the planning stage, while the study is ongoing.

Giving agents the right context for domain-specific workflows

You will: see two ways to give an agent the right context for a domain-specific job — a workflow, shown with medical monitoring; and a skill, shown with group sequential trial design.

Eric Zhang

Topic 1: A workflow for medical monitoring

Good agentic workflows start with a good harness

The harness is everything around the model: structure, memory, delegation, and checks. Build that well, and the workflow takes care of itself.

  1. Folder structure — one shape per study
  2. CLAUDE.md / AGENTS.md — standing instructions, root + per-study, read every session
  3. Audit trail — three sources together: git, session log, and the platform’s system log
  4. Git — what changed, diffable and reversible
  5. Session log — memory, why it changed, enforced by a hook
  1. Sub-agents — one per task in the workflow, split context
  2. Skills — domain knowledge, encoded once
  3. Evaluation — methodological independence, humans stay accountable
  4. Worked example — the daily dashboard workflow, step by step

One folder per study, one shape for every study

~/                             workspace root
|-- CLAUDE.md                  workspace rules, read every session
|-- STUDY-001/
|-- STUDY-002/                 one self-contained folder per study
|-- STUDY-003/
|   |-- CLAUDE.md                verified study quirks, auto-loaded here
|   |-- data/<snapshot>/         dated EDC cuts, read-only
|   |-- scripts/                 pipeline, report, weekly refresh
|   |-- output/                  dated reports, publish staging
|   `-- docs/session_log.md      append-only history
|-- .claude/
|   |-- agents/                 separate sub-agent(s) per study
|   |-- skills/                 reusable playbooks
|   `-- hooks/                  session-log enforcement
`-- ...
  • Self-contained. A study owns its data, scripts, outputs, and docs.
  • Isolated. Nothing leaks between studies: one study’s quirks, fixes, and history never enter another’s context.
  • Separate sub-agents per study. Each study gets its own agent — often more than one, e.g. build and validate — so its context only ever holds that study’s facts.
  • Repeatable. A new study is the same shape, copied.

Physical isolation is what makes the context isolation reliable: an agent working in STUDY-003/ cannot accidentally see STUDY-001/’s data, fixes, or history.

Initialize with one prompt

initialize this folder

(in Claude Code — or the built-in /init command)

Result: CLAUDE.md — what the workspace is, how it’s laid out, and the rules every session must follow.

  • Standing instructions. Read at the start of every session — no re-explaining.
  • A draft, then a contract. The prompt drafts it from what’s already in the folder; you edit it into your rules.
  • Grows with decisions. Layout, data rules, required workflows — every decision worth keeping gets written in.
  • Root + per-study. A short root file for workspace-wide rules; each study folder gets its own for that study’s quirks, auto-loaded only there.

Chat forgets. CLAUDE.md doesn’t.

The audit trail: three layers

Layer Answers Written by
Git What changed? The agent, one commit per change
Session log Why did it change? The agent, enforced by a hook
Platform system log Who was in the system, and when? The platform itself (e.g. Workbench), independent of the agent

No single layer has to be trusted alone: what the agent changed, why it changed it, and who could have touched the system at all — from three different sources.

The platform’s own system log needs no building: logins and sessions are already recorded by the infrastructure (e.g. Posit Workbench), independent of anything the agent does. Git and the session log are what you build on top of it.

Git: what changed

$ git log --oneline -5
a1b2c3d fix: detect the newest snapshot instead of a hardcoded default
9f8e7d6 feat: add a PP record to ADVS per the study SAP
7c6d5e4 fix: round AVAL per protocol convention, not language default
3b2a1f0 docs: record the baseline-derivation rule in CLAUDE.md
0e9d8c7 chore: initialize STUDY-003 from the workspace template
  • Every change is a commit. Code, CLAUDE.md, skills, specs — one history, diffable and reversible.
  • Same discipline as a human PR. The agent stages named files and writes a message; a human’s say-so is still required for anything destructive — force-push, reset --hard, skipping hooks.
  • One source of truth. The same principle behind yamaa’s version-controlled specification applies to the whole workspace.

If a change isn’t committed, it isn’t real yet.

Give the agent a memory — a log file

STUDY-003/docs/session_log.md          append-only
...
## 2026-11-09
- Snapshot mislabel: the "Nov 9" refresh read the Nov 3 cut.
  Root cause: hardcoded snapshot default; refresh passed no argument.
  Fixed: newest snapshot detected and passed explicitly.
- Rebuilt on the correct snapshot; flags 43 -> 50 (3 resolved, 4 new).
- Independent validation (10th pass): PASS -- zero errors.
- OPEN: access for the second reviewer.
...
  • Two layers. A hook auto-lists every file touched; the agent writes the why — decisions, fixes, open questions.
  • Enforced. After code changes, the hook blocks the session from ending until the entry is written.
  • Memory across sessions. The next session — or a colleague — reads what happened and why.

Hooks are your own scripts, run automatically on session events: build your own guardrails. Anything you want the agent to always do, make it a hook, not a reminder.

If it isn’t written down, the next session doesn’t know it.

Sub-agents: one per task in the workflow

  • You don’t hand-write these. Say “create a sub-agent for the STUDY-003 dashboard” and the main agent drafts it; you review and refine it.
  • Context never carries back up. The main thread gets only the final report — one validation run recomputed everything from raw data across roughly 270,000 tokens; the main thread received a two-page summary.
  • Validation gets deliberately less context than development — raw data only, never the pipeline code. That independence is what makes a PASS meaningful.

Encode domain knowledge as skills

~/.claude/skills/
|-- medical-monitoring-report/   the dashboard playbook
|   |-- SKILL.md                 when it triggers, the method
|   `-- references/              architecture, interactivity, validation
|-- recist-confirmation/         RECIST 1.1 confirmed-response rules
|-- group-sequential-design/     trial-design Q&A workflow
|-- gsdose/                      seamless phase 2/3 dose-selection designs
`-- km-digitizer/                digitize published KM curves
  • What is a skill? A playbook the agent loads whenever the task matches: triggers, method, references, scripts. Written once by the expert; followed every session after.
  • Example: ADaM derivation order. A skill encodes the order: build the intermediate ADINTDT before ADTTE, which depends on it — not the reverse.
  • Claude helps you write them. A built-in skill-creator skill interviews you, drafts the SKILL.md, and tests it.

A prompt asks for an outcome. A skill defines how it’s always done.

Evaluation: methodological independence

Evaluation runs on the same harness you just built:

A separate evaluator

Uses sub-agents. A fresh sub-agent that never built the thing — raw data only, never the pipeline code. It recomputes; it doesn’t re-read.

A validation skill

Uses skills. The same checks, every time: recompute every number from raw, check cross-output consistency, fix before shipping.

Failure patterns on file

Uses the audit trail. Past failures live in the session log; every new evaluation checks for the mistakes already made once.

The agent evaluates; a human signs off. Nothing ships on an agent’s PASS alone — we own what ships.

Worked example: the daily dashboard workflow

AI-native

skill: medical-monitoring-report

Cron scheduled trigger

Extract 🤖🛠 the day’s EDC snapshot

Build 🤖🛠 pipeline + lineage graph skill ref: architecture skill: RECIST 1.1

Assemble 🤖🛠 interactive HTML

Summary 🤖🛠 updates + data issues

Validate 🤖🔍 recompute from raw skill ref: validation

Adjudicate 🤖⚖️ resolve flagged items skill ref: validation

Publish 🤖 one stable URL

Notify 🤖 reviewers alerted

Git + session log — every step logged

Human review 👤 — outside the AI-native loop. Sign-off required.

Sub-agents for Build, Validate, and Adjudicate are each guided by a skill.

Across the trial: a TFL package that matures toward lock

The daily workflow on the previous slide doesn’t run once — it repeats for the life of the trial, giving two things in real time:

  • Real-time benefit-risk signal finding — a preliminary positive signal is tracked cycle to cycle; once it’s reinforced, that’s what triggers database lock for reporting, not a fixed calendar date.
  • Real-time TFL finalization — every cycle also adds to one persisting TFL package, so it’s already mature by the time the database locks, instead of being assembled from scratch after.
  • The harness itself can be packaged and shared. The AI-native harness you build in your working directory can naturally be packaged as a skill or an MCP server — served up as a plug-in another agency can connect to their own AI system.

Recap: context for the medical monitoring workflow

  • Context guides the agent on when to use which tool. CLAUDE.md, skills, and the session log don’t just hand the agent tools — they tell it when each one applies, so it reaches for the right one at the right step.
  • Context and tools are both areas domain experts can shape. For example, making your organization’s GitHub repo of templates, specs, and conventions available to the agent as a tool.
  • Skills, hook-written logs, and sub-agents sharpen that context — one skill per domain task, one log entry per hooked event, one sub-agent per task in the workflow — so every step runs with just the context it needs.

Topic 2: Building skills to bring domain knowledge into context

A skill packages domain knowledge once — then any agent can use it.

Where a workflow wires together the steps of one task, a skill teaches the agent what a domain expert already knows: the method, its assumptions, and when it applies. This topic shows it with group sequential design for survival trials.

  • What a skill is — not a tool, but a packaged how-to: methodology, templates, and the questions to ask before computing anything.
  • Why group sequential design — it’s a method where domain experts have their own design philosophy: how to collect design inputs, and in what order. A skill turns that philosophy into a playbook, reflecting how they’d approach the design themselves.
  • Worked example — turning a group sequential design skill into a structured Q&A the agent runs with you.

Demo: the group sequential design skill in action

🎥

Watch the skill run a design end to end

Q&A → R design script → simulation verification → Word report + reproducible R/Python scripts

A skill for group sequential design

Given a trial scenario, the group sequential design skill designs a group-sequential survival-endpoint trial — boundaries, sample size, and multiplicity strategy — through a guided Q&A, verifies the result by simulation, and delivers reproducible R/Python scripts alongside the Word report.

Skill structure

group-sequential-design/
|-- SKILL.md            Q&A workflow, design steps
|-- reference.md         design rules, spending fns, alpha recycling
|-- examples.md          R code by design pattern
|-- post_design.md       timing checks, verification
`-- scripts/
    |-- compute_event_prob.R
    `-- gsd_report_template.py

Output structure

output/gsd_{disease}_{endpoints}_{date}/
|-- gsd_design.R
|-- gsd_results.json
|-- multiplicity_diagram.png
|-- gsd_verification.R
|-- gsd_verification_log.md
|-- gsd_report.py
`-- gsd_report.docx

.R / .py scripts — rerun to reproduce every number in the report

Loaded progressively — each reference is only pulled into context when the workflow reaches that step.

The skill’s questions are my design philosophy

The skill asks its questions one at a time, in a fixed order. That order isn’t arbitrary — it’s what design inputs I need, and in what sequence, to walk through a group sequential design the way I actually would.

Benchmarking the skill: with vs. without

Any addition to an agent — skill, tool, workflow — should be benchmarked with-vs-without, not assumed to help. 21 design scenarios, skill vs. no-skill — overall scorecard, best-with-skill run per scenario:

With skill Without skill
Skill wins 16 —
Ties 4 (#3, #21, #60, #113) —
No-skill wins 1 (#139) —
Avg score (excl. #60, trivial) 90% 62%

1 · Does it add value?

16/21 wins, 4 ties, 1 loss (#139 — a structural gap, fix in progress).

2 · How good with the skill?

90% average vs. 62% without, at each scenario’s best run.

3 · Failure patterns

P0: a four-arm, contribution-of-components design.

Best practices for building a skill

1 · Structured Q&A completes the prompt

A skill’s intake isn’t boilerplate — it’s prompt engineering. Ask one question at a time, in the order that matters, and the prompt is complete before any computation starts.

2 · Document failure patterns

Record what’s gone wrong before — a structural gap, a wrong formula, a missed edge case — so the skill stops the agent from repeating a mistake it already made once.

3 · Progressive disclosure

Don’t front-load everything. Reference files load only when the workflow reaches that step — most of the skill never enters context at all.

4 · Scripts for deterministic steps

Anything with one right answer — event calculations, report generation — belongs in a script the agent runs, not in text the agent reasons through.

A skill is prompt engineering, not a prompt. Treat the intake, the failure modes, the context budget, and determinism as first-class design choices — not afterthoughts.

TrialDesignBench

The skill benchmark above is one case study inside a larger effort: evaluating whether AI agents can reproduce or design clinical trial statistical designs at all.

  • Benchmark 1 — reproduce a design, closed-book, from the SAP/protocol
  • Benchmark 2 — design the same trial with no SAP, only context (disease, treatment, regulatory, sponsor)

Full workflow spec

Building the benchmark

1 · Intake form

Statisticians write design questions and rubrics for a trial.

HuggingFace Space

2 · Benchmark review

A QA pass on submitted cases before they enter the benchmark.

HuggingFace Space

3 · Evaluation framework

Runs models against the benchmark, scores outputs against rubrics.

Built on Harbor

Example: question → rubric → grade

KN189 (KEYNOTE-189), P-004 — derivation question:

“What is the total required number of PFS events at its final analysis based on Lachin and Foulkes’ method for the variance estimation?”

Inputs used

  • Target HR = 0.7 ✓ +1
  • Randomization ratio 2:1 ✓ +1

Calculated value

  • PFS events at IA2 ≈ 468 ±3 ✗ 0

Model derived 472 — out of tolerance

Method

  • Fixed design via Lachin–Foulkes ✓ +2
  • L-F uses event fractions, not allocation fractions ✗ 0

Model scored 13 / 18 on this prompt.

Where we are

26 / 50 trials toward the target benchmark size

toward the target benchmark size

We’d love your help. Write a test case — question and rubrics — from a protocol or SAP you know well, and help stress-test AI on real trial design.

⭐ github.com/BBSW-org/TrialDesignBench

Recap: skills sharpen domain-specific context

  • Use skills to sharpen the domain-specific context an agent gets. The group sequential design skill encoded what inputs to collect, in what order, and how to compute and verify the result — my design philosophy, made reusable.
  • Agents like Claude Code ship a built-in skill for building skills. skill-creator interviews you, drafts the SKILL.md, and tests it — you don’t start from a blank page.
  • Build a benchmark for any mission-critical skill — don’t assume it helps. With-vs-without, like TrialDesignBench did here: 90% vs. 62%.

Benchmarks

You will: explore benchmark for AI evaluation

What is a benchmark?

A benchmark is a fixed task with an independently authored answer, so an agent’s work is measured, not judged by eye.

Like double programming: the agent is the production programmer, the expected output is the independent QC, and the grader compares the two datasets value by value.

Benchmark is a trial

Benchmark for SDTM and ADaM

Goal: test whether an AI agent can derive SDTM and ADaM datasets.

Expectation What each task needs Example
Relevant One real-study derivation Study day in VS, age groups in ADSL
Edge cases Inputs where two programmers could disagree Partial dates: 2024-03 (no day), 2024 (no month)
Reviewed by experts Experts confirm the task, inputs, and expected output Draft, reviewed, finalized; only people finalize

yamaa benchmarks

~ 200

benchmarks

3

execution interfaces (Python, R, CLI)

Scroll inside the frame to see the specification, inputs, and expected output.

Benchmark review progress

Each square is one benchmark; hover for its name, or select it to open the benchmark.

Harbor: run and grade an agent

Harbor runs an agent on each benchmark task and grades the result. It produced the Part 1 prompt-tier results; join yamaa on Harbor Hub to explore them.

Benchmark limitations

From benchmarks to real studies

Pilot 7: whole-study evaluation

Benchmarks test one derivation; R Consortium Pilot 7 tests whole studies.

20 / 20

ADaM datasets match the official submissions

13.7M

cells matched

Study Data flow Result
CDISC Pilot 01 SDTM -> ADaM 10 / 10 ADaM datasets match
Submission Pilots 3 and 5 SDTM -> ADaM 5 / 5 ADaM datasets match in each
CAR-T (OpenClinica sample) ODM -> SDTM -> ADaM No reference; people review
KN189, KN564 (synthetic) ODM -> SDTM -> ADaM No reference; people review

Matching data is not the end: agents flag where Define-XML and the data disagree, and people decide each finding.

yamaa reproduces R Submission Pilot 5

Real size: 254 subjects
End-to-end time, 1 core
R (yamaa)1.62 s
R (admiral)26.45 s
Peak memory
R (yamaa)0.25 GB
R (admiral)0.60 GB
Stress test: 10x subjects (2,540)
End-to-end time, 1 core
R (yamaa)12.62 s
R (admiral)181.70 s
Peak memory
R (yamaa)1.93 GB
R (admiral)4.13 GB
  • R (yamaa) vs. original R (admiral): 14–16x faster, 2.1–2.4x less memory.

Case study: reviewing benchmarks

You will: review a yamaa benchmark and an R Consortium Pilot 7 finding, and see how people finalize each one.

Two benchmarks to review

An AI-assisted draft is not truth until experts who do the work confirm it.

yamaa benchmark R Consortium Pilot 7
Scope One derivation, small input Full study
Reviewers check Task, inputs, expected output, three AI concerns Each AI-reported Define-XML vs. ADaM discrepancy
Improvement by Address reviewer feedback Fix findings confirmed by reviewers
Finalized when Two reviewers agreed Specification and derivation consistency achieved
Review guide Benchmark review Pilot 7 review guide

Reviewing a yamaa benchmark

Pilot 7: people decide on AI findings

Agents compare Define-XML with the reproduced yamaa specification and the ADaM data; people decide each finding.

Takeaways

  • Define the AI-first workflow with a clear input and output for every step.
  • Plain text first, deterministic first: keep artifacts inspectable, and let tested tools, not the model, produce the results.
  • Establish a human baseline with benchmarks before you trust an agent’s output.

Keep going after the course

  • Review a benchmark in five minutes: Start reviewing. Each review helps finalize the truth that agents are measured against.
  • Join R Consortium Pilot 7: the working group.
  • Explore a full study: the Pilot 7 synthetic data holds complete SDTM and ADaM packages; the Codespace in the appendix has yamaa and the agents ready.

References

Questions

Thank you. Post your questions in the group chat, or open an issue on github.com/elong0527/yamaa.

Appendix

Preparation

This workshop assumes familiarity with the clinical development lifecycle and CDISC standards, as well as R programming experience.

A preconfigured GitHub Codespace is ready for the workshop:

  • Open GitHub Codespaces
    • R and common R packages for data standardization.
    • OpenCode, Claude Code, and Codex.
  • Start OpenCode.
    • Run /connect to connect to OpenCode Go with the provided API key.
    • Run /models to select opencode-go/muse-spark-1.3-contributor.
    • Or sign in to the preinstalled Claude Code, Codex, or Muse code with your own account.

Verify the GitHub Codespace

Step 1. Check R and yamaa

Open the Codespace, start R from the terminal, and run:

library(yamaa)
yamaa_selftest()

Expect: the first line reads yamaa selftest: succeeded.

Step 2. Check an AI agent

Start an agent in the terminal: opencode, claude, codex, or muse. Then ask:

Explain ODM XML and its relationship to EDC systems and CRFs.

Expect: the agent answers in the terminal.

yamaa: four components

Component Purpose Links
Schema Declares the vocabulary of the specification. Repository / Introduction
Rules Defines the meaning of each specification. Repository / Rules
Engine Runs specifications in Python, R and CLI consistently following the rules. Python / R / CLI
Benchmark Provides runnable specification examples. Repository / Dashboards

Open source has reached the regulators

  • The R Consortium Submissions Working Group engages directly with the FDA.
    • Seven pilots demonstrate best practices for FDA submissions using R.

Pilot 1

2021 – 2022

Pilot 2

2022 – 2024

Pilot 3

2023 – 2024

Pilot 4a

2024 – 2025

Pilot 4b

2024 – 2025

Pilot 5

2025 – 2026

Pilot 6

2026 – ongoing

Pilot 7

2026 – ongoing

R Consortium Pilot 7: full studies

  • Builds a publicly available, agentic workflow for data standardization.
  • CDISC pilots (SDTM -> ADaM)
    • CDISCPILOT01 and R Consortium Submission Pilots 3 and 5.
  • OpenClinica real sample data (ODM -> SDTM -> ADaM)
    • Real sample data that tests the full data journey.
  • Synthetic studies (ODM -> SDTM -> ADaM)
    • ODM XML simulated from real clinical studies.

Objective: name the decision first

Intervention: the system you deploy

Evaluate what you deploy: the whole system. A trial tests the drug product as patients take it, not the molecule alone; likewise, a top leaderboard model does not guarantee a working workflow.

Endpoint: how each task is scored

Mix them: code where the truth is checkable, an LLM judge to scale the rest, and people to calibrate and sign off.