Starting from CDISC Data Standardization
81st Annual Deming Conference on Applied Statistics
2026-12-07
The course runs about three hours, with a break after Parts 1 and 2.
| Topic | Instructor | |
|---|---|---|
| Part 1 | AI agents and AI-first workflow | Yilong |
| Part 1 | Agentic CDISC data standardization | Yilong |
| Break | ||
| Part 2 | Giving agents the right context for domain-specific workflows | Eric |
| Break | ||
| Part 3 | Benchmarks | Yilong |
| Part 3 | Case study: reviewing benchmarks | Eric and Yilong |
The views and opinions expressed in this presentation are those of the individual presenters. They do not represent those of their affiliated organizations or institutions.
This workshop aims to help you:
Note
Toolchains, processes, and formats differ across organizations. This workshop shows one possible approach, not the only one.
You will: name the parts of an AI agent and say what each one can and cannot do.
Use the Pilot 5 ADSL dataset to generate a baseline characteristics table using R and gtsummary.
To finish, the agent has to:
adsl.parquet in the folder.What to check: the prompt never names the population, variables, or statistics, so the agent has to guess them. Do they match your table shell?
Muse is a personal agent from Meta. “Give Muse a goal or task and it handles the rest”: sort email, book a dinner, or fill out a form.
Muse also helps me maintain open-source projects in the background:
An LLM only predicts the next token from its context. It cannot open a file or run code: a harness does that and puts the results back into the context.
The LLM is like the CPU; the harness is like the operating system.
| Term | Meaning |
|---|---|
| Prompt | The request, with every rule you wrote into it. |
| Token | A word or part of a word: the unit a model reads and writes. |
| Context | The ordered tokens the model sees in one call: prompt, files read, tool results. |
| LLM | Predicts the next token from the context. |
| Harness | Builds the context, calls the model, runs tools, logs each step: OpenCode. |
| Tool | A program the harness runs for the model: read a file, run R. |
| Skill | Task instructions the harness loads: the agent’s SOP. |
| Agent | LLM, harness, and tools, looping until the task is done. |
You will: design an AI-first workflow and the principles that keep it trustworthy.
Shaping an AI-Native Biometrics Group from Day 0
What would you do?
You have been hired by the CMO to build a biometrics group.
Let’s brainstorm the strategy for a small- to midsize biopharma company.
AI-native is an organization whose workflows are mostly AI-first and connected through shared, inspectable artifacts.
| AI adoption | AI-first | AI-native |
|---|---|---|
| AI added to existing work | AI is the default starting point | AI is a core collaborator |
| AI helps with isolated tasks; the workflow does not change. | AI starts and executes the tasks of some workflows. | AI-human collaboration is designed across workflows. |
| Example: ChatGPT drafts an email or summarizes a document. | Example: every analysis starts with a prompt to an AI agent. | Example: AI agents run CDISC data standardization end to end. |
AI-native is a mindset, not a tool: every team member orchestrates AI agents, handles the exceptions, and keeps improving the system.
Let’s design an AI-first workflow for CDISC data standardization.
You will build an AI agent for this workflow. Where do you start, and why?
Primary goal: trustworthy assistance. The agent’s work can be inspected, repeated, challenged, and reviewed.
| Principles | Goal |
|---|---|
| Plain text first | Inspectable artifacts |
| Single source of truth | Explicit state |
| Deterministic first | Tools and evidence |
| Agent-specific identity | Attributable actions |
For CDISC data standardization, we recommend:
You will: see why agents disagree on CDISC derivations and how a reviewed, plain-text specification removes the guesswork.
From an FDA warning letter.
“you must review the AI-generated documents to ensure they were accurate and actually compliant…”
| Prompt tier | Python | R | opensas |
|---|---|---|---|
| Full | 99.3% (133/134) | 98.5% (132/134) | 86.6% (116/134) |
| Conventions | 35.8% (48/134) | 33.6% (45/134) | 34.3% (46/134) |
| Brief | 18.7% (25/134) | 17.9% (24/134) | 15.7% (21/134) |
Same 134 tasks, same agent; only the prompt changed. Writing the business rules into the prompt lifts the pass rate from under 20% to over 85%.
opensas: https://github.com/kirha-ai/opensas
Agent: opencode 1.18.33. Model: opencode-go/muse-spark-1.3-contributor, 1 attempt, Oct 12 2026
Shared Prompt:
ADVS from the provided input ADVS, following CDISC ADaM standards; one record per subject, visit, and parameter.STUDYID, USUBJID, PARAMCD, PARAM, AVAL, VISIT, DTYPE/app/input; write /app/output/advs.csv.| Rule in the prompt | Full - passes | Conventions - passes | Brief - fails |
|---|---|---|---|
| MAP identifiers | PARAMCD = "MAP"PARAM = "Mean Arterial Pressure (mmHg)" |
Same MAP identifiers | Not stated |
| Records to add | One per subject and visit with both SYSBP and DIABP; none when either is absent or missing. |
Not stated | Not stated |
AVAL |
Two-thirds DIABP plus one-third SYSBP;not rounded. |
Not rounded; formula not stated. |
Not stated |
DTYPE(sponsor convention) |
Added record: CALCULATION.Collected records: no value. |
CALCULATION or no value;assignment not stated. |
Not stated |
Brief omits both the MAP formula and the rule for adding a record.
The ADaM IG uses DTYPE for records derived differently within a parameter; setting it on a whole derived parameter is a sponsor rule the agent cannot infer.
SYSBP = systolic blood pressure; DIABP = diastolic blood pressure.
https://elong0527.github.io/yamaa/benchmark/adam-advs-map.html
Human and AI review the specification during planning; the engine executes the agreed rules.
yamaa: "1.0"
type: spec
model: ADaM
domain: ADSL
keys: [STUDYID, USUBJID]
input:
SOURCE:
path: input/adsl.csv
types: {HEIGHTCM: float, WEIGHTKG: float}
output:
path: adsl.csv
columns: [STUDYID, USUBJID, BMI]
columns:
- {name: STUDYID, type: str, label: Study Identifier, derivation: SOURCE.STUDYID}
- {name: USUBJID, type: str, label: Unique Subject Identifier, derivation: SOURCE.USUBJID}
- {name: HEIGHTCM, type: float, label: Height (cm), derivation: SOURCE.HEIGHTCM}
- {name: WEIGHTKG, type: float, label: Weight (kg), derivation: SOURCE.WEIGHTKG}
- name: BMI
type: float
label: Body Mass Index (kg/m2)
derivation: "WEIGHTKG / POWER(NULLIF(HEIGHTCM, 0) / 100, 2)"A yamaa specification has exactly one execution. Where it would have two, yamaa fails instead of choosing.
output_dataset = derive(input_datasets, spec)
The specification is a contract between people and AI agents:
The AI agent is probabilistic, so people review the specification. The engine that executes it is deterministic: the same specification and inputs give the same datasets.
Inheritance reduces duplication while keeping every resolved study specification self-contained and reviewable. (example)
People and AI agents collaborate on one version-controlled specification at the planning stage, while the study is ongoing.
You will: see two ways to give an agent the right context for a domain-specific job — a workflow, shown with medical monitoring; and a skill, shown with group sequential trial design.
Eric Zhang
Good agentic workflows start with a good harness
The harness is everything around the model: structure, memory, delegation, and checks. Build that well, and the workflow takes care of itself.
CLAUDE.md / AGENTS.md — standing instructions, root + per-study, read every session~/ workspace root
|-- CLAUDE.md workspace rules, read every session
|-- STUDY-001/
|-- STUDY-002/ one self-contained folder per study
|-- STUDY-003/
| |-- CLAUDE.md verified study quirks, auto-loaded here
| |-- data/<snapshot>/ dated EDC cuts, read-only
| |-- scripts/ pipeline, report, weekly refresh
| |-- output/ dated reports, publish staging
| `-- docs/session_log.md append-only history
|-- .claude/
| |-- agents/ separate sub-agent(s) per study
| |-- skills/ reusable playbooks
| `-- hooks/ session-log enforcement
`-- ...Physical isolation is what makes the context isolation reliable: an agent working in
STUDY-003/cannot accidentally seeSTUDY-001/’s data, fixes, or history.
initialize this folder
(in Claude Code — or the built-in /init command)
Result: CLAUDE.md — what the workspace is, how it’s laid out, and the rules every session must follow.
Chat forgets.
CLAUDE.mddoesn’t.
| Layer | Answers | Written by |
|---|---|---|
| Git | What changed? | The agent, one commit per change |
| Session log | Why did it change? | The agent, enforced by a hook |
| Platform system log | Who was in the system, and when? | The platform itself (e.g. Workbench), independent of the agent |
No single layer has to be trusted alone: what the agent changed, why it changed it, and who could have touched the system at all — from three different sources.
The platform’s own system log needs no building: logins and sessions are already recorded by the infrastructure (e.g. Posit Workbench), independent of anything the agent does. Git and the session log are what you build on top of it.
$ git log --oneline -5
a1b2c3d fix: detect the newest snapshot instead of a hardcoded default
9f8e7d6 feat: add a PP record to ADVS per the study SAP
7c6d5e4 fix: round AVAL per protocol convention, not language default
3b2a1f0 docs: record the baseline-derivation rule in CLAUDE.md
0e9d8c7 chore: initialize STUDY-003 from the workspace templateCLAUDE.md, skills, specs — one history, diffable and reversible.reset --hard, skipping hooks.If a change isn’t committed, it isn’t real yet.
STUDY-003/docs/session_log.md append-only
...
## 2026-11-09
- Snapshot mislabel: the "Nov 9" refresh read the Nov 3 cut.
Root cause: hardcoded snapshot default; refresh passed no argument.
Fixed: newest snapshot detected and passed explicitly.
- Rebuilt on the correct snapshot; flags 43 -> 50 (3 resolved, 4 new).
- Independent validation (10th pass): PASS -- zero errors.
- OPEN: access for the second reviewer.
...Hooks are your own scripts, run automatically on session events: build your own guardrails. Anything you want the agent to always do, make it a hook, not a reminder.
If it isn’t written down, the next session doesn’t know it.
~/.claude/skills/
|-- medical-monitoring-report/ the dashboard playbook
| |-- SKILL.md when it triggers, the method
| `-- references/ architecture, interactivity, validation
|-- recist-confirmation/ RECIST 1.1 confirmed-response rules
|-- group-sequential-design/ trial-design Q&A workflow
|-- gsdose/ seamless phase 2/3 dose-selection designs
`-- km-digitizer/ digitize published KM curvesADINTDT before ADTTE, which depends on it — not the reverse.SKILL.md, and tests it.A prompt asks for an outcome. A skill defines how it’s always done.
Evaluation runs on the same harness you just built:
A separate evaluator
Uses sub-agents. A fresh sub-agent that never built the thing — raw data only, never the pipeline code. It recomputes; it doesn’t re-read.
A validation skill
Uses skills. The same checks, every time: recompute every number from raw, check cross-output consistency, fix before shipping.
Failure patterns on file
Uses the audit trail. Past failures live in the session log; every new evaluation checks for the mistakes already made once.
The agent evaluates; a human signs off. Nothing ships on an agent’s PASS alone — we own what ships.
AI-native
skill: medical-monitoring-report
Cron scheduled trigger
Extract 🤖🛠 the day’s EDC snapshot
Build 🤖🛠 pipeline + lineage graph skill ref: architecture skill: RECIST 1.1
Assemble 🤖🛠 interactive HTML
Summary 🤖🛠 updates + data issues
Validate 🤖🔍 recompute from raw skill ref: validation
Adjudicate 🤖⚖️ resolve flagged items skill ref: validation
Publish 🤖 one stable URL
Notify 🤖 reviewers alerted
Git + session log — every step logged
Human review 👤 — outside the AI-native loop. Sign-off required.
Sub-agents for Build, Validate, and Adjudicate are each guided by a skill.
The daily workflow on the previous slide doesn’t run once — it repeats for the life of the trial, giving two things in real time:
A skill packages domain knowledge once — then any agent can use it.
Where a workflow wires together the steps of one task, a skill teaches the agent what a domain expert already knows: the method, its assumptions, and when it applies. This topic shows it with group sequential design for survival trials.
🎥
Watch the skill run a design end to end
Q&A → R design script → simulation verification → Word report + reproducible R/Python scripts
Given a trial scenario, the group sequential design skill designs a group-sequential survival-endpoint trial — boundaries, sample size, and multiplicity strategy — through a guided Q&A, verifies the result by simulation, and delivers reproducible R/Python scripts alongside the Word report.
Skill structure
Loaded progressively — each reference is only pulled into context when the workflow reaches that step.
The skill asks its questions one at a time, in a fixed order. That order isn’t arbitrary — it’s what design inputs I need, and in what sequence, to walk through a group sequential design the way I actually would.
Any addition to an agent — skill, tool, workflow — should be benchmarked with-vs-without, not assumed to help. 21 design scenarios, skill vs. no-skill — overall scorecard, best-with-skill run per scenario:
| With skill | Without skill | |
|---|---|---|
| Skill wins | 16 | — |
| Ties | 4 (#3, #21, #60, #113) | — |
| No-skill wins | 1 (#139) | — |
| Avg score (excl. #60, trivial) | 90% | 62% |
1 · Does it add value?
16/21 wins, 4 ties, 1 loss (#139 — a structural gap, fix in progress).
2 · How good with the skill?
90% average vs. 62% without, at each scenario’s best run.
3 · Failure patterns
P0: a four-arm, contribution-of-components design.
1 · Structured Q&A completes the prompt
A skill’s intake isn’t boilerplate — it’s prompt engineering. Ask one question at a time, in the order that matters, and the prompt is complete before any computation starts.
2 · Document failure patterns
Record what’s gone wrong before — a structural gap, a wrong formula, a missed edge case — so the skill stops the agent from repeating a mistake it already made once.
3 · Progressive disclosure
Don’t front-load everything. Reference files load only when the workflow reaches that step — most of the skill never enters context at all.
4 · Scripts for deterministic steps
Anything with one right answer — event calculations, report generation — belongs in a script the agent runs, not in text the agent reasons through.
A skill is prompt engineering, not a prompt. Treat the intake, the failure modes, the context budget, and determinism as first-class design choices — not afterthoughts.
The skill benchmark above is one case study inside a larger effort: evaluating whether AI agents can reproduce or design clinical trial statistical designs at all.
2 · Benchmark review
A QA pass on submitted cases before they enter the benchmark.
3 · Evaluation framework
Runs models against the benchmark, scores outputs against rubrics.
KN189 (KEYNOTE-189), P-004 — derivation question:
“What is the total required number of PFS events at its final analysis based on Lachin and Foulkes’ method for the variance estimation?”
Inputs used
Calculated value
Model derived 472 — out of tolerance
Method
Model scored 13 / 18 on this prompt.
26 / 50 trials toward the target benchmark size
toward the target benchmark size
We’d love your help. Write a test case — question and rubrics — from a protocol or SAP you know well, and help stress-test AI on real trial design.
SKILL.md, and tests it — you don’t start from a blank page.You will: explore benchmark for AI evaluation
A benchmark is a fixed task with an independently authored answer, so an agent’s work is measured, not judged by eye.
Like double programming: the agent is the production programmer, the expected output is the independent QC, and the grader compares the two datasets value by value.
Goal: test whether an AI agent can derive SDTM and ADaM datasets.
| Expectation | What each task needs | Example |
|---|---|---|
| Relevant | One real-study derivation | Study day in VS, age groups in ADSL |
| Edge cases | Inputs where two programmers could disagree | Partial dates: 2024-03 (no day), 2024 (no month) |
| Reviewed by experts | Experts confirm the task, inputs, and expected output | Draft, reviewed, finalized; only people finalize |
~ 200
benchmarks
3
execution interfaces (Python, R, CLI)
Scroll inside the frame to see the specification, inputs, and expected output.
Each square is one benchmark; hover for its name, or select it to open the benchmark.
Harbor runs an agent on each benchmark task and grades the result. It produced the Part 1 prompt-tier results; join yamaa on Harbor Hub to explore them.
Benchmarks test one derivation; R Consortium Pilot 7 tests whole studies.
20 / 20
ADaM datasets match the official submissions
13.7M
cells matched
| Study | Data flow | Result |
|---|---|---|
| CDISC Pilot 01 | SDTM -> ADaM | 10 / 10 ADaM datasets match |
| Submission Pilots 3 and 5 | SDTM -> ADaM | 5 / 5 ADaM datasets match in each |
| CAR-T (OpenClinica sample) | ODM -> SDTM -> ADaM | No reference; people review |
| KN189, KN564 (synthetic) | ODM -> SDTM -> ADaM | No reference; people review |
Matching data is not the end: agents flag where Define-XML and the data disagree, and people decide each finding.
You will: review a yamaa benchmark and an R Consortium Pilot 7 finding, and see how people finalize each one.
An AI-assisted draft is not truth until experts who do the work confirm it.
| yamaa benchmark | R Consortium Pilot 7 | |
|---|---|---|
| Scope | One derivation, small input | Full study |
| Reviewers check | Task, inputs, expected output, three AI concerns | Each AI-reported Define-XML vs. ADaM discrepancy |
| Improvement by | Address reviewer feedback | Fix findings confirmed by reviewers |
| Finalized when | Two reviewers agreed | Specification and derivation consistency achieved |
| Review guide | Benchmark review | Pilot 7 review guide |
Agents compare Define-XML with the reproduced yamaa specification and the ADaM data; people decide each finding.
Thank you. Post your questions in the group chat, or open an issue on github.com/elong0527/yamaa.
This workshop assumes familiarity with the clinical development lifecycle and CDISC standards, as well as R programming experience.
A preconfigured GitHub Codespace is ready for the workshop:
/connect to connect to OpenCode Go with the provided API key./models to select opencode-go/muse-spark-1.3-contributor.Step 1. Check R and yamaa
Open the Codespace, start R from the terminal, and run:
Expect: the first line reads yamaa selftest: succeeded.
Step 2. Check an AI agent
Start an agent in the terminal: opencode, claude, codex, or muse. Then ask:
Explain ODM XML and its relationship to EDC systems and CRFs.
Expect: the agent answers in the terminal.
| Component | Purpose | Links |
|---|---|---|
| Schema | Declares the vocabulary of the specification. | Repository / Introduction |
| Rules | Defines the meaning of each specification. | Repository / Rules |
| Engine | Runs specifications in Python, R and CLI consistently following the rules. | Python / R / CLI |
| Benchmark | Provides runnable specification examples. | Repository / Dashboards |
Evaluate what you deploy: the whole system. A trial tests the drug product as patients take it, not the molecule alone; likewise, a top leaderboard model does not guarantee a working workflow.
Mix them: code where the truth is checkable, an LLM judge to scale the rest, and people to calibrate and sign off.