Starting from CDISC Data Standardization
2026 Boston Pharmaceutical Statistics Symposium
2026-11-12
The course runs about three hours, with a break after Parts 1 and 2.
| Topic | Instructor | |
|---|---|---|
| Part 1 | AI agents and AI-first workflow | Yilong |
| Part 1 | Agentic CDISC data standardization | Yilong |
| Break | ||
| Part 2 | LLM and agents for statistical programming | Jimeng |
| Break | ||
| Part 3 | Benchmarks | Yilong |
| Part 3 | Case study: reviewing benchmarks | Jimeng and Yilong |
The views and opinions expressed in this presentation are those of the individual presenters. They do not represent those of their affiliated organizations or institutions.
This workshop aims to help you:
Note
Toolchains, processes, and formats differ across organizations. This workshop shows one possible approach, not the only one.
You will: name the parts of an AI agent and say what each one can and cannot do.
Use the Pilot 5 ADSL dataset to generate a baseline characteristics table using R and gtsummary.
To finish, the agent has to:
adsl.parquet in the folder.What to check: the prompt never names the population, variables, or statistics, so the agent has to guess them. Do they match your table shell?
Muse is a personal agent from Meta. “Give Muse a goal or task and it handles the rest”: sort email, book a dinner, or fill out a form.
Muse also helps me maintain open-source projects in the background:
An LLM only predicts the next token from its context. It cannot open a file or run code: a harness does that and puts the results back into the context.
The LLM is like the CPU; the harness is like the operating system.
| Term | Meaning |
|---|---|
| Prompt | The request, with every rule you wrote into it. |
| Token | A word or part of a word: the unit a model reads and writes. |
| Context | The ordered tokens the model sees in one call: prompt, files read, tool results. |
| LLM | Predicts the next token from the context. |
| Harness | Builds the context, calls the model, runs tools, logs each step: OpenCode. |
| Tool | A program the harness runs for the model: read a file, run R. |
| Skill | Task instructions the harness loads: the agent’s SOP. |
| Agent | LLM, harness, and tools, looping until the task is done. |
You will: design an AI-first workflow and the principles that keep it trustworthy.
Shaping an AI-Native Biometrics Group from Day 0
What would you do?
You have been hired by the CMO to build a biometrics group.
Let’s brainstorm the strategy for a small- to midsize biopharma company.
AI-native is an organization whose workflows are mostly AI-first and connected through shared, inspectable artifacts.
| AI adoption | AI-first | AI-native |
|---|---|---|
| AI added to existing work | AI is the default starting point | AI is a core collaborator |
| AI helps with isolated tasks; the workflow does not change. | AI starts and executes the tasks of some workflows. | AI-human collaboration is designed across workflows. |
| Example: ChatGPT drafts an email or summarizes a document. | Example: every analysis starts with a prompt to an AI agent. | Example: AI agents run CDISC data standardization end to end. |
AI-native is a mindset, not a tool: every team member orchestrates AI agents, handles the exceptions, and keeps improving the system.
Let’s design an AI-first workflow for CDISC data standardization.
You will build an AI agent for this workflow. Where do you start, and why?
Primary goal: trustworthy assistance. The agent’s work can be inspected, repeated, challenged, and reviewed.
| Principles | Goal |
|---|---|
| Plain text first | Inspectable artifacts |
| Single source of truth | Explicit state |
| Deterministic first | Tools and evidence |
| Agent-specific identity | Attributable actions |
For CDISC data standardization, we recommend:
You will: see why agents disagree on CDISC derivations and how a reviewed, plain-text specification removes the guesswork.
From an FDA warning letter.
“you must review the AI-generated documents to ensure they were accurate and actually compliant…”
| Prompt tier | Python | R | opensas |
|---|---|---|---|
| Full | 99.3% (133/134) | 98.5% (132/134) | 86.6% (116/134) |
| Conventions | 35.8% (48/134) | 33.6% (45/134) | 34.3% (46/134) |
| Brief | 18.7% (25/134) | 17.9% (24/134) | 15.7% (21/134) |
Same 134 tasks, same agent; only the prompt changed. Writing the business rules into the prompt lifts the pass rate from under 20% to over 85%.
opensas: https://github.com/kirha-ai/opensas
Agent: opencode 1.18.33. Model: opencode-go/muse-spark-1.3-contributor, 1 attempt, Oct 12 2026
Shared Prompt:
ADVS from the provided input ADVS, following CDISC ADaM standards; one record per subject, visit, and parameter.STUDYID, USUBJID, PARAMCD, PARAM, AVAL, VISIT, DTYPE/app/input; write /app/output/advs.csv.| Rule in the prompt | Full - passes | Conventions - passes | Brief - fails |
|---|---|---|---|
| MAP identifiers | PARAMCD = "MAP"PARAM = "Mean Arterial Pressure (mmHg)" |
Same MAP identifiers | Not stated |
| Records to add | One per subject and visit with both SYSBP and DIABP; none when either is absent or missing. |
Not stated | Not stated |
AVAL |
Two-thirds DIABP plus one-third SYSBP;not rounded. |
Not rounded; formula not stated. |
Not stated |
DTYPE(sponsor convention) |
Added record: CALCULATION.Collected records: no value. |
CALCULATION or no value;assignment not stated. |
Not stated |
Brief omits both the MAP formula and the rule for adding a record.
The ADaM IG uses DTYPE for records derived differently within a parameter; setting it on a whole derived parameter is a sponsor rule the agent cannot infer.
SYSBP = systolic blood pressure; DIABP = diastolic blood pressure.
https://elong0527.github.io/yamaa/benchmark/adam-advs-map.html
Human and AI review the specification during planning; the engine executes the agreed rules.
yamaa: "1.0"
type: spec
model: ADaM
domain: ADSL
keys: [STUDYID, USUBJID]
input:
SOURCE:
path: input/adsl.csv
types: {HEIGHTCM: float, WEIGHTKG: float}
output:
path: adsl.csv
columns: [STUDYID, USUBJID, BMI]
columns:
- {name: STUDYID, type: str, label: Study Identifier, derivation: SOURCE.STUDYID}
- {name: USUBJID, type: str, label: Unique Subject Identifier, derivation: SOURCE.USUBJID}
- {name: HEIGHTCM, type: float, label: Height (cm), derivation: SOURCE.HEIGHTCM}
- {name: WEIGHTKG, type: float, label: Weight (kg), derivation: SOURCE.WEIGHTKG}
- name: BMI
type: float
label: Body Mass Index (kg/m2)
derivation: "WEIGHTKG / POWER(NULLIF(HEIGHTCM, 0) / 100, 2)"A yamaa specification has exactly one execution. Where it would have two, yamaa fails instead of choosing.
output_dataset = derive(input_datasets, spec)
The specification is a contract between people and AI agents:
The AI agent is probabilistic, so people review the specification. The engine that executes it is deterministic: the same specification and inputs give the same datasets.
Inheritance reduces duplication while keeping every resolved study specification self-contained and reviewable. (example)
People and AI agents collaborate on one version-controlled specification at the planning stage, while the study is ongoing.
You will: explore benchmark for AI evaluation
A benchmark is a fixed task with an independently authored answer, so an agent’s work is measured, not judged by eye.
Like double programming: the agent is the production programmer, the expected output is the independent QC, and the grader compares the two datasets value by value.
Goal: test whether an AI agent can derive SDTM and ADaM datasets.
| Expectation | What each task needs | Example |
|---|---|---|
| Relevant | One real-study derivation | Study day in VS, age groups in ADSL |
| Edge cases | Inputs where two programmers could disagree | Partial dates: 2024-03 (no day), 2024 (no month) |
| Reviewed by experts | Experts confirm the task, inputs, and expected output | Draft, reviewed, finalized; only people finalize |
~ 200
benchmarks
3
execution interfaces (Python, R, CLI)
Scroll inside the frame to see the specification, inputs, and expected output.
Each square is one benchmark; hover for its name, or select it to open the benchmark.
Harbor runs an agent on each benchmark task and grades the result. It produced the Part 1 prompt-tier results; join yamaa on Harbor Hub to explore them.
Benchmarks test one derivation; R Consortium Pilot 7 tests whole studies.
20 / 20
ADaM datasets match the official submissions
13.7M
cells matched
| Study | Data flow | Result |
|---|---|---|
| CDISC Pilot 01 | SDTM -> ADaM | 10 / 10 ADaM datasets match |
| Submission Pilots 3 and 5 | SDTM -> ADaM | 5 / 5 ADaM datasets match in each |
| CAR-T (OpenClinica sample) | ODM -> SDTM -> ADaM | No reference; people review |
| KN189, KN564 (synthetic) | ODM -> SDTM -> ADaM | No reference; people review |
Matching data is not the end: agents flag where Define-XML and the data disagree, and people decide each finding.
You will: review a yamaa benchmark and an R Consortium Pilot 7 finding, and see how people finalize each one.
An AI-assisted draft is not truth until experts who do the work confirm it.
| yamaa benchmark | R Consortium Pilot 7 | |
|---|---|---|
| Scope | One derivation, small input | Full study |
| Reviewers check | Task, inputs, expected output, three AI concerns | Each AI-reported Define-XML vs. ADaM discrepancy |
| Improvement by | Address reviewer feedback | Fix findings confirmed by reviewers |
| Finalized when | Two reviewers agreed | Specification and derivation consistency achieved |
| Review guide | Benchmark review | Pilot 7 review guide |
Agents compare Define-XML with the reproduced yamaa specification and the ADaM data; people decide each finding.
Thank you. Post your questions in the group chat, or open an issue on github.com/elong0527/yamaa.
This workshop assumes familiarity with the clinical development lifecycle and CDISC standards, as well as R programming experience.
A preconfigured GitHub Codespace is ready for the workshop:
/connect to connect to OpenCode Go with the provided API key./models to select opencode-go/muse-spark-1.3-contributor.Step 1. Check R and yamaa
Open the Codespace, start R from the terminal, and run:
Expect: the first line reads yamaa selftest: succeeded.
Step 2. Check an AI agent
Start an agent in the terminal: opencode, claude, codex, or muse. Then ask:
Explain ODM XML and its relationship to EDC systems and CRFs.
Expect: the agent answers in the terminal.
| Component | Purpose | Links |
|---|---|---|
| Schema | Declares the vocabulary of the specification. | Repository / Introduction |
| Rules | Defines the meaning of each specification. | Repository / Rules |
| Engine | Runs specifications in Python, R and CLI consistently following the rules. | Python / R / CLI |
| Benchmark | Provides runnable specification examples. | Repository / Dashboards |
Evaluate what you deploy: the whole system. A trial tests the drug product as patients take it, not the molecule alone; likewise, a top leaderboard model does not guarantee a working workflow.
Mix them: code where the truth is checkable, an LLM judge to scale the rest, and people to calibrate and sign off.