AI-First Workflow

Starting from CDISC Data Standardization

Yilong Zhang
Jimeng Sun

2026 Boston Pharmaceutical Statistics Symposium

2026-11-12

Welcome

Outline

The course runs about three hours, with a break after Parts 1 and 2.

Topic Instructor
Part 1 AI agents and AI-first workflow Yilong
Part 1 Agentic CDISC data standardization Yilong
Break
Part 2 LLM and agents for statistical programming Jimeng
Break
Part 3 Benchmarks Yilong
Part 3 Case study: reviewing benchmarks Jimeng and Yilong

Disclaimer

The views and opinions expressed in this presentation are those of the individual presenters. They do not represent those of their affiliated organizations or institutions.

Training objectives

This workshop aims to help you:

  • Understand the building blocks of an agentic workflow, with CDISC data standardization as one example.
  • Recognize where an AI agent’s performance may be improved.
  • Get started with creating and reviewing benchmarks that assess AI agents.

Note

Toolchains, processes, and formats differ across organizations. This workshop shows one possible approach, not the only one.

Resources

  • During the workshop, we will use three projects:
    • yamaa: tools and benchmarks for CDISC data standardization.
    • R Consortium Pilot 7 project: synthetic trials that demonstrate CDISC data standardization.
    • ai4csr: the book and slides for this short course.
  • Post your questions in the group chat.

AI agents

You will: name the parts of an AI agent and say what each one can and cannot do.

Example: arena.ai

Try it at arena.ai/agent

Use the Pilot 5 ADSL dataset to generate a baseline characteristics table using R and gtsummary.

Data: https://github.com/RConsortium/submissions-pilot7-synthetic-data/tree/main/submission-pilot5/data/adam

To finish, the agent has to:

  1. Find adsl.parquet in the folder.
  2. Read it in R.
  3. Pick the population, arms, and grouping variables.
  4. Build the table with gtsummary.
  5. Return the table and its code.

What to check: the prompt never names the population, variables, or statistics, so the agent has to guess them. Do they match your table shell?

Example: Muse

Try it at muse.ai

Muse is a personal agent from Meta. “Give Muse a goal or task and it handles the rest”: sort email, book a dinner, or fill out a form.

Muse also helps me maintain open-source projects in the background:

  • Identify project gaps from published PHUSE papers.
  • Bring new benchmark ideas for CDISC standardization.

What is an AI agent?

What is a large language model?

An LLM only predicts the next token from its context. It cannot open a file or run code: a harness does that and puts the results back into the context.

What is a harness?

The LLM is like the CPU; the harness is like the operating system.

Reference: Agent vocabulary

Term Meaning
Prompt The request, with every rule you wrote into it.
Token A word or part of a word: the unit a model reads and writes.
Context The ordered tokens the model sees in one call: prompt, files read, tool results.
LLM Predicts the next token from the context.
Harness Builds the context, calls the model, runs tools, logs each step: OpenCode.
Tool A program the harness runs for the model: read a file, run R.
Skill Task instructions the harness loads: the agent’s SOP.
Agent LLM, harness, and tools, looping until the task is done.

AI-first workflow

You will: design an AI-first workflow and the principles that keep it trustworthy.

A mental game

Shaping an AI-Native Biometrics Group from Day 0

What would you do?

You have been hired by the CMO to build a biometrics group.

Blank Slate

  • No legacy systems to migrate.
  • No “we’ve always done it this way” culture.

Rare Opportunity

  • Build an AI-native group from scratch.
  • Skip the “digital transformation” pain.

Let’s brainstorm the strategy for a small- to midsize biopharma company.

What is AI-native?

AI-native is an organization whose workflows are mostly AI-first and connected through shared, inspectable artifacts.

AI adoption AI-first AI-native
AI added to existing work AI is the default starting point AI is a core collaborator
AI helps with isolated tasks; the workflow does not change. AI starts and executes the tasks of some workflows. AI-human collaboration is designed across workflows.
Example: ChatGPT drafts an email or summarizes a document. Example: every analysis starts with a prompt to an AI agent. Example: AI agents run CDISC data standardization end to end.

AI-native is a mindset, not a tool: every team member orchestrates AI agents, handles the exceptions, and keeps improving the system.

Design an AI-first workflow

Let’s design an AI-first workflow for CDISC data standardization.

Brainstorm: where to start building

You will build an AI agent for this workflow. Where do you start, and why?

Opinionated proposal

  • As a Biometrics organization, focus on defining the workflow.
  • Leverage internal and external AI infrastructure, with proper skills and tools.

Agentic workflow design principles

Primary goal: trustworthy assistance. The agent’s work can be inspected, repeated, challenged, and reviewed.

Principles Goal
Plain text first Inspectable artifacts
Single source of truth Explicit state
Deterministic first Tools and evidence
Agent-specific identity Attributable actions

For CDISC data standardization, we recommend:

  • Do not use the annotated CRF as an input; generate it as an output.
  • Do not use a specification in Excel, Word, or PDF as an input; use a plain-text specification.

Agentic CDISC data standardization

You will: see why agents disagree on CDISC derivations and how a reviewed, plain-text specification removes the guesswork.

Contributors

  • Peng Zhang CIMS Global
  • Bingjun Wang Genentech
  • Ming-chun Chen Summit Therapeutics
  • Jingwei Xiong Merck & Co., Inc.
  • Yilong Zhang BBSW AI Committee

Motivation

  • Two AI agents derive the same analysis dataset from the same specification (example from arena.ai).
  • Both report success, but their code and outputs may differ.
  • A reviewer must resolve every difference before signing off.

From an FDA warning letter.

“you must review the AI-generated documents to ensure they were accurate and actually compliant…”

One root cause

  • The user prompt leaves business rules implicit, so agents must infer them.
  • Business rules can differ from language defaults:
    • R and Python round ties to even.
    • SAS compares missing numeric values as lower than non-missing values.

Evidence: prompt completeness decides success

Prompt tier Python R opensas
Full 99.3% (133/134) 98.5% (132/134) 86.6% (116/134)
Conventions 35.8% (48/134) 33.6% (45/134) 34.3% (46/134)
Brief 18.7% (25/134) 17.9% (24/134) 15.7% (21/134)

Same 134 tasks, same agent; only the prompt changed. Writing the business rules into the prompt lifts the pass rate from under 20% to over 85%.

opensas: https://github.com/kirha-ai/opensas
Agent: opencode 1.18.33. Model: opencode-go/muse-spark-1.3-contributor, 1 attempt, Oct 12 2026

Mean arterial pressure: three prompts

Shared Prompt:

  • create an output ADVS from the provided input ADVS, following CDISC ADaM standards; one record per subject, visit, and parameter.
  • STUDYID, USUBJID, PARAMCD, PARAM, AVAL, VISIT, DTYPE
  • read from /app/input; write /app/output/advs.csv.
Rule in the prompt Full - passes Conventions - passes Brief - fails
MAP identifiers PARAMCD = "MAP"
PARAM = "Mean Arterial Pressure (mmHg)"
Same MAP identifiers Not stated
Records to add One per subject and visit with both SYSBP and DIABP; none when either is absent or missing. Not stated Not stated
AVAL Two-thirds DIABP plus one-third SYSBP;
not rounded.
Not rounded;
formula not stated.
Not stated
DTYPE
(sponsor convention)
Added record: CALCULATION.
Collected records: no value.
CALCULATION or no value;
assignment not stated.
Not stated

Brief omits both the MAP formula and the rule for adding a record.
The ADaM IG uses DTYPE for records derived differently within a parameter; setting it on a whole derived parameter is a sponsor rule the agent cannot infer.
SYSBP = systolic blood pressure; DIABP = diastolic blood pressure.
https://elong0527.github.io/yamaa/benchmark/adam-advs-map.html

Goal

  • Reduce human review effort.
  • Leverage organization templates.
  • Capture business rules that AI agents can apply.
  • Move human-AI collaboration to the planning stage.

yamaa: collaboration at the planning stage

Human and AI review the specification during planning; the engine executes the agreed rules.

yamaa: YAML specification example

yamaa: "1.0"
type: spec
model: ADaM
domain: ADSL
keys: [STUDYID, USUBJID]
input:
  SOURCE:
    path: input/adsl.csv
    types: {HEIGHTCM: float, WEIGHTKG: float}

output:
  path: adsl.csv
  columns: [STUDYID, USUBJID, BMI]

columns:
  - {name: STUDYID, type: str, label: Study Identifier, derivation: SOURCE.STUDYID}
  - {name: USUBJID, type: str, label: Unique Subject Identifier, derivation: SOURCE.USUBJID}
  - {name: HEIGHTCM, type: float, label: Height (cm), derivation: SOURCE.HEIGHTCM}
  - {name: WEIGHTKG, type: float, label: Weight (kg), derivation: SOURCE.WEIGHTKG}
  - name: BMI
    type: float
    label: Body Mass Index (kg/m2)
    derivation: "WEIGHTKG / POWER(NULLIF(HEIGHTCM, 0) / 100, 2)"

yamaa: core principle

A yamaa specification has exactly one execution. Where it would have two, yamaa fails instead of choosing.

output_dataset = derive(input_datasets, spec)

The specification is a contract between people and AI agents:

  • spec: written in the closed vocabulary of the schema.
  • derive: applies each business rule with one defined meaning.

yamaa: where the uncertainty lives

The AI agent is probabilistic, so people review the specification. The engine that executes it is deterministic: the same specification and inputs give the same datasets.

yamaa: inheritance

Inheritance reduces duplication while keeping every resolved study specification self-contained and reviewable. (example)

Summary of the workflow

People and AI agents collaborate on one version-controlled specification at the planning stage, while the study is ongoing.

Benchmarks

You will: explore benchmark for AI evaluation

What is a benchmark?

A benchmark is a fixed task with an independently authored answer, so an agent’s work is measured, not judged by eye.

Like double programming: the agent is the production programmer, the expected output is the independent QC, and the grader compares the two datasets value by value.

Benchmark is a trial

Benchmark for SDTM and ADaM

Goal: test whether an AI agent can derive SDTM and ADaM datasets.

Expectation What each task needs Example
Relevant One real-study derivation Study day in VS, age groups in ADSL
Edge cases Inputs where two programmers could disagree Partial dates: 2024-03 (no day), 2024 (no month)
Reviewed by experts Experts confirm the task, inputs, and expected output Draft, reviewed, finalized; only people finalize

yamaa benchmarks

~ 200

benchmarks

3

execution interfaces (Python, R, CLI)

Scroll inside the frame to see the specification, inputs, and expected output.

Benchmark review progress

Each square is one benchmark; hover for its name, or select it to open the benchmark.

Harbor: run and grade an agent

Harbor runs an agent on each benchmark task and grades the result. It produced the Part 1 prompt-tier results; join yamaa on Harbor Hub to explore them.

Benchmark limitations

From benchmarks to real studies

Pilot 7: whole-study evaluation

Benchmarks test one derivation; R Consortium Pilot 7 tests whole studies.

20 / 20

ADaM datasets match the official submissions

13.7M

cells matched

Study Data flow Result
CDISC Pilot 01 SDTM -> ADaM 10 / 10 ADaM datasets match
Submission Pilots 3 and 5 SDTM -> ADaM 5 / 5 ADaM datasets match in each
CAR-T (OpenClinica sample) ODM -> SDTM -> ADaM No reference; people review
KN189, KN564 (synthetic) ODM -> SDTM -> ADaM No reference; people review

Matching data is not the end: agents flag where Define-XML and the data disagree, and people decide each finding.

yamaa reproduces R Submission Pilot 5

Real size: 254 subjects
End-to-end time, 1 core
R (yamaa)1.62 s
R (admiral)26.45 s
Peak memory
R (yamaa)0.25 GB
R (admiral)0.60 GB
Stress test: 10x subjects (2,540)
End-to-end time, 1 core
R (yamaa)12.62 s
R (admiral)181.70 s
Peak memory
R (yamaa)1.93 GB
R (admiral)4.13 GB
  • R (yamaa) vs. original R (admiral): 14–16x faster, 2.1–2.4x less memory.

Case study: reviewing benchmarks

You will: review a yamaa benchmark and an R Consortium Pilot 7 finding, and see how people finalize each one.

Two benchmarks to review

An AI-assisted draft is not truth until experts who do the work confirm it.

yamaa benchmark R Consortium Pilot 7
Scope One derivation, small input Full study
Reviewers check Task, inputs, expected output, three AI concerns Each AI-reported Define-XML vs. ADaM discrepancy
Improvement by Address reviewer feedback Fix findings confirmed by reviewers
Finalized when Two reviewers agreed Specification and derivation consistency achieved
Review guide Benchmark review Pilot 7 review guide

Reviewing a yamaa benchmark

Pilot 7: people decide on AI findings

Agents compare Define-XML with the reproduced yamaa specification and the ADaM data; people decide each finding.

Takeaways

  • Define the AI-first workflow with a clear input and output for every step.
  • Plain text first, deterministic first: keep artifacts inspectable, and let tested tools, not the model, produce the results.
  • Establish a human baseline with benchmarks before you trust an agent’s output.

Keep going after the course

  • Review a benchmark in five minutes: Start reviewing. Each review helps finalize the truth that agents are measured against.
  • Join R Consortium Pilot 7: the working group.
  • Explore a full study: the Pilot 7 synthetic data holds complete SDTM and ADaM packages; the Codespace in the appendix has yamaa and the agents ready.

References

Questions

Thank you. Post your questions in the group chat, or open an issue on github.com/elong0527/yamaa.

Appendix

Preparation

This workshop assumes familiarity with the clinical development lifecycle and CDISC standards, as well as R programming experience.

A preconfigured GitHub Codespace is ready for the workshop:

  • Open GitHub Codespaces
    • R and common R packages for data standardization.
    • OpenCode, Claude Code, and Codex.
  • Start OpenCode.
    • Run /connect to connect to OpenCode Go with the provided API key.
    • Run /models to select opencode-go/muse-spark-1.3-contributor.
    • Or sign in to the preinstalled Claude Code, Codex, or Muse code with your own account.

Verify the GitHub Codespace

Step 1. Check R and yamaa

Open the Codespace, start R from the terminal, and run:

library(yamaa)
yamaa_selftest()

Expect: the first line reads yamaa selftest: succeeded.

Step 2. Check an AI agent

Start an agent in the terminal: opencode, claude, codex, or muse. Then ask:

Explain ODM XML and its relationship to EDC systems and CRFs.

Expect: the agent answers in the terminal.

yamaa: four components

Component Purpose Links
Schema Declares the vocabulary of the specification. Repository / Introduction
Rules Defines the meaning of each specification. Repository / Rules
Engine Runs specifications in Python, R and CLI consistently following the rules. Python / R / CLI
Benchmark Provides runnable specification examples. Repository / Dashboards

Open source has reached the regulators

  • The R Consortium Submissions Working Group engages directly with the FDA.
    • Seven pilots demonstrate best practices for FDA submissions using R.

Pilot 1

2021 – 2022

Pilot 2

2022 – 2024

Pilot 3

2023 – 2024

Pilot 4a

2024 – 2025

Pilot 4b

2024 – 2025

Pilot 5

2025 – 2026

Pilot 6

2026 – ongoing

Pilot 7

2026 – ongoing

R Consortium Pilot 7: full studies

  • Builds a publicly available, agentic workflow for data standardization.
  • CDISC pilots (SDTM -> ADaM)
    • CDISCPILOT01 and R Consortium Submission Pilots 3 and 5.
  • OpenClinica real sample data (ODM -> SDTM -> ADaM)
    • Real sample data that tests the full data journey.
  • Synthetic studies (ODM -> SDTM -> ADaM)
    • ODM XML simulated from real clinical studies.

Objective: name the decision first

Intervention: the system you deploy

Evaluate what you deploy: the whole system. A trial tests the drug product as patients take it, not the molecule alone; likewise, a top leaderboard model does not guarantee a working workflow.

Endpoint: how each task is scored

Mix them: code where the truth is checkable, an LLM judge to scale the rest, and people to calibrate and sign off.