PART 2

Language Models and Agents
for Statistical Programming

Jimeng Sun · Yilong Zhang

Boston Pharmaceutical Statistics Symposium · Nov 12, 2026

Part 2 at a glance

Topic Time
1 The expert’s new job 11 min
2 Agents built to be checked 17 min
3 Seven strategies beyond skills and tools 20 min
4 Verify before you deploy 7 min

Disclosure: Jimeng Sun is co-founder and CEO of Keiji AI, which builds TrialMind, the platform used in the demos.

The expert’s new job

Generating is cheap. Verifying is not.

722 manuscripts in one day

722

math manuscripts from one AI model

42%

checkable by computer (Lean)

3

withdrawn the next day, for one sign error

Generating results is cheap. Verifying them is not.

OpenAI mathematics releases, Oct 2026 · Retraction Watch, 8 Oct 2026

AI is moving the frontier in life science

The opportunity is real. What should experts do?

AlphaFold Protein Structure Database (2022); Nobel Prize in Chemistry (2024) · Xu Z, et al. Rentosertib in idiopathic pulmonary fibrosis: a randomized phase 2a trial. Nat Med (2025) · FDA draft guidance on AI for regulatory decision-making (Jan 2025); FDA launch of Elsa (Jun 2025)

Six roles for the expert

The job moves from generating to certifying.

1. Questioner

Works while you know more. Fails when wrong answers look right.

2. Learner

The test of understanding: can you derive it by hand?

3. Verifier

Double programming, with an agent as one of the programmers.

4. Manager

You can delegate the work. You cannot delegate the responsibility.

5. Editor

Hundreds of tables become the one paragraph a regulator reads.

6. Owner

The signature stays with a person.

The verifier steers the frontier

Whoever builds the verifier shapes the field.

Statisticians already have a verifier

Same protocol, one box changed: people review the disagreements.

Agents built to be checked

Three agents, each built so a reviewer can check its work.

Three agents, three ways to check

Agent How a reviewer checks it
Literature review Every number links to its source passage
TrialPanorama Every row cites its registry record
Statistical programming An independent QC program and a comparison decide

Wang Z, et al. npj Digit Med 8, 509 (2025) · Wang Z, et al. Nat Commun 16, 947 (2025) · Wang Z, et al. Nat Biomed Eng (2025)

Demo 1: systematic literature review

The reviewer no longer redoes the review. They audit the leaves.

Demo 2: TrialPanorama

An answer you can check in minutes: the query, the counts, and the record.

Demo 3: statistical programming agent

The checks, not the model, decide what is correct.

Seven strategies beyond skills and tools

What makes an agent better, and its work easier to check.

Where can an agent be improved?

Beyond skills and tools: seven strategies, by what they change.

Complete the specification

Same 134 tasks, same agent (Part 1)

Python pass rate by prompt tier

Full133 / 134
Conventions48 / 134
Brief25 / 134

5.3x

more tasks passed with the rules written down

What to do

  • Write the rules down before tuning anything else.
  • What is left out is filled by a language default or by memory of a public program.
  • Reward asking: a flagged ambiguity is a question for the statistician, not a silent choice.

Engineer the context

The model knows only its context. What the harness puts there is a design decision.

Restructure the output

Ask for what tested code can execute. With yamaa, every error became a refusal with named issues, never a wrong dataset.

Adapt the model

Design first, choose where it runs, fine-tune last.

Feed back checkable signals

Every agent loops. What matters is what comes back.

Put the SOP in code

Don’t ask the agent to follow the SOP. Make the SOP run the agent.

When the data talk to the agent

Make a successful injection harmless; don’t rely on the model to resist it.

Check independently

A review that never changes anything is not a control. Track how often reviewers return agent output.

Where AI fits in a validated environment

Validation follows one question: does a person fully check the AI’s output?

FDA draft guidance on AI for regulatory decision-making (Jan 2025) · EU GMP draft Annex 22 · ICH E6(R3) · 21 CFR Part 11

Seven strategies, in one slide

Each strategy makes the agent better and its work easier to check.

Verify before you deploy

How to check any AI tool, and how checking itself fails.

Five questions for any AI tool

Ask every vendor, including us, and your own team.

Verification has its own failure modes

82% vs 46%

radiologists correct when AI advice was right vs wrong

28% → 22%

adenoma detection without AI, after routine AI use

FAA, 2013: fly by hand more often, because autopilot is so good.

Dratsch T, et al. Radiology (2023) · Budzyń K, et al. Lancet Gastroenterol Hepatol (2025) · FAA SAFO 13002 (2013)

Five first steps

  1. Pick one piece of work you check, and write down the rule an AI would miss.
  2. Write one set of rules down so they can be checked.
  3. Test one AI output against a check you already trust.
  4. Ask every AI vendor where answers come from and what checks them.
  5. Keep sign-off with a person.

“What I cannot create, I do not understand.”
Richard Feynman

What I cannot verify, I should not deploy.

References

  • Companion tutorial: Part III (Sections 8–15) for the seven strategies and case studies.
  • Course slides: https://ai4csr.org/slides/asa-boston-2026-11-12/
  • The 722 manuscripts: Retraction Watch, 8 Oct 2026; Nature news, Oct 2026.
  • Verifiable rewards: Guo D, et al. DeepSeek-R1. Nature 645:633–638 (2025) · Arora RK, et al. HealthBench. OpenAI (2025).
  • Reproducibility: Buckheit JB, Donoho DL. WaveLab and reproducible research (1995) · Begley CG, Ellis LM. Nature 483:531–533 (2012).
  • Expert roles: Sadler M, Regan N. Game Changer (2019).

Questions

Thank you. Part 3 next: designing the evaluation of an AI agent as a study.