tracelab

Upload a CSV and ask questions in plain English. A team of LangGraph agents plans the analysis, writes and runs Python in a sandbox, and streams the whole thing back live. You watch the router, planner, parallel analysts, critic, and composer light up on the trace, with their token cost ticking up as they go. Most agent demos show you only the answer. tracelab shows the reasoning and the verification behind it.

The project treats LLM output as untrusted by default. A critic independently re-derives every number an analyst claims, and it never sees the analyst's code, only the question, the dataset profile, and the claim, so agreement actually means something. A deterministic reconciliation function, not the model's judgment, decides whether the two match: integers exactly, floats within a relative epsilon, and statistical claims on direction, significance, and whether the method was appropriate at all. A discrepancy triggers exactly one bounded retry with the critic's findings injected. Anything still unresolved ships flagged as unverified rather than quietly dropped.

The measurement layer is the other half of the project. Every run is stored as a span tree with per-agent tokens, cost, and latency, and replays offline for free, with the same result every time. A golden eval set of 33 questions is computed from the actual CSVs rather than hand-typed, then scored in two tiers: programmatic comparison, plus an LLM judge pinned to a model independent of the config under test. Every run is tagged with a config hash covering the per-role models and a content digest of every agent prompt, so you can tell which change moved the pass rate.