Normal pytest tests assume a few things: same input gives same output, there’s one right answer, and it costs nothing to run. Add LLM agents to the mix and your assumptions go out the window. LLMs are probabilistic, and for closed models you can never guarantee the same output.

In this talk, I’ll walk through my journey on building (with) LLMs, the need for tests, a framework that was built from these needs, and the features of the Pytest plugin that came from it for testing LLM agents: aggregate runs, the different kind of checks, among other things. We’ll start on “what is a test” all the way to what’s the industry standard for LLM testing, and how we can make it practical with Pytest and CI runs.