FastAPI’s BackgroundTasks looks like it solves “run this later.” It does not survive a restart, does not retry, and gives no way to check if a task finished. Even plain asyncio.create_task() has a gotcha most Python developers have hit without knowing why: if you do not hold a reference to the task, it can be garbage collected mid execution and silently vanish.

I previously spoke at PyDelhi Conference on streaming output out of a Python service. This talk is the mirror problem: keeping work alive inside your server when it takes minutes or hours, and surviving a crash halfway through.

I will walk through three rungs of solving this. First, why BackgroundTasks and fire and forget tasks quietly fail in production. Second, what task queues like ARQ, Celery, and Dramatiq add, and what they still do not give you: if step six of a ten step pipeline crashes, you restart from zero, not step six. Third, the idea behind durable execution tools like Temporal, DBOS, and Hatchet: persisting every completed step so a crashed pipeline resumes instead of repeating work.

Live on stage, I will run a real pipeline, kill the process mid run, and restart it to show it resuming exactly where it left off, using a small checkpointing library built from the Python standard library alone.

I will close with a practical framework for choosing between BackgroundTasks, a task queue, and a checkpointed pipeline based on how long your job runs.