Tidal: Co-Serving Online and Batch LLM Traffic under Deadline Contracts

A serving node sized for interactive traffic spends most of its life well below its throughput ceiling. On our GPU testbed, a vLLM node serving only online chat traffic emitted about a third of the output tokens per second that the same node sustains when saturated with offline work — and capacity left unspent in one scheduler iteration cannot be banked for the next. Commercial providers sell exactly this slack as a second product: a batch API at half the online price with a 24-hour completion window. A self-hosted deployment has no equivalent product. vLLM's own batch tooling can drain an offline workload at full speed, but only by dedicating the machine to it, and the research systems that do harvest slack behind live traffic (HyGen, ConServe) fork the engine and promise no completion time — no deadline a customer could plan around. Tidal supplies the missing product over unmodified vLLM: an OpenAI-compatible /v1/batches front end, an admission test that refuses any deadline window the system's own observed completion rate cannot meet, and per-job escalation governed by laxity — the classical real-time measure of how much time a job can still afford to lose before its deadline is at risk. On RTX A6000 nodes serving Qwen2.5-7B under a flat 20 req/s online load, a gateway sidecar recovers 69.3% of the node's steady-state offline ceiling while online request latency rises to 1.18× a matched node's online-only baseline at the median and 1.23× at the 99th percentile; under a diurnal trace at 12.1 req/s mean offered load it recovers 78.1%. We also implement the identical policy inside the engine as a scheduler plugin and compare the two placements of one policy — outside the engine versus inside it — as an explicit design axis. Code and evidence are open source.

• PDF arXiv-style preprint, 21pp download
• DOI 10.5281/zenodo.22076918 zenodo
• Code rajagurunath/tidal, Apache-2.0 source

@misc{lunkupalivenugopal2026tidal,
  title        = {Tidal: Co-Serving Online and Batch LLM Traffic under Deadline Contracts},
  author       = {Lunkupali Venugopal, Gurunath},
  year         = {2026},
  publisher    = {Zenodo},
  doi          = {10.5281/zenodo.22076918},
  url          = {https://doi.org/10.5281/zenodo.22076918}
}