~ tidal

batch + online co-serving for vLLM

your GPU's decode iterations
use ~1% of the token budget.
tidal sells the other 99%.

An OpenAI-compatible Batch API — /v1/files, /v1/batches, a 24 hour completion window, half the online price — served from the same GPU as your live traffic. Batch work fills the slack in every engine iteration and yields the instant online traffic needs the room.

  • 0.5×online price per token
  • 24 hSLA, laxity-enforced
  • no forkstock vLLM, or one plugin class
  • 69.3%of the node’s offline ceiling recovered, measured (GPU)
fig. 1  one engine iteration, decode-only schematic
A per-iteration token budget filling with batch work A horizontal bar represents one vLLM iteration's compute budget of 2048 tokens. Twenty online chats decoding occupy a 20-token sliver at the far left — about 1 percent. The remaining 2028 tokens are empty ruled space, which repeatedly floods with teal batch tokens and then drains, like a tide, while the online sliver itself swells and recedes with load. τ = max_num_batched_tokens = 2048 tok 1 step · ~10–100 ms 20 tok · 20 online chats decoding · 1.0% of τ 2,028 tok of slack · 99.0% of τ — batch fills it decode is memory-bandwidth-bound: the weights are read from HBM either way filled 1,412 / 2,048
  • online tokens (priority 0)
  • batch tokens (priority 100…1)
  • unfilled budget
A decode step costs one token per running request, and online load is a tide: twenty concurrent chats spend 20 tokens of the 2,048-token budget, a burst of four hundred still under a fifth. The weights are read from HBM regardless, so batch tokens riding the same iteration are nearly free up to the compute knee — and batch yields exactly as much room as the online tide takes. That slack is the product.

⇐how we
got here

how serving learned to batch

Each generation of batching widened what may share one engine iteration: first requests of different lengths, then requests in different phases — and Tidal widens it to two different products. Continuous batching decides which requests join an iteration; Tidal decides on whose behalf the iteration runs. All four eras animate below on one shared clock, sweeping the same iterations — the paper’s Figure 1, live.

the four eras on one clock — every grid sweeps the same iterations rows = engine slots · columns = iterations
Four batching eras animating side by side on one clock (a) static batching (b) dynamic batching (c) continuous + chunked prefill (d) + Tidal deadline contract early finishers idle till the longest ends re-formed between runs; waste remains join/leave per iteration; τ still unspent slack sold to cap X; yields to online
  • online tokens
  • prefill chunk
  • batch-job tokens (Tidal)
  • budget wasted / unspent
  • arrival queued outside
The same figure is Figure 1 of the paper. Static and dynamic batching waste inside the run; continuous batching (Orca) with chunked prefill (Sarathi-Serve) removes that waste but cannot conjure work — sparse online traffic leaves the budget unspent. The fourth era fills the remainder with deadline-contracted batch tokens under a latency cap and KV guardband.

Agateway
technique

policy outside the engine

A sidecar holds the durable Batch API and decides, once a second, how many batch items are allowed in flight. vLLM stays stock. Online traffic never touches the gateway.

co-serving spec §4, §7 · docs/plans/plan-A-gateway.md

fig. 2  dataflow & the AIMD control loop schematic
Technique A: gateway dataflow with an AIMD feedback loop Batch clients POST files and batches to the Tidal gateway, which stores them in SQLite and lets a dispatcher claim items and submit them to a stock vLLM at priority 100 down to 1. Online clients connect straight to vLLM at priority 0 and never pass through the gateway. The dispatcher scrapes vLLM's /metrics once a second; when KV cache usage crosses 0.85 or any online request is queued, it halves the allowed in-flight batch count, and the teal batch flow thins to nothing. When KV usage drops below 0.70 the target grows by one per second and batch flow resumes. batch clients lazycode · openai sdk tidal gateway :8080 · own process POST /v1/files · POST /v1/batches · bearer auth sqlite (wal) durable dispatcher target = 4 / 4 claim order: EDF across batches × prefix group within vllm serve — stock --scheduling-policy priority · V1 engine waiting heap — min(priority, arrival) lower int = sooner running — batch preempted first /metrics kv_cache_usage 0.61 num_requests_waiting 0 grow — kv < kv_low 0.70 online clients your live traffic priority = 100…1 priority = 0 — direct to vLLM, never through the gateway GET /metrics @ 1 Hz → EWMA α=0.5 → AIMD kv_high 0.85 · kv_low 0.70 · online_waiting_tolerance 0 · max_inflight 4 (mac cpu dev default; 64 on GPU)
  • batch items
  • online requests
  • metrics scrape
Two levels of control. The engine is the mechanism: priority ordering admits online first and preempts batch on KV exhaustion, at token granularity, for free. The gateway is the policy: it halves the in-flight batch target the moment KV usage passes 0.85 or an online request queues, and adds one per second once usage falls below 0.70 — so batch work never bloats the waiting queue to begin with. The 1 Hz poll is deliberately slow; the engine covers the gap between ticks.

Btidal
scheduler

policy inside the engine

A --scheduler-cls plugin gates around the stock schedule(). Online tokens fill first; batch fills whatever the interference model says is still free — every step, at token granularity.

scheduler spec §5.1–§5.3 · docs/plans/plan-B-engine.md

fig. 3  per-iteration token-budget ledger & the KV guardband schematic
Technique B: the per-iteration token budget ledger A strip of consecutive engine iterations. Each bar is one step's 2048-token compute budget. Online tokens fill from the bottom in red and are never displaced; batch tokens stack on top in teal, capped by the amber line at the largest count the latency model predicts still fits the time-between-tokens budget. Under an online burst the red grows and the teal is squeezed. When the batch queue runs dry the unfilled remainder is flagged as starved slack. A KV meter on the right rises toward the guardband; at 1 minus H over 2 the scheduler proactively evicts the resident batch request holding the least unrecoverable work, KV usage drops, and batch fill recovers. consecutive iterations → each bar = one step’s compute budget τ τ = 2048 0 kv pool block occupancy 1.0 evict · 1−H/2 = 0.94 admit stop · 1−H = 0.88 H = 12% — EWMA of online burst demand, clamped 5–30% 0.65 resident batch reqs unrecoverable work = computed − prefix-cached req b-8f21 1,920 tok req b-1c04 384 tok req b-77ae 2,560 tok mode online light P_on 24 tok X = max_batch_tokens 1,978 tok T̂(P_on+X, C) 148 ms budget = tbt_slo × (1−guard) 200 ms × 0.8 = 160 ms slack_fill_ratio 0.97 T̂(P,C) = k₁P + k₂P(P+C) + k₄(P+C) + k₅ — fitted online by least squares over a 4,000-step window. Below the R² gate of 0.85 it falls back to a static cap of 25% of τ while online traffic is present.
  • online tokens, scheduled first
  • batch tokens, filling the slack
  • cap / KV pressure / starved slack
Two budgets govern a step and they are not the same thing. Compute — max_num_batched_tokens — bounds tokens processed; KV memory bounds how many requests can be resident. Batch fill is capped by a latency model rather than a fixed quota because interference is not linear in tokens: a 2,048-token chunk costs far more against a long resident context than a fresh one. Admission stops at 1−H; above 1−H/2 the scheduler evicts the batch request with the least unrecoverable work, since preemption is recompute-only and the prefix cache survives.

§24 h
sla

laxity, not the clock

Escalation is driven by deadline risk, not elapsed time. Urgency is measured against a 6 h escalation horizon — the last stretch of slack — not against the 24 h window, so a batch that is draining fast enough never moves, no matter how much of its window has passed. A batch falling behind climbs on its own, early, without dragging every other batch up with it.

co-serving spec §7 · src/tidal/dispatcher/laxity.py

fig. 4  least-laxity-first escalation — a 6 h horizon inside a 24 h window schematic
Laxity-driven priority escalation for three batches Priority is plotted on an inverted axis, so higher on the chart means more urgent: 100 at the bottom, 1 at the top. Urgency is one minus laxity divided by the six-hour escalation horizon, clamped to zero and one, so any batch with at least six hours of slack sits at exactly priority 100. Batch A1, five thousand items draining at twelve per second, has 23 hours 53 minutes of laxity; it sits at 100 and completes after seven minutes without ever escalating. Batch A2, eighty-six thousand items at the same rate, has a constant laxity of 22 hours 1 minute and also sits at exactly 100 for the full two hours — the clock alone never moves it. Batch B also holds at 100 while online load collapses its completion rate; its laxity crosses below the six-hour horizon at 54 minutes, and from there it climbs the whole way to the priority-1 cap in 4.6 minutes, pinning at 58.5 minutes when laxity reaches zero — 4.1 percent of its 24-hour window. A dotted line above the cap marks priority 0, online parity, which escalation reaches only when sla_strict is enabled. effective priority axis inverted — up = more urgent 100 75 50 25 1 batch_priority_min = 1 — the default cap 0 priority 0 = online parity — only when sla_strict = true 0 30 m 60 m 90 m 120 m elapsed — the full window is 24 h; this is its first 8.3% B’s laxity falls below the 6 h horizon at t = 54 m — priority is exactly 100 everywhere left of it pinned at 1 — t = 58.5 m, laxity ≤ 0, only 4.1% of the 24 h window urgency > 0.9 at t = 58.1 m — token-bucket floor engages the whole 100→1 ramp is 4.6 m wide B A₁ done at 7 m · laxity 23h 53m A₂ — laxity 22h 01m, flat at 100 at the playhead t = 120 m A₂ — on track remaining0 observed rate12.00 /s projected drain— laxity22h 0m urgency0.000 priority 100 B — at risk remaining41,708 observed rate0.06 /s projected drain193h 6m laxity−171h 6m urgency1.000 priority 1 drain = remaining / rate laxity = (expires_at − now) − drain u = clamp(1 − laxity/H, 0, 1) H = escalation_horizon = 6 h priority = round(100 − u×99) A₁ (laxity 23h 53m) and A₂ (22h 01m) both sit at exactly 100: laxity ≥ the 6 h horizon ⇒ urgency 0. A₂’s laxity is constant while it drains at its observed rate — the time terms cancel, so priority never drifts. B is dashed where it coincides with A₂ at 100. Only a falling rate or a growing backlog moves a batch.
  • on track — flat at 100
  • at risk — flat, then climbing inside the horizon
Priority is recomputed from laxity — the slack left after the projected drain time — and applied when each item is submitted. Urgency ramps only inside the final 6 h of slack (escalation_horizon); scaling by the full 24 h window would escalate healthy batches on wall-clock alone. Because both time terms cancel for a batch draining at a steady rate, an on-track batch's laxity is constant — A₁ at 23h 53m and A₂ at 22h 01m, both far outside the horizon, so both sit at exactly 100 and never drift just because the day is passing. Healthy batches therefore cause zero extra interference, and at-risk batches never escalate as a synchronized flood. B holds at 100 for 54 minutes, crosses into the horizon, and pins at 1 by 58.5 minutes — 4.1% of its window, early enough that the token-bucket floor still has 23 hours to recover it. Not shown: once the floor engages, the guaranteed rate raises laxity back — a closed loop that stabilizes at the floor.

≡a vs b

which one do you want

Both ship in this repo, share the same store, the same wire format and the same SLA machinery, and are evaluated head-to-head. A is the one you can run against a vLLM you did not build.

same Batch API, two places to put the policy
dimension A — gateway B — TidalScheduler
where policy runs sidecar process, once a second inside the engine, every step (~10–100 ms)
vLLM requirement stock, --scheduling-policy priority pinned vLLM main + --scheduler-cls plugin
batch admission AIMD on /metrics watermarks per-iteration token-budget packing
interference control coarse — throttles how many items exist fine — caps tokens via a context-aware latency model
calibration none self-fitting T̂; no offline profiling sweep
KV protection back off above kv_high; engine preempts guardband H, admit stop at 1−H, proactive evict at 1−H/2
preemption victim engine default — lowest-priority running request cheapest — argmin(computed − prefix_cached)
work conservation leaves slack on the floor by design fills every step; unfilled slack is an alarmed metric
24 h SLA laxity escalation (6 h horizon) at submission time same, gateway-set; in-engine aging deferred past v1
blast radius none — the engine is untouched subclasses a non-public interface; upstream drift risk
phase P1 — usable first P2 — the throughput ceiling

Both give you durable OpenAI-wire /v1/files + /v1/batches, SQLite (WAL) → Postgres by DSN swap, EDF×prefix submission order, and metering at a 0.5× discount.

$quick
start

two terminals

Alpha, under active development. Everything below runs on a laptop against a CPU build of vLLM with a small model.

install and serve

uv venv && uv pip install -e ".[dev]"

# terminal 1 — the engine, unmodified
vllm serve Qwen/Qwen2.5-0.5B-Instruct --scheduling-policy priority

# terminal 2 — the batch tier
tidal serve

submit a batch with any OpenAI SDK

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8080/v1", api_key="tidal-dev-key")
f = client.files.create(file=open("requests.jsonl", "rb"), purpose="batch")
b = client.batches.create(input_file_id=f.id,
                          endpoint="/v1/chat/completions",
                          completion_window="24h")

or move the policy into the engine (technique B)

vllm serve Qwen/Qwen2.5-0.5B-Instruct \
  --scheduling-policy priority \
  --scheduler-cls tidal.engine.scheduler.TidalScheduler

see what it earned

tidal report   # per-batch and per-day cost at the 0.5× batch discount

⇅ladder

the A-ladder: widening the control plane

The gateway generalizes. The same contract — laxity admission, deadline windows, metrics-driven throttling — can place deferrable work over an ever wider scope: one engine, a fleet of replicas, a disaggregated deployment, and finally across time itself. Each rung carries its evidence status honestly: three panels are measured or spike-proven, the dimmed one is design.

A0 · one aggregated engine measured · gpu
Batch fill breathing against the online tide inside one token budget token budget τ harvest 69.3% of ceiling online (the tide) batch fill (AIMD) unfilled slack online price 1.18× / 1.23× p50/p99 — A6000 × Qwen2.5-7B, cross-node
Harvests one engine’s blended slack. 69.3% of the node’s steady-state offline ceiling under flat 20 req/s load; 78.1% under a diurnal trace at 12.1 req/s mean offered load. The bar breathes the way the measurement did: batch fill expands exactly as far as the online tide lets it.
A-F · fleet placement mechanism proven · local
Batch dots flow to the replica currently in its load trough Tidal gateway replica 1 at its trough replica 2 at its peak phase-shifted load · period 600 s +32% harvest vs pinning all batch to one replica
Chases uncorrelated troughs across replicas. Batch volume split almost evenly (1239/1255) but timed into each replica’s own trough: per-replica anti-phase r = −0.83 / −0.56. Proven on a shared-memory CPU rig — the latency side awaits GPU fleets with real isolation.
A-D · disaggregation-aware design only · not yet measured
Signal mapping onto NVIDIA Dynamo-style disaggregation Tidal gateway prefill workers decode workers inject when the prefill pool idles kv_usage → KVBM block metrics waiting → router queue depth observed rate → per-worker throughput
Targets phase-specific slack. Batch work is prefill-heavy while steady-state online is decode-heavy, so the two are complementary. Every signal the gateway scrapes has a direct Dynamo counterpart — this rung needs no new policy, only new signal names. Drawn dimmed because it is design, not measurement.
A-P · phase-split freeze / thaw spike-proven · local
A prefill's KV frozen to disk, thawed into a decode worker hours later prefill worker KV store ❄ frozen · 12 KiB/token decode worker t₁ prefill t₂ decode held 0 h — the 24 h SLA legalizes the gap freeze: persist the finished prefill’s KV
Prefill when prefill is free, decode when decode is free. A thawed TTFT is 31.8× faster than recomputing (7.277 → 0.229 s, median of 5) with bit-identical greedy output, across process death. Total FLOPs unchanged; only their placement in time moves, and the bill moves to storage. Local spike, debug-grade connector.

Rung status matches the paper exactly: A0 measured on GPU (§5), A-F mechanism-proven on a local rig (appendix C), A-D design plus a Dynamo metric mapping, A-P spike-proven locally. The ladder is the complement of technique B, not a competitor: B is maximum visibility over minimum scope; the ladder trades coarser signals for widening scope, ending in a decision B cannot express — parking work in time.

paper

the paper

Tidal: Co-Serving Online and Batch LLM Traffic under Deadline Contracts — the measured version of everything above. On A6000 nodes serving Qwen2.5-7B, the gateway recovers 69.3% of the node’s steady-state offline ceiling at 1.18×/1.23× online p50/p99 under flat load, and 78.1% under a diurnal trace — and the same policy is run inside the engine so both placements are compared head-to-head.

↗more

read the rest

Every figure on this page is a schematic of the designed mechanism, drawn from the specs and the configuration defaults — not a measurement. The measured results are in the paper.