1 of 4
Latency engineering
Time to first token, time to last token, and the perception gap between them.
Two metrics matter. TTFT (time to first token) is dominated by input length and prompt processing. TTLT (time to last token) is dominated by the number of output tokens, since generation is sequential. Optimising the wrong one wastes effort: shortening a 40-token answer does nothing for TTFT, and caching a huge prefix does nothing for a long generation.
- Stream for any human-facing surface — it collapses *perceived* latency even with identical total time.
- Cache the stable prefix; this is often the single largest TTFT win on long prompts.
- Ask for shorter output: concise formats, no restating the question, no preamble.
- Use a smaller model tier for latency-critical steps such as routing and classification.
- Parallelise independent calls instead of chaining them.
- Disable extended thinking where the task does not need reasoning.
Exam-ready takeaways
- TTFT ← input size and caching. TTLT ← output token count.
- Streaming changes perception, not throughput — and perception is what users report.
- Shorter outputs are usually the biggest end-to-end latency lever.