TPU 8i Adds 288GB HBM and 384MB SRAM for Agent Inference
Google's eighth-gen TPUs split into TPU 8t for training (121 ExaFlops, 9,600 chips) and TPU 8i for inference (288GB HBM, 80% better perf-per-dollar).
#inference
6 posts
NVIDIA's Custom HBM Signals Memory Bandwidth Is the New Constraint
NVIDIA's NVHBM moves memory control to the HBM stack itself, not the GPU. This architectural shift reveals memory bandwidth, not compute cores, is the real scaling bottleneck for AI inference.
vLLM 0.27 Release: New CondensePyramid V1 Attention Kernel and Multi-Modal Token Limits
vLLM 0.27 adds CondensePyramid V1 attention for multi-modal long context, new token-tier limits on templates, faster ASR CPU preprocessing via multi-threading, and CPU W4A16 INT4 MoE support.
1T-Param Model Hits 1000 tok/s Without Custom Silicon
TileRT 1.5 runs a 1T-param Xiaomi model at 1000+ tokens/s on a single 8x B200 node. Here's the exact stack, commands, and where it fits for low-latency serving.
vLLM 0.27 Removes First-Request Stalls — But Torch 2.13 Is Breaking
vLLM 0.27.0 (Aug 10) ships JIT warmup and runner-owned Triton warmup that eliminate first-request compile stalls, plus Kimi K3 support — but the torch 2.13.0 upgrade is a breaking change that needs migration testing before you bump.
Retune Draft Models Without Restarting vLLM
vLLM v0.26.0 adds runtime draft-weight updates, so you can retune speculative decoding without restarting the engine and flushing the KV cache.