HP ZBook Ultra G1a 14: 64GB Unified Memory Hits 8.6 t/s on 32B Local Models
AMD Ryzen AI Max PRO 390 with 64GB unified memory runs 32B models at 8.6 t/s — parity with a 16GB RTX 5080 laptop, no discrete GPU needed.
#local-ai
33 posts
AMD Threadripper Halo Station: 576GB HBM3E for Trillion-Parameter Workloads
AMD's Threadripper Halo Station packs a 96-core CPU with dual MI350P accelerators and up to 576GB of HBM3E — a $150K desktop that runs trillion-parameter models locally.
Your 4-Bit Model Can Now Beat Its 16-Bit Original
Quantization-Aware Healing distills a 4-bit model from the pre-compression teacher, winning 7 of 9 benchmarks over its bf16 source.
Nvidia Vera Adds 88 Olympus Cores — CPU Inference Gets a Serious Boost
Nvidia's Vera server CPU, 88 Olympus cores, targets AI inference. A credible alternative for token-heavy workloads, but software matters.
Granite 4.2 Brings Agentic RL Down to a 3B Open-Weights Model
IBM's Granite 4.2 — 3B/8B/30B, Apache 2.0, 512K context, thinking/non-thinking switch, and agentic RL on 8B/30B with real SWE, terminal, and search environments.
GLM-5.3 Hits 88.2 on Terminal Bench 2.1 as Open-Weights
Z.ai open-sourced GLM-5.3 (753B MoE). 88.2 on Terminal Bench 2.1 and 28.3 on TB 3.0, serving via vLLM/SGLang with a reasoning_effort flag and 23 existing quantizations.
1T-Param Model Hits 1000 tok/s Without Custom Silicon
TileRT 1.5 runs a 1T-param Xiaomi model at 1000+ tokens/s on a single 8x B200 node. Here's the exact stack, commands, and where it fits for low-latency serving.
Qwen 3.8-27B: 73 on Terminal Bench 2.1 and Still a 24 GB Model
Qwen 3.8-27B is the new dense flagship you can run on a single 24-32 GB GPU. It jumps to 73.0 on Terminal Bench 2.1 (up from 63.4), adds native image and hour-scale video understanding, and ships under Apache 2.0. What changed versus Qwen3.6-27B, how to run it at 4-bit, and the catches I would watch.
vLLM 0.27 Removes First-Request Stalls — But Torch 2.13 Is Breaking
vLLM 0.27.0 (Aug 10) ships JIT warmup and runner-owned Triton warmup that eliminate first-request compile stalls, plus Kimi K3 support — but the torch 2.13.0 upgrade is a breaking change that needs migration testing before you bump.
Hybrid-Attention Models Finally Run Offline in vLLM
vLLM v0.27.0 lets hybrid SWA+full attention models run through offline inference, closing the gap between eval and serving — plus a breaking torch 2.13 upgrade.
295B Parameters, 21B Active—Hy3 Proves Size Isn't Agent Performance
Tencent's Hy3 benchmarks between models 2–5x larger by parameter count. 256K context, Apache 2.0, 1 yuan per million input tokens. The catch: vendor benchmarks need independent testing.
Task-Specific Logic Compiles to 23MB Files Running Offline on 600M Models
University of Waterloo's PAW system compiles fuzzy-function specs into 23MB LoRA adapters that match 32B model accuracy on a 600M interpreter—no API calls, no cloud dependency, 30 tok/s on MacBook M3.
RTX 50 Prices Jump 19% and the 5070 Ti Is Gone — Buy Now or Wait
NVIDIA cut RTX 50 supply by 20% and sidelined the 5070 Ti. Street prices are up 19%. Here's how it changes your local AI build.
GLM-5.2 Shifts Agentic Coding Economics to 1/6th Frontier Cost
Z.AI's open-weights GLM-5.2 hits 81% on Terminal-Bench, undercutting GPT-5.5 for a fraction of the cost. Here's the benchmark breakdown.
DiffusionGemma Puts 1000+ tok/s on an RTX 5090
Google's open 26B diffusion model hits 700+ tok/s on consumer GPUs with day-zero vLLM support. Here's what the bidirectional architecture changes for local inference.
The Grid, Not the GPU, Is Your Next Inference Bottleneck
Seattle's data center ban and 3-year grid queues mean local inference and modular hardware are no longer optional.
OpenAI's $85B Burn Rate Changes the AI Cost Equation
The SEC filing exposes a nine-to-one burn-to-revenue ratio. Here is why local inference is shifting from a privacy play to a hard P&L requirement.
Your API Costs Are About to Correct
OpenAI's IPO filing ends the era of subsidized frontier AI. Here's what the pricing shift means for local inference, enterprise contracts, and your agentic stack.
NVIDIA RTX Spark: What 128 GB of Unified Memory Means for Local AI
NVIDIA's first consumer SoC brings 128 GB of unified memory and native CUDA to Windows laptops — here's what it means for running LLMs locally, and the catches.
Apple Silicon Just Cracked the 150B Wall at 56 tok/s
Rapid-MLX's new engine optimizations push DeepSeek V4 Flash 158B-A13B to 56 tok/s on a Mac. Here are the exact numbers and the tradeoffs.
On-Device Agents Just Gained a 6GB MoE That Actually Works
Liquid AI's LFM2.5-8B-A1B hits under 6GB RAM with 1.5B active parameters, native tool calling, and verified throughput across edge hardware.
Your 12GB MTP Throughput Just Jumped 23%
A community llama.cpp fork squeezes 110 tok/s out of Qwen3.6-35B-A3B MTP on a 12GB card. Here are the exact flags and the VRAM trick to make it fit.
Your Local Throughput Just Doubled (With No Accuracy Tax)
Multi-token prediction is merged into llama.cpp, delivering 1.4–2.2x throughput on Qwen3.6 with zero accuracy loss.
MTP in llama.cpp Adds 24–33% Throughput on Qwen3
llama.cpp merged Multi-Token Prediction for Qwen3. Community benchmarks show 38→47 tok/s on RTX 3090 and 63→84 tok/s on RTX 5090 — no new hardware needed.
DFlash Decode Collapses at Long Output. MTP Doesn't.
One llama.cpp flag—--spec-draft-p-min 0.75—turns MTP from a dud into a decode speed that holds flat across output lengths where DFlash falls apart.
One llama.cpp Flag Turns MTP From Dead Weight to 68% Faster
The --spec-draft-p-min filter in llama.cpp PR #22397 rescues MTP for Qwen3.6-27B: 48.9 tok/s vs 29 tok/s at 2000 tokens on a 24GB card.
Your vLLM Thinking Budget Was Doing Nothing With MTP On
vLLM 0.21.0 shipped Friday with a quiet fix: thinking_token_budget was being silently ignored when MTP speculative decoding was enabled. If you serve reasoning models with spec decode, you have been paying for it.
Your Local Qwen3.6 Throughput Probably Just Halved (and How to Fix It)
llama.cpp renamed the MTP flag on May 13. The old --spec-type mtp is silently ignored. If your tok/s dropped from 140 to 70 you are likely running without speculative decoding.
Speculative Decoding Explained: Why Your Local Model Got 2× Faster in 2026
The same Qwen3.6-27B that ran at 70 tokens/sec on a 4090 in January was running at 140 tokens/sec by April. Nothing changed about the model. Speculative decoding moved from research curiosity to default. Here is what it actually does.
The Local AI Inflection Point: May 2026
Three model releases in three weeks moved local AI from 'good enough for hobbies' to 'good enough for production'. Here's what changed and why it matters.
Running Qwen3.6-27B Locally: Hardware, Quantization, and What Actually Works
A practical guide to running Qwen3.6-27B on consumer hardware in 2026 — memory requirements per quant level, recommended runners, and the MTP trick that doubles your tokens per second.
A 27B Model on a Single GPU Is 10 Points Off Claude Opus 4.7
Qwen3.6-27B running locally now scores within 10 points of frontier closed models on SWE-bench Verified. The benchmark table, lined up side by side.
Gemma 4: Google's Open Model Family Goes Multimodal
Google released Gemma 4 on April 2, 2026 — four variants from 2B to 31B, with 256K context, native vision and audio, and Apache 2.0 licensing. Here's what it's for, where it fits, and how to run it.