Engineering

10 posts

Prefill decode disaggregation isolates compute phases across separate GPU pools in a datacenter to optimize large language model serving.

Engineering • Aug 03, 2026 •11 min read

Disaggregated GPU Inference Hits the KV Cache Wall

Disaggregated GPU inference splits prefill and decode for higher throughput, but each request moves 2.6 GB of KV cache across the datacenter.

BM25, dense, and SPLADE retrieval baselines compared for production RAG, each with distinct tradeoffs in memory footprint, query latency, and failure behavior.

Engineering • Jul 27, 2026 •13 min read

BM25 vs Dense Retrieval and SPLADE for Production RAG

BM25 vs dense retrieval on a consumer GPU. Each baseline wins one RAG scenario and breaks another. Match your index to memory, latency, and failure mode.

Financial review of AI agent cost per resolution showing why unit economics determine whether an agent ships to production.

Engineering • Jul 19, 2026 •12 min read

AI Agent Cost Per Resolution Decides If It Ships

AI agent cost per resolution, not eval accuracy, decides if your agent ships or dies. Token pricing hides the unit economics of retries and failures.

Netflix's in-house LLM serving platform illustrates what large-scale self-hosted inference infrastructure looks like when workload thresholds justify the investment.

Engineering • Jul 18, 2026 •12 min read

Self-Host LLM Inference, the Netflix Decision Framework

Learn when to self-host LLM inference instead of paying per token. Netflix's production stack reveals the real cost, latency, and control tradeoffs.

Illustration of a server network designed to orchestrate open source LLMs efficiently.

Engineering • Jul 16, 2026 •12 min read

Orchestrate Open Source LLMs vs Frontier Models

Should you orchestrate open source LLMs or call a single frontier model? We break down Sakana Fugu's claims and the real cost and latency tradeoffs.

Abstract visualization of llm agent scaffolding showing structured layers that organize and constrain software logic.

Engineering • Jul 11, 2026 •13 min read

How LLM Agent Scaffolding Fixes Failing Code Review Agents

LLM agent scaffolding constrains what models see and call. Learn why GitHub Copilot code review regressed with more tools and how routing helps.

Implementing typed schema contracts enforces strict data boundaries to successfully stop rag hallucination in automated pipelines.

Engineering • Jul 04, 2026 •12 min read

Stop RAG Hallucination with Typed Schema Contracts

Stop RAG hallucination with typed schema contracts. Build programmatic answer contracts, validate field-level citations, and handle missing data.

domain-specific LLM evaluation in a practical business context.

Engineering • Jul 03, 2026 •11 min read

Domain-Specific LLM Evaluation Demands Leakage-Free Data

GPT and Claude failed Bridgewater's private financial evals. Discover what this reveals about LLM benchmark leakage and how to build robust holdout sets for domain-specific testing.

Visual support for secure AI coding agents in an operational workflow.

Engineering • Jun 29, 2026 •12 min read

Secure AI Coding Agents From Supply Chain Attacks

Learn how to secure AI coding agents against supply chain attacks. Discover how to prevent prompt injection malware execution using sandboxing and strict file permissions.

Visual support for human in the loop systems in an operational workflow.

Engineering • Jun 25, 2026 •12 min read

Human-in-the-Loop Systems: Balancing AI Speed and Operational Safety

Discover why full automation fails at scale and learn 5 architectural patterns for human-in-the-loop systems to balance AI speed with operational safety.