Agent Systems Need Nonlinear Cost Optimization in Production

The article argues that many AI agent systems become economically unsustainable long before they look technically impressive. Teams often focus on model choice, prompt design, tool calling, and orchestration, but the deeper issue is that real-world cost scales nonlinearly with each user request. A single request can expand into routing, retrieval, reasoning, reflection, guardrail checks, tool calls, and synthesis. Because steps may repeat shared context, re-run planner decisions, or retry failed paths, the workflow behaves like a recursive, stateful computation with overlapping subproblems. The author says agent systems can “pass review” and unit tests while still being too expensive due to hidden computation that shows up in the invoice. Key mitigation ideas include “optimization layers” drawn from classical computer science: - Memoization: cache not just prompts, but repeated decisions and equivalent workflow states. - Pruning: stop reflection loops and remove unproductive branches, including cases where tools return no new information. - Dynamic programming: share work across branches with overlapping subproblems (e.g., similar questions across documents). The article also stresses that optimization must follow the agent topology: - Centralized setups should memoize planner decisions. - Decentralized and swarms should prioritize prompt caching on shared context and pruning redundant communication. - Hybrid designs need memoization across clusters and dynamic programming inside clusters. Named example coding agents include Claude Code, Codex, and Jules, but the main takeaway is operational: optimize agent systems end-to-end, or latency and cost will compound as workflows get more complex.