Inference is all about tradeoffs. Therefore choosing an inference provider means choosing a set of tradeoffs which matters a lot for every use case.
Providers serving the same model can differ in first-token latency, generation speed, caching, cost and reliability. I wanted to understand which differences matter for which workloads. So, I created an open source repo to run experiments for model-provider pairs with ease on either existing benchmarks or with my own traces.
https://github.com/aktasbatuhan/compound
I’m also interested in making those choices easier to revisit. With closed models, choosing the intelligence often means choosing the vendor. Open models give us more freedom to choose who serves them, but that freedom becomes useful only when we can measure the alternatives.
I tested DeepSeek V4 Flash across 14 serving routes: 12,795 measured calls, input sizes from 1k to 100k tokens, two output budgets, and cold and warm prompts. Temperature was zero and reasoning was off.
My favorite outcome was how much caching changed across providers. On DeepSeek’s own route, a repeated 100k prompt with a 100-token output budget cost about 15 times less than the cold condition. The cheapest option for fresh prompts wasn’t necessarily the cheapest for repeated ones.
Automatic routing adds another variable. OpenRouter can send requests to different upstreams, while cached prefixes depend on where those requests land. Auto routing still achieved substantial cache hits in this experiment, but its warm-request costs were higher than the cheapest pinned alternatives.
The report compares these tradeoffs with interactive charts and downloadable measurements. I built the experiment with Compound so others can run similar comparisons on their own prompts. The useful question is which provider meets your workload’s latency, cost and reliability requirements.
Inference is all about tradeoffs. Therefore choosing an inference provider means choosing a set of tradeoffs which matters a lot for every use case.
Providers serving the same model can differ in first-token latency, generation speed, caching, cost and reliability. I wanted to understand which differences matter for which workloads. So, I created an open source repo to run experiments for model-provider pairs with ease on either existing benchmarks or with my own traces. https://github.com/aktasbatuhan/compound
I’m also interested in making those choices easier to revisit. With closed models, choosing the intelligence often means choosing the vendor. Open models give us more freedom to choose who serves them, but that freedom becomes useful only when we can measure the alternatives.
I tested DeepSeek V4 Flash across 14 serving routes: 12,795 measured calls, input sizes from 1k to 100k tokens, two output budgets, and cold and warm prompts. Temperature was zero and reasoning was off.
My favorite outcome was how much caching changed across providers. On DeepSeek’s own route, a repeated 100k prompt with a 100-token output budget cost about 15 times less than the cold condition. The cheapest option for fresh prompts wasn’t necessarily the cheapest for repeated ones.
Automatic routing adds another variable. OpenRouter can send requests to different upstreams, while cached prefixes depend on where those requests land. Auto routing still achieved substantial cache hits in this experiment, but its warm-request costs were higher than the cheapest pinned alternatives.
The report compares these tradeoffs with interactive charts and downloadable measurements. I built the experiment with Compound so others can run similar comparisons on their own prompts. The useful question is which provider meets your workload’s latency, cost and reliability requirements.