> Instead of fixing the rate limiter that was blocking its own tests, it patched the environment detection. That's enterprise development in a nutshell.
That part made me laugh and reflects my experience when I was working on "enterprise development" teams.
I'm curious about having this run on a VPS as opposed to a local VM. What does that provide you? I understand having the VM completely disconnected from your local network etc but is there anything besides that? I ask because mentioning the cost here seemed like one of the important points but that cost wasn't necessary for the experiment itself.
Curious how the top performers compare to SOTA paid models. Also interested in understanding performance differences between running model on VPS hardware compared to a laptop. Thanks for sharing
I don't believe the models were running on the VPS itself. According to my understanding of the article, they used OpenRouter and OpenCode's service as service providers. The agent was the thing running on the VPS.
> Instead of fixing the rate limiter that was blocking its own tests, it patched the environment detection. That's enterprise development in a nutshell.
That part made me laugh and reflects my experience when I was working on "enterprise development" teams.
I'm curious about having this run on a VPS as opposed to a local VM. What does that provide you? I understand having the VM completely disconnected from your local network etc but is there anything besides that? I ask because mentioning the cost here seemed like one of the important points but that cost wasn't necessary for the experiment itself.
I’m a little confused. So you got a VPS… was that just to host the result? Surely you weren’t running the models on it?
What were the models running on? It doesn’t matter a ton but would help paint the whole picture better.
Also no mention of quants and not even sizes in many cases. I wish someone did this kind of comparison well.
Opencode doesn't publish what quantization they offer.
What VPS is $25/year? DigitalOcean seems to be about 3x that price for the low end droplets…
Curious how the top performers compare to SOTA paid models. Also interested in understanding performance differences between running model on VPS hardware compared to a laptop. Thanks for sharing
I don't believe the models were running on the VPS itself. According to my understanding of the article, they used OpenRouter and OpenCode's service as service providers. The agent was the thing running on the VPS.