Incredibly happy to see this series of CF articles. I was always so proud of devs back in the days where RAM and processing were scarce and who had to get creative to fit even the most basic stuff in the budget. It seemed to me that after RAM and processing became abundant, most gave up on optimization and focused on shipping instead which meant now that even with several cores, a basic notepad or music player failed to work. In a way, RAM becoming more expensive has ushered in a new era of forced optimizations, which I'm really happy for
I don't remember those days with a ton of fondness. Yes, the challenge was fun, but I really wanted to ship it and get my product in the hands of customers. Now I can spend more time thinking about what they want and less time about what the computer wants.
The world turned millenia ago and will turn millenia later.
The problem statement is of applications using up expensive RAM. Incidentally, expensive RAM is just one of the problems we face in the computing space. Forced obsolescence is another, when running hardware needs to be replaced because software is built for only newer CPUs.
Yes, I remember those too. The costs of manual memory management were real and were not low.
But costs on the cloud are real too, especially now. I’ve been living in JVM land for a very long time, but now it’s especially clear how important lean services are. Especially now that the bar for writing lean code is so much lower: let the borrow checker figure it out, etc.
I just spent a couple days wringing out more performance/memory efficiency for our services. Nice gains to be sure, but it’s still so immensely wasteful compared to something well written running native. If it was my money, I’d be going native for sure.
I would get rid of consistent hashing and ketama for a better system which works save an additional 600TiB.
You use the first N bits of your key hash to pick the server partition so it’s a reasonable number (eg 128 servers per partition). Then use high quality precomputed hashes (first 64 bits of sha256) for the server name as N in H(K + N). Use wymum from wyhash as the H so that you do o(n) integer multiplications while retaining a result that’s still a good hash statistically.
Now you’re using a tournament hash, the small N means O(N) vs O(N log N) doesn’t matter, and also this O(N) is also going to be much less CPU than computing 160 hashes per key as they do now, so much less latency added per request.
Cloudflare is truly amazing, they have made so much possible for my main side-project at a price and performance that I can’t really take credit for (http://sourcelibrary.org), I don’t care if their text was written with AI, I just wish I could get my own AI to sing so well about hashing… but wait.. today I noticed Claude trying to use hashing when a timestamp would honestly do, and now I’m really doubting myself, hmm…
This is pleasant coincidence. Really like your site and it's queued to send in my newsletter in the morning! Just happened to see your comment here while I was reading. Great work.
> I don’t care if their text was written with AI, I just wish I could get my own AI to sing so well about hashing… but wait.. today I noticed Claude trying to use hashing when a timestamp would honestly do, and now I’m really doubting myself, hmm…
Tried out the first 1000 words in Pangram, and it seemed happy it was human written. Not surprised either, it has been some of the better writing I've seen out of Cloudflare recently.
These optimizations are impressive, but it gets me thinking: at what point does a company become a collection of impenetrable siloes, where nothing really does what you expect? Maybe know with AI this is less of an issue as exploring a codebase is also much faster.
I feel like any company whose products have RESTful interfaces are already there…
One wants to turn on an indicator on a remote device. A simple Boolean value. But we need networking, TLS, authentication plugins, certificate validation, distributed logging, containers, orchestration, HTTP client/server, interprocess communication, daemon dependency management, …
Sure, one can say each of these layers and abstractions has an important and justifiable purpose. But one can also step back and start wondering - what the hell are we really doing???
At some level, it seems like each layer of abstraction has to manage others, only simply because they exist.
Imagine the simplicity of 1800s telegraph signaling - no software!
Too often we build systems with Fortune-50 style hierarchies when a 5-person team could do the whole job.
I mean you can just pause and refactor regularly. I tend to use every ~4th session as an opportunity to refactor, adjust interfaces, break overly large modules into smaller ones and suchlike.
And at what point does a company start to care about performance? 100TB of RAM is expensive as hell, but getting products to market faster was worth the cost
This sort of thing makes me thing that we're about to enter an era where software development is going to be where most of the jobs fallout will be. You can't one-shot vibe code your way to this. But for proper Software Engineering, those jobs are safe where more and more problems are going to actually need solving by creatively using math because all the problems individuals deliver are just going to be larger. People are just mourning the loss of the low hanging fruit.
I think you're speaking like a software engineer, which is understandable, and not like a historian or economist. There's no reason to believe math based jobs would survive. The models regularly do well on math problems. You can auto-research loop ways to optimize memory usage for any particular program.
The point isn't that "doing math" is safe. Auto-research solves one target variable in one system, doing it at scale where say one developer is SME for the agentically manged 200 microservices down the line, heh I mean you certainly can, but good luck with that token cost of auto-research when that problem space is O(microservice^2). I point at that example yesterday of that optimized database memory with the comments pointing out that the specific problem fit in memory, over optimized and didn't generalize. The problem isn't the work, but the rework. A historian should know that new solutions to problems doesn't lead to "no problems ever again" but only problems with barriers that the new solution doesn't solve.
The argument is probably that LLMs can find those optimizations cheaper than a human expert. Since LLM cost at fixed capability seems to be going down you either expect humans to be completely replaced or human wages to be lowered by LLMs.
The only Rust section is the one on storage improvements about the struct that stores the hash, but do they really need that many hashes that 2 bytes makes that big of a difference? Article doesn't expand, but I guess it's a hash for every task on every computer, so maybe yes.
That's exactly his point/the area of cost saving - that they didn't actually need as many hashes as they had started with. The trick was in finding out how many hashes they could cull without degrading load balance.
Bang up article! As someone who doesn't get to do enough (almost any) calculus in my daily programming assignments, I thoroughly enjoyed reading about Kevin's dive into that derivation (linked in the supplemental article). All the people being negative here can swallow raisins
CPU DRAM can't really be used for inference efficiently -- inference mostly wants memory bandwidth, not memory capacity, and GPU DRAM has >10x more bandwidth. The fabs can switch between them but you can't switch after the fact.
Those machines with GPUs still need RAM of their own, and they generally want large caches to avoid SSD penalties. You even see this spill out in the form of costs for KV cache in <1min, 5m, 1hr rates etc.
Incredibly happy to see this series of CF articles. I was always so proud of devs back in the days where RAM and processing were scarce and who had to get creative to fit even the most basic stuff in the budget. It seemed to me that after RAM and processing became abundant, most gave up on optimization and focused on shipping instead which meant now that even with several cores, a basic notepad or music player failed to work. In a way, RAM becoming more expensive has ushered in a new era of forced optimizations, which I'm really happy for
I don't remember those days with a ton of fondness. Yes, the challenge was fun, but I really wanted to ship it and get my product in the hands of customers. Now I can spend more time thinking about what they want and less time about what the computer wants.
And its this obsession with shipping things as fast as possible quality be dammed that got us basic weather apps that eat a gigabyte+ of RAM.
And yet the world still turned
The world turned millenia ago and will turn millenia later.
The problem statement is of applications using up expensive RAM. Incidentally, expensive RAM is just one of the problems we face in the computing space. Forced obsolescence is another, when running hardware needs to be replaced because software is built for only newer CPUs.
Yes, I remember those too. The costs of manual memory management were real and were not low.
But costs on the cloud are real too, especially now. I’ve been living in JVM land for a very long time, but now it’s especially clear how important lean services are. Especially now that the bar for writing lean code is so much lower: let the borrow checker figure it out, etc.
I just spent a couple days wringing out more performance/memory efficiency for our services. Nice gains to be sure, but it’s still so immensely wasteful compared to something well written running native. If it was my money, I’d be going native for sure.
(Cat reading newspaper) I should build a database out of pointers
Different people are different.
Some of us find production and optimization more interesting than marketing and distribution.
incredible way to put it!!
I would get rid of consistent hashing and ketama for a better system which works save an additional 600TiB.
You use the first N bits of your key hash to pick the server partition so it’s a reasonable number (eg 128 servers per partition). Then use high quality precomputed hashes (first 64 bits of sha256) for the server name as N in H(K + N). Use wymum from wyhash as the H so that you do o(n) integer multiplications while retaining a result that’s still a good hash statistically.
Now you’re using a tournament hash, the small N means O(N) vs O(N log N) doesn’t matter, and also this O(N) is also going to be much less CPU than computing 160 hashes per key as they do now, so much less latency added per request.
Cloudflare is truly amazing, they have made so much possible for my main side-project at a price and performance that I can’t really take credit for (http://sourcelibrary.org), I don’t care if their text was written with AI, I just wish I could get my own AI to sing so well about hashing… but wait.. today I noticed Claude trying to use hashing when a timestamp would honestly do, and now I’m really doubting myself, hmm…
This is pleasant coincidence. Really like your site and it's queued to send in my newsletter in the morning! Just happened to see your comment here while I was reading. Great work.
> I don’t care if their text was written with AI, I just wish I could get my own AI to sing so well about hashing… but wait.. today I noticed Claude trying to use hashing when a timestamp would honestly do, and now I’m really doubting myself, hmm…
Tried out the first 1000 words in Pangram, and it seemed happy it was human written. Not surprised either, it has been some of the better writing I've seen out of Cloudflare recently.
An AI would have known that saying, “Hi, mom” in a professional post was a bad idea.
What does Cloudflare make possible for your project?
These optimizations are impressive, but it gets me thinking: at what point does a company become a collection of impenetrable siloes, where nothing really does what you expect? Maybe know with AI this is less of an issue as exploring a codebase is also much faster.
I feel like any company whose products have RESTful interfaces are already there…
One wants to turn on an indicator on a remote device. A simple Boolean value. But we need networking, TLS, authentication plugins, certificate validation, distributed logging, containers, orchestration, HTTP client/server, interprocess communication, daemon dependency management, …
Sure, one can say each of these layers and abstractions has an important and justifiable purpose. But one can also step back and start wondering - what the hell are we really doing???
At some level, it seems like each layer of abstraction has to manage others, only simply because they exist.
Imagine the simplicity of 1800s telegraph signaling - no software!
Too often we build systems with Fortune-50 style hierarchies when a 5-person team could do the whole job.
Build a system as simple as possible but no simpler.
An 1800s telegraph system doesnt work in the modem world, there is far too much communication and the system would just collapse into molten slag.
All those things you've listed are because we live in an adversarial world and I'd steal all your money off the telegraph wire if you tried it.
Having worked in hardware for a moment, everything we do in software is like this. Even C.
There are a whole series of blog posts from the Fishworks guys explaining why it could possibly be so hard to turn on one LED.
But Oracle probably deleted then so you'll have to find them on archive.org.
like this one? https://eschrock.dtrace.org/2008/07/
My intuition around larger companies is they are already impenetrable silos, and AI makes it worse
Actually, AI creates spaghetti faster than any human ever could before.
The steelman argumentation is probably that spaghetti doesn't matter to LLMs and no humans will read that code anyway.
Do you feel like this is a very meaningful comment? Will someone go "mm yes, actually it's spaghetti, I didn't think of that.."
Are you trying to succinctly say AL'S spaghetti code outweighs the benefits of what it produces quickly?
If you're not saying it outweighs it, what are you saying?
I mean you can just pause and refactor regularly. I tend to use every ~4th session as an opportunity to refactor, adjust interfaces, break overly large modules into smaller ones and suchlike.
And at what point does a company start to care about performance? 100TB of RAM is expensive as hell, but getting products to market faster was worth the cost
This sort of thing makes me thing that we're about to enter an era where software development is going to be where most of the jobs fallout will be. You can't one-shot vibe code your way to this. But for proper Software Engineering, those jobs are safe where more and more problems are going to actually need solving by creatively using math because all the problems individuals deliver are just going to be larger. People are just mourning the loss of the low hanging fruit.
I think you're speaking like a software engineer, which is understandable, and not like a historian or economist. There's no reason to believe math based jobs would survive. The models regularly do well on math problems. You can auto-research loop ways to optimize memory usage for any particular program.
The point isn't that "doing math" is safe. Auto-research solves one target variable in one system, doing it at scale where say one developer is SME for the agentically manged 200 microservices down the line, heh I mean you certainly can, but good luck with that token cost of auto-research when that problem space is O(microservice^2). I point at that example yesterday of that optimized database memory with the comments pointing out that the specific problem fit in memory, over optimized and didn't generalize. The problem isn't the work, but the rework. A historian should know that new solutions to problems doesn't lead to "no problems ever again" but only problems with barriers that the new solution doesn't solve.
The argument is probably that LLMs can find those optimizations cheaper than a human expert. Since LLM cost at fixed capability seems to be going down you either expect humans to be completely replaced or human wages to be lowered by LLMs.
The only Rust section is the one on storage improvements about the struct that stores the hash, but do they really need that many hashes that 2 bytes makes that big of a difference? Article doesn't expand, but I guess it's a hash for every task on every computer, so maybe yes.
That's exactly his point/the area of cost saving - that they didn't actually need as many hashes as they had started with. The trick was in finding out how many hashes they could cull without degrading load balance.
Bang up article! As someone who doesn't get to do enough (almost any) calculus in my daily programming assignments, I thoroughly enjoyed reading about Kevin's dive into that derivation (linked in the supplemental article). All the people being negative here can swallow raisins
Thanks! Maybe dial it back or people are going to think I paid you
I'm surprised to see no mention of hierarchical rendezvous hashing in either the article or the comments here.
It is purpose-designed for exactly this type of proxy/cache load-balancing scenario!
I definitely enjoyed this writing style more than a lot of the recent cf blog posts. Cool article!
Someone really needed a few hundred TB to waste on inference and went looking under the rugs…
CPU DRAM can't really be used for inference efficiently -- inference mostly wants memory bandwidth, not memory capacity, and GPU DRAM has >10x more bandwidth. The fabs can switch between them but you can't switch after the fact.
Those machines with GPUs still need RAM of their own, and they generally want large caches to avoid SSD penalties. You even see this spill out in the form of costs for KV cache in <1min, 5m, 1hr rates etc.
The majority of inference actually does happen in cpu.
It’s nice to see Cloudflare is letting humans write the blog posts again after all the fallout from their LLM slop blogs
anyone remember 100tb hosting company?
does this mean RAM prices can go down now? Please?