> Five months ago, we rebuilt our warehouse on DuckDB instead of ClickHoue
They switched from ClickHouse to DuckDB five months ago, then switched from DuckDB to their own Trino derivative?
The article then ends before saying if the new setup solved their problems.
It’s great that we can now make these migrations with little downtime (I assume?) and try different things, but I wish we could have read more about what testing they did before jumping. The fact that they jumped to DuckDB without realizing it wasn’t going to work at their scale and spent only a few months with it before jumping again is not convincing.
I hoe there’s more to this story that just didn’t make it into the blog posts. As written it’s giving me flashbacks to some teams I worked with who changed tech stacks every six months because they liked migrating more than tuning and maintaining.
A while back, a client called me in to look at a system whose author has left years ago. It was a mess. And for no good reason, it used Docker extensively, and would only work with a specific pre-release version. I searched and discovered that the guy had stayed just long enough to do a talk on this then-hot new technology before getting a better job elsewhere.
It was the clearest example of Resume Driven Development I'd ever seen, but no means the only one.
I worked at a startup that locked all the architects cronies in a room for a number of months to build every white paper he was interested in. Told rest of company to adopt this new world order… within a year of the rewrite they laid off 2/3 of engineering and sold the company off shortly thereafter. Architect accepted a role at FAANG for experience gathered. I remember a lunch with him where he joked “I usually leave companies worse off than when they started.” Self-indulgent psychopath.
The things the original commenter wants to know are exactly the things that will count in an interview. The reasons behind the decisions. Not the actual implementation.
Implementing anything, even something of significant complexity is trivial with frontier agents now. So build your experience / knowledge in a technology-agnostic way. The language/app/stack is superficial.
Edit: Speaking anecdotally; it’s what I’d look for in a candidate if I was hiring today
Things like this are a part of why I push back on people telling me we can do important things later. I don’t want to swap the engine out of a machine that’s moving that fast. There’s no time to learn about how it behaves with real versus synthetic traffic and what the failure modes are when you’re doing many thousands of transactions per second.
Why wouldn’t you want to sort stuff like that out when the problem is still relatively small?
The only other reasonable option is to wait until the problem gets so big that it splits into several domains. And then you can move one domain over, see how that goes, and move more over if it goes well.
If you can’t know for certain how the system will behave under organic load that may mean migrating traffic onto the new system and then back to the old one several times, which means you need migrations in both directions and each transition is an opportunity for human error to destroy data or take the service offline.
> Things like this are a part of why I push back on people telling me we can do important things later. I don’t want to swap the engine out of a machine that’s moving that fast. There’s no time to learn about how it behaves with real versus synthetic traffic and what the failure modes are when you’re doing many thousands of transactions per second.
Counter-point - the longer you delay a crucial decision the more information you have to make it a correct one, so you might avoid migrating DB engine twice a year
I suspect you and the person you are responding to are actually in agreement, making that an additional "point", rather than "counter-point".
The crucial choice is not (just) the one made in a pivot: choosing any DB (or any similarly foundational stratum of your system) is always a crucial decision. Ultimately, you have two choices: put off making an informed, optimal choice, or make an informed, optimal choice now - or if not now, then as soon as you have enough data/experience to make the optimal choice for your needs.
I think you might be attributing the bulk of potential cost to the act of migrating away from an initial choice. But the cost of migrating away from something is proportional to the extent to which it is intertwined/coupled with your system, which in turn is largely a function of the scale at which you are operating. The problem, at least as I have seen it, is that organizations often pick something with minimal (if any) thought put into that initial choice, wait until scale reaches a point that switching away is matter of life and death, and then have to desperately fight against inertia. I think that's what the parent of your post is getting at.
Once the avalanche starts it is too late for the pebbles to vote.
Yes, when decision can be delayed, that is powerful. But we are discussing decisions that have already been made which need to be unmade or made again. Where it breaks is when the tech debt is growing faster than our expertise. The usual stall is not to allow time to apply resources to the problem but to continue to not apply resources to the problem. So our knowledge base is growing at a negligible rate. Particularly if you’re a first mover. Nobody will solve your problems better than you have until it’s far too late to migrate to their solutions.
Founder here -- this is about our warehouse product which is separate from our main analytics DB. Our DuckDB warehouse was used by a handful of pilot customers. Our clickhouse database is serving all of our analytical queries.
There are many good reasons why the industry largely moved away from having DBAs as a specialist position. But one downside I've seen is that it's increasingly common to have teams decide to just switch platforms without fully exploring whether that's really necessary.
I'm not really surprised. Data storage and data access technology is a pretty large subject all by itself, and a "jack of all, master of none" developer might never develop the expertise needed to properly optimize a database.
with the caveat being that I came into the industry in 2014 so I may have missed the golden age of the DBA fully - I worked for a DBMS vendor on sales/solutions side for a while and most DBA’s I spoke to were slightly more knowledgeable business analysts. I met very few who really understood IO and Paging and Locks and how they’d impact a DB’s performance. Which kind of makes sense because if you were really obsessively into that you’d probably do something different or you’d work for a business dealing in storage systems. The value a DBA has been very low in my career outside of forcing process improvement just by saying no to things.
>The article then ends before saying if the new setup solved their problems.
This reads to me like a recruiting post; thats why they are open if it solved their problems. I'm not a fan of the way they framed it, it seems they have made their architecture decision and now need help implementing this.
I'd imagine if you are the kind of person that wants to work on large scale data query problems this is an invite for you to apply. That said, if you are that person it might not be ideal to join after all the fun decisions have been made.
I don’t know? They have blog posts about it but I don’t know how much I’d trust them because their chosen replacement only lasted 5 months before the next replacement.
I think it was closer to what they need; it comes down to keeping TB of data on local SSD and having a query engine next to it. That takes persistent, shared CPU, caching, sharding, resource management, etc.
Duckdb is an interesting approach to resource management (all ephemeral), but TB's are TB's. I doubt it was finding data where it needed it.
Clickhouse seems slow at getting to the top end of scaling; it's stuck with the 99% of its userbase doing 1 GB queries. I'm hoping they put some of their money into it.
DuckDB is now close to competitive with Trino. As the post shares, large distributed multi-tenant data systems require a ton of work no matter what technology you use, but as a huge DuckDB fan, I do have to agree that it is not an out of the box data warehouse, though check back in a year or so when DuckLake and DuckDB 2.0 have had time to mature a bit more for this use case.
As a huge duckdb fan, I can’t think of a workload set I have where the two overlap, so I don’t see them as competitive - I would either pick Trino/Presto or DuckDB as the obvious solution.
> We have no infrastructure, servers or people in Russia. The original engineering team successfully moved out of Russia and are now working in our Amsterdam office.
If I was a user of this product I'd be really worried about the "We rewrote our software 5 months ago, but that didn't solve the issue we wanted to solve so we're doing it again, maybe this time we'll fix it" message of this post.
[Author] it was more like: we underestimated the problem and complexity of workloads our customers want to run. We did not rewrite anything but developing a new stack we can give to our customers doing warehousing. Agreed that this context is not really clear from the blog post.
Unironically, I think we will see a lot more of this given how engineering is being done currently. I have been involved in 2 green field projects in the last 2 years. Both have been rewritten or had massive "ship of theseus" type changes multiple times over that time window.
Engineers working on new products, thanks to LLMs, have very little understanding of the technical details of the product. They have the same level of understanding a traditional high level manager had of products they owned. If you ever worked in an engineering org with an M2 or M3 who always attended your outage review, and in every outage review something has to involve some issue with Postgres or your Kafka queue, that disconnected M3 would conclude:
"Almost every outage involves some problem with Postgres. My conclusion, observed over the last 6-12 months, Postgres is bad. I see this new FancyDB or DynamoDB thing is popular on Twitter. We should try that"
I have lost count of the number of idiotic takes like that from M2s or M3s who have no idea how things work but instead focus on keywords they can latch onto that seems to be in the intersection of all issues. Then the logical solution is just to remove that and see what happens. This used to be very expensive, but if you can throw an LLM at a code base and say "Replace Postgres with FancyDB, split it to a list of logical checkpoints adding regression testing when missing to each checkpoint and open stacked PRs for all checkpoints", let it churn for an after noon, and 46k-line change PR (and $800 token bill) later you are ready to try FancyDB, why would you not give it a try? Then if you have issues with FancyDB, are you gonna spend the time to understand what's going on or just say "lets try DynamoDB next"?
I worked at so many companies that operated this way, the tool is new but the approach is not. Those failed startups you hear about usually did a couple of these before hitting the ground
Honestly... to me, this reads like nothing new. I had similar experiences a decade ago (and was myself the problem), long before Claude Code was a twinkle in anyone's eye.
This is what this new feature called “loops” are for. Tell your agent to identify a list of databases and try them all, in prod, until it finds a good one. Be sure to tell it “no outages”.
I have lost count of the number of idiotic takes like that from M2s or M3s who have no idea how things work but instead focus on keywords they can latch onto that seems to be in the intersection of all issues.
This phenomenon is at the root of the hardest organizational & architectural challenges I’ve been involved in.
It really needs a name so it can be tagged as a leadership anti-pattern.
Good leaders find real connections to explore & strengthen.
Bad leaders reach conclusions at the fuzzy pattern matching stage.
To me to key “insight” is that if you are going to spin up single VMs (or even groups) per-customer and then route queries, you need to be able to estimate the query cost first.
I know snowflake does this, they do the query optimization before sending it for execution. So it knows about how many resources will be needed (it can obviously be wrong).
[Author] Basically yes, but we spun up pods on EKS. It was hard for the customer to estimate how much they might need and hard for us to know how much capacity to provide. Given our plans on pricing on CPUHours (or similar) the logic was kinda broken.
How does Snowflake use that information? You still need to pick the warehouse. It would be cool if there was an "elastic warehouse" where heavier queries are routed to bigger Warehouses, but that's not the case
[Author] Basically what we are doing with the help of Trino: a distributed query engine which can use as many resources as the query needs dynamically.
Right, there is an option to scale horizontally, if a lot of queries are queued, Snowflake can spin up another node. But that doesn't help if you query is spilling to remote storage. You need a bigger node to speed up those queries.
Snowflake has an option to add more resources, which is supposed to be vertical scaling, but it doesn't work as well as increasing the size of the warehouse
We engineered, then we measured and then we tried to make it work, probably for a too long of a time before we decided to completely scrap our current architecture and go back to the drawing board. Happy to answer more questions.
Why didn't you just use Iceberg and Trino from the get go (or Paimon if you are fancy)? This has been the accepted answer for quite some time now. The fast OLAP stores just don't scale as well as going object store to a proper query engine like Trino.
With DuckDB and the queries we got, we had to provision such huge pods/nodes that we made it work but could never get to an economical viability. In the end, most of our customers had pods running 46 vCPUs / 360GB and asked for more. DuckDB is amazing, it really does everything right on a single node to make your query run successfully to the end almost always but it just doesn't work in our cloud env. Imagine when a query is running for 12h with this since and then another 10 from the same customer come in :)
Trino on the other hand has a much more convenient distributed query concept, so you can much more dynamically assign resources.
I don't know about their development or business practices, but if you think that their CEO posts on LinkedIn are serious and not satire I'm sorry to say the jokes are flying above your head a bit.
The entire website is a bit of a red flag to me, starting with the cookie banner featuring an image of Ursula von der Leyen and some misinformation (cookie banners are not legally required).
The intro to that post is so bad: "PostHog's original data warehouse had a single goal: delay the moment a company needs to hire their first data engineer."
It is not to serve high quality and timely data to make good decisions, is to avoid hiring a specific role, lol. So bad.
no ones upset, I think given the quality of technical write ups we see on HN with very detailed analysis that this article felt like a faux tech article to promote a product? But it kind of is a self own as it just looks like they didn't do due diligence with their tech choices.
I mean, mostly it just comes off as egregious self-indulgence to try one thing and then jump to "we built the solution ourselves!" when the problem is not novel at all. These things are just tiresome.
Author here. We went past the POC stage to actually make real customer workloads work, which we did. Probably yes, we spent too much time not rethinking our approach but sunken cost fallacy so that's why.
The job of management is to help manage that. Typically, the desire to experiment is constrained to lab environments and so on. Experimenting on your actual users carries a lot of risks - to product stability, customer relations, market perception, etc.
Author here. I admit I kinda left out the context which is important here. We did not replace Clickhouse. Instead we want to move our warehouse product from Clickhouse to another stack which suits the long modeling jobs and multi-tenancy better. We learned through all our early design partners running huge workloads what we need to change.
As much as I'd love to claim victory, we are not there yet :)
Sometimes a company just isn't able to manage a data warehouse; I've worked for a few.
I even phone screened with this one but couldn't cover their crazy quilt of requirements; at that time they were also looking for someone to rewrite the clickhouse optimizer, and they were stuck trying to reshard their cluster, which should take an afternoon.
I still like the company because their business prop reminds me of what we were doing in 2009, a kind of federated analytics organization on a centralized data warehouse that enables a lot of tools to be developed quickly at large scale.
But goddammit they keep picking these half-working platforms and flip-flopping from one to another. We were crushing millions of queries against 2T-row tables on Vertica in 2010, and it's only gotten better for customer-facing data apps, which is what Posthog is. The company doesn't seem to have a baseline of what to expect; the market is too saturated with slow tools that lower expectations.
Author here. This is not about our internal warehouse but the warehouse product we offer to customers. This is still running on our centralized Clickhouse. Long-running data modelling queries and multi-tenancy is a challenge on Clickhouse.
I worked at a place with a couple of overly optimistic principals who were so happy about their computationally convoluted system responding to queries in 600 ms, but it practically meant we needed almost half a CPU per request, and then pre-autoscaling, a strange requirement to be able to failover from taking a region offline so we ran each at around 60% utilization, putting us at about 1 CPU per req/s. Which is just fucking embarrassing.
The first big system I worked on is the one where I learned not to generate expensive responses from a read mostly system, instead you should make writes expensive so reads are cheap. That project had an almost equally convoluted responses, and yet we were doing 30 req/s per instance on, IIRC, 4 core boxes. Adjusted for hot spare capacity, that ancient system was servicing 4x as many requests per core, 8x as configured.
A big part of the problem, besides selling a “premium” service to customers, was drinking the flavor of Knuth’s “premature optimization” misquoted koolaid that convinces people that all substantial speed improvements come from major architectural shifts. Which is how in the time I was on that team of 6-9 people, I delivered over half of the response time and cluster size improvements. If you count the work done to support upgrading the runtime, which brought perf improvements of its own, then more like 2/3.
Sounds a bit like this team might have the same disease.
The problem with internal service teams, of which this seems to be one, is that dealing with noisy neighbors is always a bigger problem than anyone is willing to tackle because it involves confrontation. Team X gets punished every time Team Y introduces a new feature that leans on the shared resource harder. It’s a multi tenant system without the fairness guarantees that we usually build into proper multi tenant systems.
Sometimes the easiest way to solve this is sharding. You end up deploying the same service many times and routing different classes of requests to different machines so if Team Y keeps trying to wreck the Commons they slow themselves down faster than they slow everyone else down. Those sped up consequences create a feedback loop that almost works instead of an amorphous one with consequences so far removed from actions as to become academic.
On that second team I mentioned we had an unfortunate situation of introducing things that took 100 ms off of response time and then three months later we were right back where we had been due to new features and creeping complexity of user content. It was demotivating for at least half of the team. My biggest contributions lowered the slope of that line, which allowed us to finally carry improvements over from year to year instead of being on the Red Queen problem.
I wonder if they have considered moving back to ClickHouse. ClickHouse Iceberg support has improved a lot over the past few months. You can easily run stateless ClickHouse on top of Iceberg. This solves the tenancy issue, and it can easily handle petabytes scale [1].
Author here. We didn't leave Clickhouse. It still drives most of what Posthog offers. I missed that bit of context in my post I admit. Instead we are looking into a new stack we can give customers so they can run their warehousing workloads on it instead of the centralized CH.
Stateless CH on Iceberg sounds interesting, we looked into it but Delta/Iceberg had other downsides for us.
I kind of liked the originality. It's what happens when you give software developers free reign on the marketing pages as well, I guess.
But it seems to have gotten a re-skin recently, and I preferred the previous more pixelated version. Also, there used to be a way to render a "normal" version of the website, but I can't find it anymore.
I actually like the menus. They work great on mobile too.
We built up all these UI conventions from decades of desktop development, then abandoned them for the web (and worse, electron). It's annoying learning a new convention for every website. We could do a lot worse than bringing back menus.
And what's with that bs cookie banner?
If the only cookies are functional/essential, they don't need to show a cookie banner.
If it's for analytics, then I should be able to reject it.
Otherwise, it's just a dick-over.
The key insight I am getting from this is that as migrations get easier to do with agents, there will be a lot more of them.
But more importantly a lot of them will happen because we will all get more trigger happy to start one. On the other hand I also don't know if at scales like PostHog, a successful migration can be tested without doing all of it.
Author here. I agree. We spent most of the time to make it work at the scale of our large design partners and probably should have stepped back earlier. And yes, with an agent you can have a much larger scope hashed out faster than before.
Hah I suppose it’s all relative on fast - Trino’s certainly intended to support ‘interactive analytics’ vs being a pure ETL platform. Def seconds vs milliseconds for Druid/clickhouse/duckdb in general though.
This reminds me so much of when we would bring a new marketing leader and they would plant their flag by nuking all of the existing marketing automation processes to bring in theirs, with no net impact on the business. Bike shedding.
I would have appreciated more technical details with methodologies and explicit failures observed. Oh and a human voice would be nice for a change.
> Customers ran more complex queries than we expected
Did the queries change after DuckDB was introduced? Did they do a study of query patterns that could have anticipated this ahead of time?
> Predicting CPU and memory for a specific query was even harder.
What attempts / methodologies were used to try this? What was hard? What failed?
> We'd rather have contributed to the existing community and helped it grow. But we couldn't land changes as fast as we needed, so we ended up carrying patches in a fork
This isn't a BAD approach per se. IME it's common (and necessary) for the large scale users of a platform to maintain a fork for their needs and work with the community and the maintainers to upstream changes when/if the community is ready to accept. That's called influencing. On the other hand, if you toss of a few slop grenades to the community and they reject you shouldn't be surprised. (Not suggesting that's what happening here but the defeatest conclusion may imply it?)
> Missing reconnects and managing TTLs and idle connections at scale is hard
Listen, lad. I built this database up from nothing. When I started here, all there was was swamp. Other kings said I was daft to build a database on a swamp, but I built it all the same, just to show 'em. It sank into the swamp. So, I built a second one. That sank into the swamp. So, I built a third one. That burned down, fell over, then sank into the swamp, but the fourth one... stayed up! And that's what you're gonna get, lad: the strongest database in these islands.
Now they have their "own flavor" of Trino and a brand new open table format called "HogLake". This gives the impression of making excuses with little introspection.
It would be great to see their ducklake fork, their patches and github issues so the community can learn.
Author here. Our fork, like most of our work is public on Github: https://github.com/postHog/ducklake I think it does not include 100% of all changes since a few we upstreamed directly (to a few extensions related to Ducklake too) but if you are really interested, you can find them too.
This idiotic churn used to be restricted to front end devs and backend folks were spared. Thanks for Agents now every part of the stack is going to be in constant churn thanks to resume builders. At no point in the article did you get a sense they have a handle on what their problem is and why a certain technology and architecture would be a better fit. Just try it out, churn out, try it out.
Certainly frontend folks churned like mad by picking one package after another to build on. I saw it first hand at a startup. I was the backend guy, just wrote it once. That doesn't mean it was perfect or he wasn't doing fine, but I didn't need to rewrite it so many times. I'm not ready for a world where everything gets rewritten on some new toolset just because. Hope you've got good tests.
6 months from now: We are moving our warehouse off of Trino + HogLake to newer technology XXX.
This is ridiculous. Why can you not foresee such issues? Do you not do testing with mock data at the size you need? Migrating production between stacks so often means you are not getting new features in. This is pure churn.
There are two kinds of companies: those who know how to build low-level tools that serve as foundation and moat for business growth, and those whose middle managers wring their hands about what their people will be able to maintain and constantly chase off-the-shelf solutions to stitch together.
One thing I really appreciate about CloudFlare blog posts is that often they show their work, by showing what tickets they opened, what improvements they land upstream. With this march across clickhouse, duckdb, now trino, I'd love to know what if any places the team worked with upstream, what improvements got tried.
I'm pretty ok with the march across products myself. It felt like the big shift was from shared giant clickhouse to lots of duckdbs. I wonder what juggling many more clickhouses would have been like. Excited and hopeful to here more back on Trini, with hope!
It's wild how many really really good data stacks are out there. Apache has been collecting some absolutely stellar projects. And it feels like the rest of world meaning everyone but China is kind of sleeping on a lot of these options. Maybe it's just that these projects often go out of their way to have a place for emblems, for who is using the project, but a lot of these have pretty wide ranges of companies using them.
I do wish we were still in an age of company blogging, where people did talk more to the tech getting played with, talk to what's cooking in the shop. It's great to have these outreaches. There's so much tech and we are just so quiet about it! There's so much less signal to go off of, so much less shared awareness than the aughts and 10's.
> Five months ago, we rebuilt our warehouse on DuckDB instead of ClickHoue
They switched from ClickHouse to DuckDB five months ago, then switched from DuckDB to their own Trino derivative?
The article then ends before saying if the new setup solved their problems.
It’s great that we can now make these migrations with little downtime (I assume?) and try different things, but I wish we could have read more about what testing they did before jumping. The fact that they jumped to DuckDB without realizing it wasn’t going to work at their scale and spent only a few months with it before jumping again is not convincing.
I hoe there’s more to this story that just didn’t make it into the blog posts. As written it’s giving me flashbacks to some teams I worked with who changed tech stacks every six months because they liked migrating more than tuning and maintaining.
> The article then ends before saying if the new setup solved their problems.
They got to put ClickHouse, DuckDB and Trino on their resume & LinkedIn. Problem solved!
A while back, a client called me in to look at a system whose author has left years ago. It was a mess. And for no good reason, it used Docker extensively, and would only work with a specific pre-release version. I searched and discovered that the guy had stayed just long enough to do a talk on this then-hot new technology before getting a better job elsewhere.
It was the clearest example of Resume Driven Development I'd ever seen, but no means the only one.
I worked at a startup that locked all the architects cronies in a room for a number of months to build every white paper he was interested in. Told rest of company to adopt this new world order… within a year of the rewrite they laid off 2/3 of engineering and sold the company off shortly thereafter. Architect accepted a role at FAANG for experience gathered. I remember a lunch with him where he joked “I usually leave companies worse off than when they started.” Self-indulgent psychopath.
Screw that company mate. I was in a company with startup-y mindset. It was just like that and I quit after 3 days, without taking any money
Honestly that shouldn’t even matter anymore.
The things the original commenter wants to know are exactly the things that will count in an interview. The reasons behind the decisions. Not the actual implementation.
Implementing anything, even something of significant complexity is trivial with frontier agents now. So build your experience / knowledge in a technology-agnostic way. The language/app/stack is superficial.
Edit: Speaking anecdotally; it’s what I’d look for in a candidate if I was hiring today
Things like this are a part of why I push back on people telling me we can do important things later. I don’t want to swap the engine out of a machine that’s moving that fast. There’s no time to learn about how it behaves with real versus synthetic traffic and what the failure modes are when you’re doing many thousands of transactions per second.
Why wouldn’t you want to sort stuff like that out when the problem is still relatively small?
The only other reasonable option is to wait until the problem gets so big that it splits into several domains. And then you can move one domain over, see how that goes, and move more over if it goes well.
If you can’t know for certain how the system will behave under organic load that may mean migrating traffic onto the new system and then back to the old one several times, which means you need migrations in both directions and each transition is an opportunity for human error to destroy data or take the service offline.
> Things like this are a part of why I push back on people telling me we can do important things later. I don’t want to swap the engine out of a machine that’s moving that fast. There’s no time to learn about how it behaves with real versus synthetic traffic and what the failure modes are when you’re doing many thousands of transactions per second.
Counter-point - the longer you delay a crucial decision the more information you have to make it a correct one, so you might avoid migrating DB engine twice a year
I suspect you and the person you are responding to are actually in agreement, making that an additional "point", rather than "counter-point".
The crucial choice is not (just) the one made in a pivot: choosing any DB (or any similarly foundational stratum of your system) is always a crucial decision. Ultimately, you have two choices: put off making an informed, optimal choice, or make an informed, optimal choice now - or if not now, then as soon as you have enough data/experience to make the optimal choice for your needs.
I think you might be attributing the bulk of potential cost to the act of migrating away from an initial choice. But the cost of migrating away from something is proportional to the extent to which it is intertwined/coupled with your system, which in turn is largely a function of the scale at which you are operating. The problem, at least as I have seen it, is that organizations often pick something with minimal (if any) thought put into that initial choice, wait until scale reaches a point that switching away is matter of life and death, and then have to desperately fight against inertia. I think that's what the parent of your post is getting at.
Once the avalanche starts it is too late for the pebbles to vote.
Yes, when decision can be delayed, that is powerful. But we are discussing decisions that have already been made which need to be unmade or made again. Where it breaks is when the tech debt is growing faster than our expertise. The usual stall is not to allow time to apply resources to the problem but to continue to not apply resources to the problem. So our knowledge base is growing at a negligible rate. Particularly if you’re a first mover. Nobody will solve your problems better than you have until it’s far too late to migrate to their solutions.
Founder here -- this is about our warehouse product which is separate from our main analytics DB. Our DuckDB warehouse was used by a handful of pilot customers. Our clickhouse database is serving all of our analytical queries.
If this is a pilot program for a small number of users, the blog post would feel a lot more logical.
As it was written, it looks like the engineering team is migrating the core infrastructure of the company around.
Try VictoriaLogs for your warehouse product next time. It provides efficient storing and querying petabytes of wide events with hundreds of fields - see https://docs.victoriametrics.com/victorialogs/keyconcepts/ , and it provides very easy scaling path from single-node setup to cluster setup, so you can start with an instance per customer and then gradually scale to many storage nodes as workload grows - https://docs.victoriametrics.com/victorialogs/cluster/#singl... .
It reminds of people who use ORMs, never pop the hood to analyze the queries, indexes, etc - then decide it's the database that's the problem.
There are many good reasons why the industry largely moved away from having DBAs as a specialist position. But one downside I've seen is that it's increasingly common to have teams decide to just switch platforms without fully exploring whether that's really necessary.
I'm not really surprised. Data storage and data access technology is a pretty large subject all by itself, and a "jack of all, master of none" developer might never develop the expertise needed to properly optimize a database.
with the caveat being that I came into the industry in 2014 so I may have missed the golden age of the DBA fully - I worked for a DBMS vendor on sales/solutions side for a while and most DBA’s I spoke to were slightly more knowledgeable business analysts. I met very few who really understood IO and Paging and Locks and how they’d impact a DB’s performance. Which kind of makes sense because if you were really obsessively into that you’d probably do something different or you’d work for a business dealing in storage systems. The value a DBA has been very low in my career outside of forcing process improvement just by saying no to things.
Yeah. I’d say that by 2014 a lot of the shift had already happened.
>The article then ends before saying if the new setup solved their problems.
This reads to me like a recruiting post; thats why they are open if it solved their problems. I'm not a fan of the way they framed it, it seems they have made their architecture decision and now need help implementing this.
I'd imagine if you are the kind of person that wants to work on large scale data query problems this is an invite for you to apply. That said, if you are that person it might not be ideal to join after all the fun decisions have been made.
What's wrong with ClickHouse again?
I don’t know? They have blog posts about it but I don’t know how much I’d trust them because their chosen replacement only lasted 5 months before the next replacement.
I think it was closer to what they need; it comes down to keeping TB of data on local SSD and having a query engine next to it. That takes persistent, shared CPU, caching, sharding, resource management, etc.
Duckdb is an interesting approach to resource management (all ephemeral), but TB's are TB's. I doubt it was finding data where it needed it.
Clickhouse seems slow at getting to the top end of scaling; it's stuck with the 99% of its userbase doing 1 GB queries. I'm hoping they put some of their money into it.
ClickHouse is fast OLAP. Just do Iceberg with Trino like a normal big data person. WTF?
DuckDB is now close to competitive with Trino. As the post shares, large distributed multi-tenant data systems require a ton of work no matter what technology you use, but as a huge DuckDB fan, I do have to agree that it is not an out of the box data warehouse, though check back in a year or so when DuckLake and DuckDB 2.0 have had time to mature a bit more for this use case.
As a huge duckdb fan, I can’t think of a workload set I have where the two overlap, so I don’t see them as competitive - I would either pick Trino/Presto or DuckDB as the obvious solution.
Unless something has changed in the last year, a lot of teams have serious problems with stability.
PostHog has listed a lot of job openings in the last couple of years. I figured growth but maybe it’s turnover.
I am curious to learn more about those issues, anyone have shared their experience publicly? would love to read
No support for real S3 (zero copy replication) in the open source version.
Russian ties?
https://news.ycombinator.com/item?id=33082224
That's where it started very long ago, but they are based out of the Netherlands mostly now.
https://clickhouse.com/blog/we-stand-with-ukraine
> We have no infrastructure, servers or people in Russia. The original engineering team successfully moved out of Russia and are now working in our Amsterdam office.
You can take the guy out of Russia.. :)
I kid but that's probably what people have a problem with.
Could say that about almost any country in the world, especially USA :)
Sure can. But some countries are more different like China, Russia, North Korea. A lot of people would include USA in that list of course. :)
Sounds like a common problem in the AI era.
It doesn’t look like planning and then executing.
It looks like executing and simultaneously reacting to the periodic “novel” information that would have been surfaced during the planning.
Burn enough tokens and it gets harder to tell them apart. Sometimes. But it’s appearance more than anything else.
Failure to plan can’t be repaired with even more tokens.
Am I missing something here, or is the only technical detail in the post "a trillion rows is a lot of data, maybe trying trinio will work this time"?
It feels like a lot of churn and not a lot of measurements or engineering.
If I was a user of this product I'd be really worried about the "We rewrote our software 5 months ago, but that didn't solve the issue we wanted to solve so we're doing it again, maybe this time we'll fix it" message of this post.
[Author] it was more like: we underestimated the problem and complexity of workloads our customers want to run. We did not rewrite anything but developing a new stack we can give to our customers doing warehousing. Agreed that this context is not really clear from the blog post.
Unironically, I think we will see a lot more of this given how engineering is being done currently. I have been involved in 2 green field projects in the last 2 years. Both have been rewritten or had massive "ship of theseus" type changes multiple times over that time window.
Engineers working on new products, thanks to LLMs, have very little understanding of the technical details of the product. They have the same level of understanding a traditional high level manager had of products they owned. If you ever worked in an engineering org with an M2 or M3 who always attended your outage review, and in every outage review something has to involve some issue with Postgres or your Kafka queue, that disconnected M3 would conclude:
I have lost count of the number of idiotic takes like that from M2s or M3s who have no idea how things work but instead focus on keywords they can latch onto that seems to be in the intersection of all issues. Then the logical solution is just to remove that and see what happens. This used to be very expensive, but if you can throw an LLM at a code base and say "Replace Postgres with FancyDB, split it to a list of logical checkpoints adding regression testing when missing to each checkpoint and open stacked PRs for all checkpoints", let it churn for an after noon, and 46k-line change PR (and $800 token bill) later you are ready to try FancyDB, why would you not give it a try? Then if you have issues with FancyDB, are you gonna spend the time to understand what's going on or just say "lets try DynamoDB next"?I worked at so many companies that operated this way, the tool is new but the approach is not. Those failed startups you hear about usually did a couple of these before hitting the ground
Honestly... to me, this reads like nothing new. I had similar experiences a decade ago (and was myself the problem), long before Claude Code was a twinkle in anyone's eye.
> lets try DynamoDB next
This is what this new feature called “loops” are for. Tell your agent to identify a list of databases and try them all, in prod, until it finds a good one. Be sure to tell it “no outages”.
I have lost count of the number of idiotic takes like that from M2s or M3s who have no idea how things work but instead focus on keywords they can latch onto that seems to be in the intersection of all issues.
This phenomenon is at the root of the hardest organizational & architectural challenges I’ve been involved in.
It really needs a name so it can be tagged as a leadership anti-pattern.
Good leaders find real connections to explore & strengthen.
Bad leaders reach conclusions at the fuzzy pattern matching stage.
To me to key “insight” is that if you are going to spin up single VMs (or even groups) per-customer and then route queries, you need to be able to estimate the query cost first.
I know snowflake does this, they do the query optimization before sending it for execution. So it knows about how many resources will be needed (it can obviously be wrong).
[Author] Basically yes, but we spun up pods on EKS. It was hard for the customer to estimate how much they might need and hard for us to know how much capacity to provide. Given our plans on pricing on CPUHours (or similar) the logic was kinda broken.
How does Snowflake use that information? You still need to pick the warehouse. It would be cool if there was an "elastic warehouse" where heavier queries are routed to bigger Warehouses, but that's not the case
[Author] Basically what we are doing with the help of Trino: a distributed query engine which can use as many resources as the query needs dynamically.
There is an elastic option, I think . Also, even with a specific warehouse, not every query uses every node in the warehouse.
Right, there is an option to scale horizontally, if a lot of queries are queued, Snowflake can spin up another node. But that doesn't help if you query is spilling to remote storage. You need a bigger node to speed up those queries.
Snowflake has an option to add more resources, which is supposed to be vertical scaling, but it doesn't work as well as increasing the size of the warehouse
The query planner / optimizer does not run in the warehouse. One of its primary functions is to predict this stuff.
Hi! Author here.
We engineered, then we measured and then we tried to make it work, probably for a too long of a time before we decided to completely scrap our current architecture and go back to the drawing board. Happy to answer more questions.
Why didn't you just use Iceberg and Trino from the get go (or Paimon if you are fancy)? This has been the accepted answer for quite some time now. The fast OLAP stores just don't scale as well as going object store to a proper query engine like Trino.
We tried Iceberg and it was too slow writing data is the short story.
>The fast OLAP stores just don't scale as well as going object store to a proper query engine like Trino
this exactly why we started with this project.
> Happy to answer more questions.
Ah yay, I appreciate it! I feel like its rare to get a look at how these decisions cone about.
What did you measure? How'd you realise there was a problem? And did you measure anything to give you confidence that Trino won't hit the same issues?
With DuckDB and the queries we got, we had to provision such huge pods/nodes that we made it work but could never get to an economical viability. In the end, most of our customers had pods running 46 vCPUs / 360GB and asked for more. DuckDB is amazing, it really does everything right on a single node to make your query run successfully to the end almost always but it just doesn't work in our cloud env. Imagine when a query is running for 12h with this since and then another 10 from the same customer come in :)
Trino on the other hand has a much more convenient distributed query concept, so you can much more dynamically assign resources.
And I presume Trino has some way of avoiding the slow writes that hurt Iceberg?
Its not Trino, its the catalog you use in the back.
I don't get the blog post either.
This sounds more like: shoot first ask later. (OMG!)
> It feels like a lot of churn and not a lot of measurements or engineering.
They are big on shipping slop at lightnigg speed. So this is not surprising. Checkout posts by their ceo on cringedin.
I don't know about their development or business practices, but if you think that their CEO posts on LinkedIn are serious and not satire I'm sorry to say the jokes are flying above your head a bit.
The entire website is a bit of a red flag to me, starting with the cookie banner featuring an image of Ursula von der Leyen and some misinformation (cookie banners are not legally required).
> Five months ago, we rebuilt our warehouse [1] on DuckDB instead of ClickHouse.
Why did they write this post if they had not even allowed things to settle. It sounds like someone wanted to claim victory (and promo?)
https://posthog.com/blog/why-we-rebuilt-our-data-warehouse
The intro to that post is so bad: "PostHog's original data warehouse had a single goal: delay the moment a company needs to hire their first data engineer."
It is not to serve high quality and timely data to make good decisions, is to avoid hiring a specific role, lol. So bad.
Ironically a good data engineer should probably let them avoid needing 3 different data warehouse systems within a year.
Engineers like to experiment and we get the benefit of learning from their experiences. Not sure why you’re upset.
Maybe it's the "me first" mindset you're bringing to engineering?
no ones upset, I think given the quality of technical write ups we see on HN with very detailed analysis that this article felt like a faux tech article to promote a product? But it kind of is a self own as it just looks like they didn't do due diligence with their tech choices.
I mean, mostly it just comes off as egregious self-indulgence to try one thing and then jump to "we built the solution ourselves!" when the problem is not novel at all. These things are just tiresome.
This sounds more like a 5 month POC that the engineer should have been able to prove sooner that DucksDB didn’t fit the use case.
Author here. We went past the POC stage to actually make real customer workloads work, which we did. Probably yes, we spent too much time not rethinking our approach but sunken cost fallacy so that's why.
> Engineers like to experiment
The job of management is to help manage that. Typically, the desire to experiment is constrained to lab environments and so on. Experimenting on your actual users carries a lot of risks - to product stability, customer relations, market perception, etc.
Author here. I admit I kinda left out the context which is important here. We did not replace Clickhouse. Instead we want to move our warehouse product from Clickhouse to another stack which suits the long modeling jobs and multi-tenancy better. We learned through all our early design partners running huge workloads what we need to change.
As much as I'd love to claim victory, we are not there yet :)
This is the same company with intentionally annoying full-wrap bus ads on Muni. They are trying to capture mindshare.
Posthog is a relatively small company, and they have victory-claiming people already.
NGMI
Sometimes a company just isn't able to manage a data warehouse; I've worked for a few.
I even phone screened with this one but couldn't cover their crazy quilt of requirements; at that time they were also looking for someone to rewrite the clickhouse optimizer, and they were stuck trying to reshard their cluster, which should take an afternoon.
I still like the company because their business prop reminds me of what we were doing in 2009, a kind of federated analytics organization on a centralized data warehouse that enables a lot of tools to be developed quickly at large scale.
But goddammit they keep picking these half-working platforms and flip-flopping from one to another. We were crushing millions of queries against 2T-row tables on Vertica in 2010, and it's only gotten better for customer-facing data apps, which is what Posthog is. The company doesn't seem to have a baseline of what to expect; the market is too saturated with slow tools that lower expectations.
Author here. This is not about our internal warehouse but the warehouse product we offer to customers. This is still running on our centralized Clickhouse. Long-running data modelling queries and multi-tenancy is a challenge on Clickhouse.
I worked at a place with a couple of overly optimistic principals who were so happy about their computationally convoluted system responding to queries in 600 ms, but it practically meant we needed almost half a CPU per request, and then pre-autoscaling, a strange requirement to be able to failover from taking a region offline so we ran each at around 60% utilization, putting us at about 1 CPU per req/s. Which is just fucking embarrassing.
The first big system I worked on is the one where I learned not to generate expensive responses from a read mostly system, instead you should make writes expensive so reads are cheap. That project had an almost equally convoluted responses, and yet we were doing 30 req/s per instance on, IIRC, 4 core boxes. Adjusted for hot spare capacity, that ancient system was servicing 4x as many requests per core, 8x as configured.
A big part of the problem, besides selling a “premium” service to customers, was drinking the flavor of Knuth’s “premature optimization” misquoted koolaid that convinces people that all substantial speed improvements come from major architectural shifts. Which is how in the time I was on that team of 6-9 people, I delivered over half of the response time and cluster size improvements. If you count the work done to support upgrading the runtime, which brought perf improvements of its own, then more like 2/3.
Sounds a bit like this team might have the same disease.
The problem with internal service teams, of which this seems to be one, is that dealing with noisy neighbors is always a bigger problem than anyone is willing to tackle because it involves confrontation. Team X gets punished every time Team Y introduces a new feature that leans on the shared resource harder. It’s a multi tenant system without the fairness guarantees that we usually build into proper multi tenant systems.
Sometimes the easiest way to solve this is sharding. You end up deploying the same service many times and routing different classes of requests to different machines so if Team Y keeps trying to wreck the Commons they slow themselves down faster than they slow everyone else down. Those sped up consequences create a feedback loop that almost works instead of an amorphous one with consequences so far removed from actions as to become academic.
On that second team I mentioned we had an unfortunate situation of introducing things that took 100 ms off of response time and then three months later we were right back where we had been due to new features and creeping complexity of user content. It was demotivating for at least half of the team. My biggest contributions lowered the slope of that line, which allowed us to finally carry improvements over from year to year instead of being on the Red Queen problem.
I wonder if they have considered moving back to ClickHouse. ClickHouse Iceberg support has improved a lot over the past few months. You can easily run stateless ClickHouse on top of Iceberg. This solves the tenancy issue, and it can easily handle petabytes scale [1].
[1] https://clickhouse.com/blog/jump-trading-uses-clickhouse-wit...
Author here. We didn't leave Clickhouse. It still drives most of what Posthog offers. I missed that bit of context in my post I admit. Instead we are looking into a new stack we can give customers so they can run their warehousing workloads on it instead of the centralized CH.
Stateless CH on Iceberg sounds interesting, we looked into it but Delta/Iceberg had other downsides for us.
But why even bother with ClickHouse? What's wrong with Trino?
It’s the third time you’ve probed this question bud… Trino w/Iceberg is a good answer but not the only one here
Haven't seen Posthog's website in a while. Man, it's hideous.
I kind of liked the originality. It's what happens when you give software developers free reign on the marketing pages as well, I guess.
But it seems to have gotten a re-skin recently, and I preferred the previous more pixelated version. Also, there used to be a way to render a "normal" version of the website, but I can't find it anymore.
yeah nah no engineer gonna bother replicating desktop vibe on a marketing page
I actually like the menus. They work great on mobile too.
We built up all these UI conventions from decades of desktop development, then abandoned them for the web (and worse, electron). It's annoying learning a new convention for every website. We could do a lot worse than bringing back menus.
I use it as a free web analytics, the UX is so bad and convoluted, it's like a bad joke.
And what's with that bs cookie banner? If the only cookies are functional/essential, they don't need to show a cookie banner. If it's for analytics, then I should be able to reject it. Otherwise, it's just a dick-over.
I actually love this design but I nevertheless agree it's atrocious.
really? I think its one of the best websites going these years.
The key insight I am getting from this is that as migrations get easier to do with agents, there will be a lot more of them.
But more importantly a lot of them will happen because we will all get more trigger happy to start one. On the other hand I also don't know if at scales like PostHog, a successful migration can be tested without doing all of it.
Author here. I agree. We spent most of the time to make it work at the scale of our large design partners and probably should have stepped back earlier. And yes, with an agent you can have a much larger scope hashed out faster than before.
Not sure their singlenode customers will be happy with Trino
https://benchmark.clickhouse.com/#system=+Hoaus|ucqs|Tuts&ty...
Author here. Good point. Trino is not made for fast response times. We are using it for long running data modelling jobs or nightly ETLs.
Hah I suppose it’s all relative on fast - Trino’s certainly intended to support ‘interactive analytics’ vs being a pure ETL platform. Def seconds vs milliseconds for Druid/clickhouse/duckdb in general though.
This reminds me so much of when we would bring a new marketing leader and they would plant their flag by nuking all of the existing marketing automation processes to bring in theirs, with no net impact on the business. Bike shedding.
Is Quack, the DuckDB client-server protocol, something that could have helped them with the scaling problems they encountered?
Author here. Good question. AFAIK you cannot use Quack for anything that looks remotely like distributed querying.
I would have appreciated more technical details with methodologies and explicit failures observed. Oh and a human voice would be nice for a change.
> Customers ran more complex queries than we expected
Did the queries change after DuckDB was introduced? Did they do a study of query patterns that could have anticipated this ahead of time?
> Predicting CPU and memory for a specific query was even harder.
What attempts / methodologies were used to try this? What was hard? What failed?
> We'd rather have contributed to the existing community and helped it grow. But we couldn't land changes as fast as we needed, so we ended up carrying patches in a fork
This isn't a BAD approach per se. IME it's common (and necessary) for the large scale users of a platform to maintain a fork for their needs and work with the community and the maintainers to upstream changes when/if the community is ready to accept. That's called influencing. On the other hand, if you toss of a few slop grenades to the community and they reject you shouldn't be surprised. (Not suggesting that's what happening here but the defeatest conclusion may imply it?)
> Missing reconnects and managing TTLs and idle connections at scale is hard
Wonder if they tried pgdog?
The 2026 equivalent of 2016's "Why Uber Engineering Switched from Postgres to MySQL" ! https://www.uber.com/us/en/blog/postgres-to-mysql-migration/
Listen, lad. I built this database up from nothing. When I started here, all there was was swamp. Other kings said I was daft to build a database on a swamp, but I built it all the same, just to show 'em. It sank into the swamp. So, I built a second one. That sank into the swamp. So, I built a third one. That burned down, fell over, then sank into the swamp, but the fourth one... stayed up! And that's what you're gonna get, lad: the strongest database in these islands.
Now they have their "own flavor" of Trino and a brand new open table format called "HogLake". This gives the impression of making excuses with little introspection.
It would be great to see their ducklake fork, their patches and github issues so the community can learn.
Author here. Our fork, like most of our work is public on Github: https://github.com/postHog/ducklake I think it does not include 100% of all changes since a few we upstreamed directly (to a few extensions related to Ducklake too) but if you are really interested, you can find them too.
Feels off that only 5 months ago the company just migrated to DuckDB only to now move to something else.
Who did the growth estimation of the data? This looks like an example of bad planning.
Interestingly, it seems that a data lake is the hot new thing to build in house. Hell, we did it too. If you don't mind, I wrote up exactly what these new data lakes tend to look like: https://telemetrymachine.com/blog/the-new-state-of-the-art-i...
I'm surprised at this. That is a lot of engineering effort without revealing why they chose they own format and chose to query it via Trino.
"engineering effort" - its all just agents spinning in the background now
This idiotic churn used to be restricted to front end devs and backend folks were spared. Thanks for Agents now every part of the stack is going to be in constant churn thanks to resume builders. At no point in the article did you get a sense they have a handle on what their problem is and why a certain technology and architecture would be a better fit. Just try it out, churn out, try it out.
Certainly frontend folks churned like mad by picking one package after another to build on. I saw it first hand at a startup. I was the backend guy, just wrote it once. That doesn't mean it was perfect or he wasn't doing fine, but I didn't need to rewrite it so many times. I'm not ready for a world where everything gets rewritten on some new toolset just because. Hope you've got good tests.
after 3 months, how we switched back to forked clickhouse.
The git repo [0] has a lot more of the technical details.
[0] https://github.com/PostHog/hoglake
This post is pointless. Tells us how it went, show performance numbers, show benefits, resource usage, ease of use, etc.
Translation: frontier models are so good that we can rewrite our own warehouse infra.
6 months from now: We are moving our warehouse off of Trino + HogLake to newer technology XXX.
This is ridiculous. Why can you not foresee such issues? Do you not do testing with mock data at the size you need? Migrating production between stacks so often means you are not getting new features in. This is pure churn.
HogLake is quite the name ..
There are two kinds of companies: those who know how to build low-level tools that serve as foundation and moat for business growth, and those whose middle managers wring their hands about what their people will be able to maintain and constantly chase off-the-shelf solutions to stitch together.
I know which company I'd rather pick as a vendor.
OK, but I think I would have tested "will it scale?" before switching to DuckDB in the first place.
>We rebuilt everything around our own flavor of Trino and created our own open table format, HogLake.
Ah yes, making the same mistake again.
Ooipiol
One thing I really appreciate about CloudFlare blog posts is that often they show their work, by showing what tickets they opened, what improvements they land upstream. With this march across clickhouse, duckdb, now trino, I'd love to know what if any places the team worked with upstream, what improvements got tried.
I'm pretty ok with the march across products myself. It felt like the big shift was from shared giant clickhouse to lots of duckdbs. I wonder what juggling many more clickhouses would have been like. Excited and hopeful to here more back on Trini, with hope!
It's wild how many really really good data stacks are out there. Apache has been collecting some absolutely stellar projects. And it feels like the rest of world meaning everyone but China is kind of sleeping on a lot of these options. Maybe it's just that these projects often go out of their way to have a place for emblems, for who is using the project, but a lot of these have pretty wide ranges of companies using them.
I do wish we were still in an age of company blogging, where people did talk more to the tech getting played with, talk to what's cooking in the shop. It's great to have these outreaches. There's so much tech and we are just so quiet about it! There's so much less signal to go off of, so much less shared awareness than the aughts and 10's.