Instead of spending all of this effort trying to optimize an OLAP data lake for an OLTP use case, why not ust use a regular OLTP database like Postgres or MySQL?
Even if the data size is "infinite" you can put it on PlanetScale or something like that. If the tables are well optimized with covering indexes, you can pull out thousands of rows in 100ms for point queries, which is plenty for "point" queries.
1PB of EBS volumes on sc1-class HDD is roughly $15k/month.
Let's say you need at least 2 copies of each byte for cross-AZ availability and durability, that's 30k per PB for JUST the drives.
These disks also need compute attached to them, coupling their scaling needs. Now you have a fleet of (expensive) VMs, regardless of how many queries end up hitting that database.
Of course, those VMs have to run pretty complex software such as PlanetScale, which has to take care of replication, backups, concurrency control, sharding, indexing, vaccuuming, and a whole bunch of other things that add operational complexity and require expertise.
That's the fixed costs of an OLTP database. A lot of that cost is derived from the need to, well, process transactions.
Thing is, they already have a data lake. So the fixed cost to store a PB of data on S3 is already paid. They mention the index is roughly 3 orders of magnitude smaller than the data set (1PB of data is roughly 1TB of index).
With a marginally less complex compute stack, I believe this becomes cost effective pretty quickly.
Pretty clever to keep the existing parquet files as is (assuming daily partitioned) to not break any analytical queries. Also creating this index for the first time would make me sweat. With exabytes in the lake, the backfill bill will be $$$ (even just scanning the key column).
I'm not sure about the AI agent angle though, why would it need that specific context on a user level? I guess they are exploring a "chat with my spotify history" or user generated playlists?
(also the article has several LLM smells. It's technically interesting but reads like slop at times)
Instead of spending all of this effort trying to optimize an OLAP data lake for an OLTP use case, why not ust use a regular OLTP database like Postgres or MySQL?
Even if the data size is "infinite" you can put it on PlanetScale or something like that. If the tables are well optimized with covering indexes, you can pull out thousands of rows in 100ms for point queries, which is plenty for "point" queries.
costs?
1PB of EBS volumes on sc1-class HDD is roughly $15k/month. Let's say you need at least 2 copies of each byte for cross-AZ availability and durability, that's 30k per PB for JUST the drives.
These disks also need compute attached to them, coupling their scaling needs. Now you have a fleet of (expensive) VMs, regardless of how many queries end up hitting that database.
Of course, those VMs have to run pretty complex software such as PlanetScale, which has to take care of replication, backups, concurrency control, sharding, indexing, vaccuuming, and a whole bunch of other things that add operational complexity and require expertise. That's the fixed costs of an OLTP database. A lot of that cost is derived from the need to, well, process transactions.
Thing is, they already have a data lake. So the fixed cost to store a PB of data on S3 is already paid. They mention the index is roughly 3 orders of magnitude smaller than the data set (1PB of data is roughly 1TB of index). With a marginally less complex compute stack, I believe this becomes cost effective pretty quickly.
Pretty clever to keep the existing parquet files as is (assuming daily partitioned) to not break any analytical queries. Also creating this index for the first time would make me sweat. With exabytes in the lake, the backfill bill will be $$$ (even just scanning the key column).
I'm not sure about the AI agent angle though, why would it need that specific context on a user level? I guess they are exploring a "chat with my spotify history" or user generated playlists?
(also the article has several LLM smells. It's technically interesting but reads like slop at times)