The guide is very detailed, so I've saved some for later. I've been using NATS I think since around 2018-2019 (can't recall). I've only used its pub:sub feature as it was MUCH lighter than Kafka.
It's interesting that the platform has grown so much. I paused reading at the inbox feature, so there's more to dig in to. I enjoyed reading the topic guide, and I think it was pretty intuitive when I started using it.
Outside of work projects, I maintain a public transit info site, where I either estimate or process telemetry feeds to generate trip updates, alerts, vehicle positions.
NATS pub:sub works so well for me (along with tidwall:tile38 for geofencing). The site isn't that large, but a large volume of messages pass through NATS every few seconds. It's really been a great reliable small piece of technology.
Oh what a huge article and a good read. Thank you. Ok, let's go.
Why people use Kafka? Because you can throw lava at it and it will eat it.
RabbitMQ is also very good, don't have production experience on it, so I leave it for a specialist. Redis (or valkey.io) is also amazing and very fast.
But for a tram system or any system that needs persistent record for a period of time, I use a Kafka cluster with minimum of 3 servers. I might need to save that data for days or weeks or a single day.
Most of the time the connectors will bring that data, some streams transform it and put it back to Kafka and somewhere down the road, a Postgres connector will save that out. And Kafka works very well on a cluster. For what I've seen it is pretty much limited by network bandwith. It is not that hard to configure even with TLS. Documentation is amazing and you know that you get a rock solid system.
The guide is very detailed, so I've saved some for later. I've been using NATS I think since around 2018-2019 (can't recall). I've only used its pub:sub feature as it was MUCH lighter than Kafka.
It's interesting that the platform has grown so much. I paused reading at the inbox feature, so there's more to dig in to. I enjoyed reading the topic guide, and I think it was pretty intuitive when I started using it.
Outside of work projects, I maintain a public transit info site, where I either estimate or process telemetry feeds to generate trip updates, alerts, vehicle positions. NATS pub:sub works so well for me (along with tidwall:tile38 for geofencing). The site isn't that large, but a large volume of messages pass through NATS every few seconds. It's really been a great reliable small piece of technology.
Oh what a huge article and a good read. Thank you. Ok, let's go.
Why people use Kafka? Because you can throw lava at it and it will eat it.
RabbitMQ is also very good, don't have production experience on it, so I leave it for a specialist. Redis (or valkey.io) is also amazing and very fast.
But for a tram system or any system that needs persistent record for a period of time, I use a Kafka cluster with minimum of 3 servers. I might need to save that data for days or weeks or a single day. Most of the time the connectors will bring that data, some streams transform it and put it back to Kafka and somewhere down the road, a Postgres connector will save that out. And Kafka works very well on a cluster. For what I've seen it is pretty much limited by network bandwith. It is not that hard to configure even with TLS. Documentation is amazing and you know that you get a rock solid system.
the ecosystem around Kafka is so crazy specially in regards to data engineering.
CDC capture Kafka (Redpanda) these days - just works and simplifies a bunch of stuff.
Recent big discussion on NATS. https://news.ycombinator.com/item?id=46196105 (3 months ago)
It's very interesting topic, too bad it's an ai;dr for me
[dead]
[dead]