This is obviously slightly exaggerated, but I do feel like this whenever people dismiss Kubernetes as either too complicated or not needed.
The response I always got when suggesting Kubernetes is "you can do all those things without Kubernetes"
Sure, of course. There are a million different ways to do everything Kubernetes does, and some of them might be simpler or fit your use case more perfectly. You can make different decisions for each choice Kubernetes makes, and maybe your decisions are more perfect for your workload.
However, the big win with Kubernetes is that all of those choices have been made and agreed upon, and now you have an entire ecosystem of tools, expertise, blog posts, AI knowledge, etc, that knows the choices Kubernetes made and can interface with that. This is VERY powerful.
Kubernetes is a complicated solution to a complicated problem. A lot of companies have different problems and should look for different solutions. But if you are facing this particular problem, Kubernetes is the way to go. The trick is to understand which problem you are facing.
Kubernetes can be a sign that are you making things more complicated than they should be, too early. But if you actually have made things complicated enough (whether through essential or accidental complexity) that you have problems that k8s is good at solving, I really hope you have it instead of some hand rolled solution.
I feel the same way about commercial APM tools. Obviously in a perfect world, you would have software so simple and fast that they’re unnecessary. Maybe every month or two someone has to grep some logs that are already in place. Once you’ve gotten yourself in situation where this is obviously not true, having Datadog, New Relic or similar set up (or using k8s instead of 100 unversioned shell scripts by someone who doesn’t work there anymore) will make your inevitable distributed microservice snafu get resolved in hours rather than a longer business-risking period.
> But if you actually have made things complicated enough [...]
The only problem I see in this case is that complexity doesn't come all at once. By the time you reach a problem that k8s is good at solving, you've probably already accidentally made a k8s alongside your piece of software.
In my(quite short) SWE career, I've seen software evolve, even ones with a proper design stage. Maybe I just don't have enough experience to have seen a properly designed project, but I don't know what I don't know after all.
> all of those choices have been made and agreed upon
Have they really? I have a few apps deployed on k8s and I feel like every time I need something, it turns out it doesn't do that and I'm into some exotic extension or plugin type ecosystem.
Something as simple as service autoscaling (this was a few years ago) was an adventure into DIY. Moving from google cloud to AWS was a complete writeoff almost - just build it again.
I'm sure it captures some layer of abstraction that's useful but my personal experience is it seems very thin and elusive.
This, and because of that, claiming your app "runs in kubernetes" is completely meaningless.
Concretely: Take your app. With one button click, or apt-get install ??? on all your machines, configure k8s. Now, run your app.
The idea that this could work has been laughable for any k8s production environment I've seen, which means you can't do things like write automated tests that inject failures into the etcd control plane, etc.
(Yes, I know there are chaos-monkey things, but they can't simulate realistic failures like kernel panics or machine reboots, because that'd impact other tenants of the Kubernetes cluster, which, realistically, is probably single tenant, but I digress..)
If your configuration is megabytes of impossible to understand YAML, and is also not portable to other environments, then what's the point?
(I understand the point for vendors in the ecosystem: People pay them for things like CNI and CSI, which replace Linux's network + storage primitives with slower, more complicated stuff that has worse fault tolerance semantics. Again, I digress...)
> If your configuration is megabytes of impossible to understand YAML, and is also not portable to other environments, then what's the point?
If almost all your configuration is about getting Kubernetes set up, and not about your application setup inside Kubernetes, there probably isn't a point. But being able to use roughly the same config inside different Kubernetes is quite good.
But I've never seen portable kubernetes configs (except for vendor software that probably wouldn't be needed outside of kubernetes).
If you just tell kubectl to dump your pod configs, then load them on some other cluster, that definitely won't work.
If you use the management software that generated the pod setup somewhere else, that probably won't work either because the somewhere else is going to be missing the CSI and CNI you targeted. Even if those match, it'll be missing the CRDs. God help you if you want to run two programs on one Kubernetes, and there's a CRD versioning conflict in their two dependency sets.
> Moving from google cloud to AWS was a complete writeoff almost - just build it again.
Yep. Kubernetes is not just kubernetes when moving between clouds, it becomes a very opinionated product (for better or worse) with lots of vendor addons. Could someone that is familiar with one pick up on the other? Sure! But there are gotchas. And then kubernetes on prem adds the hardware lifecycle piece, and potential data locality issues, etc.
There are differences across vendors, but there’s a way to build with k8s where the benefit far outweighs the cost.
We run a bunch of services in two very different cloud vendors (one of which used to be DIYed with kubeadm), and also on dev machines with k3s. Takes a while to figure this out and to draw the kustomize boundaries in the right place, but once you do, it’s actually really nice.
Two things work in our favor:
- we’ve been at this for around 8 years, so we didn’t have to deal with all the gotchas at once
- we aggressively avoid tech that isn’t universal (so S3 is OK, but SQS or DynamoDB is not; use haproxy instead of ingress controllers; etc)
> Kubernetes is not just kubernetes when moving between clouds, it becomes a very opinionated product (for better or worse) with lots of vendor addons.
I think this is gradually getting better. Networking with Gateways is better than with Ingress in this sense. Things like autoscaling groups need to get better, as they are (or were a couple of years ago) very bespoke.
I wouldn’t really call it “DIY” per se, k8s has the resource API and you can create whatever scaling policies you want to with it, but I do see how that’s not obvious when it’s advertised as ‘batteries included’
> However, the big win with Kubernetes is that all of those choices have been made and agreed upon, and now you have an entire ecosystem of tools, expertise, blog posts, AI knowledge, etc, that knows the choices Kubernetes made and can interface with that. This is VERY powerful.
Yep! I am now using k8s even for small / 'single purpose' clusters just so I can keep renovate/argo/flux in the loop. Yes, I _could_ wire renovate up to some variables in a salt state or chef cookbook and merge that to `main` and then have the chef agent / salt minion pick up the new version(s) and roll them out gradually... but I don't need to, now!
Agree. For years I had developed my own preferred way of deploying Rails apps large and small on VMs: haproxy, nginx, supervisord, ufw, the actual deploy tooling (capistrano and other alternatives) and so on... and if those tools are old or defunct now it's because my knowledge of that world basically halted 8 years ago because I've never had to configure anything but k8s since then.
I've used it every day since then so I have the luxury of knowing it well. So the frustrations that the new or casual user may have are not the same for me.
I just feel like "you can do this with Kubernetes" is a slippery slope.
"You can do X with Y, so use Y" is a great way to add a dependency, especially if it is "community vetted" already.
Sometimes simple is better - you don't need to add anything that implements some of you logic as a dependency to stay DRY or whatever you want to call it.
It really feels like we are drowning in self-imposed tech debt and keep adding layers to try and hold it for just a while longer.
Now that being said, there is no reason not to add Kubernetes once a sufficient overlap is achieved.
Kubernetes handles so many layers you are going to need for every app, though… deployments, networking, cert management, monitoring, logging, server maintenance, horizontal scaling… this isn’t a slippery slope, it is just what you need.
You have to pick and then configure those components, just like you would have had to pick and configure apps doing those things if you were not using k8s, so the only thing k8s actually brings to the table is a common configuration format (yaml).
The thing about Kubernetes is its a standardization of deployment. Kubernetes is complicated because deploying software is complicated. You might try to YAGNI hand wave it away, but as the article points out, over time, you end up building Kubernetes anyway
You can use k8s on $2/mo digital ocean projects. It probably even works on the free tier of a lot of providers.
And there's zero setup. Just a deployment yaml that specifies exactly what you want deployed, which has the benefit of easy version control.
I don't get why people are so bent on hating Kubernetes. The mental cost to deploy a 6-line deployment yaml is less than futzing around with FTP and nginx.
Kube is the new LAMP stack. It's easier too. And portable.
If you're talking managed kube vs one you're taking the responsibility of self-managing, sure. But that's no different than self-managing your stack in the old world. Suddenly you have to become Sysadmin/SRE.
This made me audibly guffaw. Kubernetes is a lot of things, but "portable" is not one of them. GKE, EKS, AKS, OCP, etc., portability between them is nowhere near guaranteed.
It is if you stick to standard Kubernetes resources, and it has gotten even easier with better storage class and load balancer support. All of the cloud providers now give you default storage classes and ingresses when you provision a cluster on them, so you can use the exact same deployment on any of them an automatically get those things provisioned in the right way out of the box.
>It is if you stick to standard Kubernetes resources
"If you stick to standard C..."
No one does, that's the issue. Helm charts that only support certain cloud providers, operators and annotations that end up being platform specific, etc.
>now give you default storage classes and ingresses
Ingress is being deprecated, it's Gateway now! Welcome to hell, er, Kubernetes.
If you're using it after it's dead, you're at risk of further problems of this nature that aren't in the underly nginx reverse proxy but in the code wrapping it.
That's one reason I've always used Traefik as my Ingress (I work mostly with K3S, which uses it by default). Which appears to have had its own security issues too, but it still looks like an implementation issue, not a weakness designed in by the spec.
On EKS I'm using whatever AWS has brewed up to integrate ELB/ALB, but I'll tend to trust it ... though maybe I shouldn't, given all the troubles I have with other integrations like secrets management.
Would love to use Gateway! Every time I spin up a new cluster it goes like this:
- New cluster setup, time to use gateway! Yay!
- Oh crap, like 80% of the helm chart and other existing configurations I need for the softwares I'm trying to deploy STILL doesn't use gateway, this new API that's been out for... like half a decade at least.
- Even core networking things like Istio/Envoy only have limited gateway support compared to ingress
- Sigh. Ingress again.
It's been like this since gateway's inception and every time I check the needle has moved like 2% towards gateway. So I'm looking forward to year 2050 when I can use gateway!
The problem, as CNCF knows, if they pushed Gateway and deprecated ingress the world would revolt due to the amount of work involved to migrate stuff. Therefore, they leave it up to "the people" to do the extra work themselves, who have no incentive to do so since for many usecases it's not materially better.
I use Kubernetes every day, and have worked with dozens of helm charts, and have yet to encounter cloud specific helm charts. Are these internal helm charts for your company?
Obviously you can lock yourself in if you choose, but I have yet to see third party tools that assume a specific provider (unless you are using tools created BY that provider).
At my previous spot, we were running dozens of clusters, with some on prem and some in the cloud. It was easy to move workloads between the two, the only issue was ACLs, but that was our own choice.
I know they are pushing the new gateway api, but ingresses still work just fine.
"When deploying a JFrog application on an AWS EKS cluster, the AWS EBS CSI Driver is required for dynamic volume provisioning. However, this driver is not included in the JFrog Helm Charts."
"JFrog validates compatibility with core Kubernetes distributions. Some Kubernetes vendors apply additional logic or hardening (for example, Rancher), so JFrog Platform deployment on those vendor-specific distributions might not be fully supported."
I'm a Kubernetes user and advocate but to call it "portable" just tells me you've never actually tried to deploy the similar thing on multiple different clouds. Even the standardized kubernetes resources behave differently due to various cloud idiosyncracies. You can of course make the situation easier, but to call it entirely portable is probably a misnomer.
I don't you made that argument but could a valid conclusion of your comment be that, because Kubernetes is so ubiquitous, using it frees you from being a Sysadmin/SRE?
> If you can solve the same problem in a simpler way without using k8s
I think I disagree with this, or at least the implication. I think it is true you can solve EACH OF THISE PROBLEMS INDIVIDUALLY in a simpler way than Kubernetes, the fact that you are going to have to solve at least 5-10 of those problems individually makes the sum total more complicated than Kubernetes, not to mention bespoke. The Kubernetes solutions are all designed to work together, and when they fail to work together, you are more likely to find answers when you search for it because everyone is using the same thing.
I think it is fair to say k8s is not a zero cost abstraction, but nothing you use instead is going to be, either, and when you do run into a situation where that abstraction breaks, it will be easier to find a solution for kubernetes than it will for the random 5 solutions you pieced together yourself.
Ephemeral user accounts were agreed upon before that. The OG container
Docker and k8s are just wrappers around namespaces, cgroups, file system ACLs, some essential cli commands, which can also be configured per user.
We may be headed back there. Have seen some experiments leveraging Linux kernels BPF and sched_ext to fire off just the right sized compute schedule in response to sequences of specific BPF events.
Future "containers" may just be kernel processes and threads... again. Especially if enough human agency looks away from software as AI makes employment for enough people untenable. Why would those who remain want to manage kernels and k8s complexity?
Imo its less we agreed on k8s specifically and more we agreed to let people use all the free money to develop whatever was believed to make the job easier; but if the jobs go away then it's just more work for the few left
> Docker and k8s are just wrappers around namespaces, cgroups, file system ACLs, some essential cli commands, which can also be configured per user.
Docker, yes, but kubernetes is way more than that the instant you have more than one physical machine node. (If you only have one node in any deploy, sure, it's likely overkill, but that seems like a weird enough case to not be worth too much ink.)
If you silently replaced all my container images with VM images and nodes running containers with nodes running VMs, I think the vast majority of all my Kubernetes setup would be essentially unchanged. Heck, replace it all with people with hands on keyboard in a datacenter running around frantically bringing up new physical servers, slapping hard drives in them, and re-configuring the network, and I don't think the user POV of how to describe it would change that much.
I've seen some places advertise it but I have not tried it.
But, honestly, more generally in my head I wasn't thinking much about it since I consider that as a "cost optimization" thing than a "core kubernetes function." E.g. the addition (or not) of limits is just a couple lines, compared to all the rest of the stuff that I'd be managing specification of (replicas, environment, resource baseline, scheduling constraints, deployment mode...) that would translate seamlessly.
(And there are a lot of parts of kubernetes that annoy me, especially around the hoops it puts up to customize certain things if you reaalllly actually need to, but it would never cross my mind in a hundred years to characterize it as just a wrapper around cgroups etc like the OP.)
Something often underappreciated is that, in the possible future you're describing, you can use all of these new fangled "what's old is new again" approaches by continuing to just use Kubernetes. Kubernetes is, in a way, designed to replace itself.
"Kubernetes is beautiful. Every Concept Has a Story, you just don't know it yet... So you use a Deployment... So you use a Service... So you use Ingress... So..."
lol, the big problem with kubernetes is that none of the choices have been made, it's not opinionated at all, there's no conventions. It's all configuration and choices all the way down. There's way too much yaml, and way to many choices for ever tiny component, it's just too much.
I do run a k3s cluster for home stuff...
But I really wish I could get what it provides in a much simpler solution.
My dream solution would effectively do the same as k3s + storage, but with a much simpler config, zero yaml, zero choices for components, very limited configuration options, it should just do the right ting by default.
Storage (both volume and s3), networking, scale to zero, functions, jobs, ingress, etc... should all just be built in.
Well... we have k8s for that... I do not wish to take k8s away from those who like it, I am asking for a new solution that's very opinionated, and as close to zero config as practical.
I have limited ram and want scale to zero for apps that use a lot of ram, but I only use one of at a time like game servers, or things that can be done over night while I sleep like media encoding.
The main reason I went to k8s, is for the not having to think about what machine will have enough resources to run an app, just throw it at the cluster and it figures out where there's capacity.
And, I want hardware failing/getting replaced to be a non issue.
edit: I wanted to add that my hobby is not systems admin, I want it to be as hands off as possible. Self-hosting is a means to an end. I have so far saved over $200/month in subscriptions by replacing subscriptions I was using with self-hosted alternatives. I can now use that money on my actual hobbies.
Yeah, I spent quite a bit of time learning Kubernetes, but now I'd use it to host a static webpage on a single server, over alternatives. It's so awesome.
I am not the person you asked this question to, but I would probably do the same so I will answer:
Once you get used to it, it just makes managing things simple if you always use it for everything. I have a personal harbor service that I run on my local cluster that has all my helm charts and images, and i can run a single script that sets up my one node cluster, then run a helm install that installs cert-manager and my external-dns, and now I can deploy my app with whatever subdomain I want and I immediately get DNS set up and certs automatically provisioned and rotated. It will just work.
1. Assuming managed service, it frees me from host OS management. So basically the same proposition, as good old "PHP+MySQL" hosters. You upload your website, they make sure it works. But without limitations and with much better independence.
2. It allows me to configure everything using standard manifests. I need to provision the cluster itself initially, then everything could be done with gitops of various automation levels. I don't need to upload my pages via FTP. My CI will build OCI image, publish it to some registry, then I'll change image tag of my deployment and it'll be updated.
3. It allows to start simple, and extend seamlessly in the future. I can add new services. I can add new servers. I can add new replicas of existing services. I can add centralized logging, metrics, alerts. It'll get more complicated but I can manage the complexity and stop where I feel comfotable.
4. One big thing that's solved even with the simplest Kubernetes deployment is new version deployment with zero downtime. When I'll update image tag of my deployment, by default kubernetes will start new pod, will wait for it to answer to liveness checks, then redirect traffic to new pod, let old pod to gracefully stop and then remove it. With every alternative technology, configuring the same requires quite a bit of friction. Which naturally restricts you to deploy new versions only at blessed times. With Kubernetes, I started to trust it enough, I don't care about deployment time, I can deploy new version of heavily loaded service in the middle of the day and nobody notices.
5. There are various "add-ons" to Kubernetes which solve typical issues. For example Ingress Controller allows the developer to describe Ingress of the application. It's a set of declarative HTTP routes which will be visible outside and which will be reverse-proxied to the service inside. Simplest route is https://www.example.com/ -> http://exampleservice:8080, but there's a lot more to it, basically you can think about it as nginx config done differently. Another example is certificate manager, you install it, you configure it once to work with letsencrypt and you forget about TLS, it just works. Another example is various database controllers, for example cloudnativepg allows you to declaratively describe postgres. Controller will create pod for database, will initialize it, will create second pod, will configure it as replica, will perform continuous backup to S3, will monitor its availability and switch master to replica if necessary, will handle database upgrades. A lot of moving parts (which might be scary, tbh), all driven by a simple declarative configuration. Another example is monitoring solutions, which allow to install prometheus instance and configure it to capture all metrics from everything in cluster along with some useful charts in grafana, all with very little configuration.
6. There are various "packages" for Kubernetes which essentially package some useful software, usually in a helm charts. You can think about `apt-get` but for a more complicated set of services, mostly pre-configured and typically useful for web applications. The examples above are all installable with helm, but they add new kubernetes manifest types, which is why I called them "add-ons", but there are also simpler applications.
Just for the record, I don't suggest that to everyone. I spent quite a bit of time tinkering with Kubernetes. It definitely brings a lot of gotchas for a new user and it also requires quite a bit of self-restrictions for experienced users to not implement every devops good practice in the world. Sometimes maybe you don't even want to start with ingress, I saw cluster which used manually configured nginx reverse proxy instead and it worked for them. You can be very simple with Kubernetes.
Honestly the main problem is people using k8s for something that's like... a database, and an app, and maybe a second app, that all could be containers or just a systemd service.
And then they hit all the things that make sense in big company with like 40 services but very little in their context and complain that complex thing designed for complex interactions isn't simple
But if you want some redundancy, k8s let's you just say run 4 of this, 6 of this on these 3 machines. At least I find it quite straight forward.
The database is more complex since there is storage affinity (I use cockroachDB with local persistent volumes for it) - but stateful is always complicated.
Most of the time you don't need redundancy. You need regular backups for exceptional circumstances. And k8s gives you more complexity, and more problems through more moving parts, to give you the possibility of using a feature you'll never need, and if you do start to use it it'll probably be instead of fixing performance problems downstream
Are we talking for personal projects where there are no expectations, or small startups where you don’t have much scale but you still care about down time and data loss?
Personal projects are one thing, but even the smallest startup wants to be able to avoid data loss and downtime. If you are running everything on one server, how do you do kernel patches? You need to be able to move your workload to another server to reboot for that, even if you don’t want redundancy. Kubernetes does this for you. Bring in another node, drain one (which will start up new instances on the new node and shift traffic before bringing down the other instance, all automatically for you out of the box), and then reboot the old one.
Again, you could do all of this with other tech, but it is just standard with Kubernetes.
> but even the smallest startup wants to be able to avoid data loss
Seems true at a glance!
> and downtime.
Maybe less so - I think there’s plenty out there, where they’re not chasing nines and care more about building software instead of some HA setup. Probably solve that issue when you have enough customers to actually justify the engineering time. A few minutes of downtime every now and then isn’t the end of the world if it buys you operational simplicity.
A while back when the agents got hyped I was looking into the whole "give it a VM / docker container" I realized the safest and simplest option was just to give it its own machine.
Then I realized giving it root on a $3 VPS is functionally equivalent. If it blows it up, you just reset the VM.
It sounds bad but I can't see an actual difference.
No argument there. The Toyota 5S-FE non-interference engine is a near indestructible 4 cylinder engine that's well documented, popular and you can purchase parts for pennies. It has powered 10 models of Camrys and Lexus and battle proven. You can expect any mechanic who has been a professional mechanic for the last 3 years know exactly what to do when it starts acting up. 1 out of 4 cars on the road have this engine or a close clone of it.
It's not what any reasonable person would use for a weedwhacker, lawnmower, pool pump or an air compressor.
Sure, but to extend your metaphor, Kubernetes HAS smaller engine models that you can use in those situations, and still gain all the benefits of being in the same ecosystem. You can use K3s, for example, and get all the benefits without having a giant engine in your weedwhacker.
IMO, Kubernetes isn't inevitable, and this seems to paint it as such.
K8s is well suited to dynamically scaling a SaaS product delivered over the web. When you get outside this scenario - for example, on-prem or single node "clusters" that are running K8s just for API compatibility, it seems like either overkill or a bad choice. Even when cloud deployed, K8s mostly functions as a batteries-not-included wrapper around the underlying cloud provider services and APIs.
There are also folks who understand the innards of K8s very well that have legitimate criticisms of it - for example, this one from the MetalLB developer: https://blog.dave.tf/post/new-kubernetes/
Before you deploy something, actually understand what the pros/cons are, and what problem it was made to solve, and if your problem isn't at least mostly a match, keep looking.
This is a need it fails at miserably. k8s reminds me of the raid recentralization anti pattern problem where you fix a hardware failure that never occurs in exchange for knowing simple higher level mistakes or security problems will tank something now too large to fail again.
Kubernetes, in the form of k3s, was a critical success factor for us with the onprem deployment of our SaaS product.
What's the problem with a single-node cluster? We use that for e.g. dev environments, as well as some small onprem deployments.
> Even when cloud deployed, K8s mostly functions as a batteries-not-included wrapper around the underlying cloud provider services and APIs.
Which batteries are not included? The "wrapper around the underlying cloud provider services and APIs" is enormously important. Why would you prefer to use a less well-designed, more vendor-specific set of APIs?
I seriously don't get these criticisms of k8s. K8s abstracts away, and standardizes, an enormous amount of system complexity. The people who object to it just don't have the requirements where it starts making sense, that's all.
> Kubernetes, in the form of k3s, was a critical success factor for us with the onprem deployment of our SaaS product.
What surprises and gotchas did you have to deal with using k3s as a Kubernetes implementation?
Did you use an LB? Which one? I'm assuming all your onprem nodes were just linux servers with very basic equipment (the fanciest networking equipment you used were 10GbE PCIe cards, nothing more special than that?)
We sell to enterprise customers. All of them deploy our solution on internal cloud-style VM clusters. We use the Traefik ingress controller by default.
There really weren't any particular surprises or gotchas at that level.
In this context, I've never had to deal with anything at the level of the type of Ethernet card. That's kind of the point: platforms like k8s abstract away from that.
It's also difficult for data pipelines or data intensive things. At several companies we've run into the "Need to put ML model behind API and pods get killed because health checks via API are basically not compatible with container fully under load but still working"
The saddest part about Kubernetes is… after you set it all up, you still need a hacky deploy.sh to sed in the image tag to deploy! And pretty soon you’re back to “my dear friend you have built a Helm”. And so the configuration clock continues ticking…
Claude Code has essentially fixed this perpetual annoyance for me. Doesn't matter if it's a hacked up deploy.sh that mixes sed, envsubst and god knows what or a non-idiomatic Helm chart that was perpetually on my backlog to fix... today I just say "make this do this thing and also fix any bash bugs along the way" and it just does it. Its effectiveness for these thousand-little-cuts type DevOps tasks is underrated IMO.
Now the actual CI/CD/thing-doers tools that all suck... I'm still stuck with those.
This is why we don't take advice from randos on internet.
I manage 100+ variations on a single helm chart and 50+ such helm charts at work daily for 7 years across 11 datacenters/kubernetes clusters. And I have team members who swear by kustomize. The number of kustomize typo errors and issues that I have deal with is unimaginable. Whereas if I test and deploy a helm chart, I know it will work everywhere in every variation.
Kustomize is just plain terrible and backwards as a solution. It doesn't scale, it is half assed. It tries to basically require you to build your own compiler and parser and transform. With kustomize + envsubst: dear friend, you have built helm.
How do you handle cleanups and hooks? The best way to do helm, at least for me, seems to be about limiting its use to simple templating use cases; if you end up needing an if, you've probably done something terribly wrong.
My personal theory is that Helm may be ok for distributing a pre-packaged solution to other people. Then people mistook it for a tool that should be used in-house to deploy a company’s own systems, where it makes much less sense.
It makes absolute sense. You can use no variables and still deploy helm chart. It is a directory of plain old yaml objects. And add customization when you need as you evolve. Good luck doing that with kustomize.
Cleanups: I want to do a `helm uninstall` and have all the manifests go away at once instead of looking around for N different resources.
Hooks: I want to apply my database migrations and populate the database with static datasets before I deploy my application, without having my CI connect to the database cluster (at places I've worked, the CI cluster and K8s cluster were completely separate).
Regarding cleanups: I'm using flux CD with kustomize. It tracks resources that it created. If I delete manifest from my repository, flux will delete resources that were created from these manifests. For me that's pretty much the ideal workflow.
Regarding hooks: I don't know. All applications that I've used, implemented migrations internally (it's usually Java with Flyway), so I don't need to think about it. One possible approach could be to use flux CD with Job definition. I think that Flux will re-create Job when it changes. So if you change image tag, it'll re-create Job and it'll trigger Pod execution. But I didn't try this approach, so not sure if that would work for you.
> I want to apply my database migrations and populate the database with static datasets before I deploy my application, without having my CI connect to the database cluster
A Job feels like a good fit for this. CI deployes the Job without connecting to DB, Job runs migrations using the same connectivity as the application.
> apply my database migrations and populate the database with static datasets before I deploy my application
You could a) have the app acquire a lock in the db and do its own migrations, or b) create a k8s job that runs the migration tool, but make sure the app waits for the schema to be updated or at least won't do anything bad.
There are a multitude of cases of operations which need to be performed before and after specific actions in K8s. It depends on the resource, operator, operational changes, state, bugs, order of operations, and more.
Or if your colleagues are "smarter" than you they make it in Clojoure instead, with an EDN-but-with-subroutines config language, so that not only yaml-aware editors are useless, but EDN-aware editors cannot make heads or tails of the macros.
For very simple deployments, you don't need anything at all. Just write manifests and use `kubectl apply`. You can write `deploy.sh` but it'll be trivial.
If you want templating, there are many options. You can use `sed` for the most simple templating needs. You can use `cpp`, `m4`, `helm` or `kustomize`. I, personally, like `kustomize`, but `helm` probably not the worst template engine out there.
Kustomize is even somewhat included into basic kubernetes tooling, so if you want something "opinionated", it is there for you. It works.
https://fluxcd.io/ + helm + with a CI pipeline that pushes the docker images to a registry means that after the setup, anytime you push a new image and tag, k8s can automatically update without needing to do anything manual.
Anyone remembers the GitOps thingy called flux? Weave was the company name.
Git and Kubetnetes configuration cannot go hand in hand. You cannot go back in past indefinitely because cluster state might not be that reversible. If so, git is useless.
And no, doesn't apply for database migrations. You can mostly run migrations backwards if each migration was written carefully.
I'm starting to use it for my self hosted services.
I have a "simple" representation of services using CUE, that generates the yaml manifests and flux deploys them.
I hesitated a while before going the k8s route but before that I had a overly error-prone Ansible configuration and I got sick of manual templating (hence the move to CUE for type safety).
There's also the fact that I wanted my services to be as plug and play as possible, so for example automatically generated openid credentials and very easily configurable central SSO, along with the easily configurable reverse-proxy.
If anyone thinks that k8s is not the best tool for this, I'm always interested in advice.
(Also a lot of complexity in my setup is due to self hosting, I have Istio, MetalLB, proxmox CSI, and all other kind of stuff that your cloud provider would already have, and these are the things that take most of the configuration files in my repo)
I've used it in the past and personally loved it. Just bumping a yaml file in a git repo to the image tag I wanted deploying was a godsend and nearly automated. I can't speak to your experience though which I am certain is valid and a real problem. We just never had those kind of issues so we could either revert to an earlier tag that worked or publish a new image with the required resolution steps.
And if you want your Helm to run on certain deploys, and maintain a declarative set of the variables given to charts over time, thinking you can use Helmfile and some custom GitHub Actions… “my dear friend you have built a GitOps.”
(I tend to think this one is acceptable in the beginning, but certainly doesn’t scale.)
They have built an orchestrator, not Kubernetes. There is one key difference: they know this thing, end-to-end, down to every single bolt and piece of duct tape (with possible exception for Docker internals)
And that's a very important distinction when it comes to maintaining complex systems. This could've changed with LLMs (I'm still adjusting to what new capabilities mean for various decision-making logic), but before machine intelligence debugging an issue with Kubernetes could've been a whole world of pain.
And chances are only they know it. If my role has enough cluster access, I can muddle through pretty much any helm chart (with lots of cursing, yes) but it might take me days to set up whatever elaborate bespoke environment and script invocations are needed to replicate the current production setup maybe.
As someone rolling their self-hosted stuff via Compose and shell scripts instead of K8s specifically for the simplicity of the experience, this is 100% why you need to understand what Kubernetes solves before writing it off entirely.
I'm not doing overlay networks, I'm using a single bare-metal host, and I value the hands-on Linux administration experience versus the K8s cluster admin experience. All of these are reasons I specifically chose not to use Kubernetes.
The second I want HA, or want to shift from local VLANs to multi-cloud overlays, or I don't need the local Linux sysadmin experience anymore? Yeah, it's K8s at the top of the list. Until then, my solution works for exactly what I need.
I run K8s at home. I used to do docker-compose - and I'd still recommend that to most people - but even for my 1 little NUC with 4vcpu / 16Gi Homelab, I still love deploying with K8s. It's genuinely simpler for me.
If anyone's looking for inspiration, my setup:
* ArgoCD pointed to my GitLab repos
* GitLab repos contain Helm charts
* Most of the Helm charts contain open-source charts as subcharts, with versions set like (e.g.) `version: ~0` - meaning I automatically receive updates for all major version until `1`
* Updating my apps usually consists of logging into the UI, reviewing the infrastructure and image tag updates, and manually clicking sync. I do this once every few months
My next little side project: Autoscaling into the cloud (via a secure WireGuard tunnel) when I want to expand past my current hardware limitations
A reason not to run k8s is if you want your server to reach C10 idle states. The k8s control plain with its polling and checking are quite heavy on the mostly idle server.
I have reverted to just use Nixos and oci podman containers. Everything is declarative and reproducible
another one is swap. UnlimitedSwap was deprecated and you can now only use LimitedSwap which restricts how much swap you can use, so you can't take full advantage of zram, which sucks for those looking to run lean
I can tell you how vendors deliver a software solution that runs on Kubernetes: very poorly.
The needed tweaks, the ability to customize things, basically goes to zero because the support staff is technical about the software, but NOT about Kubernetes.
I am not joking: a recent deployment required 3x VMs for Kubernetes, each VM having 256 gigabytes of RAM; then a separate 3x VMs for a different piece. 1.5TB of RAM to manage less than 1200 network devices (routers etc. that run BGP).
No one knew, for instance, how to lower the MongoDB (because of course you need it!) resource usage, despite the fact that the clustered VMware install is using a very fast SSD storage solution and thus MongoDB is unlikely accelerate anything; so over 128GB RAM is being burned on caching the results coming back from SSDs that are running at many-GB/s throughput.
Whether this is deployed via Helm charts or a native controller, there's almost certainly some overlay where you can override resource values, unless this is just a very crappy vendor.
Kubernetes was overkill (I do that all day, 5 days a week); Kamal was too restrictive, so I found myself rolling out Yoink. Just what I need from k8s, but simple enough I can point it to a baremetal machine on Hertzner that can easily run all my workloads.
- using caddy-docker-proxy for ingress is brilliant
What do you use for:
- service discovery
- secret store (EDIT: Crap you use Infisical. No shade, I just have this horrible foreboding it will end up like Hashicorp. I use Conjur Secretless Broker but am tracking: https://news.ycombinator.com/item?id=47903690)
- backing up and restoring state like in a DB
PS: Have you been having issues with Hetzner the last few weeks?
Service discovery is basically just Docker's internal DNS. Caddy-docker-proxy can use it to find healthy upstreams.
For secrets, I self-host Infisical on the box -- easy to plug in whatever secret manager, should make it pair nicely with https://github.com/tellerops/teller or something similar
Had no problems with Hertzner so far, just enjoying the raw CPU power of bare metal. The plan is to roll out more boxes across different providers, using Tailscale for the backplane network and Cloudflare to load-balance between them. All in due time What issues have you been having ?
I have a suspicion you're using Headscale? If so, I urge you to consider Ionscale. I use it with Authentik as the IdP.
Personally commiting to using Tailscale as a core foundation of my infrastructure and Ionscale is my hedge against getting Hashicorped.
> Service discovery is basically just Docker's internal DNS. Caddy-docker-proxy can use it to find healthy upstreams
Do you have a writeup of this somewhere? I'm unaware of being able to manage Docker's internal DNS over some kind of an API (would appreciate if you know a way to). The only way I know is to manipulate network aliases via Docker Engine API. As a result I use Hickory DNS with RFC 2136. That coupled with Caddy-docker-proxy gets me extremely close.
All it would take to make this post actually good would be to replace "Kubernetes" with "orchestrator"; that would also keep the symmetry with the post it's riffing on, about building compilers (it's not "Dear friend you have built a GHC").
Criticisms of Kubernetes generally come from a few places:
- People who would prefer their way of doing this, whether that's deployments on VMs, or use some sort of simpler cloud provider.
I had the same opinion a few years ago, but have kind of come to like it, because I can cleanly deploy multiple applications on a cluster in a declarative fashion. I still don't buy the "everything on K8s", and my personal setup is to have a set of VMs bought from a infrastructure provider, setup a primary/replica database on two of them, and use the rest as Kubernetes nodes.
- People who run Kubernetes at larger scales and have had issues with them.
This usually needs some custom scaling work; the best way to work around this if you're managing your own infra[1] is to split the cluster into many small independent clusters, akin to "cellular deployments"[2]/"bulkhead pattern"[3]. Alternatively, if you are at the point where you have a 500+ node cluster, it may not be a bad idea to start using a hyperscaler's service as they have typically done some of the scaling work for you, typically in form of replacing etcd and the RPC layer through something more stable.
- People who need a deep level of orchestration
Examples of such use cases may be to run a CI system or a container service like fly.io; for such use cases, I agree that K8s is often overkill, as you need to keep the two datastores in sync and generate huge loads on the kube-apiserver and the cluster datastore in the process, and it might be often better to just bring up Firecracker MicroVMs or similar yourself.
Although, I should say that teams writing their first orchestration process almost always run to Kubernetes without realizing this pitfall, though I have learned to keep my mouth shut as I started a small religious war recently at my current workplace by raising this exact point.
[1] Notice how I don't say "on-prem", because the hyperscaler marketing teams would rather have you believe in two extremes of either using their service or running around in a datacenter with racks, whereas you can often get bog-standard VMs from Hetzner or Vultr or DigitalOcean and build around that.
Another case: People who want to run workloads that are inherently incompatible with Kubernetes networking model.
For example:
* For some cursed reasons you want to make sure every single one instance of a large batch job see just one NIC in its container and they are all the same IP and you NAT to the outside world. Ingress? What ingress? This is a batch job!
* Like the previous point, except that your "batch job" somehow has multiple containers in one instance now, and they should be able to reach each other by domain.
That is indeed a weirdly cursed requirement. Why? Black box of legacy stuff? A system that was never designed to be run in multiple does so if all the nodes think they’re the same machine? Defeating a license restriction?
Unless you’re in Erlang world (Elixir, Gleam..) and all that is already baked into OTP and the BEAM. You can go on holiday knowing it will be a while longer before you need to break out the pods (and at that scale, you will be able to afford a colleague or two to help you).
I've experienced something like this at work but with data warehouse instead, and it happened multiple times (to be fair, data engineering is still fairly new where I'm from).
One example was an engineer wanted to build an API that accepts large CSV (GBs of credit reports) to extract some data and perform some aggregations. He was in the process of discussing with SREs on the best way to process the huge CSV file without using k8s stateful set, and the solution he was about to build was basically writing to S3 and having a worker asynchronously load and process the CSV in chunks, then finally writing the aggregation to db.
I stepped in and told him he was about to build a data warehouse. :P
If it was less than 100 gb, he probably should have just loaded the whole thing in RAM on a single machine, and processed it all in a single shot. No S3, no network round trips, no chunking, no data warehouse.
So much defense of Kubernetes as the only solution to problems like this. Nomad and Incus exist as solid and less complicated alternatives. They also have a larger scaling ceiling.
I see docker as a way to avoid having a standard dev platform for everyone in the company so that the infra team don't have to worry about patch xyz for library abc, only run docker.
But, with all the effort put in place to coordinate docker, k8s and all the shebang, isn't it finally easier to force a platform and let it slowly evolve over time?
Is docker another technical tool that tries to solve a non-technical problem?
Docker is a solution to one specific problem: the need for "a user" to run 10 different potentially conflicting apps, all at the same time, on one machine, and abstract away anything which might make those apps conflict if they ran on a single OS. It provides a dozen different solutions in one package.
K8s is a way to take that and make it scale up for a large number of applications on a large number of hosts in a production business in a way that's automated and resilient to failure.
I do not follow you. Every app has different needs. Containers encode them in a shareable way. You can evolve the image over time. So what more do you want?
Why both posts mention docker compose and not mentioning docker swarm. Being using it for my projects for long time. And it's so nice. Similar syntax, easy networking, rollout strategy, easy to add nodes to cluster.
You can have one template docker-compose.yaml file and separate deployment files for different envs, like: docker-compose.dev.yaml, docker-compose.prod.yaml
I've been there. We still ended up with messy deploy scripts written in Ruby and the only debugging solution was "just comment out everything then run line by line".
'docker stack deploy' covers most of the cases. But yeah, there is still some problems like: "update a config or a secret", that require manually invoking additional commands (or via scripts)
Literally just finished building a personal orchestrator system I wanted and had this very much in back of my mind.
Ended up doing a mix. Built on compose for now but in a manner that’ll lift and shift to k8s easily enough. Its containers talking over network either way
After reading this and remembering an old hobby project, I decided to switch the deploy from a systemd service to PM2, which apparently has rolling deployments without needing Docker engine (for those of us minmaxing instance RAM).
Partially true, I would argue. Try running kubernetes on an on-prem infra and you'll soon realize that it was much easier to run the apps on VMs themselves.
I worked with a guy who fits this description. If there’s something he can reinvent poorly, he’d do it. A big part of what I did while working on the same team as him was slowly chipping away at his weird domain by introducing standard tools. I always had to advocate for them in such a way that they were solving a different problem, and then slowly work towards a point where some of his homegrown stuff became unnecessary.
As to the job security idea: the only people who do this are people who aren’t good at creating real value, so they have to try to create niches where they’re needed.
k3s (lightweight k8s) is basically almost as easy as docker compose and then you have access to the full open source CNCF ecosystem in form of helm charts.
im just about giving OPs premise another go. compose just feels so much better as abstraction especially with small and medium setups looking close to the optimum of expressiveness without boilerplate to describe what is needed. The missing pieces seem to also be in the compose compatible “docker stack” aka new docker swarm, which i ignored for probably too long as i assumed it was the discontinued old swarm. Even if new swarm mode sucks how hard can it be to make something compose shaped vs running k8?
> Ah, but wait! Inevitably, you find a reason to expand to a second server
>> The real problem is that programmers have spent far too much time worrying about efficiency in the wrong places and at the wrong times; premature optimization is the root of all evil (or at least most of it) in programming.
-- Donald Knuth, Computer Programming as an Art (1974)
EDIT:
> Except if you quit or go on vacation, who will maintain this custom pile of shell scripts?
Honestly? I don't care. There is a reason why I quit and 99% of time it's the pay. And if the company doesn't pay me enough to bother then why should I? Why should I bother about some company future in the first place?
Shit just gets really weird when your network isn’t split for k8s in an equivalent way to what GCP/AWS expect. Like, if you have other services running on the nodes that you want things inside k8s to talk to, or if the nodes are in a flat subnet with other stuff in it, things get annoying. Those are worst practices for a reason, but pretty common in environments with home rolled k8s clusters.
I am a big fan which is why I am saying this: you're dismissing the kernel and ABI surface is a huge assumption that must hold true for your comment to hold
stavros.
If you had said "unikernels" I would have had no arguments to make.
right - that's precisely what I meant. I read your comment "Containers are just statically-linked programs for the rest of us." as "containers can be replaced by statically-linked programs".
If you didn't imply that, I apologize.
If you did mean that, I disagree with you precisely because your point works if you only care about dependency management - it falls apart on system state. A static binary is a process on the host and shares the same process space, network stack, and filesystem.
OTOH, a container is a jail (the primary usecase): I can't cgroup a static binary's memory usage or give it a virtual network interface without reimplementing "container lite". Containers aren't just 'statically linked programs' - they allow me to use the kernel as a hypervisor for isolated environments.
What they are though, a messy but practical compromise to Unikernels - which was my last point in our GP.
Oh, no, I meant that containers serve the same purpose as statically linked programs for languages that can't do that. Eg if you want to deploy a Python codebase, a container is a good way to include all dependencies.
I didn't mean "containers don't have any advantages compared to statically linked programs".
NOOO you have to use my shitpile of nested yaml with the same dependency sprawl cancer as modern javascript. You can't just upload a binary to your own servers and host it there you need to overthink everything and make an extremely simple process overcomplicated just install one more side car and fifty more dependencies on your helm chart bro and then we can move on to figuring out CSI it should only take like a month to get it working properly I promise!!!!!
Not sure how to describe this, but in a small/medium size company you have a few chips with talent.
And somehow k8s tends to use one of the most valuable chips. The smartest guy who from then on only works with maintaining and updating k8s. Instead of building the best and smartest products.
More like "Dear friend, you have built an Application Container", even more so now with the VCs looking for money in powering WebAssembly containers as pods.
Not as well as they can reason (or others can google) something as standardized as kubernetes. There’s just less context (in both senses of the term) needed to understand something running on a common substrate versus something bespoke, even if the bespoke thing is itself comprised of standardized parts.
For a project set up by a qualified engineer, there would be little difference to the end user in practice. The LLM would work out a solution with a negligible difference in speed. Maybe debugging would also be faster for the LLM without the abstraction layers and low level access?
This is obviously slightly exaggerated, but I do feel like this whenever people dismiss Kubernetes as either too complicated or not needed.
The response I always got when suggesting Kubernetes is "you can do all those things without Kubernetes"
Sure, of course. There are a million different ways to do everything Kubernetes does, and some of them might be simpler or fit your use case more perfectly. You can make different decisions for each choice Kubernetes makes, and maybe your decisions are more perfect for your workload.
However, the big win with Kubernetes is that all of those choices have been made and agreed upon, and now you have an entire ecosystem of tools, expertise, blog posts, AI knowledge, etc, that knows the choices Kubernetes made and can interface with that. This is VERY powerful.
Kubernetes is a complicated solution to a complicated problem. A lot of companies have different problems and should look for different solutions. But if you are facing this particular problem, Kubernetes is the way to go. The trick is to understand which problem you are facing.
Kubernetes can be a sign that are you making things more complicated than they should be, too early. But if you actually have made things complicated enough (whether through essential or accidental complexity) that you have problems that k8s is good at solving, I really hope you have it instead of some hand rolled solution.
I feel the same way about commercial APM tools. Obviously in a perfect world, you would have software so simple and fast that they’re unnecessary. Maybe every month or two someone has to grep some logs that are already in place. Once you’ve gotten yourself in situation where this is obviously not true, having Datadog, New Relic or similar set up (or using k8s instead of 100 unversioned shell scripts by someone who doesn’t work there anymore) will make your inevitable distributed microservice snafu get resolved in hours rather than a longer business-risking period.
> But if you actually have made things complicated enough [...]
The only problem I see in this case is that complexity doesn't come all at once. By the time you reach a problem that k8s is good at solving, you've probably already accidentally made a k8s alongside your piece of software.
In my(quite short) SWE career, I've seen software evolve, even ones with a proper design stage. Maybe I just don't have enough experience to have seen a properly designed project, but I don't know what I don't know after all.
> all of those choices have been made and agreed upon
Have they really? I have a few apps deployed on k8s and I feel like every time I need something, it turns out it doesn't do that and I'm into some exotic extension or plugin type ecosystem.
Something as simple as service autoscaling (this was a few years ago) was an adventure into DIY. Moving from google cloud to AWS was a complete writeoff almost - just build it again.
I'm sure it captures some layer of abstraction that's useful but my personal experience is it seems very thin and elusive.
Yep, this is my main problem with k8s, it really feels like none of the choices have been made, it's all chose and configure components.
This, and because of that, claiming your app "runs in kubernetes" is completely meaningless.
Concretely: Take your app. With one button click, or apt-get install ??? on all your machines, configure k8s. Now, run your app.
The idea that this could work has been laughable for any k8s production environment I've seen, which means you can't do things like write automated tests that inject failures into the etcd control plane, etc.
(Yes, I know there are chaos-monkey things, but they can't simulate realistic failures like kernel panics or machine reboots, because that'd impact other tenants of the Kubernetes cluster, which, realistically, is probably single tenant, but I digress..)
If your configuration is megabytes of impossible to understand YAML, and is also not portable to other environments, then what's the point?
(I understand the point for vendors in the ecosystem: People pay them for things like CNI and CSI, which replace Linux's network + storage primitives with slower, more complicated stuff that has worse fault tolerance semantics. Again, I digress...)
> If your configuration is megabytes of impossible to understand YAML, and is also not portable to other environments, then what's the point?
If almost all your configuration is about getting Kubernetes set up, and not about your application setup inside Kubernetes, there probably isn't a point. But being able to use roughly the same config inside different Kubernetes is quite good.
But I've never seen portable kubernetes configs (except for vendor software that probably wouldn't be needed outside of kubernetes).
If you just tell kubectl to dump your pod configs, then load them on some other cluster, that definitely won't work.
If you use the management software that generated the pod setup somewhere else, that probably won't work either because the somewhere else is going to be missing the CSI and CNI you targeted. Even if those match, it'll be missing the CRDs. God help you if you want to run two programs on one Kubernetes, and there's a CRD versioning conflict in their two dependency sets.
> Moving from google cloud to AWS was a complete writeoff almost - just build it again.
Yep. Kubernetes is not just kubernetes when moving between clouds, it becomes a very opinionated product (for better or worse) with lots of vendor addons. Could someone that is familiar with one pick up on the other? Sure! But there are gotchas. And then kubernetes on prem adds the hardware lifecycle piece, and potential data locality issues, etc.
There are differences across vendors, but there’s a way to build with k8s where the benefit far outweighs the cost.
We run a bunch of services in two very different cloud vendors (one of which used to be DIYed with kubeadm), and also on dev machines with k3s. Takes a while to figure this out and to draw the kustomize boundaries in the right place, but once you do, it’s actually really nice.
Two things work in our favor:
- we’ve been at this for around 8 years, so we didn’t have to deal with all the gotchas at once
- we aggressively avoid tech that isn’t universal (so S3 is OK, but SQS or DynamoDB is not; use haproxy instead of ingress controllers; etc)
> Kubernetes is not just kubernetes when moving between clouds, it becomes a very opinionated product (for better or worse) with lots of vendor addons.
I think this is gradually getting better. Networking with Gateways is better than with Ingress in this sense. Things like autoscaling groups need to get better, as they are (or were a couple of years ago) very bespoke.
I wouldn’t really call it “DIY” per se, k8s has the resource API and you can create whatever scaling policies you want to with it, but I do see how that’s not obvious when it’s advertised as ‘batteries included’
> However, the big win with Kubernetes is that all of those choices have been made and agreed upon, and now you have an entire ecosystem of tools, expertise, blog posts, AI knowledge, etc, that knows the choices Kubernetes made and can interface with that. This is VERY powerful.
Yep! I am now using k8s even for small / 'single purpose' clusters just so I can keep renovate/argo/flux in the loop. Yes, I _could_ wire renovate up to some variables in a salt state or chef cookbook and merge that to `main` and then have the chef agent / salt minion pick up the new version(s) and roll them out gradually... but I don't need to, now!
Agree. For years I had developed my own preferred way of deploying Rails apps large and small on VMs: haproxy, nginx, supervisord, ufw, the actual deploy tooling (capistrano and other alternatives) and so on... and if those tools are old or defunct now it's because my knowledge of that world basically halted 8 years ago because I've never had to configure anything but k8s since then.
I've used it every day since then so I have the luxury of knowing it well. So the frustrations that the new or casual user may have are not the same for me.
I just feel like "you can do this with Kubernetes" is a slippery slope. "You can do X with Y, so use Y" is a great way to add a dependency, especially if it is "community vetted" already. Sometimes simple is better - you don't need to add anything that implements some of you logic as a dependency to stay DRY or whatever you want to call it.
It really feels like we are drowning in self-imposed tech debt and keep adding layers to try and hold it for just a while longer. Now that being said, there is no reason not to add Kubernetes once a sufficient overlap is achieved.
Kubernetes handles so many layers you are going to need for every app, though… deployments, networking, cert management, monitoring, logging, server maintenance, horizontal scaling… this isn’t a slippery slope, it is just what you need.
But k8s does not do almost any of those things!
You have to pick and then configure those components, just like you would have had to pick and configure apps doing those things if you were not using k8s, so the only thing k8s actually brings to the table is a common configuration format (yaml).
The thing about Kubernetes is its a standardization of deployment. Kubernetes is complicated because deploying software is complicated. You might try to YAGNI hand wave it away, but as the article points out, over time, you end up building Kubernetes anyway
You can use k8s on $2/mo digital ocean projects. It probably even works on the free tier of a lot of providers.
And there's zero setup. Just a deployment yaml that specifies exactly what you want deployed, which has the benefit of easy version control.
I don't get why people are so bent on hating Kubernetes. The mental cost to deploy a 6-line deployment yaml is less than futzing around with FTP and nginx.
Kube is the new LAMP stack. It's easier too. And portable.
If you're talking managed kube vs one you're taking the responsibility of self-managing, sure. But that's no different than self-managing your stack in the old world. Suddenly you have to become Sysadmin/SRE.
>And portable.
This made me audibly guffaw. Kubernetes is a lot of things, but "portable" is not one of them. GKE, EKS, AKS, OCP, etc., portability between them is nowhere near guaranteed.
It is if you stick to standard Kubernetes resources, and it has gotten even easier with better storage class and load balancer support. All of the cloud providers now give you default storage classes and ingresses when you provision a cluster on them, so you can use the exact same deployment on any of them an automatically get those things provisioned in the right way out of the box.
>It is if you stick to standard Kubernetes resources
"If you stick to standard C..."
No one does, that's the issue. Helm charts that only support certain cloud providers, operators and annotations that end up being platform specific, etc.
>now give you default storage classes and ingresses
Ingress is being deprecated, it's Gateway now! Welcome to hell, er, Kubernetes.
> Ingress is being deprecated
Do you have any links about Ingress being deprecated?
Official docs here: https://kubernetes.io/docs/reference/kubernetes-api/service-...
There are no mentions about this API being deprecated.
Ingress resource is bassicaly "implementation specific" and isn't portable. It's not deprecated now, but there plans to retire ingress-nginx: https://kubernetes.io/blog/2025/11/11/ingress-nginx-retireme...
Anyway, Ingress resource been in "Migrate to Gateway" state for awhile.
> but there plans to retire ingress-nginx
To clarify, it’s already retired and the repo has been archived since 24th of March.
Mentally, I'm still in January. For some reason the fact that March 24th, 2026 already passed didn't click in my head.
https://kubernetes.io/blog/2025/11/11/ingress-nginx-retireme...
NGINX Ingress is deprecated, not the Ingress resource itself.
Ingress is frozen, not deprecated. Gateway does more, but Ingress isn’t going anywhere. It’s a stable API, which is the opposite of churn.
Til there's a security issue, right? Nginx is a big target.
The API of Ingress is not Nginx's API. The spec itself is basically a yaml schema, it's hard to have a vulnerability in that.
There have been critical vulns in nginx-ingress (the part which is deprecated) like this: https://kubernetes.io/blog/2025/03/24/ingress-nginx-cve-2025...
If you're using it after it's dead, you're at risk of further problems of this nature that aren't in the underly nginx reverse proxy but in the code wrapping it.
That's one reason I've always used Traefik as my Ingress (I work mostly with K3S, which uses it by default). Which appears to have had its own security issues too, but it still looks like an implementation issue, not a weakness designed in by the spec.
On EKS I'm using whatever AWS has brewed up to integrate ELB/ALB, but I'll tend to trust it ... though maybe I shouldn't, given all the troubles I have with other integrations like secrets management.
Would love to use Gateway! Every time I spin up a new cluster it goes like this:
- New cluster setup, time to use gateway! Yay!
- Oh crap, like 80% of the helm chart and other existing configurations I need for the softwares I'm trying to deploy STILL doesn't use gateway, this new API that's been out for... like half a decade at least.
- Even core networking things like Istio/Envoy only have limited gateway support compared to ingress
- Sigh. Ingress again.
It's been like this since gateway's inception and every time I check the needle has moved like 2% towards gateway. So I'm looking forward to year 2050 when I can use gateway!
The problem, as CNCF knows, if they pushed Gateway and deprecated ingress the world would revolt due to the amount of work involved to migrate stuff. Therefore, they leave it up to "the people" to do the extra work themselves, who have no incentive to do so since for many usecases it's not materially better.
I use Kubernetes every day, and have worked with dozens of helm charts, and have yet to encounter cloud specific helm charts. Are these internal helm charts for your company?
Obviously you can lock yourself in if you choose, but I have yet to see third party tools that assume a specific provider (unless you are using tools created BY that provider).
At my previous spot, we were running dozens of clusters, with some on prem and some in the cloud. It was easy to move workloads between the two, the only issue was ACLs, but that was our own choice.
I know they are pushing the new gateway api, but ingresses still work just fine.
Tell me you haven't managed Kubernetes at scale without telling me you haven't managed Kubernetes at scale.
Helm charts may not support a cloud platform like Rancher, Azure, etc. or may have platform specific issues. First one I checked: https://docs.jfrog.com/installation/docs/helm-chart-requirem...
"When deploying a JFrog application on an AWS EKS cluster, the AWS EBS CSI Driver is required for dynamic volume provisioning. However, this driver is not included in the JFrog Helm Charts."
"JFrog validates compatibility with core Kubernetes distributions. Some Kubernetes vendors apply additional logic or hardening (for example, Rancher), so JFrog Platform deployment on those vendor-specific distributions might not be fully supported."
I'm a Kubernetes user and advocate but to call it "portable" just tells me you've never actually tried to deploy the similar thing on multiple different clouds. Even the standardized kubernetes resources behave differently due to various cloud idiosyncracies. You can of course make the situation easier, but to call it entirely portable is probably a misnomer.
> And there's zero setup. Just a deployment yaml that specifies exactly what you want deployed
Writing this yaml is hours and hours of setup if you can't ctrl+c/v from your last project
AI is especially good at writing IaC. Most small projects I let it write a Dockerfile too.
> Suddenly you have to become Sysadmin/SRE
I don't you made that argument but could a valid conclusion of your comment be that, because Kubernetes is so ubiquitous, using it frees you from being a Sysadmin/SRE?
Frees you from being a sysadmin, but burndens you with being a k8s operator, still an SRE.
If you can solve the same problem in a simpler way without using k8s, that means k8s is not a zero cost abstraction.
It's not obligated to be, but it's also obvious why people would want it to be.
> If you can solve the same problem in a simpler way without using k8s
I think I disagree with this, or at least the implication. I think it is true you can solve EACH OF THISE PROBLEMS INDIVIDUALLY in a simpler way than Kubernetes, the fact that you are going to have to solve at least 5-10 of those problems individually makes the sum total more complicated than Kubernetes, not to mention bespoke. The Kubernetes solutions are all designed to work together, and when they fail to work together, you are more likely to find answers when you search for it because everyone is using the same thing.
I think it is fair to say k8s is not a zero cost abstraction, but nothing you use instead is going to be, either, and when you do run into a situation where that abstraction breaks, it will be easier to find a solution for kubernetes than it will for the random 5 solutions you pieced together yourself.
Ephemeral user accounts were agreed upon before that. The OG container
Docker and k8s are just wrappers around namespaces, cgroups, file system ACLs, some essential cli commands, which can also be configured per user.
We may be headed back there. Have seen some experiments leveraging Linux kernels BPF and sched_ext to fire off just the right sized compute schedule in response to sequences of specific BPF events.
Future "containers" may just be kernel processes and threads... again. Especially if enough human agency looks away from software as AI makes employment for enough people untenable. Why would those who remain want to manage kernels and k8s complexity?
Imo its less we agreed on k8s specifically and more we agreed to let people use all the free money to develop whatever was believed to make the job easier; but if the jobs go away then it's just more work for the few left
> Docker and k8s are just wrappers around namespaces, cgroups, file system ACLs, some essential cli commands, which can also be configured per user.
Docker, yes, but kubernetes is way more than that the instant you have more than one physical machine node. (If you only have one node in any deploy, sure, it's likely overkill, but that seems like a weird enough case to not be worth too much ink.)
If you silently replaced all my container images with VM images and nodes running containers with nodes running VMs, I think the vast majority of all my Kubernetes setup would be essentially unchanged. Heck, replace it all with people with hands on keyboard in a datacenter running around frantically bringing up new physical servers, slapping hard drives in them, and re-configuring the network, and I don't think the user POV of how to describe it would change that much.
> nodes running VMs,
huh, but how would bursting work then? Do VMs support it nowadays?
I've seen some places advertise it but I have not tried it.
But, honestly, more generally in my head I wasn't thinking much about it since I consider that as a "cost optimization" thing than a "core kubernetes function." E.g. the addition (or not) of limits is just a couple lines, compared to all the rest of the stuff that I'd be managing specification of (replicas, environment, resource baseline, scheduling constraints, deployment mode...) that would translate seamlessly.
(And there are a lot of parts of kubernetes that annoy me, especially around the hoops it puts up to customize certain things if you reaalllly actually need to, but it would never cross my mind in a hundred years to characterize it as just a wrapper around cgroups etc like the OP.)
Something often underappreciated is that, in the possible future you're describing, you can use all of these new fangled "what's old is new again" approaches by continuing to just use Kubernetes. Kubernetes is, in a way, designed to replace itself.
Kubernetes is software. It cannot do anything "itself" let alone "replace itself". Don't anthropomorphize software
Inevitably it will be a human replacing it with what they define is the best method
https://x.com/livingdevops/status/2034957580750266632
"Kubernetes is beautiful. Every Concept Has a Story, you just don't know it yet... So you use a Deployment... So you use a Service... So you use Ingress... So..."
(Full text at the link)
lol, the big problem with kubernetes is that none of the choices have been made, it's not opinionated at all, there's no conventions. It's all configuration and choices all the way down. There's way too much yaml, and way to many choices for ever tiny component, it's just too much.
I do run a k3s cluster for home stuff... But I really wish I could get what it provides in a much simpler solution.
My dream solution would effectively do the same as k3s + storage, but with a much simpler config, zero yaml, zero choices for components, very limited configuration options, it should just do the right ting by default. Storage (both volume and s3), networking, scale to zero, functions, jobs, ingress, etc... should all just be built in.
You're going to have to write some sort of config. It not being opinionated it a good thing. It lets you deploy just about anything under the sun.
Well... we have k8s for that... I do not wish to take k8s away from those who like it, I am asking for a new solution that's very opinionated, and as close to zero config as practical.
In what cases do you need autoscaling on your home stuff?
I have limited ram and want scale to zero for apps that use a lot of ram, but I only use one of at a time like game servers, or things that can be done over night while I sleep like media encoding.
The main reason I went to k8s, is for the not having to think about what machine will have enough resources to run an app, just throw it at the cluster and it figures out where there's capacity. And, I want hardware failing/getting replaced to be a non issue.
edit: I wanted to add that my hobby is not systems admin, I want it to be as hands off as possible. Self-hosting is a means to an end. I have so far saved over $200/month in subscriptions by replacing subscriptions I was using with self-hosted alternatives. I can now use that money on my actual hobbies.
My take is that k8s the idea is okish the implementation on the other hand not that much.
This comes down to that fact that software architecture is not great in general and very few people care about simplicity.
Yeah, I spent quite a bit of time learning Kubernetes, but now I'd use it to host a static webpage on a single server, over alternatives. It's so awesome.
The question is, how do we outsiders differentiate Stockholm syndrome from something truly being awesome?
This is truly interesting to me. Why?
I am not the person you asked this question to, but I would probably do the same so I will answer:
Once you get used to it, it just makes managing things simple if you always use it for everything. I have a personal harbor service that I run on my local cluster that has all my helm charts and images, and i can run a single script that sets up my one node cluster, then run a helm install that installs cert-manager and my external-dns, and now I can deploy my app with whatever subdomain I want and I immediately get DNS set up and certs automatically provisioned and rotated. It will just work.
1. Assuming managed service, it frees me from host OS management. So basically the same proposition, as good old "PHP+MySQL" hosters. You upload your website, they make sure it works. But without limitations and with much better independence.
2. It allows me to configure everything using standard manifests. I need to provision the cluster itself initially, then everything could be done with gitops of various automation levels. I don't need to upload my pages via FTP. My CI will build OCI image, publish it to some registry, then I'll change image tag of my deployment and it'll be updated.
3. It allows to start simple, and extend seamlessly in the future. I can add new services. I can add new servers. I can add new replicas of existing services. I can add centralized logging, metrics, alerts. It'll get more complicated but I can manage the complexity and stop where I feel comfotable.
4. One big thing that's solved even with the simplest Kubernetes deployment is new version deployment with zero downtime. When I'll update image tag of my deployment, by default kubernetes will start new pod, will wait for it to answer to liveness checks, then redirect traffic to new pod, let old pod to gracefully stop and then remove it. With every alternative technology, configuring the same requires quite a bit of friction. Which naturally restricts you to deploy new versions only at blessed times. With Kubernetes, I started to trust it enough, I don't care about deployment time, I can deploy new version of heavily loaded service in the middle of the day and nobody notices.
5. There are various "add-ons" to Kubernetes which solve typical issues. For example Ingress Controller allows the developer to describe Ingress of the application. It's a set of declarative HTTP routes which will be visible outside and which will be reverse-proxied to the service inside. Simplest route is https://www.example.com/ -> http://exampleservice:8080, but there's a lot more to it, basically you can think about it as nginx config done differently. Another example is certificate manager, you install it, you configure it once to work with letsencrypt and you forget about TLS, it just works. Another example is various database controllers, for example cloudnativepg allows you to declaratively describe postgres. Controller will create pod for database, will initialize it, will create second pod, will configure it as replica, will perform continuous backup to S3, will monitor its availability and switch master to replica if necessary, will handle database upgrades. A lot of moving parts (which might be scary, tbh), all driven by a simple declarative configuration. Another example is monitoring solutions, which allow to install prometheus instance and configure it to capture all metrics from everything in cluster along with some useful charts in grafana, all with very little configuration.
6. There are various "packages" for Kubernetes which essentially package some useful software, usually in a helm charts. You can think about `apt-get` but for a more complicated set of services, mostly pre-configured and typically useful for web applications. The examples above are all installable with helm, but they add new kubernetes manifest types, which is why I called them "add-ons", but there are also simpler applications.
Just for the record, I don't suggest that to everyone. I spent quite a bit of time tinkering with Kubernetes. It definitely brings a lot of gotchas for a new user and it also requires quite a bit of self-restrictions for experienced users to not implement every devops good practice in the world. Sometimes maybe you don't even want to start with ingress, I saw cluster which used manually configured nginx reverse proxy instead and it worked for them. You can be very simple with Kubernetes.
Honestly the main problem is people using k8s for something that's like... a database, and an app, and maybe a second app, that all could be containers or just a systemd service.
And then they hit all the things that make sense in big company with like 40 services but very little in their context and complain that complex thing designed for complex interactions isn't simple
But if you want some redundancy, k8s let's you just say run 4 of this, 6 of this on these 3 machines. At least I find it quite straight forward.
The database is more complex since there is storage affinity (I use cockroachDB with local persistent volumes for it) - but stateful is always complicated.
Most of the time you don't need redundancy. You need regular backups for exceptional circumstances. And k8s gives you more complexity, and more problems through more moving parts, to give you the possibility of using a feature you'll never need, and if you do start to use it it'll probably be instead of fixing performance problems downstream
Are we talking for personal projects where there are no expectations, or small startups where you don’t have much scale but you still care about down time and data loss?
Personal projects are one thing, but even the smallest startup wants to be able to avoid data loss and downtime. If you are running everything on one server, how do you do kernel patches? You need to be able to move your workload to another server to reboot for that, even if you don’t want redundancy. Kubernetes does this for you. Bring in another node, drain one (which will start up new instances on the new node and shift traffic before bringing down the other instance, all automatically for you out of the box), and then reboot the old one.
Again, you could do all of this with other tech, but it is just standard with Kubernetes.
> but even the smallest startup wants to be able to avoid data loss
Seems true at a glance!
> and downtime.
Maybe less so - I think there’s plenty out there, where they’re not chasing nines and care more about building software instead of some HA setup. Probably solve that issue when you have enough customers to actually justify the engineering time. A few minutes of downtime every now and then isn’t the end of the world if it buys you operational simplicity.
Agreed. Upgrading just one piece, and ensuring every committed write survives is critical in most commercial applications.
Luckily since I met this guy named Claude most of that complexity has gone away.
A while back when the agents got hyped I was looking into the whole "give it a VM / docker container" I realized the safest and simplest option was just to give it its own machine.
Then I realized giving it root on a $3 VPS is functionally equivalent. If it blows it up, you just reset the VM.
It sounds bad but I can't see an actual difference.
> This is VERY powerful
No argument there. The Toyota 5S-FE non-interference engine is a near indestructible 4 cylinder engine that's well documented, popular and you can purchase parts for pennies. It has powered 10 models of Camrys and Lexus and battle proven. You can expect any mechanic who has been a professional mechanic for the last 3 years know exactly what to do when it starts acting up. 1 out of 4 cars on the road have this engine or a close clone of it.
It's not what any reasonable person would use for a weedwhacker, lawnmower, pool pump or an air compressor.
Sure, but to extend your metaphor, Kubernetes HAS smaller engine models that you can use in those situations, and still gain all the benefits of being in the same ecosystem. You can use K3s, for example, and get all the benefits without having a giant engine in your weedwhacker.
IMO, Kubernetes isn't inevitable, and this seems to paint it as such.
K8s is well suited to dynamically scaling a SaaS product delivered over the web. When you get outside this scenario - for example, on-prem or single node "clusters" that are running K8s just for API compatibility, it seems like either overkill or a bad choice. Even when cloud deployed, K8s mostly functions as a batteries-not-included wrapper around the underlying cloud provider services and APIs.
There are also folks who understand the innards of K8s very well that have legitimate criticisms of it - for example, this one from the MetalLB developer: https://blog.dave.tf/post/new-kubernetes/
Before you deploy something, actually understand what the pros/cons are, and what problem it was made to solve, and if your problem isn't at least mostly a match, keep looking.
> K8s is well suited to dynamically scaling a SaaS product delivered over the web
It’s well suited to other things as well, people are just in denial about some of them.
“I need to run more than two containers and have a googleable way to manage their behavior” is a very common need.
This is a need it fails at miserably. k8s reminds me of the raid recentralization anti pattern problem where you fix a hardware failure that never occurs in exchange for knowing simple higher level mistakes or security problems will tank something now too large to fail again.
Kubernetes, in the form of k3s, was a critical success factor for us with the onprem deployment of our SaaS product.
What's the problem with a single-node cluster? We use that for e.g. dev environments, as well as some small onprem deployments.
> Even when cloud deployed, K8s mostly functions as a batteries-not-included wrapper around the underlying cloud provider services and APIs.
Which batteries are not included? The "wrapper around the underlying cloud provider services and APIs" is enormously important. Why would you prefer to use a less well-designed, more vendor-specific set of APIs?
I seriously don't get these criticisms of k8s. K8s abstracts away, and standardizes, an enormous amount of system complexity. The people who object to it just don't have the requirements where it starts making sense, that's all.
> Kubernetes, in the form of k3s, was a critical success factor for us with the onprem deployment of our SaaS product.
What surprises and gotchas did you have to deal with using k3s as a Kubernetes implementation?
Did you use an LB? Which one? I'm assuming all your onprem nodes were just linux servers with very basic equipment (the fanciest networking equipment you used were 10GbE PCIe cards, nothing more special than that?)
We sell to enterprise customers. All of them deploy our solution on internal cloud-style VM clusters. We use the Traefik ingress controller by default.
There really weren't any particular surprises or gotchas at that level.
In this context, I've never had to deal with anything at the level of the type of Ethernet card. That's kind of the point: platforms like k8s abstract away from that.
It's also difficult for data pipelines or data intensive things. At several companies we've run into the "Need to put ML model behind API and pods get killed because health checks via API are basically not compatible with container fully under load but still working"
The saddest part about Kubernetes is… after you set it all up, you still need a hacky deploy.sh to sed in the image tag to deploy! And pretty soon you’re back to “my dear friend you have built a Helm”. And so the configuration clock continues ticking…
Claude Code has essentially fixed this perpetual annoyance for me. Doesn't matter if it's a hacked up deploy.sh that mixes sed, envsubst and god knows what or a non-idiomatic Helm chart that was perpetually on my backlog to fix... today I just say "make this do this thing and also fix any bash bugs along the way" and it just does it. Its effectiveness for these thousand-little-cuts type DevOps tasks is underrated IMO.
Now the actual CI/CD/thing-doers tools that all suck... I'm still stuck with those.
I agree, I'm not great at devops, but my setup.sh and deploy.py have been game changers. Just vibe coding those was good enough.
Same with build.sh and doing it in such a way that I can use all the build.sh in my ci.yml for Github Actions.
I have been using Kubernetes for 7 or 8 years now, and have nearly 100% stayed away from Helm.
Some Kustomize, a little bit of envsubst and we're good to go thank you very much.
This is why we don't take advice from randos on internet.
I manage 100+ variations on a single helm chart and 50+ such helm charts at work daily for 7 years across 11 datacenters/kubernetes clusters. And I have team members who swear by kustomize. The number of kustomize typo errors and issues that I have deal with is unimaginable. Whereas if I test and deploy a helm chart, I know it will work everywhere in every variation.
Kustomize is just plain terrible and backwards as a solution. It doesn't scale, it is half assed. It tries to basically require you to build your own compiler and parser and transform. With kustomize + envsubst: dear friend, you have built helm.
How do you handle cleanups and hooks? The best way to do helm, at least for me, seems to be about limiting its use to simple templating use cases; if you end up needing an if, you've probably done something terribly wrong.
That's my main gripe with Helm.
For the simple use case you're describing, Helm is not required. Plenty of other solutions around.
For use cases where it starts getting useful, we both agree that something has gone terribly wrong.
I still don't know why Helm exists. It's a solution that created lots of problems that didn't exist.
My personal theory is that Helm may be ok for distributing a pre-packaged solution to other people. Then people mistook it for a tool that should be used in-house to deploy a company’s own systems, where it makes much less sense.
It makes absolute sense. You can use no variables and still deploy helm chart. It is a directory of plain old yaml objects. And add customization when you need as you evolve. Good luck doing that with kustomize.
> And add customization when you need as you evolve.
Using one of the most horrible templating languages since ASP. Helm is what happens when a devops team decides to yolo into software development.
What's the issue with kustomize? It works well for us.
You can rely purely on kubectl with something like:
cat manifests.yaml | kubectl apply -f - --server-side --field-manager "$FIELD_MANAGER" --prune --applyset "$APPLYSET" --namespace "$NAMESPACE"
Seems to be a case of the XY problem. What do you need cleanups and hooks for?
Cleanups: I want to do a `helm uninstall` and have all the manifests go away at once instead of looking around for N different resources.
Hooks: I want to apply my database migrations and populate the database with static datasets before I deploy my application, without having my CI connect to the database cluster (at places I've worked, the CI cluster and K8s cluster were completely separate).
Regarding cleanups: I'm using flux CD with kustomize. It tracks resources that it created. If I delete manifest from my repository, flux will delete resources that were created from these manifests. For me that's pretty much the ideal workflow.
Regarding hooks: I don't know. All applications that I've used, implemented migrations internally (it's usually Java with Flyway), so I don't need to think about it. One possible approach could be to use flux CD with Job definition. I think that Flux will re-create Job when it changes. So if you change image tag, it'll re-create Job and it'll trigger Pod execution. But I didn't try this approach, so not sure if that would work for you.
> Cleanups: I want to do a `helm uninstall` and have all the manifests go away at once instead of looking around for N different resources.
> I want to apply my database migrations and populate the database with static datasets before I deploy my application, without having my CI connect to the database clusterA Job feels like a good fit for this. CI deployes the Job without connecting to DB, Job runs migrations using the same connectivity as the application.
> apply my database migrations and populate the database with static datasets before I deploy my application
You could a) have the app acquire a lock in the db and do its own migrations, or b) create a k8s job that runs the migration tool, but make sure the app waits for the schema to be updated or at least won't do anything bad.
There are a multitude of cases of operations which need to be performed before and after specific actions in K8s. It depends on the resource, operator, operational changes, state, bugs, order of operations, and more.
This is a BS claim with no proof. This is the strength of helm.
Going on 10 years now for me, tried Helm a bit and yep - all I've really needed was a package.json deploy script with sed to bump the image version.
Or if your colleagues are "smarter" than you they make it in Clojoure instead, with an EDN-but-with-subroutines config language, so that not only yaml-aware editors are useless, but EDN-aware editors cannot make heads or tails of the macros.
Fun times.
I don't understand you.
For very simple deployments, you don't need anything at all. Just write manifests and use `kubectl apply`. You can write `deploy.sh` but it'll be trivial.
If you want templating, there are many options. You can use `sed` for the most simple templating needs. You can use `cpp`, `m4`, `helm` or `kustomize`. I, personally, like `kustomize`, but `helm` probably not the worst template engine out there.
Kustomize is even somewhat included into basic kubernetes tooling, so if you want something "opinionated", it is there for you. It works.
https://fluxcd.io/ + helm + with a CI pipeline that pushes the docker images to a registry means that after the setup, anytime you push a new image and tag, k8s can automatically update without needing to do anything manual.
Anyone remembers the GitOps thingy called flux? Weave was the company name.
Git and Kubetnetes configuration cannot go hand in hand. You cannot go back in past indefinitely because cluster state might not be that reversible. If so, git is useless.
And no, doesn't apply for database migrations. You can mostly run migrations backwards if each migration was written carefully.
I'm starting to use it for my self hosted services.
I have a "simple" representation of services using CUE, that generates the yaml manifests and flux deploys them.
I hesitated a while before going the k8s route but before that I had a overly error-prone Ansible configuration and I got sick of manual templating (hence the move to CUE for type safety).
There's also the fact that I wanted my services to be as plug and play as possible, so for example automatically generated openid credentials and very easily configurable central SSO, along with the easily configurable reverse-proxy.
If anyone thinks that k8s is not the best tool for this, I'm always interested in advice.
(Also a lot of complexity in my setup is due to self hosting, I have Istio, MetalLB, proxmox CSI, and all other kind of stuff that your cloud provider would already have, and these are the things that take most of the configuration files in my repo)
I've used it in the past and personally loved it. Just bumping a yaml file in a git repo to the image tag I wanted deploying was a godsend and nearly automated. I can't speak to your experience though which I am certain is valid and a real problem. We just never had those kind of issues so we could either revert to an earlier tag that worked or publish a new image with the required resolution steps.
And if you want your Helm to run on certain deploys, and maintain a declarative set of the variables given to charts over time, thinking you can use Helmfile and some custom GitHub Actions… “my dear friend you have built a GitOps.”
(I tend to think this one is acceptable in the beginning, but certainly doesn’t scale.)
Use a CD solution like Spinnaker, BunnyShell, or Kargo.
skaffold. It'll also wait for rollout to stabilize.
If few lines of scripting is your problem you shouldn't be programming
They have built an orchestrator, not Kubernetes. There is one key difference: they know this thing, end-to-end, down to every single bolt and piece of duct tape (with possible exception for Docker internals)
And that's a very important distinction when it comes to maintaining complex systems. This could've changed with LLMs (I'm still adjusting to what new capabilities mean for various decision-making logic), but before machine intelligence debugging an issue with Kubernetes could've been a whole world of pain.
And chances are only they know it. If my role has enough cluster access, I can muddle through pretty much any helm chart (with lots of cursing, yes) but it might take me days to set up whatever elaborate bespoke environment and script invocations are needed to replicate the current production setup maybe.
As someone rolling their self-hosted stuff via Compose and shell scripts instead of K8s specifically for the simplicity of the experience, this is 100% why you need to understand what Kubernetes solves before writing it off entirely.
I'm not doing overlay networks, I'm using a single bare-metal host, and I value the hands-on Linux administration experience versus the K8s cluster admin experience. All of these are reasons I specifically chose not to use Kubernetes.
The second I want HA, or want to shift from local VLANs to multi-cloud overlays, or I don't need the local Linux sysadmin experience anymore? Yeah, it's K8s at the top of the list. Until then, my solution works for exactly what I need.
PREACH!
I run K8s at home. I used to do docker-compose - and I'd still recommend that to most people - but even for my 1 little NUC with 4vcpu / 16Gi Homelab, I still love deploying with K8s. It's genuinely simpler for me.
If anyone's looking for inspiration, my setup:
* ArgoCD pointed to my GitLab repos
* GitLab repos contain Helm charts
* Most of the Helm charts contain open-source charts as subcharts, with versions set like (e.g.) `version: ~0` - meaning I automatically receive updates for all major version until `1`
* Updating my apps usually consists of logging into the UI, reviewing the infrastructure and image tag updates, and manually clicking sync. I do this once every few months
My next little side project: Autoscaling into the cloud (via a secure WireGuard tunnel) when I want to expand past my current hardware limitations
A reason not to run k8s is if you want your server to reach C10 idle states. The k8s control plain with its polling and checking are quite heavy on the mostly idle server. I have reverted to just use Nixos and oci podman containers. Everything is declarative and reproducible
another one is swap. UnlimitedSwap was deprecated and you can now only use LimitedSwap which restricts how much swap you can use, so you can't take full advantage of zram, which sucks for those looking to run lean
I can tell you how vendors deliver a software solution that runs on Kubernetes: very poorly.
The needed tweaks, the ability to customize things, basically goes to zero because the support staff is technical about the software, but NOT about Kubernetes.
I am not joking: a recent deployment required 3x VMs for Kubernetes, each VM having 256 gigabytes of RAM; then a separate 3x VMs for a different piece. 1.5TB of RAM to manage less than 1200 network devices (routers etc. that run BGP).
No one knew, for instance, how to lower the MongoDB (because of course you need it!) resource usage, despite the fact that the clustered VMware install is using a very fast SSD storage solution and thus MongoDB is unlikely accelerate anything; so over 128GB RAM is being burned on caching the results coming back from SSDs that are running at many-GB/s throughput.
Whether this is deployed via Helm charts or a native controller, there's almost certainly some overlay where you can override resource values, unless this is just a very crappy vendor.
The setting is there, somewhere, I agree; but no one can tell you how to change it or what the implications might be.
So unless I want to dig into their YAML files that are not documented…
I somehow feel that it's actually the opposite. It should be "Dear kubernetes user, you have just built a shell".
Kubernetes is a powerful tool for complicated problems. If if it seems complicated, you probably don’t have a complicated deployment problem.
But really this applies to any powerful tool. If you need to measure a voltage, an 4 channel oscilloscope also probably seems too complicated.
Discussed at the time:
Dear friend, you have built a Kubernetes - https://news.ycombinator.com/item?id=42226005 - Nov 2024 (277 comments)
Found this the same day I published this: https://github.com/oddur/yoink
Kubernetes was overkill (I do that all day, 5 days a week); Kamal was too restrictive, so I found myself rolling out Yoink. Just what I need from k8s, but simple enough I can point it to a baremetal machine on Hertzner that can easily run all my workloads.
> found myself rolling out Yoink
- using Tailscale SSH is brilliant
- using caddy-docker-proxy for ingress is brilliant
What do you use for:
- service discovery
- secret store (EDIT: Crap you use Infisical. No shade, I just have this horrible foreboding it will end up like Hashicorp. I use Conjur Secretless Broker but am tracking: https://news.ycombinator.com/item?id=47903690)
- backing up and restoring state like in a DB
PS: Have you been having issues with Hetzner the last few weeks?
Service discovery is basically just Docker's internal DNS. Caddy-docker-proxy can use it to find healthy upstreams.
For secrets, I self-host Infisical on the box -- easy to plug in whatever secret manager, should make it pair nicely with https://github.com/tellerops/teller or something similar
Had no problems with Hertzner so far, just enjoying the raw CPU power of bare metal. The plan is to roll out more boxes across different providers, using Tailscale for the backplane network and Cloudflare to load-balance between them. All in due time What issues have you been having ?
I have a suspicion you're using Headscale? If so, I urge you to consider Ionscale. I use it with Authentik as the IdP.
Personally commiting to using Tailscale as a core foundation of my infrastructure and Ionscale is my hedge against getting Hashicorped.
> Service discovery is basically just Docker's internal DNS. Caddy-docker-proxy can use it to find healthy upstreams
Do you have a writeup of this somewhere? I'm unaware of being able to manage Docker's internal DNS over some kind of an API (would appreciate if you know a way to). The only way I know is to manipulate network aliases via Docker Engine API. As a result I use Hickory DNS with RFC 2136. That coupled with Caddy-docker-proxy gets me extremely close.
All it would take to make this post actually good would be to replace "Kubernetes" with "orchestrator"; that would also keep the symmetry with the post it's riffing on, about building compilers (it's not "Dear friend you have built a GHC").
Criticisms of Kubernetes generally come from a few places:
- People who would prefer their way of doing this, whether that's deployments on VMs, or use some sort of simpler cloud provider.
I had the same opinion a few years ago, but have kind of come to like it, because I can cleanly deploy multiple applications on a cluster in a declarative fashion. I still don't buy the "everything on K8s", and my personal setup is to have a set of VMs bought from a infrastructure provider, setup a primary/replica database on two of them, and use the rest as Kubernetes nodes.
- People who run Kubernetes at larger scales and have had issues with them.
This usually needs some custom scaling work; the best way to work around this if you're managing your own infra[1] is to split the cluster into many small independent clusters, akin to "cellular deployments"[2]/"bulkhead pattern"[3]. Alternatively, if you are at the point where you have a 500+ node cluster, it may not be a bad idea to start using a hyperscaler's service as they have typically done some of the scaling work for you, typically in form of replacing etcd and the RPC layer through something more stable.
- People who need a deep level of orchestration
Examples of such use cases may be to run a CI system or a container service like fly.io; for such use cases, I agree that K8s is often overkill, as you need to keep the two datastores in sync and generate huge loads on the kube-apiserver and the cluster datastore in the process, and it might be often better to just bring up Firecracker MicroVMs or similar yourself.
Although, I should say that teams writing their first orchestration process almost always run to Kubernetes without realizing this pitfall, though I have learned to keep my mouth shut as I started a small religious war recently at my current workplace by raising this exact point.
[1] Notice how I don't say "on-prem", because the hyperscaler marketing teams would rather have you believe in two extremes of either using their service or running around in a datacenter with racks, whereas you can often get bog-standard VMs from Hetzner or Vultr or DigitalOcean and build around that.
[2] https://docs.aws.amazon.com/wellarchitected/latest/reducing-...
[3] https://learn.microsoft.com/en-us/azure/architecture/pattern...
Another case: People who want to run workloads that are inherently incompatible with Kubernetes networking model.
For example:
* For some cursed reasons you want to make sure every single one instance of a large batch job see just one NIC in its container and they are all the same IP and you NAT to the outside world. Ingress? What ingress? This is a batch job!
* Like the previous point, except that your "batch job" somehow has multiple containers in one instance now, and they should be able to reach each other by domain.
That is indeed a weirdly cursed requirement. Why? Black box of legacy stuff? A system that was never designed to be run in multiple does so if all the nodes think they’re the same machine? Defeating a license restriction?
You can configure k8s so pod to pod networking works just fine so I'm not even sure what complaint here is
Unless you’re in Erlang world (Elixir, Gleam..) and all that is already baked into OTP and the BEAM. You can go on holiday knowing it will be a while longer before you need to break out the pods (and at that scale, you will be able to afford a colleague or two to help you).
How does Elixir renew my certificates, mount networked storage mounts and create a VIP and internal DNS entries for my valkey instance?
Scaling is a sidenote, that it becomes easy is a result of hoisting everything else onto one control plane and a set of coherent APIs.
I've experienced something like this at work but with data warehouse instead, and it happened multiple times (to be fair, data engineering is still fairly new where I'm from).
One example was an engineer wanted to build an API that accepts large CSV (GBs of credit reports) to extract some data and perform some aggregations. He was in the process of discussing with SREs on the best way to process the huge CSV file without using k8s stateful set, and the solution he was about to build was basically writing to S3 and having a worker asynchronously load and process the CSV in chunks, then finally writing the aggregation to db.
I stepped in and told him he was about to build a data warehouse. :P
If it was less than 100 gb, he probably should have just loaded the whole thing in RAM on a single machine, and processed it all in a single shot. No S3, no network round trips, no chunking, no data warehouse.
So much defense of Kubernetes as the only solution to problems like this. Nomad and Incus exist as solid and less complicated alternatives. They also have a larger scaling ceiling.
Docker swarm is good enough for most cases if the black plague called Microservices isn't in your stack.
Basecamp is running successfully with Kamal and surely making more money than all the Kubernetes experts combined on this platform.
Small companies should not behave and pretend like big enterprises. Maybe Kubernetes is justified in those scenarios.
But for 95% of the population, it's an overkill and a constantly moving target.
I need clarifications.
I see docker as a way to avoid having a standard dev platform for everyone in the company so that the infra team don't have to worry about patch xyz for library abc, only run docker.
But, with all the effort put in place to coordinate docker, k8s and all the shebang, isn't it finally easier to force a platform and let it slowly evolve over time?
Is docker another technical tool that tries to solve a non-technical problem?
Docker is a solution to one specific problem: the need for "a user" to run 10 different potentially conflicting apps, all at the same time, on one machine, and abstract away anything which might make those apps conflict if they ran on a single OS. It provides a dozen different solutions in one package.
K8s is a way to take that and make it scale up for a large number of applications on a large number of hosts in a production business in a way that's automated and resilient to failure.
I do not follow you. Every app has different needs. Containers encode them in a shareable way. You can evolve the image over time. So what more do you want?
Porque no los dos. Force a platform. Deploy non containerized. Build a dockerized version of the forced platform for cross-platform local dev.
Why both posts mention docker compose and not mentioning docker swarm. Being using it for my projects for long time. And it's so nice. Similar syntax, easy networking, rollout strategy, easy to add nodes to cluster.
You can have one template docker-compose.yaml file and separate deployment files for different envs, like: docker-compose.dev.yaml, docker-compose.prod.yaml
I think swarm is really underrated
It’s in a death spiral - not widely used so people aren’t incentivised to put time into it so not widely used
How do you solve persistence with swarm? Can I deploy postgres with network storage that will mount automatically on node where container is launched?
> Can I deploy postgres with network storage that will mount automatically on node where container is launched?
Yes (if I'm getting your question right). here is an example of nfs volume: https://docs.docker.com/reference/compose-file/volumes/#driv...
I've been there. We still ended up with messy deploy scripts written in Ruby and the only debugging solution was "just comment out everything then run line by line".
'docker stack deploy' covers most of the cases. But yeah, there is still some problems like: "update a config or a secret", that require manually invoking additional commands (or via scripts)
Some days, it would be better to build a working not-kubernetes than debugging the not-working kubernetes.
Literally just finished building a personal orchestrator system I wanted and had this very much in back of my mind.
Ended up doing a mix. Built on compose for now but in a manner that’ll lift and shift to k8s easily enough. Its containers talking over network either way
After reading this and remembering an old hobby project, I decided to switch the deploy from a systemd service to PM2, which apparently has rolling deployments without needing Docker engine (for those of us minmaxing instance RAM).
Partially true, I would argue. Try running kubernetes on an on-prem infra and you'll soon realize that it was much easier to run the apps on VMs themselves.
I'll just say the quiet part out loud: A pile of shell scripts that no one else understands is job security.
If your work is easily googleable/parseable by an AI, why would anyone pay you?
I worked with a guy who fits this description. If there’s something he can reinvent poorly, he’d do it. A big part of what I did while working on the same team as him was slowly chipping away at his weird domain by introducing standard tools. I always had to advocate for them in such a way that they were solving a different problem, and then slowly work towards a point where some of his homegrown stuff became unnecessary.
As to the job security idea: the only people who do this are people who aren’t good at creating real value, so they have to try to create niches where they’re needed.
k3s (lightweight k8s) is basically almost as easy as docker compose and then you have access to the full open source CNCF ecosystem in form of helm charts.
im just about giving OPs premise another go. compose just feels so much better as abstraction especially with small and medium setups looking close to the optimum of expressiveness without boilerplate to describe what is needed. The missing pieces seem to also be in the compose compatible “docker stack” aka new docker swarm, which i ignored for probably too long as i assumed it was the discontinued old swarm. Even if new swarm mode sucks how hard can it be to make something compose shaped vs running k8?
See also Greenspuns tenth law:
> Any sufficiently complicated C or Fortran program contains an ad-hoc, informally-specified, bug-ridden, slow implementation of half of CommonLisp.
https://wiki.c2.com/?GreenspunsTenthRuleOfProgramming
> Ah, but wait! Inevitably, you find a reason to expand to a second server
>> The real problem is that programmers have spent far too much time worrying about efficiency in the wrong places and at the wrong times; premature optimization is the root of all evil (or at least most of it) in programming.
-- Donald Knuth, Computer Programming as an Art (1974)
EDIT:
> Except if you quit or go on vacation, who will maintain this custom pile of shell scripts?
Honestly? I don't care. There is a reason why I quit and 99% of time it's the pay. And if the company doesn't pay me enough to bother then why should I? Why should I bother about some company future in the first place?
The rule is to build everything such that there is either 0, 1, or infinite of anything
Kubernetes networking is an inefficient mess.
Some CNIs are definitely better than others. Unfortunately it seems 99% of people want to work against the k8s networking model.
Shit just gets really weird when your network isn’t split for k8s in an equivalent way to what GCP/AWS expect. Like, if you have other services running on the nodes that you want things inside k8s to talk to, or if the nodes are in a flat subnet with other stuff in it, things get annoying. Those are worst practices for a reason, but pretty common in environments with home rolled k8s clusters.
If you manage to build kubernetes again I recommend you burn it down and start again.
> I know you wanted to "choose boring tech" to just run some containers.
The people advocating for boring tech generally aren't interested in containers.
You can just run programs.
Legacy apps are far nicer if they are containerized.
If your app is just a blob that can be run it is fine, but many languages make it more complicate.
I wonder if just putting app into .appimage + using systemd for some of the separation would be a sweet spot ?
Containers are just statically-linked programs for the rest of us.
I am a big fan which is why I am saying this: you're dismissing the kernel and ABI surface is a huge assumption that must hold true for your comment to hold stavros.
If you had said "unikernels" I would have had no arguments to make.
What do you mean? Statically linked programs depend on the kernel too.
right - that's precisely what I meant. I read your comment "Containers are just statically-linked programs for the rest of us." as "containers can be replaced by statically-linked programs".
If you didn't imply that, I apologize.
If you did mean that, I disagree with you precisely because your point works if you only care about dependency management - it falls apart on system state. A static binary is a process on the host and shares the same process space, network stack, and filesystem.
OTOH, a container is a jail (the primary usecase): I can't cgroup a static binary's memory usage or give it a virtual network interface without reimplementing "container lite". Containers aren't just 'statically linked programs' - they allow me to use the kernel as a hypervisor for isolated environments.
What they are though, a messy but practical compromise to Unikernels - which was my last point in our GP.
Oh, no, I meant that containers serve the same purpose as statically linked programs for languages that can't do that. Eg if you want to deploy a Python codebase, a container is a good way to include all dependencies.
I didn't mean "containers don't have any advantages compared to statically linked programs".
I 100% thought you were yelling a the clouds lol. see it way too often on HN
The problem is that running programs isn't always trivial.
I have no interest in going back to the hell that is deploying a java app.
NOOO you have to use my shitpile of nested yaml with the same dependency sprawl cancer as modern javascript. You can't just upload a binary to your own servers and host it there you need to overthink everything and make an extremely simple process overcomplicated just install one more side car and fifty more dependencies on your helm chart bro and then we can move on to figuring out CSI it should only take like a month to get it working properly I promise!!!!!
Not sure how to describe this, but in a small/medium size company you have a few chips with talent.
And somehow k8s tends to use one of the most valuable chips. The smartest guy who from then on only works with maintaining and updating k8s. Instead of building the best and smartest products.
Kafka also competes with this.
More like "Dear friend, you have built an Application Container", even more so now with the VCs looking for money in powering WebAssembly containers as pods.
"Except if you quit or go on vacation, who will maintain this custom pile of shell scripts?" LLMs can reason about and fix them quite well.
Not as well as they can reason (or others can google) something as standardized as kubernetes. There’s just less context (in both senses of the term) needed to understand something running on a common substrate versus something bespoke, even if the bespoke thing is itself comprised of standardized parts.
For a project set up by a qualified engineer, there would be little difference to the end user in practice. The LLM would work out a solution with a negligible difference in speed. Maybe debugging would also be faster for the LLM without the abstraction layers and low level access?
[dead]