We've run Kubernetes inside Redgate for some research projects (like Spawn) and we are building some skills running this orchestrator. At the same time, we've had no shortage of challenges in keeping the clusters up at times, patching, fixing issues, upgrading to new configurations, etc. Like any software, there is work involved with managing the orchestrator.
I've watched Andrew Pruski and Anthony Nocentino write about containers and Kubernetes and overall they've made me view the clusters like email. It's useful and I want to use it, but I don't want to manage or administer it. I'd want some service like AKS or EKS instead. Let someone else build expertise.
If you use containers, do you have an orchestrator running in your data center? Mercedes does, with over 900 clusters. They found value early on with container technologies and built in-house expertise within their research arm. I think a large organization like Mercedes likely can make this investment pay off, especially as they likely don't depend on any one person to understand and manage Kubernetes. They can afford someone like Andrew or Anthony quitting and taking another position.
The rest of us can't really do that, certainly not without our organization feeling containers and orchestration is a core competency.
The key for Mercedes is automation. They note that if they added 500 more clusters, they'd need just one more engineer. That's a key for any of us that want to manage growing numbers of systems without spending a lot of our time reviewing resumes and hoping we can hire good staff. Hiring is hard, and finding good people even harder. When you find them, set them free codifying their knowledge using DevOps, scripting, automation, and more.
Then educate others and teach them what your talented engineer is doing. Mercedes notes that finding people is hard, and educating existing people is easier. DevOps, better coding, understanding APIs and declarative scripting are not hard skills, but they are something people need to practice to develop familiarity and skill. We want staffers to be able to easily pick up the work of another, understand it, and extend or improve it. We don't want to depend on the person that wrote it.
The way Mercedes has attacked this technology is the way I'd have developers and administrators tackle DevOps. Take advantage of the power of modern software development and infrastructure tools and empower your staff to make things better. They are likely to enjoy their jobs more and remain employed, reducing your need to struggle with the vagaries of finding and hiring good people, a problem no one has solved well.