Container Orchestration Choices
We need to make a move.
September 16, 2019
Here at Brand New Box we're big fans of containers. But keeping hundreds of discrete containers spinning across dozens of environments is too great of a challenge to tackle individually. We need orchestration!
Why use containers in production? And how?
Containers give us a great way to reduce specific configuration needed for each server. All of our projects are unique and each one requires a custom mix of software running together on the server to do its job. Configuring the servers used to be a time consuming headache but containers make that a thing of the past. Now all we need is a server with Docker installed and we can easily run all of our software as prepackaged containers on that host.
Now that the decision to run containers in production is made challenges still remain. How do you deploy updates to your software? How do you restart containers if they crash? If an application is big enough to require containers across more than one server how do you keep all the containers communicating smoothly and securely? The answer is to use a container orchestrator.
We’ve been using software called Rancher to run our containers in production. Specifically we’ve been using the “Cattle” orchestrator that was available in the 1.x versions of Rancher. I say was available because the Cattle orchestrator is being put out to pasture. I feel like I might be mixing my metaphors. It was a homegrown product developed by Rancher and going forward in version 2.x they have adopted an orchestrator with more widespread industry adoption. The end days for Cattle are here, Rancher 1.x is no longer under active development and even security updates will stop in December. Brand New Box needs a new orchestrator.
What are our choices?
The obvious answer is to upgrade to Rancher 2.x and adopt their new orchestrator, Kubernetes. Kubernetes is the superstar of the container orchestration world with companies like Spotify, Comcast, and IBM using it. We’re not quite at that scale, but we still need a solution.
Deployed my blog on Kubernetes pic.twitter.com/XHXWLrmYO4
— DevOps Thought Liker (@dexhorthy) April 24, 2017
Kubernetes and Rancher 2.x will absolutely work for our use case but it’s not a slam dunk.
We already have a few projects up and running on Rancher 2.x and we’ve found there is a big step up in complexity from where we are now. The terminology is new and there is a lot of clicking around to move from one project to the next, especially because each of our projects is hosted on its own Kubernetes cluster. Rancher and Kubernetes are an extremely powerful combination, with almost limitless capacity and possibilities. Not only does Rancher orchestrate our containers with Kubernetes, it can monitor them, help us install additional software in the clusters, give us fine grained security controls, and has all kinds of other bells and whistles.
For us this choice is the status quo but it might be a bit much for our use case. I started looking for a less complex solution and I’ve come across a couple of possibilities, Docker Swarm and HashiCorp Nomad.
The contenders!
Docker Swarm is the Docker native orchestrator, and is really just a mode of running the Docker software. If you have a recent version of Docker installed, you already have Swarm. Swarm uses similar commands to the Docker commands we already use in development, so it should be easy to pick up and use. The only big feature of modern orchestrators it is missing is that we can’t use it to automatically scale any part of our infrastructure. If we want more or less capacity for anything, we’ll have to tell it so. We don’t use this currently on any of our projects, but it would be nice to have occasionally (I’m looking at you Merchtable ).
Nomad is another orchestrator that can run containers. It is lightweight in terms of resource requirements and designed to be simple to learn and operate. It is unique among our considered solutions in that it can also run workloads that aren’t containerized at all. Once you have your configuration ready to go Nomad can also dry run itself, showing the user exactly what will happen and eliminating surprises. Because it’s not Docker or container native, it will have a whole new paradigm and configuration to learn to use. It also lacks native autoscaling.
A shared downside to both Docker Swarm and Nomad is that they aren’t Kubernetes, and Kubernetes has huge upward momentum in the industry right now. There are thousands of blog posts, examples, and code snippets all over the internet for working with Kubernetes. Those resources exist but are harder to find for our other choices. Unlike Rancher with the Cattle orchestrator, Docker and HashiCorp continue to stand behind Swarm and Nomad and have given no sign that development on those products will end anytime soon.
Looking at the choices above, I think I’ll take a harder look at Docker Swarm. Using it we can leverage our existing Docker knowledge and hopefully get up and running quickly. We’ll just have to figure out an easy way to provision servers and create Swarm clusters, but that is a blog post for another day.
