K8s Is Not the Platform – Or Is It and We All Misunderstood?

Note: this content was originally posted on The New Stack here.

Just a few years ago on the main stage of KubeCon North America, Kelsey Hightower talked about how Kubernetes is just a low-level tool and not a (developer) platform.

Companies building internal platforms to run their business would have to build abstractions and machinery above Kubernetes to reduce developers’ cognitive load. Even more so, if done right, Kubernetes would become invisible and just an implementation detail.

For those of us deeply invested in Kubernetes and the giant ecosystem around it, this last sentence sounds like a push back. If that statement was true, we would be working on making components invisible eventually.

Now a few years later, it feels like we all misunderstood. Kubernetes is everywhere, and people are building platforms more than ever with Kubernetes as the core component. So, what has changed? Let’s explore where we have ended up and what is different today.

From Container Orchestration to More

Kubernetes was originally a container orchestration platform, foremost built for this single use case. But while building up the most advanced container orchestrator available in the open source community, the project did not end with only orchestrating pods. The project very early on built extension points into the system, most prominently custom resource definitions (CRDs). Originally, CRDs were used to extend a compute cluster with auxiliary resources. Think of configuration resources, CertManager CAs and CertificateRequests, third-party networking layers like service meshes. The possibilities are endless, and a flourishing ecosystem evolved around compute.

But custom resource definitions opened the door to a lot more. Many concepts in Kubernetes are not constrained to container orchestration. When asked what will persist most likely from Kubernetes the longest, many thought leaders agree: the object model, the highly opinionated CRUD API server model will likely live on the longest.

What does this mean? It means we have learned much more than how to run pods most efficiently since Kubernetes 1.0 was announced in 2015. People love the uniformity of APIs that Kubernetes offers. People love that they can use one tool like kubectl to talk to very different objects, one GitOps tool speaking the Kubernetes API language to roll out very different resources, one controller library like controller-runtime to build reconcile loops for very different domains. You see the pattern.

We all remember discussions about building solutions on Kubernetes and building APIs using CRDs, where the question arose: Why Kubernetes? Any other API machinery would do. This is true, obviously. But at the same time, you can ask the inverse: Why not Kubernetes and its API machinery with its rich ecosystem? Why shouldn’t we use CRDs to build, like developer platforms?

This discussion is kind of superfluous and quickly replaced with reality where people are doing just that. People love the Kubernetes ecosystem, the tooling built around CRDs and Kubernetes APIs. They know the tools and know how to write controllers. The question “What else?” is answered simply by “Kubernetes, because we have the knowledge, the tooling, and we are productive with it.” Plus, there is a giant community around it with the Cloud Native Computing Foundation and all its projects.

Let’s look at how Kubernetes helps to build the platform every business today is after. At the center of Kubernetes there is the API, an API that allows CRUD of very different objects, such as managing their life cycle in a uniform way. What do we mean by that? Kubernetes serves (clusterwide and namespaced) objects called resources. Objects have a common shape that we all know:

This common shape makes universal solutions possible, such as solutions for GitOps that work with all kinds of resources offered by a Kubernetes cluster. If our developer platform speaks this language of Kubernetes APIs, all those tools will just work and can immediately be part of our toolbox.

Traditionally, Kubernetes resources eventually need real code, usually Golang code. While the CRDs can be declaratively defined with a bunch of YAML lines, the actual logic requires real programming. Programming means cost, maintenance and more. It’s usually what platform builders avoid rather than embrace.

Enter Crossplane

The Kubernetes object model can support other ways to define logic for implementing APIs. The Crossplane project is a great example. Crossplane, another CNCF project, was born in 2018 from the insight that the Kube API is perfectly suited to orchestrate arbitrary cloud resources. A year or two into the project, a concept called Compositions was invented. Compositions implement a pattern without a single line of code that we all know from Kubernetes: think of deployments, ReplicaSets and pods. They form a hierarchy: Deployments spawn ReplicaSets, and ReplicaSets spawn pods. Compositions implement this kind of hierarchy defining how an instance unfolds into other Kubernetes objects.

A developer platform needs higher-level abstractions to minimize cognitive load for developers. A deployment is an abstraction of pods. In a developer platform, think of a BigCorp Application, an abstraction BigCorp wants to offer to its developers: highly opinionated, highly custom for BigCorp, all API based. Crossplane makes it super easy to define such an abstraction without a single line of code:

A Composite Resource Definition (short XRD) defines a custom resource definition with composite semantics. Concretely, a Composition object defines how an (namespaced) Application claim object spawns a XApplication, spawning further resources when instantiated:

Imagine a developer in BigCorp using this API by creating an instance for the new customer feedback application to be deployed at a subdomain of the company website:

The Composite Resource Definition (XRD) defines the schema of this API. The Composition xapplication-webapp defines which resources this unfold into and how the specified domain value eventually ends up in the ingress definition (through Patch and Transforms, very much like templating).

The Composition specifies two normal Kubernetes objects in this example but also one infrastructure resource for a cloud provider database. The Kubernetes resources can both land on the same Kubernetes cluster as the claim itself or be distributed to other clusters like your us-east1 application production cluster, or another one in Europe.

Note that the Application customer-feedback/marketing-team is just another Kubernetes API object, compatible with all the tools available in the ecosystem. For instance, the platform team wants to define guardrails about what this developer can do with an Application. Choose one of Kyverno, OpenPolicyAgent or jsPolicy. The finance team wants to reduce cloud spend and aggregate cost onto BigCorp’s Application objects to make cost visible to the application owner? Choose OpenCost or KubeCost. And so on.

The developer, with all this platform tooling in place, will use kubectl or the GitOps tool of choice to create the BigCorp Application claim:

For Kubernetes, this is just another resource: Kubernetes will manage its life cycle, define its API semantics, etc. For Crossplane, this is an instance of the xapplication-webapp composition. Crossplane knowns from the Composition and XRD above how to give life to the claim:

• Crossplane will create an XApplication composite.
• It will select the xapplication-webapp composition for the XApplication as implementation.
• It will unfold the XApplication into a database, the ingress and deployment object according to the selected composition.
• It will create the SQL database at the cloud provider of your choice.
• It will deploy the Kubernetes objects onto the same cluster or on production clusters of your choice.
• Or, it will create another dedicated cluster with Argo CD as the GitOps tool of choice automatically installed and connected to your application git repository.
• It will register the application and the specified dependencies in the company’s Backstage instance.
• It will watch the status of all of these steps and report back to the developer in the status of the BigCorp Application object.
• It will manage the life cycle of all the resources created.
• It will tear down everything when the developer decides to delete the application.

All of this happens without code. It happens on Kubernetes, and possibly involves multiple Kubernetes clusters. It is implementing a piece of a developer platform. The developer platform is not Kubernetes in the sense of the abstractions Kube as container orchestrator provides. But the developer platform is built on Kubernetes technology and the language of Kubernetes APIs and controllers. It implements high-level abstractions. It gives BigCorp a toolkit of building blocks and a rich ecosystem.

Kubernetes offers compute, merely as a side-effect. BigCorp Application could choose to deploy to some other compute service, for instance by creating a lambda.aws.crossplane.io/v1beta1 object for the customer-feedback application instead of a Kubernetes deployment. Compute is just a piece of the developer platform and potentially invisible to the developer. Compute is only a piece of Kubernetes, the potential it has as a foundation to solve real business problems.

So where are we today? Kubernetes and tools like Crossplane give us a powerful toolbox to build platforms. Kubernetes is not the platform. We misunderstood what this means. Kubernetes container orchestration is not a “platform as is” for developers, and it might even be invisible or unused. But Kubernetes as technology is bigger and has become the language of choice to build platforms.

Note: this content was originally posted on The New Stack here.