A Platform Engineer’s Guide to Platform Engineering

What is Platform Engineering?

Platform Engineering is a subset of software engineering that focuses on accelerating developer productivity by reducing the cognitive load on development teams through a centrally managed technology platform. The platform should contain all the components to help developers build, deploy, observe, and support their services and corresponding infrastructure. In addition to making developer's lives better, the platform should also be easy to build, modify, and maintain.

Although the term has garnered a lot of buzz lately, platform engineering has been around for a while. Companies like Google, Facebook, and Netflix have had internal developer platforms for over a decade. The cloud providers themselves are platforms that enable individuals to provision hardware. The rise in popularity began around 2020 when enterprises were forced to work remotely due to a global pandemic.

I'm entering my fourth year of working on a platform team and over ten years in the industry. Before joining Upbound, I worked on a platform team at Guidewire that enabled developers and externally facing support teams to provision various stateful components of our platform like Aurora, S3, Dynamo, etc. Our mission at Guidewire was to build THE platform for the property and casualty insurance industry. Not only were we building a platform for our internal developers, but we were also supporting the insurance industry's journey to the cloud.

Currently at Upbound, the responsibility is not as scoped. The platform team at Upbound supports the whole company: developers, marketing, sales, etc. Our mission as a company is to continue to usher in the new era of control planes and be THE universal cloud platform.  Although the scope of my responsibilities have broadened, the idea remains the same: accelerating productivity. 

Lessons Learned

In this article, I will cover some of the lessons that I've learned over the years as a platform engineer.

Platform Engineering is Similar Yet Distinct from Modern Software Engineering

Platform engineering is similar to modern software development when it comes to building and maintaining APIs for cloud infrastructure. My platform experience revolves around writing APIs that provision cloud resources via Crossplane compositions. Although there aren't as many distinct layers in the platform API, my experience with writing APIs as a Java developer has helped tremendously with writing platform APIs.

The core concepts still applied:

1. Design the API
2. Write failing tests based on the desired output of the design
3. Create an API that passes those tests
4. Deploy 
5. Iterate

As you can see, these steps employ test-driven development (TDD), which is commonly used in modern software development shops. TDD is invaluable in platform engineering because it ensures that APIs are reliable and meet specified requirements. At Upbound, we employ kuttl tests to unit test our compositions, maintaining a high standard of quality and functionality.

However, platform engineering differs significantly in that it lacks compilers and other guardrails that protect you from careless mistakes. Composing platform APIs consists of writing an abstraction layer on top of an abstraction layer that interacts with different types of hardware and other software. The existence of a compiler for a single platform language would be very difficult to author and maintain. So development of a platform tends to feel like putting together a puzzle of different tools or in-house solutions to increase developer and operational productivity.

The good news is that the tools are getting better and are reducing the amount of tools that are required to support platform features. Here at Upbound, our developers have created a product that not only manages your infrastructure but has features that you would normally have to use other tools to assist with infrastructure management and monitoring. For example, we currently have a control plane per environment with a one to one relationship to a kubernetes cluster. With Upbound we will reduce our infrastructure overhead significantly by leveraging Upbound.

And if you’re not ready to go all in on Upbound, a great tool to use is Crossplane, a leading cloud native control plane framework. It’s a tool that enables you to orchestrate applications and infrastructure no matter where they run. Based on the Kubernetes API, it has a highly configurable backend that lets you define the declarative API it offers.

In Crossplane version v1.16.0, there are a plethora of features that didn't exist just a few minor versions ago: from insight into resource management to streamlined ways to clean up resources. I've been a user of Crossplane since version v1.3.0 and it's been exciting to see how the project has matured.

The Platform Should Be Rigid, But the Platform Engineer Nimble

Since platform engineering is like putting together a complex puzzle, it is crucial for the engineer to be nimble and open to new tools and ideas. The platform is supposed to be a stable, almost rigid surface for the company to launch new services and ideas. However, the engineer must be malleable, open to new ideas. Being receptive to real-time feedback such as attending developer stand-ups or sprint-planning is essential for understanding their needs and workflows. Real-time interactions provide valuable insights into the challenges developers face, allowing platform engineers to identify areas for optimization and improvement. This proactive engagement fosters a collaborative environment where you're more than likely to build a platform that they want to use.

Resource constraints often require platform engineers to be creative and adaptive. Sometimes, you have to be like Leto II from the Dune series and merge with the sandtrout—in this case, tools—and become an integral part of the platform, implementing Golden Path to production. This metaphor illustrates the necessity of deeply integrating with the tools and technologies at hand, sometimes even becoming the de facto solution yourself. While the shift-left movement advocates for developers to broaden the scope of their role, expecting developers to fully own both software and hardware expertise is unrealistic. Therefore, platform engineers must be prepared to be a mediary, configuring lower-level details to ensure successful outcomes.

Have a Maintenance Plan for Your Platform Components

As I mentioned earlier, a platform is essentially a conglomerate of tools, whether it be artisanal or otherwise. Maintaining a platform is not a "set it and forget it" endeavor, like cooking a rotisserie chicken in the Showtime Rotisserie. Platforms, like any software product, require ongoing attention and care. Platform neglect can lead to unexpected failures and vulnerabilities, striking at the most inopportune times. At Upbound, our developers are building a product that leverages bleeding edge technology. Our developers are constantly exploring new technologies to implement innovative features. In order to continue to support innovation at our company, the platform team needs to also make sure that the platform components are up-to-date with the latest features. And besides, the longer you wait to upgrade, the potential for more breaking changes accumulate, making the upgrade process more complex and time-consuming. For instance, we delayed upgrading Kubernetes and some cluster components, which resulted in a month-long effort to update three environments to the latest version.

Conclusion

Platform engineering is a dynamic and evolving discipline that requires a unique blend of software development skills, creativity, and adaptability. It involves not just building and maintaining an interface for cloud infrastructure, but also understanding and addressing the needs of various stakeholders within an organization. My journey from a Java developer to a platform engineer has taught me the importance of leveraging core software engineering principles, like test-driven development, while also being flexible enough to integrate new tools and ideas.

As a platform engineer, your goal is to create a stable and reliable platform that accelerates productivity and supports the organization's goals. This involves continuous maintenance, regular updates, and staying attuned to the latest advancements in technology. At Upbound, we've seen firsthand the benefits of staying ahead of the curve, whether through innovative tools like Crossplane or by ensuring our platform components are always up-to-date.

Whether you're just starting your journey in platform engineering or looking to refine your skills, these lessons can serve as a guide to navigating this complex and rewarding field. Keep learning, stay flexible, and continue to build platforms that empower developers and drive innovation.

If you’re interested in trying out Upbound as a tool that can help streamline your platform processes like I mentioned earlier, you can try it out for yourself in a trial here.