Guides and Tutorials

Opinions and Insights

News and Announcements

Customer and Partner Stories

Featured

Press Releases

Subscribe to our Newsletter

Upbound

Crossplane

Kubernetes

Marketplace

Terraform

Solutions

<p>Today we're releasing a new multicloud Kubernetes Crossplane configuration to the <a href="https://cloud.upbound.io/registry/upbound/platform-ref-multi-k8s" target="_blank">Upbound Registry</a>. Using this Configuration, SREs, and the application teams they support, can easily deploy Kubernetes clusters into multiple cloud environments. Upbound Cloud enables this through a single API or console interface so teams no longer have to duplicate time consuming setup or maintain multiple Infrastructure-as-Code scripts.</p><p>Read more below about how customers are deploying Kubernetes across clouds, and a short tutorial showing you how to use this configuration in Upbound Cloud today.</p><h2>Background</h2><p>A key promise of Kubernetes is having a common application runtime across infrastructure environments, making the goal of workload portability a reality. While this sounds simple, truth be told, things are typically more complicated. Thankfully, Upbound Cloud helps you manage cluster configuration in a way that's cloud agnostic and lets application developers safely provision clusters on their own without needing to become infrastructure experts.</p><p>Despite the rise in popularity of Kubernetes managed services like EKS or GKE, deploying and managing Kubernetes clusters across those environments has become an increasingly challenging problem. Infrastructure operators need to define and manage configuration for multiple environments, repeating work for every cloud each time a setting or policy changes.</p><p>Additionally, during development and prototyping, application teams typically need the ability to quickly provision new Kubernetes environments on their own. Without a quick and safe way of doing this, operators get bogged down in tickets and emails from developers requesting new resources.</p><p>Thankfully Upbound Cloud can enable quick, safe, and cloud agnostic Kubernetes cluster deployment in just a few minutes right out of the box.</p><h2>Requirements</h2><p>To get started, first <a href="https://cloud.upbound.io/register" target="_blank">create an account</a> on Upbound Cloud. While you're at it, go ahead and create a free Organization.</p><p>Next, clone <a href="https://github.com/upbound/platform-ref-multi-k8s/" target="_blank">this example repository</a>. We'll be referencing some files and code found in the <code>example</code> directory, so it will be easier to follow along if you have a copy.</p><p>Finally, install the <a href="https://crossplane.io/docs/v0.14/getting-started/install-configure.html#install-crossplane-cli" target="_blank">Crossplane CLI</a> to make it easier to configure your Crossplane instance we'll create in Upbound Cloud.</p><h2>Getting Started</h2><p>Inside the repository you just downloaded, you'll find several directories containing YAML for Crossplane <a href="https://github.com/crossplane/crossplane/blob/master/design/design-doc-composition.md" target="_blank">compositions</a>. This YAML gets packaged up into what we call a configuration and installed into a Crossplane cluster which uses it to provide a custom API to the composite resources defined. The easiest way to this of this is that configurations act like a "blueprint" for your cloud API.</p><p>The <code>cluster</code> and <code>network</code> directories each contain their own composition, and can all be thought of as separate API endpoints defined by this configuration.</p><p>What's interesting is when you look into each, you can how we've defined configuration for both GCP and AWS infrastructure. Here's the anatomy of the <code>cluster</code> API. There's a composition defining GKE cluster setup, another for EKS setup, and finally a <code>services</code> directory which leverages <code>provider-helm</code> to install on-cluster services into both clouds.</p>

<p><span style="background-color:rgb(255,255,255);color:rgb(3,7,36);font-family:Avenir-Roman, Arial, sans-serif;">In the </span><code>definition.yaml</code><span style="background-color:rgb(255,255,255);color:rgb(3,7,36);font-family:Avenir-Roman, Arial, sans-serif;"> file, which renders a dynamic UI inside of Upbound Cloud which you'll see later on in this post, we've gone ahead and hidden this complexity behind a simple dropdown for users to choose which cloud they want their composite cluster resource to run inside.</span></p>

<p>To save some time later on in this tutorial, we've <a href="https://cloud.upbound.io/registry/upbound/platform-ref-multi-k8s" target="_blank">uploaded this configuration to the Upbound Registry</a> where you can install it into a platform using the rest of this guide.</p><h2>Multicloud Cluster Deployment</h2><h3>Create a New Platform</h3><p>First, you'll need to create a new Platform in Upbound Cloud. This is just a hosted Crossplane cluster we run and manage on your behalf.</p>

<h3>Connect to Your Platform's Crossplane</h3><p>UI install is coming soon, but today, CLI access to the hosted Crossplane cluster powering your Platform is required to install the Configuration from your Repository</p><p>To get started, we need to create a user token which will allow you to connect and authenticate to Upbound Cloud via <code>kubectl</code>.</p><p>Afterwards, go back to the Console, select your Platform, click "Connect via CLI" on the bottom left of the screen. Follow those instructions using your newly created Access Token:</p>

<p><span style="background-color:rgb(255,255,255);color:rgb(3,7,36);font-family:Avenir-Roman, Arial, sans-serif;">Once connected, we'll need to install the </span><code>provider-aws</code><span style="background-color:rgb(255,255,255);color:rgb(3,7,36);font-family:Avenir-Roman, Arial, sans-serif;">, </span><code>provider-gcp</code><span style="background-color:rgb(255,255,255);color:rgb(3,7,36);font-family:Avenir-Roman, Arial, sans-serif;"> and </span><code>provider-helm</code><span style="background-color:rgb(255,255,255);color:rgb(3,7,36);font-family:Avenir-Roman, Arial, sans-serif;"> your configuration requires. Automatic dependency resolution is coming later this year.</span></p>

<p><span style="background-color:rgb(255,255,255);color:rgb(3,7,36);font-family:Avenir-Roman, Arial, sans-serif;">Use your AWS account credentials to authenticate </span><code>provider-aws</code><span style="background-color:rgb(255,255,255);color:rgb(3,7,36);font-family:Avenir-Roman, Arial, sans-serif;"> by downloading a AWS token locally into a file and using that to configure the provider:</span></p>

<p>Next we'll need to configure your GCP provider by storing your GCP credentials into a secret it can read.</p><p>Set up your GCP account keyfile by following the instructions on <a href="https://crossplane.io/docs/v0.14/getting-started/install-configure.html#select-provider" target="_blank">https://crossplane.io/docs/v0.14/getting-started/install-configure.html#select-provider</a></p><p>Ensure that the following roles are added to your service account:<br>- <code>roles/compute.networkAdmin</code><br>- <code>roles/container.admin</code><br>- <code>roles/iam.serviceAccountUser</code></p><p>Create the <code>ProviderConfig</code>, ensuring to set the <code>projectID</code> to your specific GCP project:</p><p><code>kubectl apply -f examples/provider-default-gcp.yaml</code></p><p>Finally, we're ready to install the configuration whose repository you just downloaded.</p>

<p>After the Configuration installs, you should see each Composition show up under the Global Resources view.</p>

<p><span style="background-color:rgb(255,255,255);color:rgb(3,7,36);font-family:Avenir-Roman, Arial, sans-serif;">Once added, you'll create a </span><code>network</code><span style="background-color:rgb(255,255,255);color:rgb(3,7,36);font-family:Avenir-Roman, Arial, sans-serif;"> object for your desired cloud in order to spin up shared networking resources for your teams.</span></p>

<p>Finally, we'll go ahead and create a cluster which will use that composite network resource you just created.</p>

<p>So far, you've done everything inside of the Upbound Cloud Admin Console. However, if you go to your organization picker and select User Console, you'll be taken to the experience your developers get with Upbound Cloud.</p>

<p>Here developers see a paired down interface and are greeted to the cloud console dynamically generated from the configurations you've installed. It's tailor-fit to their needs and designed to make it easier for them to safely get the resources they need, whenever they need them.</p><h2>Conclusion</h2><p>That's it! You're now able to deploy Kubernetes clusters across cloud environments using a single source of truth for cluster configuration.</p><p>Upbound Cloud lets you define and manage the API for this composite cluster resource, and then give your developers self-service access to the API through their console.</p><p>If this was interesting to you, <a href="https://upbound.io/" target="_blank">sign up</a> for our Community Preview and try this out today, or reach out to learn more and we'll give you a live demo.</p><p>&nbsp;</p><hr><p>&nbsp;</p><p>PS: Powering every Upbound Cloud account is the CNCF popular open source project, <a href="https://crossplane.io/" target="_blank">Crossplane</a>. Crossplane is hosted on <a href="https://github.com/crossplane" target="_blank">Github</a>, and we have a thriving community on <a href="https://slack.crossplane.io/" target="_blank">Slack</a> and <a href="https://twitter.com/crossplane_io" target="_blank">Twitter</a>. Come join the fun!</p>

Multicloud Kubernetes Deployments with Upbound Cloud

<p>Upbound’s cloud-native infrastructure management platform is now available in the <a href="https://aws.amazon.com/marketplace/pp/prodview-ksv3yjcqod3kg?sr=0-3&amp;ref_=beagle&amp;applicationId=AWSMPContessa" target="_blank">AWS Marketplace</a>, empowering AWS customers to build, deploy and manage cloud services from a single point of control using Crossplane control planes. AWS customers use Upbound for multi-cluster management, cloud service provisioning, and building complete internal developer platforms.</p><p>Customers can now access the Upbound platform directly from the cloud provider using the new integration with AWS Marketplace. Like other products available on the AWS Marketplace, Upbound meets AWS’s security and solution design requirements.</p>

<h2>Key customer benefits of purchasing Upbound through AWS Marketplace</h2><ul><li>Use existing AWS accounts and budgets to purchase Upbound software, accelerating planned projects and strategic cloud migration objectives</li><li>Flexible purchasing and consumption program options</li><li>Negotiate Upbound software procurement pricing using AWS Marketplace Private Offers, where the customer negotiates pricing and terms directly with Upbound to transact through marketplace</li><li>Cost-effective, faster procurement: AWS Marketplace purchases count towards customer commitments under the terms of their Enterprise Agreements. Purchasing Upbound through AWS Marketplace simplifies approvals by enabling “Click to Buy” through your Master Payer Account with AWS</li><li>Simplified billing: Customers get a single consolidated AWS invoice for all the purchases made through the AWS Marketplace with complete visibility over their spending in a single location</li></ul>

<h2>Upbound features</h2><ul><li>Creates a platform for deploying clusters, databases, networks, AI/ML, and other workloads that can be used as part of a internal developer platform to increase velocity, meet compliance, and reduce risk</li><li>Deploy and manage Crossplane control planes at scale, with Upbound managing the underlying infrastructure and auto-scaling control plane resources</li><li>Manage major and minor upgrades of core components on behalf of the user, including automatic snapshotting of platform state for disaster recovery</li><li>Command and Control Center for Day 2 Crossplane operations</li><li>Console enabling users to view control plane usage, manage operations, debug API calls passing through users control planes, and integrate with logging and monitoring solutions</li><li>Available for consumption as both SaaS and self-managed deployments</li></ul>

<h2>Resources</h2><ul><li>Upbound is available as a <a href="https://www.upbound.io/register/a" target="_blank">free trial</a></li><li>Upbound is available as an <a href="https://aws.amazon.com/blogs/awsmarketplace/deploy-third-party-software-add-ons-aws-marketplace-amazon-eks-clusters/#:~:text=The%20following%20third-party%20software%20products%20are%20ready%20to,Agent%204%20Factorhouse%20%28KPOW%29%205%20Tetrate%206%20Teleport" target="_blank">EKS add-on</a> from the EKS management console</li><li>AWS’s blueprints for deploying &amp; using <a href="https://aws.amazon.com/blogs/opensource/introducing-aws-blueprints-for-crossplane/" target="_blank">EKS</a> and <a href="https://docs.aws.amazon.com/lambda/latest/dg/with-kubernetes.html" target="_blank">Lambda</a> with Crossplane</li><li><a href="https://youtu.be/0U51OeJJHQ0?feature=shared" target="_blank">Best practices</a> for architecting an internal developer platform with AWS and Crossplane</li><li>Upbound’s Open Source Marketplace, with <a href="https://marketplace.upbound.io/configurations/upbound/platform-ref-aws/v0.8.0" target="_blank">AWS reference architectures</a> to get started</li><li>Read about the new OSS standard<a href="https://cnoe.io/" target="_blank"> CNOE approach</a> to building internal developer platforms, and try the <a href="https://marketplace.upbound.io/configurations/upbound/platform-ref-aws-cnoe/v0.1.0" target="_blank">Upbound CNOE reference platform</a></li></ul><h2>What’s Next</h2><p>Check out the <a href="https://aws.amazon.com/marketplace/pp/prodview-ksv3yjcqod3kg?sr=0-3&amp;ref_=beagle&amp;applicationId=AWSMPContessa" target="_blank">Upbound listing</a> on the AWS Marketplace, sign up for an <a href="https://www.upbound.io/register/a" target="_blank">Upbound free trial</a> and learn more about deployment and purchasing options in the AWS marketplace. If you are interested in talking with Upbound please <a href="https://www.upbound.io/contact" target="_blank">contact us here</a>.</p>

Upbound is Now Available in AWS Marketplace

<p>The previous<a href="https://blog.upbound.io/upbound-day-0"><u>&nbsp;blog post</u></a>&nbsp;in this series showcased how Upbound makes it easy for organizations to get up and running with their first instance of Crossplane in minutes. Today’s entry in the series covers a typical Day 1 operation for users of Upbound: configuring your control planes for&nbsp;<a href="https://docs.upbound.io/xp-arch-framework/interface-integrations/git-and-gitops/"><u>GitOps flows</u></a>.</p><p>GitOps is an approach for managing a system by declaratively describing desired resources’ configurations in Git and using controllers to realize the desired state. Crossplane is compatible with this pattern and it’s strongly recommended you integrate GitOps in your platforms built on Crossplane–<strong>and you can do this with&nbsp;</strong><a href="https://www.upbound.io/product/upbound"><strong><u>Upbound managed control planes</u></strong></a>!</p><p><i>Want to learn more? Here are the links to other blog posts showcasing new features for platform operators:</i></p><ul style="list-style-type:disc;"><li><a href="https://blog.upbound.io/2023-06-09-product-updates"><i><u>New Changes in Upbound: ProviderConfigs, Control Plane Explorer</u></i></a></li><li><a href="https://blog.upbound.io/2023-06-30-product-updates"><i><u>Upbound Identity with Azure, search in the Marketplace, and expanding the Configuration catalog</u></i></a></li><li><a href="https://blog.upbound.io/2023-07-28-product-updates"><i><u>New from Upbound: Configuration &amp; Provider Events in the Console, Crossplane Architecture Framework</u></i></a></li></ul><h2>GitOps and Crossplane</h2><p>Every managed control plane in Upbound comes with a&nbsp;<a href="https://docs.upbound.io/concepts/ctp-portal/"><u>control plane portal</u></a>&nbsp;by default. This portal is a convenient CRUD (create, read, update, and delete) graphical interface to interact with the APIs on your control plane. Think of using the control plane portal as talking&nbsp;<i>directly&nbsp;</i>to your control plane. The GitOps flows we will describe in this post are yet another flow you can use to interact with your control plane, and likely one you will want to use for production scenarios.</p><p>Integrating GitOps flows with Crossplane is useful in three ways:</p><ol style="list-style-type:decimal;"><li><strong>You can use git to store the definition of your control plane configuration.&nbsp;</strong>This is something users of Upbound get as a built-in feature of Upbound, since the platform supports automatically syncing the definition of your control plane’s configuration from GitHub. Read about&nbsp;<a href="https://docs.upbound.io/concepts/control-plane-configurations/#how-configurations-work-on-upbound"><u>how control plane configurations work</u></a>&nbsp;on Upbound.</li><li><strong>You can use Git as an interface to invoking your control plane’s APIs</strong>. Whenever a user needs a resource from your control plane, a claim must get created. Claims themselves are resources with a configuration (a spec), and as such can and should be declaratively stored in Git. This is the one we’ll demonstrate in today’s blog.</li><li><strong>Use Git to store the infrastructure definition that backs your control plane.&nbsp;</strong>In Upbound’s&nbsp;<a href="https://docs.upbound.io/xp-arch-framework/"><u>Crossplane Architecture Framework</u></a>, we call this out as a third way you can integrate with git. In the case of Upbound, this is not relevant because managed control planes by nature shift this responsibility away from you and over to Upbound.</li></ol><p><br>By the end of this blog, our goal is to have:</p><ol style="list-style-type:decimal;"><li>A managed control plane with a custom API running in Upbound.</li><li>A GitHub repository that is the source of all the claims I want to have against my control plane’s APIs.</li><li>An engine that observes when new claims get committed to my Git repository and syncs them over to my control plane. Conversely, when a claim is removed from the Git repository, it will get cleaned up from my control plane.</li><li>Once a claim is on the control plane, the control plane will fulfill its API contract and create all the resources that need composing.</li></ol><h2>Prerequisites</h2><p>If you want to follow along with this blog post, it requires that you have an account on&nbsp;<a href="https://www.upbound.io/register/a"><u>Upbound</u></a>&nbsp;and have already created a managed control plane. The configuration you choose for your control plane doesn’t matter; pick whichever one you like. You should also make sure you have installed the&nbsp;<a href="https://docs.upbound.io/cli/"><u>up CLI</u></a>&nbsp;on your machine.</p><h2>Set up a GitOps engine</h2><p>If you are already familiar with Kubernetes, it's likely you’ve heard of the GitOps pattern. There is already popular tooling available in the Kubernetes ecosystem to solve these problems.&nbsp;<a href="https://argo-cd.readthedocs.io/en/stable/"><u>Argo CD</u></a>&nbsp;and&nbsp;<a href="https://fluxcd.io/"><u>Flux</u></a>&nbsp;are two examples of projects in the Kubernetes ecosystem commonly used for GitOps. You can use them in tandem with Upbound managed control planes to achieve GitOps flows.</p><p>Today, we demonstrate how to integrate Flux. Check back to our blog or documentation for an update when Argo CD integration becomes available.</p><p>In order to integrate Flux with a managed control plane, you first need an instance of Flux installed into a Kubernetes cluster. Flux is not installable inside of Upbound, so you need to provision it on your own cloud infrastructure. For production scenarios, it’s recommended you designate a cluster for running Flux. For the purpose of this blog, we installed Flux into a local&nbsp;<a href="https://kind.sigs.k8s.io/"><u>kind</u></a>&nbsp;cluster.</p><p>You install Flux into a cluster by&nbsp;<a href="https://fluxcd.io/flux/installation/#github-and-github-enterprise"><u>bootstrapping</u></a>&nbsp;it and pointing it at the repo it should monitor. You would run something like this (assuming you are connecting to GitHub):</p>

<p>Once Flux is installed, you should see pods deployed in the flux-system namespace.</p>

<h2>Connect Flux to your managed control plane</h2><p>When Flux observes a change in a Git repository, it syncs those changes to a target. In a traditional installation of Flux on Kubernetes, the assumption is that Flux is installed as an add-on in the same cluster you wish to sync changes to. That’s not the case for a managed control plane on Upbound, so you need to tell Flux about the managed control plane. You do this by providing the managed control plane’s API server endpoint.</p><p>All managed control planes have a deterministic Kubernetes API server endpoint in the following form:</p><blockquote><p>https://proxy.upbound.io/v1/controlPlanes/&lt;account&gt;/&lt;control-plane-name&gt;/k8s</p></blockquote><p>Fetch your managed control plane’s API server endpoint so you can provide it to Flux. In the step below, use the&nbsp;<a href="https://docs.upbound.io/concepts/managed-control-planes/#connect-directly-to-your-mcp"><u>up CLI</u></a>&nbsp;to fetch the kubeconfig of your managed control plane and write it to a file. The up CLI uses personal access tokens to authenticate to Upbound. You can&nbsp;<a href="https://docs.upbound.io/concepts/console/#create-a-personal-access-token"><u>generate a personal access token</u></a>&nbsp;from the Upbound Console. This is the token passed in the following command:</p>

<p>Next, create a secret on the external Kubernetes cluster where you installed Flux. This secret should contain the kubeconfig from the previous step.</p>

<p>Then, define a new Flux Kustomization resource representing your managed control plane and apply to your Flux cluster. The example manifest below assumes you store claims in your Git repository under a claims folder.</p>

<p>Finally, apply a Kustomization to the cluster where Flux is running to tell Flux about this resource.</p>

<h2>Use Git to drive control plane interactions</h2><p>You can now commit claims in the Git repository that Flux is monitoring. When you do, Flux will sync them to your managed control plane running in Upbound. Your control plane will create and compose resources accordingly.</p>

<h2>What’s Next?</h2><p>To get the full rundown on the end-to-end flow of building an internal developer platform using control planes, I explain it in depth in this webinar with the Linux Foundation, “<a href="https://resources.upbound.io/all-content/todo-enable-developer-self-service-using-kubernetes-and-control-planes-powered-by-crossplane" target="_blank">//TODO Enable Developer Self-Service Using Kubernetes and Control Planes Powered by Crossplane</a>”.</p>

<p>To follow along with this blog series as well as the webinar above, be sure to&nbsp;<a href="https://www.upbound.io/register/a"><u>sign up for a free trial of Upbound</u></a>&nbsp;today. You will be able to create and configure your first managed control plane in minutes!</p>

How to Build a Self-Service Platform on Upbound: Day 1

<p>Upbound announces a practical application of Crossplane's capabilities through a new reference implementation for <a href="https://cnoe.io/" target="_blank">Cloud Native Operational Excellence (CNOE)</a>. This example demonstrates how <a href="https://www.crossplane.io/" target="_blank">Crossplane</a>, a powerful open source control plane, serves as a valuable starting point for organizations navigating the complexities of cloud-native development.</p><h2>CNOE + Crossplane</h2><p>CNOE, supported by industry leaders such as Adobe, AWS, Autodesk, Salesforce, and Twilio, aims to simplify developer tooling and guide informed technology choices. However, current tools for building Internal Developer Platforms (IDPs) lack practicality, leaving organizations with the challenge of assembling components for effective cloud-native operational excellence.</p><p>Enter Crossplane, a robust open-source control plane created by Upbound and governed under the Cloud Native Computing Foundation (CNCF). This tailored example provides a seamless entry point for platform teams eager to initiate their <a href="https://www.upbound.io/solutions/internal-developer-platform" target="_blank">IDP</a> journey, showcasing Crossplane's potential in simplifying cloud-native operations.</p>

<p>Crossplane is a key component of CNOE, and with today’s announcement, engineers have an easy way to get started with best practices in building a control plane as the foundation of their IDPs.</p><h2>Benefits Simplified</h2><p>By highlighting this Crossplane configuration, organizations can explore the potential of Crossplane and the Upbound platform in a straightforward manner. Standardizing on Crossplane can help build a robust foundation for your cloud-native stack and accelerate the creation of IDPs. This is a practical illustration, inviting organizations to delve into the unique benefits and ease of use that Crossplane brings to the cloud-native operational landscape.</p>

<p>While this example is demonstrated with AWS, it's essential to note that it's designed as a starting point. Modifying basic configurations for Azure or Google Cloud is a small task with predefined configuration packages.</p><h2>Hands-On Overview in 3 Steps</h2><p>Ready to get started with Crossplane and CNOE? Follow these simple steps:</p><ol><li>Go to the <a href="https://marketplace.upbound.io/" target="_blank">Upbound Marketplace</a> and search for <a href="https://marketplace.upbound.io/configurations/upbound/platform-ref-aws-cnoe" target="_blank">platform-ref-aws-cnoe</a></li><li>Install this configuration to your cluster with "Install Manifest"</li><li>In just minutes, you'll have your first IDP up and running. This includes a Kubernetes Cluster with Network, ArgoCD, Backstage, and Keycloak as SSO.</li></ol><p>Now, you can start spinning up more resources within this powerful stack, customizing it to fit your specific needs.</p><p>If you want to contribute to the project you can head over to <a href="https://github.com/upbound/platform-ref-aws-cnoe/" target="_blank">platform-ref-aws-cnoe in GitHub</a>.</p><p>Unlock the potential of Crossplane and kickstart your cloud-native journey today with Upbound!</p>

Upbound Announces New Crossplane Reference Implementation for CNOE

<p>Crossplane was one of the most discussed topics of <a href="https://www.cncf.io/kubecon-cloudnativecon-events/?utm_content=inline-mention" target="_blank">KubeCon + CloudNativeCon North America</a> 2022. It was mentioned in keynotes and talks, and the surrounding chatter was all over the show floor. &nbsp;Crossplane is being deployed in production across Global 2000 organizations, and at KubeCon companies like Grupo Boticário, Adobe, and Autodesk shared how they are adopting Crossplane.</p><p>For those who might not be familiar, <a href="https://crossplane.io/" target="_blank">Crossplane</a> is a framework to build internal cloud platforms using <a href="https://thenewstack.io/gitlab-deploys-the-crossplane-control-plane-to-offer-multicloud-deployments/" target="_blank">control planes</a>.</p><h2>Control Plane-Powered Platforms</h2><p>You may be asking, “Why should I use control planes?” Let’s start with the big picture. Every bigger company has some kind of internal “cloud, DevOps, platform” organization. Let’s call them the “platform team.” They manage the operations and consumption of public clouds and internal services. After the first wave of automation with Infrastructure as Code (IaC) tools, these platform teams have realized that a pure IaC approach is not sufficient, as they often fail to address two fundamental problems:</p><ul><li>How do we govern infrastructure and services in a central place to ensure they are used efficiently and in compliance with our security and other standards?</li><li>How do we unblock development teams and enable self-service that doesn’t require deep Infrastructure as Code knowledge?</li></ul><p>Various approaches have attempted to tackle these objectives and promised self-service with a single pane of glass at reduced costs. But all the offerings have trade-offs: They lock platform teams into a particular technology or infrastructure vendor, and they are too opinionated about the layer of abstraction provided.</p>

<p>The missing ingredient here is a control plane. A control-plane-based architecture gives you a declarative approach to defining and managing resources, a continuous reconciliation to eliminate configuration drift and offer a single point of control for concerns like policy enforcement and permission management.</p><p>Which brings us back to Crossplane, an <a href="https://thenewstack.io/crossplane-a-kubernetes-control-plane-to-roll-your-own-paas/" target="_blank">open source framework</a> that runs on top of Kubernetes to build platforms using a control-plane-based architecture. Let’s see how that is actually done.</p><h2>Building a Platform with Crossplane</h2><p>Building and using a Crossplane-based platform involves three major steps: <strong>Connect</strong> to the resources you want to manage, <strong>Compose</strong> the different resources into a custom API for your team, and finally <strong>Consume</strong> those resources by developers:</p>

<h3><strong>Connect to the Cloud with Providers</strong></h3><p>The first step for a platform team is to connect to the Cloud APIs they would like to offer internally. With Crossplane, this involves creating a custom resource definition (CRD) and running a controller that makes the respective API calls. These specific CRDs are called <a href="https://crossplane.io/docs/v1.9/concepts/managed-resources.html" target="_blank">managed resources</a> and are bundled into a specific <a href="https://crossplane.io/docs/v1.9/concepts/providers.html" target="_blank">Crossplane provider</a>.</p><p>It’s been a consistent message in the community that the quality and coverage of providers is important to their adoption of Crossplane. There are hundreds of contributions and open source listings available, but it’s often unclear which ones are the highest quality, and it’s becoming increasingly more difficult to discover the latest or more relevant contributions.</p><p>It’s because of this that <a href="https://www.upbound.io/?utm_campaign=2022_Q3_EVER_GBL_The-NewStack-GENERAL&amp;utm_medium=The-New-Stack&amp;utm_source=referral&amp;utm_content=inline-mention" target="_blank">Upbound</a> created <a href="https://marketplace.upbound.io/" target="_blank">The Upbound Marketplace</a> as the one-stop shop for all things Crossplane, including official providers and other listings. It’s a great place to see what’s happening with the project. The marketplace offers users:</p><ul><li>A catered place to host Crossplane packages.</li><li>Auto-generated API documentation for packages published to it.</li><li>An easy way to find and learn about different Crossplane extensions and how to use them.</li><li><a href="https://blog.upbound.io/__GHOST_URL__/first-official-providers/">Upbound Official providers</a>: a set of Crossplane providers that are maintained, tested and supported by Upbound.</li></ul><p>As of today, the marketplace offers 1,182 cloud resources to choose from. This goes from the usual suspects of Amazon Web Services (AWS), Google Cloud Provider (GCP) and Microsoft Azure over to smaller cloud providers such as <a href="https://marketplace.upbound.io/providers/civo/provider-civo/v0.1" target="_blank">Civo</a>, <a href="https://marketplace.upbound.io/providers/digitalocean/provider-digitalocean/v0.2.0" target="_blank">DigitalOcean</a> and <a href="https://metal.equinix.com/?utm_content=inline-mention" target="_blank">Equinix,</a> but also other “special” providers such as Kubernetes, Helm or even <a href="https://marketplace.upbound.io/providers/upbound/provider-terraform/v0.6.0" target="_blank">Terraform</a> and <a href="https://marketplace.upbound.io/providers/crossplane-contrib/provider-ansible" target="_blank">Ansible</a>. Upbound has released a first set of <a href="https://blog.upbound.io/__GHOST_URL__/first-official-providers/">official providers</a> and made them available to the whole Crossplane community.</p><p>Additionally <a href="https://www.upbound.io/learn/resources/upbound-marketplace-launch-partners" target="_blank">the community</a> keeps contributing many providers. We will continue to support this effort and if you want to generate a provider for your API, please see <a href="https://github.com/upbound/upjet" target="_blank">Upjet</a> as a great tool to generate a provider.</p><h3><strong>Compose by Patching Resources</strong></h3><p>With so many services at our disposal, it’s now time to move to our next step: compose. With Crossplane, you don’t expose the raw API to developers. Instead, you provide a tailored API in what we call a <a href="https://crossplane.io/docs/v1.9/concepts/composition.html" target="_blank">Composite Resource Definition</a> (XRD). So instead of giving a developer a raw <a href="https://marketplace.upbound.io/providers/upbound/provider-aws/v0.18.0/resources/rds.aws.upbound.io/Instance/v1beta1" target="_blank">PostgresSQLInstance</a>, you set guardrails by providing a tailor-made <a href="https://marketplace.upbound.io/configurations/upbound/platform-ref-aws/v0.4.1/resources/aws.platformref.upbound.io/XPostgreSQLInstance/v1alpha1" target="_blank">XPostgreSQLInstance</a>. Now the developer can only change the options you’ve enabled and the platform team has permitted; the rest is pre-defined and ready to use.</p><p>For this new API to come into reality, you need to build a composition by configuring a set of resources and combining them. This is what we call patching, and you can see the <a href="https://marketplace.upbound.io/configurations/upbound/platform-ref-aws/v0.4.1/compositions/xpostgresqlinstances.aws.platformref.upbound.io/aws.platformref.upbound.io/XPostgreSQLInstance" target="_blank">XPostgreSQLInstance</a> as an example. Combining a set of resources is very powerful and fun (as fun as YAML goes ;-). So, the next big step for the Crossplane community is to come up with new and exciting APIs by creating some patches.</p><p>&nbsp;</p>

<p>The last patch I’ve created was a&nbsp;<a href="https://github.com/luebken/platform-example-logistics/blob/main/demo.md"><u>digital twin for a ship</u></a>. So, I’d like to challenge you to create something more fun and usable. If you are looking for a starting point, I’d like to point to our reference platforms:&nbsp;<a href="https://github.com/upbound/platform-ref-aws"><u>upbound/platform-ref-aws</u></a>,&nbsp;<a href="https://github.com/upbound/platform-ref-gcp"><u>upbound/platform-ref-gcp</u></a>&nbsp;or&nbsp;<a href="https://github.com/upbound/platform-ref-azure"><u>upbound/platform-ref-azure</u></a>. You can also start from scratch by following our quick starts from&nbsp;<a href="https://marketplace.upbound.io/providers/upbound/provider-aws/latest/docs/quickstart"><u>AWS</u></a>,&nbsp;<a href="https://marketplace.upbound.io/providers/upbound/provider-gcp/v0.30.0/docs/quickstart"><u>GCP</u></a>, or&nbsp;<a href="https://marketplace.upbound.io/providers/upbound/provider-azure/latest/docs/quickstart"><u>Azure</u></a>. And be sure to join&nbsp;<a href="https://slack.crossplane.io/"><u>https://slack.crossplane.io/</u></a>&nbsp;and ping me&nbsp;<a href="https://twitter.com/luebken"><u>@luebken.</u></a></p><p>This article was&nbsp;<a href="https://thenewstack.io/composing-a-platform-by-patching-crossplane-resources/"><u>originally posted on The New Stack.</u></a></p>

Composing a Platform by Patching Crossplane Resources

<p>Today, we are announcing a new feature of <a href="https://www.upbound.io/product/upbound" target="_blank">Upbound</a>: Spaces. Upbound Spaces allows teams to run their own Upbound managed control planes in any hyperscale cloud provider or data center where compliance or data sovereignty is a concern.</p><p>Previously, when users wanted to run Crossplane, they had to spin up an entire Kubernetes cluster per instance of Crossplane and manage its lifecycle. With managed control planes, users can offload that management to Upbound’s software. Now, with Spaces, <strong>users can self-host</strong> many managed control planes in a single Kubernetes cluster running <strong>in their own environment</strong> in addition to the Upbound managed environment that we offer today.</p><p>It’s been incredible to see the ways customers are building platforms powered by managed control planes on Upbound. Variphy is an analytics platform for Unified Communications who’ve found a lot of value building on Upbound. Dylan Moore, a Principal Engineer at Variphy shared, “By using Upbound, we are able to shift complex cloud-native management and deployment to a highly scalable service rather than trying to manage it all ourselves. Upbound's new self-hosted feature gives us more options for where we deploy managed control planes. This helps us be more efficient while allowing flexibility for future deployment requirements.”</p><p>For other users, they want to offload the toil of managing Crossplane at scale to Upbound, but <strong>requirements such as compliance and data sovereignty</strong> mean they need to run control planes in their own environments instead of as SaaS. Spaces let you extend the Upbound platform into your own environment–whether that’s a cloud account in AWS, Azure, or Google Cloud or an on-prem data center–which provides the data separation needed for certain environments.</p><p>Nuno Guedes, cloud compute lead at <a href="https://www.upbound.io/why-upbound/customers/millennium-bcp" target="_blank">Millennium bcp</a> shared, “As Portugal’s largest private bank, we invest in the best technologies to provide us with business value while enabling us to adhere to our compliance standards. Upbound helped us to modernize and standardize our workflow backed by control planes, leading our platform team to save thousands of hours yearly, reducing human error and improving compliance and visibility. Upbound Spaces enables us to reach new scale and service continuity targets, while providing a tailored control plane experience to our internal customers.”</p><p>Business critical customers can begin using the Spaces feature today. Interested users can get trial access to the feature by <a href="https://www.upbound.io/contact" target="_blank">contacting Upbound</a>.</p><h2>The power of managed control planes</h2><p>At the heart of the Upbound platform is managed control planes. Our managed control planes were built from the ground up for performance, scalability, and full lifecycle management:</p><ul><li><strong>Lower operational complexity:</strong> You no longer have to manage individual Kubernetes clusters to use Crossplane, making it practical for organizations to successfully operate tens–or even hundreds–of control planes. Organize your control planes by team, business unit, or application. You choose.</li><li><strong>Highly scalable:</strong> each managed control plane runs a hyper-optimized Crossplane instance, dynamically scaling resource consumption. Scale to arbitrary sizes with your organization by running many of them.</li><li><strong>Git integration out of the box:</strong> Drive the configuration of your control planes from Git without needing to set up additional tools.</li></ul><p>Managed control planes commoditize creating new instances of Crossplane and unlock new ways to build multi-tenant solutions that are highly secure.</p><h2>Fully managed control planes in your own environment</h2><p>Think of an Upbound Space as a self-managed slice of the Upbound platform in your own environment, whether that’s a hyperscale cloud provider or on-prem. Spaces supplement our SaaS service by enabling a new deployment option: to have control planes that run in your preferred hosting environment <i>and</i> Upbound’s own SaaS. Spaces let you:</p><ul><li>Increase scale and cost efficiency by running 50 or more control planes per Kubernetes Cluster instead of just 1.</li><li>Each control plane configuration is fully managed from git</li><li>Integrate with Kubernetes ecosystem tooling to manage the full lifecycle of the control planes.</li></ul><p>All that’s required to bootstrap a Space is a Kubernetes cluster. Popular managed Kubernetes services including Amazon EKS, Google GKE, and Microsoft AKS are fully supported.</p>

<p>After you bootstrap an Upbound Space into a Kubernetes Cluster, you can use the <i>up</i> CLI or interact with its API directly to create managed control planes in your Space. The control planes’ lifecycles are automatically managed by Upbound software running in the Kubernetes cluster.</p><p>Today at launch, we have full support for interacting with a Space using the <i>up</i> CLI. It’s on our roadmap to add the ability to optionally connect a Space back to Upbound’s global SaaS service and Console. This will give you a unified point of control across all your managed control planes. You’ll be able to use the same great control plane explorer views and debugging tools on a managed control plane running in a Space.</p><h2>Declarative Interfaces</h2><p>We’re also introducing new declarative interfaces for interacting with resources in Upbound. Customers can use the new Controlplane API to declaratively create new managed control planes. Users can use popular GitOps tools combined with Git to declaratively create new managed control planes. This capability will first be available for control planes running in a Space and make its way back to the broader Upbound platform soon.</p>

<h2>Enhanced Git integration</h2><p>All managed control planes have a single source of truth for their configuration stored in a Git repository. In Upbound, GitHub integration is available out-of-the-box. This enables platform teams to define their control plane’s API definitions and compositions in Git. Upbound handles packaging and deploying the configuration to the managed control plane.</p><p>We’re expanding the capabilities of Upbound’s built-in Git integration to support deploying all Crossplane resource types. In addition to deploying XRDs and compositions, you can now deploy “runtime” configurations like ProviderConfigs, EnvironmentConfigs, and more. Control planes also now have a ControlplaneSource property that allows them to source configuration from <i>any</i> Git repository–not just GitHub–provided the version control service offers <i>SSH</i>, <i>BearerToken</i>, <i>Basic</i>, or <i>None</i> as auth methods. This enhancement is available for managed control planes when running in a Space today and will make its way back into the broader Upbound platform soon.</p>

<h2>Bring managed control planes into your own Space today</h2><p>You can get started building with Spaces today. Spaces is a feature of Upbound and available in our business critical tier. A trial is available if you contact Upbound. Head over to <a href="https://docs.upbound.io/" target="_blank">Upbound's docs</a> to find the quickstart guides. For pricing and other commercial information, <a href="https://www.upbound.io/contact" target="_blank">contact Upbound sales</a>.</p><p>What to learn more? <a href="https://www.addevent.com/event/dg18380407" target="_blank">Watch the livestream</a> on Thursday, September 7 at 8 AM PT, Unveiling Upbound Spaces: Unlock the Power of Managed Control Planes in Your Own Hosted Environment. Register for <a href="https://www.upbound.io/resources/events/webinars/upbound-spaces-webinar-sep28?utm_campaign=2023_Q3_9.6_GBL_CT_EM_Spaces-Announcement-Email&amp;utm_medium=email&amp;utm_source=webinar&amp;utm_term=webinar" target="_blank">this webinar</a> September 28 @ 8 AM PT to hear more about why running Upbound Spaces might be right for you, especially if you have rigorous compliance and data sovereignty requirements for your internal developer platforms (IDPs) and cloud-native Infrastructure-as-Code (IaC)!</p>

Introducing Spaces: Bring Upbound Managed Control Planes into your own environment

<p>This article was <a href="https://vmblog.com/archive/2024/01/25/upbound-2024-predictions-the-standardization-of-ai-tooling-and-platforms.aspx" target="_blank">previously published on VMblog here</a>.</p><p>With the rise of hot artificial intelligence (AI) tools, every company and person seemed to be experimenting with AI in 2023. There is no going back: AI will continue to play an evolving role.</p><h2>AI and platforms</h2><p>I predict that 2024 will be more about standardization on AI tooling and platforms. It's still the Wild Wild West with AI, but with more maturity will come more standardization and out of the box solutions.&nbsp;</p><p>There will be a great reckoning. We're going to see a lot of the "garbage" that exists today-like low-value UX/UI on top of other platforms (like on top of ChatGPT)-will basically go away. This will happen since it'll get much easier for any platform engineer to run these AI stacks themselves.</p><p>Here's what other members of the Upbound team contemplated regarding the potential impact of AI on businesses this year.</p><h2>AI and candidate-driven job matching</h2><p>Lem Diaz, head of people at Upbound, says:</p><blockquote><p>"I predict that AI will become more prevalent in conversational job matching. Job seekers will begin to see more interactive tools to help them navigate the front end of their job-seeking journeys. This may include things like asking questions about the company's culture, strategy, competitive landscape, tech stack, funding, and other factors and getting a list of matches. These interactive models will enrich a job seeker's understanding of their potential fit and whether or not to pursue a position at a company. This will begin the shift from employer-driven job matching to candidate-driven job matching. It'll still be nascent in 2024, but I believe it will mark the beginning of this shift powered by AI."</p></blockquote><h2>AI makes marketing less tedious</h2><p>Kelsey Havens, head of marketing at Upbound, notes:</p><blockquote><p>"AI will become less of a buzzword in marketing and more of an integrated feature in existing marketing tools and technology. Tedious and repetitive tasks like ad sizing, sequence writing, tagging and task management will be offloaded to AI, giving marketers more mental space and time to be creative and strategic."</p></blockquote><h2>Cloud-native, multi-cloud infrastructure powers AI and ML</h2><p>Tom Anthony, vice president of global sales and customer success at Upbound, adds:&nbsp;</p><blockquote><p>"In 2024, AI will make its way deeper into enterprises. I see this evolving in two ways. Cloud-native, multi-cloud infrastructure will be the fabric that enterprises standardize on to power their AI/ML initiatives. And, "AI-as-a-Service"- the ability for businesses to access AI tools and capabilities without having to invest in their own AI infrastructure or develop expertise-will become the way to drive faster innovation, productivity and cost savings within the enterprise."</p></blockquote><p>It is clear the teams building the underlying infrastructure powering platforms, including AI and ML, have played an increasingly important role in companies. Here at Upbound, we're about making the lives of platform teams easier while helping them meet security, compliance, and policy standards. I'm convinced that creating platforms powered by control planes is a crucial step towards enhanced efficiency-a need that becomes more pressing as AI permeates all facets of the business. I talk more about this in a video on how to <a href="https://resources.upbound.io/community/unleash-the-potential-of-your-aiml-platform-with-a-crossplane-control-plane" target="_blank">unleash the potential of your AI/ML platform with a Crossplane control plane</a>.</p><p>To learn more about how Upbound can enable you to standardize AI/ML infrastructure with reference architectures that are easily deployed and meet security, compliance, and policy requirements, <a href="https://blog.upbound.io/ai-ml/">view this blog here</a>.</p>

The Standardization of AI Tooling and Platforms

<p>Organizations are increasingly tasking a central team to build self-service developer platforms. These platforms are built above AWS, Azure, or GCP with the goal to accelerate app teams’ velocity by enabling them to select from a list of templates that have already baked together a set cloud infrastructure, preconfigured and compliant to an organization’s internal policies and requirements.</p><p>Creating a resource in the cloud can seem simple at first, but quickly the complexity compounds: what VPC, IAM roles, and security groups need to be set up? What does the security team think about opening an API gateway ingress point? What policy concerns are there from naming to access? Suddenly that need for a cloud resource has become many steps across several teams with multi-week approval cycles. Crossplane allows you to define a new abstraction—your own custom API—that provides a simple interface. A central platform team can define and implement those concerns, alleviating app teams from needing to think about these things and cutting down on cross-team approvals in the process.</p><p>Many organizations, such as&nbsp;<a href="https://www.upbound.io/why-upbound/customers/millennium-bcp"><u>Millennium Bank</u></a>, are choosing&nbsp;<a href="https://blog.upbound.io/what-is-crossplane"><u>Crossplane</u></a>&nbsp;to power their self-service platforms. Crossplane is a flexible, Kubernetes-native framework that allows users to build their own abstractions above the cloud. In this post, we will demonstrate how you can get started building a self-service platform above GCP&nbsp;<a href="https://www.upbound.io/register/a"><u>using Upbound</u></a>, which allows you to offload the job of operating Crossplane to Upbound so you can focus on what matters—building your platform!</p><h2>Create your first control plane</h2><p>Get started by&nbsp;<a href="https://www.upbound.io/register/a"><u>creating an account</u></a>&nbsp;on Upbound (which offers a free 30 day trial). Upbound will prompt you to create a new organization and then start your free trial. After you start your free trial, Upbound offers you a few control plane configurations as a starting point. Select the GCP-based configuration.</p>

<p>Complete the remainder of the flow:</p><ul><li>Connect your Upbound account to GitHub so it can clone a repo that holds the source definition of your control plane’s configuration.</li><li>Create your first control plane, which will be configured with the package you selected earlier.</li><li>Connect your control plane to your GCP cloud account via&nbsp;<a href="https://docs.upbound.io/knowledge-base/oidc/"><u>Upbound Identity</u></a>, an OIDC-based auth mechanism. For a step-by-step guide on how to do this, see the&nbsp;<a href="https://docs.upbound.io/quickstart/gcp-deploy/#connect-to-gcp-with-oidc"><u>GCP quickstart</u></a>.</li></ul><p>Once you have finished setup, launch into the Upbound Console.</p>

<h2>Taking inventory</h2><p>At this point, you have:</p><ul><li>Created and configured your first control plane, with a definition that is sourced directly from git.</li><li>It is connected to your GCP account.</li><li>In your git repo, you will find a “starter” API already defined and which has been already installed on your control plane. You can use this API to create and manage PostgreSQL instances on GCP CloudSQL.</li></ul><p>The Upbound Console is currently intended as a first-class experience for platform teams. Since our objective is to build a “self-service” platform, we need app teams to be able to use the API(s) offered by our control plane to provision resources on their own. The next two questions we will tackle are: “What if I want to offer APIs besides databases?” and “How can I enable my app teams to use these APIs?”</p><h2>Expand your control plane’s capabilities</h2><p>We will begin by answering the first question. Our control plane can provision PostgreSQL instances in GCP; what if you want to do more? You can add as many APIs and abstractions as you want to a given control plane. The configuration you picked in the “Get Started with Upbound” flow is meant to be a starting point. You are expected and encouraged to customize your control plane’s configuration by modifying what was scaffolded in your git repo. You can navigate to the git repo which sources your control plane’s configuration by going to the Dashboard &gt; “Configurations” pane in the lower half of the screen &gt; click the git repo hyperlink.</p>

<p>Clone your git repo to your machine. Go to the root of the cloned repo, where you will find a folder called APIs. This is where you can define new APIs with Crossplane–new <a href="https://docs.crossplane.io/v1.12/concepts/#composite-resources" target="_blank">CompositeResourceDefinitions</a> (XRDs) and compositions (implementations for those XRDs). For this blog, we’ll define new APIs to create new Compute Engine instances and the associated network resource required. For brevity, you can copy the <a href="https://github.com/upbound/gcp-example-platform/commit/d4ab9a2955c33d6da765ebf018d22503617396a2" target="_blank">same changes</a> made in <a href="https://github.com/upbound/gcp-example-platform" target="_blank">this repo</a>. The repo contains the following:</p><ul><li>Addition of two XRDs defining new virtual machine and network resources</li><li>Two compositions that implement those resources, respectively</li><li>Moved the postgres resource into its own folder for consistency</li></ul><p>Once those changes have been committed to your synced repository, Upbound will automatically detect the commit and build a new package so you can update the definition installed on your control plane. Go back to your <a href="https://docs.upbound.io/concepts/console/#control-plane-explorer" target="_blank">control plane explorer</a> by clicking the control plane tile on the dashboard. Click the “Update available” button and confirm the update of your control plane’s configuration.</p>

<p><span style="background-color:rgb(255,255,255);color:rgb(3,7,36);font-family:Avenir-Roman, Arial, sans-serif;">Once your control plane’s configuration has been updated, refresh the page to see the new resource card offered by your control plane.</span></p>

<p><span style="background-color:rgb(255,255,255);color:rgb(3,7,36);font-family:Avenir-Roman, Arial, sans-serif;">⚠️ </span><strong>Very important next step:</strong><span style="background-color:rgb(255,255,255);color:rgb(3,7,36);font-family:Avenir-Roman, Arial, sans-serif;"> the quickstart had you create a GCP Service Account configured with role access to CloudSQL (which your control plane assumes when making requests to GCP). You need to make sure to enable role access so the Service Account can create Compute Engine resources, too. After you’ve updated your Service Account’s role access, it's helpful to confirm auth is working by using your </span><a href="https://docs.upbound.io/concepts/ctp-portal/" target="_blank">control plane’s portal</a><span style="background-color:rgb(255,255,255);color:rgb(3,7,36);font-family:Avenir-Roman, Arial, sans-serif;"> to directly CRUD resources.</span></p><h2>Self-service</h2><p>Now that you have a control plane configured to offer and manage the resources you need for your organization, the next step is to build out how you want to enable self-service with it. There are a few ways you could do this.</p><h3>Use a CLI</h3><p>For organizations who are comfortable providing direct access of the control plane directly to their app teams, you can use a CLI directly (such as with kubectl). You can configure a kubeconfig for <a href="https://docs.upbound.io/concepts/managed-control-planes/#connect-directly-to-your-mcp" target="_blank">connecting directly</a> to a control plane and afterwards run `kubectl apply -f claim.yaml` to create resources directly.</p><h3>Use the MCP Connector</h3><p>Upbound ships with a feature called the <a href="https://docs.upbound.io/concepts/control-plane-connector/" target="_blank">managed control plane (MCP) connector</a>. This allows you to connect one or more Kubernetes app clusters (running somewhere in your own cloud environment) to your control plane (running in Upbound’s cloud environment). This feature was built for organizations that have app teams deploy to Kubernetes. It allows teams to bundle resource claims alongside their app deployment configurations and use GitOps to drive communication with your control plane. You can find instructions for how to set this up in this <a href="https://docs.upbound.io/concepts/control-plane-connector/#connecting-to-managed-control-planes" target="_blank">section</a> of the Upbound docs.</p><h3>BYO frontend</h3><p>The MCP Connector approach involves setting up external Kubernetes clusters with supporting tooling to have a functional self-service flow. Another common approach that many organizations are adopting is building their own frontend, either custom-built in-house or by starting from a project such as <a href="https://github.com/backstage/backstage" target="_blank">Backstage</a>. The control planes in Upbound are not opinionated; each control plane <a href="https://docs.upbound.io/concepts/managed-control-planes/#connect-directly-to-your-mcp" target="_blank">offers a Kubernetes API server</a> endpoint that you can communicate with. You will need to connect the frontend for your self-service platform to the API endpoint of your control plane.</p><p>In a future blog post, we will explore the steps for how to implement these methods.</p>

How to Build a Self-Service Platform on Upbound: Day 0

<h2>Introduction</h2><p>Crossplane Compositions are a powerful abstraction layer that enables platform teams to create a custom API for their internal customers to simplify and standardize the infrastructure management process. The Composition authors wrestle with the complexity of cloud infrastructure and at the same time need to ensure a stable and user-friendly API surface for the platform consumers.</p><p>Historically, Compositions were intended to support only very simple resources’ manipulation to allow the API calls to patch and transform the information passed from a Claim or Composite Resource (XR) to the Composition engine. The lack of touring complete language supporting the required transformations resulted in a very verbose YAML files with lots of repetitions, increasing the toil of authoring Compositions.</p><p>With the addition of <a href="https://github.com/crossplane/crossplane/releases/tag/v1.11.0" target="_blank" rel="noopener noreferrer">Compositions Functions</a> in the Crossplane v.1.11 it is possible to use a programming language, like Go or Python (more to come) or in our case a capability of Go called <a href="https://pkg.go.dev/text/template" target="_blank" rel="noopener noreferrer">Go-templating</a> to enrich and transform the authoring of Compositions.</p><p>This blog will guide you through the process of rewriting a simple internal platform Composition in the Go templating style using the <a href="https://marketplace.upbound.io/functions/crossplane-contrib/function-go-templating" target="_blank" rel="noopener noreferrer">go-templating-function</a>.</p><h2>Use Case: Collapsing two Compositions into one</h2><p>Since traditional Composition's engine does not allow using conditionals, there is no way to render one Composition or another using a parameter from a Claim. To illustrate this example, let’s look at the two Compositions that render a GCP Bucket:</p><h3>Standard bucket</h3><p>This very simple composition renders a single GCP bucket</p>

<h3>Versioned Bucket</h3><p>Another Composition needs to be created to accommodate for a versioning setting.</p>

<p>In this case, we have to have two Compositions and need to use a Composition selector in a claim just for this one field.</p><h3>Using go-templating-function</h3><p>The two Compositions can be transformed into just one using the go-templating-function. Follow these steps</p><p>1. Install <a href="https://marketplace.upbound.io/functions/crossplane-contrib/function-go-templating" target="_blank" rel="noopener noreferrer">go-templating-function</a> and <a href="https://marketplace.upbound.io/functions/crossplane-contrib/function-auto-ready" target="_blank" rel="noopener noreferrer">function-auto-ready</a> from the marketplace.</p>

<p>2. Convert the Composition to use the go templating. The initial conversion gives us one to one translation between the standard “Patch and Transform” style Composition and the go-templating style. We are going to improve this design.</p>

<p>Using Composition Functions is very well documented in the <a href="https://docs.crossplane.io/v1.14/concepts/composition-functions/" target="_blank" rel="noopener noreferrer">Crossplane docs</a>. Read the details to learn more about it.</p><p>The current implementation of our Composition renders the <i>non-versioned</i> Bucket. Next steps are to change the XRD and template to accommodate for the versioning field.</p><p>Using the new Crossplane beta trace command is helpful in checking the latest events on the composed resources from the Claim/XR:</p>

<h3>Adding the versioning field</h3><p>1. Modify the XRD to set the versioning field with a boolean flag:</p>

<p>2. Now let’s modify the Composition template to add the conditional for the versioning field.</p>

<p>3. Now we are ready to apply the Claim of the versioning enabled Bucket to the preconfigured GCP project.</p>

<h3>Default values</h3><p>Not all the schema fields are required, and for those the Composition provides default values. location and storageClass are not required in the schema, and the Composition provides default values for those fields when omitted. Let’s add them to the template. Now supplying a Claim without the location and storageClass fields will result in the default values being rendered like so.</p>

<h3>Bringing back PatchSets</h3><p>This specific Composition is very simple, but there are multiple others where the same patch needs to be applied to multiple resources. In the original Composition we have the ownerAndServiceLabels that are applied to all resources that support labels.</p>

<p>With the go-templating-function, it’s easy to define the <i>patchSet-like</i> behavior</p><p>Here is the modified Composition template part with the labels:</p>

<h3>Template Variables</h3><p>One small quality of life improvement is to define variables in the template so there is no need to type .observed.composite.resource.spec.parameters all the time. Here is the final version of the Composition:</p>

<h2>Conclusion</h2><p>In conclusion, the use of the go-templating-function in Compositions authoring significantly simplifies the process of creating and managing resources with varying configurations and makes it easier to manage complex Compositions.</p><p>It allows for the creation of Compositions with conditional fields, reducing the need for multiple compositions for each variation of a resource. The re-introduction of PatchSets-like behavior makes the portability from “Patch and Transform” style Compositions easier. Adding variables scoped to Composite Resource paths makes it easier to reason about the template.</p><p>Additionally, using the new Crossplane CLI functionality to render the resources based on the Claim, Composition and functions and also trace the state of resources is a very helpful development tool. Here’s a tip: running the commands with watch makes the development loop even better.</p>

<p>If you are interested in technical details and design of the go-templating-function, check out the <a href="https://github.com/crossplane/crossplane/pull/4935" target="_blank" rel="noopener noreferrer">go-templating-function one-pager</a>. Big thanks to <a href="https://github.com/ezgidemirel" target="_blank" rel="noopener noreferrer">@ezgidemirel</a> for creating the function! It significantly improves the Compositions authoring process.</p><p>You can find various functions in our <a href="https://marketplace.upbound.io/functions" target="_blank" rel="noopener noreferrer">Marketplace</a> and <a href="https://docs.crossplane.io/knowledge-base/guides/write-a-composition-function-in-go/" target="_blank" rel="noopener noreferrer">create your own</a> to take the Crossplane Compositions to the next level.</p>

Improve Crossplane Compositions Authoring with go-templating-function

<h4>Bringing structure and security to the Kubernetes type system with Crossplane packages.</h4><p>Organizations running on Kubernetes have been steadily moving to a multi-cluster approach. The life-cycle of any individual cluster is relatively short and provisioning new clusters is a common occurrence. There are a variety of factors leading to this shift, including:</p><ol style="list-style-type:decimal;"><li><strong>Availability</strong>: more clusters means that losing any one cluster is not catastrophic.</li><li><strong>Isolation</strong>: while RBAC and Namespaces provide isolation within a single cluster, utilizing multiple clusters allows for cluster-scoped isolation, and physical isolation when necessary.</li><li><strong>Scalability</strong>: depending on the nature of your workloads, the various components that make up a Kubernetes cluster may not scale to accommodate your use case. One such example is the API server struggling to handle a large number of Custom Resource Definitions (CRDs), as detailed in a <a href="https://blog.upbound.io/__GHOST_URL__/scaling-kubernetes-to-thousands-of-crds/">recent post</a> by my colleague <a href="https://twitter.com/internegz" target="_blank" rel="noopener noreferrer">Nic Cope</a>.</li></ol><p>This is not a comprehensive list, but the reasons listed are broadly applicable to many organizations. Mitigating these issues has been compelling enough to drive them to adopt a more complex architecture.</p><p>As organizations have moved into this new world, there has simultaneously been an explosion in the number of CRDs and controllers that are installed in a given cluster. This greatly enhances the capabilities of Kubernetes, but also increases the management burden of replicating the same custom API across many clusters. As a result, interacting with a Kubernetes cluster has become more like writing a program, but without any of the safety or guarantees offered by a modern programming language. In this post, we’ll explore why a lack of API compatibility can lead to dangerous and unpredictable behavior, and how Crossplane packages make it possible to present a verifiably consistent programming interface across multiple clusters.</p><h2>The Perils of an Unknown API</h2><p>What can actually go wrong when API’s diverge across clusters? There are multiple possible outcomes, listed below in order of increasing severity.</p><ol style="list-style-type:decimal;"><li><strong>The “Early Conflict”</strong>: an object being created violates the type definition (CRD) in one cluster, but not the other. This is the best case scenario, but still presents a poor user experience to the developer.</li><li><strong>The “Hope I’m Not On-Call”</strong>: The object is successfully created in each cluster, but only reaches a healthy status in one due to divergence in reconciliation logic, which is decoupled from the definition of the type.</li><li><strong>The “Sleeper Agent”</strong>: The object is successfully created in each cluster, and both reach a healthy status, but reconciliation logic has performed different operations in response.</li></ol><p>In each of these scenarios, a stronger type system could surface these issues prior to creating the object. We are specifically looking for the following attributes:</p><ul><li>Verification that type definitions are equivalent across clusters.</li><li>Verification that reconciliation logic is equivalent across clusters.</li></ul><p><span style="background-color:rgb(255,255,255);color:rgb(3,7,36);font-family:Avenir-Roman, Arial, sans-serif;">Crossplane packages can offer these guarantees, but before exploring how, let’s examine how the Kubernetes type system compares to that of popular programming languages.</span></p><h2>What’s a type?</h2><p>Despite the ever-growing API surface area and the movement to a multi-cluster world, we have not seen a fundamental change in Kubernetes architecture. This is mostly a good thing; the stability and predictability of Kubernetes is a major factor in its widespread adoption. However, when we drastically change how we consume Kubernetes, we must also consider the impact that has on how we configure it. One of the most pronounced differences in configuring a Kubernetes cluster in a multi-cluster world is the scoping of types.</p><p>Kubernetes comes with some built-in types - <code>Pods</code>, <code>Deployments</code>, etc. - and allows you to define your own with CRDs. This is not dissimilar from most programming languages: there are built-in primitive types, such as <code>boolean</code>, <code>integer</code>, etc., but there are also user-defined types, which may be defined via a built-in type, such as a <code>struct</code>.</p><blockquote><p>I <a href="https://danielmangum.com/posts/the-missing-kubernetes-type-system/" target="_blank" rel="noopener noreferrer">have argued</a> that types defined by CRDs directly are, or should be, considered built-in types, but that’s a conversation for another day.</p></blockquote><p>User-defined types are typically scoped - they are identified relative to where they are defined. For example, in the Go programming language I can define a <code>MyType</code> in the <code>cooltypes</code> package. I can also define a <code>MyType</code> in the <code>okaytypes</code> package. The identifiers for these types internally are <code>cooltypes.MyType</code> and <code>okaytypes.MyType</code>, allowing them to be distinguished when consumed.</p>

<p>Zooming out even further, the packages are hosted somewhere on the internet, or at a location on my local machine. This adds further specificity to their scope, guarding against any collisions with another <code>cooltypes</code> package that may be hosted elsewhere. However, with these two types having identical definitions, do we really need to distinguish one from the other? As with many topics in computing: it depends.</p><h2>Defining Equivalence</h2><p>There are two popular mechanisms for defining equality in a type system: <strong>nominal equivalence</strong> and <strong>structural equivalence</strong>. Nominal equivalence defines two types as being the same if they have the same name. For example, in the following example, the types of variables <code>a</code> and <code>b</code> are considered equivalent, but the type of <code>c</code> is not considered equivalent to <code>a</code> or <code>b</code>.</p>

<p>This is because Go, like many languages, uses <strong>nominal equivalence</strong> when evaluating equality of types. Importantly, the <i>name</i> of the type in this context is referring to the fully qualified name. If we compile this program and look at the symbol table, we can see each of the two unique type names defined:</p>

<p>In a language that uses <strong>structural equivalence</strong>, such as TypeScript, OCaml, Haskell, and more, <code>cooltypes.MyType</code> and <code>okaytypes.MyType</code> would be considered equivalent because their members (i.e. <code>Message </code>and <code>ID</code>) are equivalent. In fact, Go uses structural equivalence when evaluating whether a given type implements an <code>interface</code> (i.e. the <i>structure</i> of the methods are equivalent).</p><p>In Go, dependencies are pinned at the <a href="https://pkg.go.dev/cmd/go#hdr-Modules__module_versions__and_more" target="_blank" rel="noopener noreferrer">module level</a> (we are going to skip over the new <a href="https://go.dev/blog/get-familiar-with-workspaces" target="_blank" rel="noopener noreferrer">workspaces</a> flow added in 1.18 for this post). When a new dependency is added, the go.mod file is updated with a reference to the location of the module, and the <a href="https://go.dev/ref/mod#go-sum-files" target="_blank" rel="noopener noreferrer">digest of the content of the package</a> (and any transitive dependencies) is added to the <code>go.sum</code> file. This makes it such that you, or anyone else who is adding code to the module, are able to verify that the type you are creating an instance of is exactly the same as it was last time, barring any change in the <code>go.mod</code> or <code>go.sum</code> files. It also means that we can easily verify whether two types in separate modules are equivalent by ensuring that they are nominally equivalent, then comparing the checksums of their source modules. This is not something that is typically required when writing programs because creating instances of types is typically scoped to a single code base (i.e. module).</p><p>Kubernetes also uses nominal equivalence in its type system, with each type having a name consisting of its Group, Version, and Kind (GVK). However, unlike a statically compiled binary, Kubernetes has types dynamically added and removed at runtime. This means when you are programming against a cluster’s API, the name of the type you are creating an instance of does not fully reflect the type’s identity—the API may have been modified since the last time you interacted with the cluster, and changing the source or content of the type is decoupled from creating new instances of the type. Furthermore, unlike writing a program, there are typically multiple personas involved when configuring and consuming a Kubernetes cluster.</p>

<p>This becomes an even larger problem when multiple clusters are involved, as the constraint of only one type existing for a single GVK is no longer upheld. When we are treating clusters as ephemeral, the inability to verify whether two types with the same name are equivalent becomes an issue, as the semantics of the “language” we are using is changing beneath our feet. A multi-cluster environment is akin to programming across multiple Go modules, where someone else may be changing types, and the only way to ensure consistency is by a literal byte-to-byte comparison.</p><p>In fact, the situation is even more complex than described thus far. While we have primarily been discussing their structure (i.e. definition), types also have associated behavior. In programming languages this is frequently represented as methods that may be invoked imperatively on an instance of the type. In a Kubernetes cluster, associated behavior takes the form of one or more controllers that are invoked based on the declarative life-cycle of the type. Despite these controllers being dependent upon the existence of the types they reconcile. Kubernetes does not natively create a strong association between types and their controllers. Because of this, even comparing the content of a CRD with another with the same GVK does not necessarily indicate equivalence in both structure and behavior.</p><p>In an ideal world, we could verify whether creating an instance of a type in one cluster would result in the same operations being performed as in another cluster.</p><h2>Crossplane Packages and OCI Images</h2><p><a href="https://crossplane.io/docs/v1.9/concepts/packages.html" target="_blank" rel="noopener noreferrer">Crossplane packages</a> (xpkgs) are how new types are added to Kubernetes clusters where Crossplane is installed. Their <a href="https://crossplane.io/docs/v1.9/reference/xpkg.html" target="_blank" rel="noopener noreferrer">specification</a> is a subset of the <a href="https://github.com/opencontainers/image-spec/blob/main/spec.md" target="_blank" rel="noopener noreferrer">OCI image specification</a> (i.e. xpkgs impose additional requirements), which allows for users to take advantage of the vast ecosystem of tooling and distribution.</p><p>OCI images are made up of content and manifests. Traditionally, the content includes filesystem changeset layers (i.e. <i>blobs</i>), which can be extracted to form a complete filesystem for a container at runtime. Manifests tie these layers together, specifying how to obtain the content, as well as the order in which the changesets should be applied. Crossplane takes advantage of this flexible model by assigning significance to content in the blobs, specifically a <code>package.yaml</code> file that exists in the root of the final filesystem. This file is a <a href="https://yaml.org/spec/1.2.2/#92-streams" target="_blank" rel="noopener noreferrer">YAML stream</a> containing package metadata, and all the types that Crossplane should install. Crossplane is indifferent as to the content of the rest of the image.</p>

<p>A Provider or Configuration package can be installed by creating an instance of the <code>Provider</code> or <code>Configuration</code> type in the cluster with a reference to the package image.</p>

<p>When installing, Crossplane first fetches the digest of the package image manifest, and creates either a <code>ProviderRevision</code> or <code>ConfigurationRevision</code>, depending on the package type, with a name constructed from the manifest digest. Next, the JSON content of the manifest is fetched. As mentioned previously, the manifest contains a list of content blobs in the form of descriptors. A descriptor contains the media type, size, and digest of the layer. It may also include annotations, among other fields, with additional information.</p><blockquote><p>Note that both the manifest, and descriptors in the manifest, may include annotations. In this case we are talking about annotations on descriptors in the manifest.</p></blockquote>

<p>If the one of the layer descriptors in the package image manifest has a <code>io.crossplane.xpkg</code>: base annotation, Crossplane only fetches that layer, then extracts the <code>package.yaml</code> file from it. If none of the layer descriptors in the manifest have the annotation, all layers in the manifest are fetched, the full filesystem is constructed, then the <code>package.yaml</code> is extracted from the root directory.</p>

<blockquote><p>Note that this manifest also includes an <code>io.crossplane.xpkg: upbound</code> annotation, which is an xpkg extension. We’ll cover extensions in a future post, but you can read about how to implement your own in the <a href="https://crossplane.io/docs/v1.9/reference/xpkg.html#manifests" target="_blank" rel="noopener noreferrer">specification</a>.</p></blockquote><p>Crossplane parses the contents of the <code>package.yaml</code> file, caches it, then begins installing the package. If the package is a Provider, the <code>package.yaml</code> contains metadata and a stream of CRDs, <a href="https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/#validatingadmissionwebhook" target="_blank" rel="noopener noreferrer">Validating Webhook Configuration</a>, and <a href="https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/#mutatingadmissionwebhook" target="_blank" rel="noopener noreferrer">Mutating Webhook Configurations</a>. If the package is a Configuration, it contains metadata and a stream of Composite Resource Definitions (XRDs) and Compositions. Before applying the objects to the cluster, Crossplane first adds the package revision to a singleton cluster-scoped object of kind <code>Lock</code>. <strong>This object plays a similar role to </strong><code><strong>go.mod</strong></code><strong> and </strong><code><strong>go.sum</strong></code><strong>, defining the packages that are installed in the cluster, the dependency relationships between them, and information about their content.</strong></p>

<p>For example, installing <code>xpkg.upbound.io/upbound/provider-aws:v0.11.0</code> would result in the following entry in the singleton <code>Lock</code>, where the package revision reference, type, and name are defined. The name includes the resolved short digest of the referenced image (<code>e1dde2d0d249</code>).</p>

<p>Installing objects from a package into the cluster consists of first verifying that the given revision can obtain sole control of each object, and, if so, applying them to the cluster with a <a href="https://kubernetes.io/docs/concepts/overview/working-with-objects/owners-dependents/#owner-references-in-object-specifications" target="_blank" rel="noopener noreferrer">controller owner reference</a> to the package revision. This ensures that in a cluster where the Kubernetes API surface is <i>only expanded via Crossplane packages</i>, we can ensure the <i>only one package</i> can ever act on a given type.</p><p>This is especially important with Providers, as installing a Provider not only adds new types to the cluster, but also implements each type’s corresponding functionality in the form of a <code>Deployment</code> that runs controllers to watch and take action based on the lifecycle of instances of the installed types. Unless overridden, Crossplane and the controller image from the same package are the only system components that are given permission on any installed type.</p><h2>(Content) Addressing the Issue</h2><p>A more subtle advantage of being built on top of OCI is the usage of a content-addressable API. Just like any OCI image, Crossplane packages can be installed by digest rather than by tag (e.g. <code>v0.11.0</code>) to ensure that the content that is installed matches what was expected.</p><blockquote><p>If you don’t think pulling by digest is important, I encourage you to <a href="https://www.youtube.com/watch?v=Ve_CFyVYo3M&amp;t=1066s" target="_blank" rel="noopener noreferrer">listen to Jon Johnson</a> describe all of the reasons why you can’t trust a registry when pulling by tag.</p></blockquote>

<p>As seen below, the package revision still uses the short digest in the name (e.g. <code>e1dde2d0d249</code>), but the version corresponds to the full digest of the package image.</p>

<p>Pulling by digest guards against a number of potential issues and vulnerabilities, including tags being updated to point at a different manifest, or a malicious party intercepting requests. However, it does make utilizing semantic versioning for upgrades and dependency resolution more difficult. It would be nice if we could have the convenience of using versions when referencing packages, but ensure that the content is still what we expect, similar to the guarantees we get with a programming language like Go. Fortunately, C<strong>rossplane does not force you to choose between convenience and security</strong>. When installing a Provider or Configuration, users can choose a <code>revisionActivationPolicy</code> of either <code>Automatic</code> or <code>Manual</code>.</p>

<p>If set to <code>Manual</code>, Crossplane resolves the image tag to its digest, but waits until the revision is manually updated from <code>Inactive</code> to <code>Active</code> before continuing with installation. This allows the human to interact with human-readable identifiers (tags), while still relying on the machine to verify the content.</p>

<p>Using this strategy is particularly useful when installing or updating packages across clusters where a consistent API is desired. It also means that if package contents are lost from the cache and the package source has updated the tag to point at a different manifest, Crossplane will wait for approval before rolling to a new revision of the package.</p><blockquote><p>Combining <code>Manual</code> revision activation with <code>packagePullPolicy</code>: Always can be used in interesting ways, such as pinning to a channel tag (e.g. <code>latest</code>) and letting Crossplane create new <code>Inactive</code> revisions any time the tag is updated to point at a new manifest. It is highly recommended that users consult the <code>packagePullPolicy</code> <a href="https://crossplane.io/docs/v1.9/concepts/packages.html#specpackagepullpolicy" target="_blank" rel="noopener noreferrer">documentation</a> before enabling this functionality.</p></blockquote><p><strong>Most importantly, using a content-addressable API and a unit of installation (a package) that contains both type definitions and their corresponding reconciliation logic enables the verifiable equivalence of types across clusters we are seeking.</strong> However, though we do want to couple definition and behavior, sometimes it is advantageous to be able to verify them independently. Let’s take a look at a scenario where this would be useful.</p><h2>Case Study: Multi-Arch Bundled Provider Packages</h2><p>In the <a href="https://github.com/crossplane/crossplane/releases/tag/v1.6.0" target="_blank" rel="noopener noreferrer">v1.6.0 release</a>, Crossplane added support for <a href="https://github.com/crossplane/crossplane/pull/2758" target="_blank" rel="noopener noreferrer">bundled Provider packages</a>. Prior to this change, Provider package metadata was required to include a reference to the controller image for the types included in the package image. Bundled packages allow for omitting this reference, and instead building the package content and controller machinery into a single image. The previous system of decoupling the images had a number of advantages:</p><ol style="list-style-type:decimal;"><li>Crossplane only needed to download and cache the package contents from the package image, while the often larger controller image was only pulled by the node that the <code>Deployment</code> was scheduled to.</li><li>If <i>only</i> the content of the package changed, or only the content of the controllers changed, the digest of the other would remain the same, making it simple to determine if changes occurred to the API surface, the controller logic, or both. This is especially useful when evaluating whether an upgrade includes breaking changes.</li></ol><blockquote><p>Actually performing the verification in (2) proved to be cumbersome, as the digest of the package image would change if its controller image reference changed. The ability to c<a href="https://doc.crds.dev/github.com/crossplane/crossplane/pkg.crossplane.io/ControllerConfig/v1alpha1@v1.9.0#spec-image" target="_blank" rel="noopener noreferrer">hange the controller image reference at runtime</a> via a <code>ControllerConfig</code> somewhat mitigated this issue, but is not a recommended pattern in production settings.</p></blockquote><p>However, the flexibility of the OCI image specification, as well as the formalization of the xpkg specification in the subsequent <a href="https://github.com/crossplane/crossplane/releases/tag/v1.7.0" target="_blank" rel="noopener noreferrer">v1.7.0 release</a>, made it possible to achieve the above benefits while also reducing the complexity of managing multiple images. As previously described, Crossplane packages may annotate a descriptor with the <code>io.crossplane.xpkg</code>: base annotation to indicate that the package manager should only consider that single layer when extracting package contents. In doing so, Crossplane does not increase the amount of data it maintains in its package cache, and, importantly, can easily verify if two packages install an equivalent set of type definitions.</p><p>This functionality is useful in a number of scenarios, such as verifying whether a patch release of a Provider presents the same API as previous patches for the same minor version. One area where this is especially useful is multi-architecture Provider images. Prior to supporting bundled Provider images, there was no need to publish variants of package images for multiple platforms as they only contained YAML content. However, when the controller and package images are bundled into a single image, ensuring that the package content layer is consistent across variants for each platform is critical. It would be highly confusing if installing a package on a <code>linux/arm64</code> host resulted in different types being installed than when installed on a <code>linux/amd64</code> host.</p><p>Multi-architecture OCI images are supported by <a href="https://github.com/opencontainers/image-spec/blob/main/image-index.md" target="_blank" rel="noopener noreferrer">image indexes</a> (sometimes referred to as manifest lists). When you pull a multi-architecture image, your client is first hitting an image index, then resolving to the manifest for your host platform, before finally downloading the layers specified in that platform-specific manifest. The image index for the <code>xpkg.upbound.io/upbound/provider-aws:v0.11.0</code> image we have been using throughout this post looks like this:</p>

<p>As expected, the <code>digest</code> for the <code>linux/amd64</code> image manifest does not match the <code>linux/arm64</code> image manifest <code>digest</code>. However, if we were to examine the layers for each of these manifests, we would see the final two layers are identical:</p>

<p>The ability to both couple the definition and behavior of types (image digest), as well as decouple the verification of the definitions (<code>base</code> layer digest) is critical in ensuring the integrity of multi-architecture images. When pushing multi-architecture packages to Upbound’s registry (<code>xpkg.upbound.io</code>), different digests between the <code>base</code> layers in each variant cause packages to be rejected during verification.</p><h2>Where We’re Going</h2><p>This post outlines ways in which Crossplane enables a more robust Kubernetes type system, but the current functionality only serves as a foundation for future tooling and machinery, similar to how OCI images have helped enable more secure distribution of containerized applications, and as we’ve seen today, any content. A small sample of improvements that can be made to this system includes:</p><ul><li>Supporting installing by type, rather than by package.</li><li>Verifying controller logic on a per-type basis.</li><li>Verifying which types are not equivalent across packages, rather than just that some types are not.</li><li>Supplementing the <code>apiVersion</code> and <code>kind</code> of objects with the digest of the type and its corresponding controller logic.</li><li>Supporting <a href="https://github.com/crossplane/crossplane/issues/3048" target="_blank" rel="noopener noreferrer">package image signature verification</a>.</li></ul><p>If you are interested in joining <a href="https://www.upbound.io/">Upbound</a> in doing our part to enable this future, check out our <a href="https://upbound-www.vercel.app/blog">open roles</a>.</p><p>—</p><p>Upbound is the company that invented the open source Crossplane project. Our mission is to help customers build Internal Cloud Platforms using control planes. Every Upbound customer gets access to Upbound Universal Crossplane (UXP) and Official Providers so they can build their Internal Cloud Platforms with Crossplane confidently. UXP is Upbound’s downstream distribution of Crossplane, and Official Providers are production ready versions of Providers available exclusively to Upbound customers. Both are maintained, tested, and supported by Upbound on behalf of our customers, and are included with an Upbound subscription.</p>

Verifiable Kubernetes API Compatibility

<p>Provisioning infrastructure should never be seen as a barrier to developer velocity. In fact, platform teams often seek to deploy Kubernetes Clusters-as-a-Service to provide developers with a streamlined and efficient environment for deploying and managing their applications. One significant advantage of offering Cluster-as-a-Service is the abstraction of complex infrastructure management.&nbsp;<strong>Upbound makes</strong>&nbsp;<a href="https://www.upbound.io/solutions/cluster-as-a-service"><strong><u>Cluster-as-a-Service</u></strong></a>–and many other self-service deployments of cloud-native infrastructure and applications–<strong>easy for developers</strong>&nbsp;and manageable for the platform team. Let’s explore why it’s essential for any platform team and how it’s done!</p><h2>Complexity is the enemy of developer velocity</h2><p>When dealing with complexity,&nbsp;<strong>abstractions are super important</strong>. A good abstraction will allow ease-of-use while still providing control over the complexity that lies underneath the surface. In the case of Cluster-as-a-Service, developers can focus on writing code and building applications without the need to worry about the underlying cluster setup, scaling, or maintenance. This abstraction reduces the learning curve for developers, allowing them to quickly leverage the power of Kubernetes for container orchestration without becoming Kubernetes experts.</p><p>Furthermore, enabling Clusters-as-a-Service offers scalability and resource optimization. Platform teams can configure and manage the clusters to suit the specific needs of different applications and services, ensuring optimal resource utilization. Developers can request and provision Kubernetes clusters on-demand, which leads to more efficient resource allocation, reducing costs, and minimizing infrastructure waste. By standardizing the deployment process with Upbound, platform teams can also enforce best practices, security policies, and compliance standards across all applications, enhancing the overall stability and security of the platform. In short, Clusters-as-a-Service, when done correctly, means&nbsp;<strong>faster development cycles and more reliable applications</strong>.</p><h2>Control planes offer the best of both worlds</h2><p>Upbound started the&nbsp;<a href="https://www.crossplane.io/"><u>Crossplane project</u></a>&nbsp;and now also offers the&nbsp;<a href="https://www.upbound.io/product/upbound"><u>Upbound platform</u></a>&nbsp;to solve this very problem. By using control planes, developers get a simple API to interact with based on the configurations and definitions that the platform team specify and maintain. Since these configurations can be reconciled when changes are made, the platform team can obfuscate all the gory details while&nbsp;<strong>reducing</strong>&nbsp;<a href="https://blog.upbound.io/true-cost-of-configuration-drift"><strong><u>configuration drift</u></strong></a><strong>&nbsp;and tickets for failed deployments</strong>.</p><h2>Let's try it!</h2><p>Let’s say you are a member of a platform team. One of your goals is to improve the speed and reliability of infrastructure deployments by offering a self-service option for your developers. Perhaps you have the following requirements:</p><ul><li>Create an API for fast, easy deployment</li><li>Multiple deployment options (different cloud providers, regions, business units, etc.)</li><li>Changes must be easily reconciled</li><li>Control of every aspect behind the scenes</li><li>No scripts since they cause configuration drift!</li></ul><p>We will use the Upbound platform to get everything set up in just a few minutes. If you are following along, you will first need to&nbsp;<a href="https://upbound.io/register"><u>sign up for a free trial</u></a>&nbsp;or use an existing Upbound account. If you are signing up for a free trial, you can follow this&nbsp;<a href="https://docs.upbound.io/quickstart/multicloud-deploy/"><u>detailed Quickstart Guide</u></a>&nbsp;if needed. You’ll need to create an organization and add your GitHub credentials at some point. If you’re a returning user, you can get started by heading over to the Configurations section of Console and hit "create new control plane".</p>

<p>You can fill out a few details to clone the example configuration into your Upbound account so you can use it. As part of this process, Upbound automatically creates a Git repository in GitHub on your behalf. After Upbound creates your repository, it watches for new commits to the main branch. For every commit, Upbound automatically attempts to build the Configuration package for you. If it’s successful, you can then upgrade the definition on your managed control planes.</p>

<p>After a few seconds the configuration will be cloned and a window will appear prompting you to create a new control plane to install this configuration onto.</p>

<p>That’s it! Sit back and watch as your new control plane and APIs are configured. You can review the information about the control plane that you just created.</p>

<p>The control plane will take only 5–10 minutes to provision. While you are waiting, be sure to check out the documentation to learn more about control planes and how to interact with them.</p>

<p>After a few minutes, your control plane with the example configuration will be ready to use&nbsp;<strong>without writing a single script</strong>. As you can see,&nbsp;<strong>we now have Cluster-as-a-Service APIs</strong>&nbsp;created for AWS, Azure, and GCP.</p>

<p>Of course, before you can start rolling out infrastructure you need to configure access to whichever cloud provider you will be using. To do that, click on the Providers tab, choose one of the cloud providers and create a ProviderConfig. This will be different for each cloud service so we have put together&nbsp;<a href="https://docs.upbound.io/quickstart/multicloud-deploy/#authenticate-with-providers"><u>some documentation</u></a>&nbsp;that covers the process.</p>

<p>The next step would be to enable these APIs to be used in whatever developer frontend you are using. This could be Backstage, Port, or anything else. Just as an example, you could click on “Self Service” and choose Control Plane Portal.</p>

<p>Click on one of the APIs on the left of the screen and then click the “Create New” button. This will bring up all the required fields that were defined in this example that are necessary to deploy a new cluster. However, outside of this demonstration, this could be whatever you require. For example: namespace, region, number of required nodes, instance size, etc. The beauty of a control plane is that it can be defined however you need it to be. You are not locked into someone else's vision for how things should work.</p>

<p>We will stop here as we have met all the objectives so far:</p><ul><li>Setup an API for fast, easy deployment</li><li>Multiple deployment options (different cloud providers, regions, business units, etc.)</li><li>Changes must be easily reconciled</li><li>Control of every aspect behind the scenes</li><li>No scripts since they cause configuration drift!</li></ul><p>However, there is so much more you can do with Upbound and control planes. For a more complete look at setting up an internal developer platform in your organization be sure to check out&nbsp;<a href="https://blog.upbound.io/7-core-elements-of-an-internal-developer-platform" target="_blank">this blog</a>. Don’t forget that you can&nbsp;<a href="https://www.upbound.io/register/a" target="_blank">try this for free right now for 30 days</a>&nbsp;and be sure to contact us with any questions you might have.</p>

Using Upbound to Roll Out Cluster-as-a-Service to Your Developers

<h4>Upbound, creator of Crossplane, is now a certified VMware Tanzu Application Platform partner</h4><p>The developer experience can mean a lot of things. To level-set and simplify, let's go with the notion that it is all of the feelings software developers have while producing and publishing their code. Mostly, it measures their level of aggravation during the process. In other words, how many interruptions in their work flow they have to endure. In today's environment, where there is the desire to move fast and securely, the developer experience is usually disappointing.</p><p>Today, Upbound and VMware are working together for the better and celebrating a new partnership for the benefit of the Crossplane community and enterprise users. Let's see how Upbound and VMware are teaming up to build a completely satisfying developer experience.</p><h2>About Upbound &amp; UXP</h2><p><a href="https://www.upbound.io/" target="_blank">Upbound</a> is the company behind open-source CNCF <a href="https://www.crossplane.io/" target="_blank">Crossplane</a>. Crossplane utilizes the Kubernetes resource model to expose reusable APIs to build cloud services. Essentially, it is a framework to create control planes to deliver a standardized experience to consume cloud services. <a href="https://www.upbound.io/product/universal-crossplane" target="_blank">Universal Crossplane (UXP)</a>, is Upbound's fully tested and supported version of OSS Crossplane.</p><h2>About VMware Tanzu Application Platform</h2><p>VMware <a href="https://tanzu.vmware.com/application-platform" target="_blank">Tanzu Application Platform</a> is an end-to-end integrated platform solution that enables companies to build and deploy more software, more quickly and securely, through a rich set of developer tooling and a pre-paved path to production. Teams can build golden paths to production from the first lines of code, to delivery, to production. VMware Tanzu Application Platform is modular and flexible, with the ability to plug in multiple solutions into your organization’s custom golden paths.</p><h2>Before Tanzu Application Platform Met UXP</h2><p>Individually, UXP and Tanzu Application Platform help improve the developer experience. UXP provides APIs to build standardized cloud services and Tanzu Application Platform enables all the appropriate tools to build and ship software to be plugged in. However, until now, there was still aggravation when a developer needed to build a supporting service to test or deliver their code. Not only did they need to know how to build their services, they also needed to know the approved ways and how to connect the services to their application. Unfortunately, this aggravation was what caused a lot of ad-hoc creation of services leading to cloud sprawl—an increase in potential security holes and a decrease in overall quality.</p><h2>The Integrated Experience</h2><p>Today VMware has added Upbound as a Tanzu Application Platform partner. With UXP and the <a href="https://tanzu.vmware.com/content/blog/tanzu-application-platform-1-5-ga" target="_blank">release of Tanzu Application Platform 1.5</a>, developers can now utilize the UXP powered APIs within their existing workflow with no additional headaches. Cloud services can be automatically provisioned and connected to their code in a platform-team approved way automatically! Current UXP or Crossplane users will be able to utilize any CompositeResourceDefinition (XRD) created by UXP in Tanzu Application Platform workflows.</p><blockquote><p>“We’re excited to be partnering with Upbound and build on the innovation provided by Crossplane as we continue to meet customers where they are in their platform journey. VMware Tanzu Application Platform and Upbound can support Platform Engineering and Application Development teams with a seamless experience to provision and connect to the data services they need. Together, we’re able to help customers embrace the OSS tools and cloud services of their choice, while providing enterprise support and security best practices," Samuel des Baux, Director Tanzu Application Platform – Cloud and Services, VMware.</p></blockquote><p>Currently, in order to use both UXP and Tanzu Application Platform together, they must be installed and managed on the same Kubernetes cluster. However, with the release of <a href="https://blog.upbound.io/upbound-general-availability">Upbound and managed control planes</a>, Tanzu Application Platform will be able to consume APIs available on Upbound.</p><h2>What’s Next?</h2><p>To find out more about the UXP integration with Tanzu Application Platform check out this joint webinar, <a href="https://resources.upbound.io/all-content/achieve-a-fully-automated-devops-experience-with-universal-crossplane-and-vmware-tanzu-application-platform" target="_blank">"Achieve a Fully Automated DevOps Experience with Universal Crossplane and VMware Tanzu Application Platform"</a> or learn more in the <a href="https://docs.vmware.com/en/VMware-Tanzu-Application-Platform/1.5/tap/crossplane-about.html" target="_blank">tech docs</a>.</p><p>If you’re curious about Upbound and what we can offer, try our latest product offering, Upbound, for free. Enable SRE teams with a more efficient workflow and the building blocks of platform engineering by setting up your first cloud platform powered by Upbound’s managed control planes. <a href="https://upbound.io/register?ref=blog.upbound.io" target="_blank">Sign up for your 30 day free trial here</a>!</p>

Upbound and VMware Tanzu Application Platform Partner to Deliver a Completely Satisfying Developer Experience

<p>In this sprint’s product update, we cover improvements to the Marketplace that will make it much easier to find the Crossplane content you need, the arrival of support for the managed control plane connector in a Spaces-managed control plane, and some additional features now in the <code>up</code> cli to make interacting with control planes easier.</p><h2>Navigating the Upbound Marketplace faster</h2><p>Browsing available providers, configurations, and API definitions just got a lot faster. Users should expect to see up to 50x faster page load times in the <a href="https://marketplace.upbound.io/" target="_blank">Upbound Marketplace</a>, thanks to some recent changes we pushed. Whether you’re browsing the list of <a href="https://marketplace.upbound.io/providers" target="_blank">Crossplane providers</a> or exploring the spec of a managed resource, it should be considerably faster to get where you need to go in the Marketplace.</p><p>We’ve also updated how search results in Marketplace to allow for more content to be shown on screen, depending on the size of device viewing the Marketplace.</p>

<h2>Managed control plane connector now supported in an Upbound Spaces</h2><p>Spaces is a recent feature of Upbound that allows customers to operate self-hosted environments for Upbound’s managed Crossplane control planes. When a managed control plane runs in a Space, you can now use the same MCP connector that we support in our SaaS environment.</p><p>The MCP connector is one of many interfaces you can use to interact with a Spaces control plane. It’s specifically designed to help you connect a Kubernetes app cluster to a managed control plane, so your dev team can bundle their app helm chart (destined for their app cluster) alongside Crossplane claims (destined for a managed control plane) and deploy it to their single app cluster context.</p><p>Learn more about how to set up the MCP connector in a Spaces-managed control plane by reading <a href="https://docs.upbound.io/mcp/ctp-connector/" target="_blank">the documentation</a>.</p>

<h2>New capabilities in up CLI</h2><p>We’ve made it simpler to connect and disconnect from a control plane’s API server. <code>up ctp connect</code> was re-introduced, with different experience. Invoking the command will now connect your kubectl directly to the target control plane. Any kubectl commands run in that terminal will be executed against the target control plane. To return to the previous context stored in your kubeconfig, simply run <code>up ctp disconnect</code>. Learn about this more in the latest up CLI <a href="https://docs.upbound.io/reference/cli/rel-notes/#v0210" target="_blank">release notes</a>.</p>

<h2>Improvements &amp; Fixes</h2><ul><li>Improved the control plane dashboard and configuration list by making the table headers sticky, allowing you to scroll through longer lists without losing context</li><li>Resolved an issue affecting Configurations with dependencies on the latest releases of family providers that prevented users from creating new ProviderConfigs via the UI</li><li>The console now shows UI create hints on the control plane and configuration list screens for organizations with no control planes or configurations created.</li><li>We fixed the background color of the main navigation tabs to match the page.</li><li>The UI tabs in a managed control plane dashboard now link via url hash. For example, the settings tab renders as&nbsp;<code>console.upbound.io/org/controlPlanes/org:ctp-name#resources</code>)</li></ul><p>If you have feedback, a feature you’d like to see, or want to chat with the Product team, you can find us in the <a href="https://crossplane.slack.com/archives/C01TRKD4623" target="_blank">#Upbound channel</a> of the Crossplane community slack.</p><h2>Want to Learn More?</h2><p>To understand the full picture of Spaces, check out this webinar, "<a href="https://resources.upbound.io/all-content/managed-crossplane-in-your-own-environment-with-upbound-spaces" target="_blank">Managed Crossplane in your own environment with Upbound Spaces</a>" that I put together to give a great overview and detailed rundown.</p>

<p>On behalf of the Upbound team,<br>Craig D Wilhite</p>

New from Upbound: Finding Content Faster in the Marketplace, MCP Connector Comes to Spaces

<p>Remote work exploded during the pandemic, with 35% of US workers having the option of working fully remotely in 2022 according to <a href="https://www.mckinsey.com/industries/real-estate/our-insights/americans-are-embracing-flexible-work-and-they-want-more-of-it" target="_blank" rel="noopener noreferrer">McKinsey</a>. Formerly office-based companies have had to shift their ways of collaborating and working to deal with this new normal, with varying degrees of success and hand-wringing. Some companies have given up on this whole remote work thing and are demanding a return to the office… and many of the product managers I know are celebrating this.</p><p>Product people have had a particularly hard time making the adjustment to being suddenly remote.</p><blockquote><p>Product management is about leading through influence, making it a communication-heavy role where building relationships and driving alignment is key.</p></blockquote><p>We spend our time understanding our users, translating the company’s vision into actionable strategies, and representing the customer in the decisions our teams make while building the product. For the typical PM, that responsibility has historically resulted in meetings—meetings with their devs, design, the field, other PMs, customers and users… all day, every day.</p><p>Defaulting to meetings as your primary communication method is a PM antipattern that has been laid bare thanks to the rise of remote work. Trying to replicate the all-day-wall-of-meetings approach in an office-free company is like playing product management on hard mode. Besides being a recipe for the dreaded <a href="https://hbr.org/2020/04/how-to-combat-zoom-fatigue" target="_blank" rel="noopener noreferrer">Zoom fatigue</a> and eventual burnout, it’s a missed opportunity to unlock the full potential and flexibility of being on a remote dev team. Add in any kind of geographic complexity, like a team spread across multiple time zones, and your insistence on getting everyone onto a video call will result in bottlenecks, irritated engineers, and someone—perhaps the same someone, repeatedly—having to join meetings outside of their working hours if they want to be included in the process. <strong>But what if I told you that there are product managers out there working on fully remote teams </strong><i><strong>and loving it?</strong></i></p><p>As a PM at Upbound, I’m helping platform teams around the world to <a href="https://www.upbound.io/whitepaper/platform-teams-solution?utm_campaign=rise-above-the-clouds&amp;utm_medium=linkedin&amp;utm_source=organic-social&amp;utm_term=whitepaper&amp;utm_content=platform-for-platform-teams">build internal cloud platforms</a> that help them deliver amazing experiences to their own end users. We’re a global remote-first company—as of the writing of this post, Upbounders <a href="https://www.upbound.io/about">work fully remotely</a> from 10 countries on five continents.</p><p><strong>A huge advantage to working in a fully remote company is that you can form the best team around the problems you’re trying to solve and the value you’re going to deliver to your customers.</strong> For my squad, optimizing for skills and experience rather than location has resulted in a team spanning 4 countries from the Pacific (UTC-8) to Central Europe (UTC+1) time zones. Collaborating successfully with such a wide geographic spread is impossible if you’re a PM with a meeting addiction. If you’re stuck in the PM meeting trap, read on!</p><h2>Tips for getting out of the remote PM meeting trap</h2><p>Getting out of the meeting trap as a PM means rethinking the role that meetings play in your communication. It requires you to do the up-front work of establishing trust and working relationships remotely, being empathetic to ensure flexibility is distributed fairly within the team, and determining the right balance between asynchronous and synchronous collaboration for your context.</p><h3>Establish your foundation of trust</h3><p>Building up relationships based on trust and mutual respect is essential in any PM role, but requires a more intentional approach in a fully remote setting than you might have taken as an office worker.</p><h4>My tips for building trust with your team as a remote PM:</h4><ol style="list-style-type:decimal;"><li>Be sure you can fulfill your role as the “voice of the customer” on-demand and asynchronously through documentation</li><li>Create clear and transparent records of the decisions you make and rationale behind them</li><li>Dedicate time for casual one-on-one conversations with your teammates and peers</li></ol><p>Trust as a PM is earned by being a reliable subject matter expert on your customers and their problems as well as the data surrounding usage of your products. The classic approach for sharing that knowledge might involve throwing together some slides and spreadsheets that you walk people through (perhaps more than once) to make the customer problem and priority understandable. In a fully remote setting, it’s essential that you document that customer knowledge in a way that is more digestible and discoverable so that your squad can self-service access to the insights and understanding that you’ve developed by talking to users and researching the market. One lo-fi way to accomplish this is to put together a “cheat sheet” before kicking off feature work that lays out who you’re building for, what problem they’re trying to solve, why solving that problem is important to users and the business, and your supporting evidence (i.e. hard data and research snippets).</p><p>Related to this is being explicit and clear with the criteria you’re using to make product decisions—and getting that documented in a way that’s visible to the team async. This can be as simple as commenting on a Github issue with the rationale for a decision that was made live during daily standup, or as complex as building out a scorecard for prioritizing feature work. If product decisions are a black box for your team, async collaboration falls apart.</p><p>For me, building up trust is also rooted in dedicating time to get to know my teammates as whole people. Developing camaraderie in a fully remote setting can be tricky if your schedule as a PM is 100% dedicated to “official” meetings—the office apologists are onto something when they mourn the loss of hallway chats and coffee machine banter. I make a habit of throwing 20-minute casual coffee chats on people’s calendars once a month or so, being clear in the invitation that the goal is to get to know one another better rather than discuss a specific agenda. I’m lucky at Upbound to have regular opportunities to meet up IRL with others from the company—be it by traveling to a conference like KubeCon or a company event. If you have similar travel opportunities in your remote PM role, be sure to take full advantage of them!</p><h3>Strive for fairly-distributed flexibility and an economy of meetings</h3><p><strong>Flexibility</strong> is one of the biggest reasons I choose to work in a fully remote organization—from the flexibility to choose where I live, to avoiding time wasted on commutes, to setting custom work hours based on my needs. As a PM on a global remote team, it’s especially important to me that the decisions I make do not result in my teammates being unable to enjoy that same flexibility. In other words, if I am scheduling a meeting, I stop to consider how it will impact the people I am inviting and their own work-life balance. This might seem obvious, but in other companies, I’ve had colleagues consistently have to decide whether to join meetings outside of their working hours or risk being excluded from important discussions and decisions (especially when someone’s time zone or schedule made them the “odd one out”).</p><p>Our squad’s budget for synchronous communication with the full team hovers around 10 hours per 2-week sprint due to time zone differences. Having such an explicit limit on the amount of time we can all sit together on a call is a healthy mechanism for preventing meeting sprawl. Just like we prioritize our backlog of issues, we prioritize our backlog of conversations and collaboration points to try to ensure meeting time is spent wisely. Rather than defaulting to a meeting, communication and alignment might happen around a document, a Github issue, or an online whiteboard.</p><p>In a typical sprint, we spend 5 hours on squad rituals – daily standups, story writing/tech breakdowns, retros, and sprint planning. These timeboxed touchpoints form the heart of our squad operations and keep the tactical day-to-day work flowing when combined with all of the async communication happening within Github or Slack. The remaining budget, if we decide to spend it, is available for group discussions to brainstorm and share ideas for the future, align design and engineering approaches in a dynamic way, and so on. Our meetings are intentional, with a specific agenda and outcome expected, and we record them to share with others whenever possible.</p><h3>In the end, working remotely will make you a better, more empowered product manager</h3><p>Succeeding as a product manager in a fully remote company means breaking free of the PM meeting trap. Working on a remote team opens up new possibilities to build amazing software together with teammates and users from around the world, but it requires a different mindset than the one that served you well back in your office days. From recording product walkthroughs to creating self-serve knowledge bases of customer insights, remote product people are forever finding ways to serve as the voice of the customer even when they can’t be present for every discussion. Whether your foray into remote product management is temporary or permanent, reducing your meeting footprint creates new opportunities to build up your communication muscles while freeing up valuable time for strategic thinking and learning from your users.</p>

Succeeding as a Fully Remote Product Manager Means Breaking Your Meeting Dependence

<h2>Overview</h2><p>Today, we're announcing the Azure Reference Platform for Kubernetes + Data Services configuration package release in <a href="https://cloud.upbound.io/registry/upbound/platform-ref-azure">Upbound Registry</a>. The Upbound Platform Reference Architectures (<a href="https://github.com/upbound/platform-ref-aws" target="_blank" rel="noopener noreferrer">AWS</a>, <a href="https://github.com/upbound/platform-ref-gcp" target="_blank" rel="noopener noreferrer">GCP</a>, <a href="https://github.com/upbound/platform-ref-azure" target="_blank" rel="noopener noreferrer">Azure</a>, <a href="https://github.com/upbound/platform-ref-cloud-native" target="_blank" rel="noopener noreferrer">Cloud-Native</a>, and <a href="https://github.com/upbound/platform-ref-multi-k8s" target="_blank" rel="noopener noreferrer">Multi-K8s</a>) are foundations, intended to help accelerate understanding and application of Crossplane within your organization. We love to see how people build on these references, and encourage you to customize and publish your own configuration packages to the Upbound Registry.</p><p>Crossplane enables platform teams to offer simple and flexible self-service interfaces (via the Kubernetes API server) to provision any combination of API fronted resources, without sacrificing governance or control. The Azure Reference Platform for Kubernetes + Data Services is a great example of this. With this configuration, your platform team can offer a pair of resource kinds called "Cluster" and "PostgreSQLInstance" that allow your developers to provision the infrastructure they need in a self-service manner, while only having access to a limited set of configuration values.</p><p>With those limited values, we enable the requestor to provision a fully functional AKS cluster, with supporting infrastructure (e.g. Resource Group, Virtual Network, Subnet, and Azure DB) while the platform team maintains control of finer-grained attributes like Region, Network CIDR, and VM shapes. We also enforce Service Endpoint configuration to ensure the Azure DB instances are securely connected to the AKS nodes. As a bonus, we're using the Helm provider to include Prometheus in the provisioned cluster.</p><blockquote><p>Note: Because Crossplane providers operate as Kubernetes controllers, the deployed infrastructure is continuously reconciled. If someone changes a value in the cloud provider portal, Crossplane will automatically revert it to our intended state.</p></blockquote><p>The following manifests represent all that is required from a requestor. For Cluster, we allow node count, node size (This is transformed via a Composition from small/medium/large to predefined Azure VM types, the requestor cannot override those mappings), and Prometheus Helm chart version.</p>

<p>For PostgreSQLInstance, we expose only provisioned storage size. We additionally require a reference to the AKS cluster ID to correlate the secure networking configuration. Notice there are no inputs required for network provisioning. Those details are abstracted from the requestor and entirely controlled by the platform team.</p>

<blockquote><p>Note: In the two sample manifests, you'll notice the 'writeConnectionSecretToRef' block. For resources that generate sensitive connection details, Crossplane will automatically write these to Kubernetes secrets. This block enables the requestor to specify a predictable name for future reference.</p></blockquote><p>Using this Configuration, platform teams (and the application teams they support) can easily deploy Kubernetes clusters with securely connected database services to Azure Cloud, unlocking a lot of velocity and flexibility to deliver their software to multiple environments.</p><h2>Summary</h2><p>In the Kubernetes + Data Services Deployments on Azure platform reference architecture, we are offering requestors a semi-flexible self-service API (via the Kubernetes kubeapi). The platform team maintains control over the node sizes/mappings, regions, networking, and other supporting infrastructure configuration, while developers get a self-service experience that increases their deployment velocity. We encourage you to create an Upbound Cloud account and try it out for yourself with this <a href="https://github.com/upbound/platform-ref-azure" target="_blank" rel="noopener noreferrer">quick start</a>.</p>

Kubernetes + Data Services Deployments on Azure with Upbound Cloud

<p>At <a href="https://akuity.io/" target="_blank">Akuity</a>, we are <a href="https://www.upbound.io/why-upbound/partners" target="_blank">collaborating with Upbound</a> to streamline our infrastructure using <a href="https://akuity.io/what-is-argo/" target="_blank">Argo CD</a> and <a href="https://www.crossplane.io/" target="_blank">Crossplane</a>. This blog post will explore how Argo CD users can implement Crossplane to manage infrastructure resources beyond Kubernetes to simplify infrastructure management and create a clear sense of the dependencies for an application.</p><h2>Introduction</h2><p>Argo CD is a prevalent tool for implementing GitOps, the practice of declaratively storing the desired state with immutable versions and using tooling to reconcile the live state, in Kubernetes. Argo CD enables GitOps by implementing the Application resource, which provides a declarative approach to managing config management tooling for Kubernetes resources (e.g., Helm, Kustomize) and a controller for handling the reconciliation loop for syncing the resources into the cluster (i.e., updating the live state to match the desired state).</p><p>A basic example of an Application looks like this:</p>

<p>This 'guestbook' Application will create a Deployment and a Service in Kubernetes based on the manifests in the `guestbook` folder of the `akuity/argo-cd-crossplane` repository.</p>

<p>What is not represented by the manifests in the repository is that the fictitious guestbook application relies on a bucket for storing images uploaded by guests. But since the bucket isn’t a Kubernetes resource, Argo CD can’t manage it without the help of another tool.</p><h2>Before a Universal Control Plane</h2><p>The existing process to create the storage bucket is to log in to the GCP console and create a new bucket, specifying its name, location, storage class, access controls, configure versioning, and the lifecycle rules. Once the bucket is created, you need to configure the application to use it.</p>

<p>This whole process is done once, with none of it represented declaratively. So the next time the application is deployed to a new environment, another bucket must be created manually. Hopefully, the same person is creating it, or the documentation is good. Otherwise, you will waste time figuring out what settings are needed for the bucket (probably by comparing it to the existing one).</p><p>The downside of this approach is that it requires the application developers to have direct access to GCP or rely on an operator to create it, and it’s a manual imperative process prone to human error. For example, the bucket access could be easily misconfigured, compromising the bucket's contents. Additionally, if the bucket configuration needs to be updated, the change must be made manually, potentially leading to inconsistencies between the desired and actual state.</p><h2>Introducing Crossplane</h2><p>Crossplane is an open-source Kubernetes-native framework for building cloud-native control planes using a consistent API without writing code. Crossplane facilitates creating resources on various providers, including GCP, AWS, and Azure, using configurations defined as Kubernetes resources, making it easier to manage infrastructure using familiar Kubernetes tooling.</p><p><a href="https://www.upbound.io/" target="_blank">Upbound</a> created Crossplane to democratize control plane technology for all and continues to drive it forward as maintainers and contributors. Upbound also offers <a href="https://www.upbound.io/product/upbound" target="_blank">commercial products</a> and <a href="https://www.upbound.io/product/universal-crossplane" target="_blank">support</a> for those organizations that want to take Crossplane even further.</p><h3>Using Crossplane to Manage the Storage Bucket</h3><p>To use Crossplane to manage the storage bucket, you must install <a href="https://marketplace.upbound.io/providers/upbound/provider-gcp/latest" target="_blank">the GCP provider</a> (`provider-gcp`), which requires authentication to a service account with permission to manage the resources on the provider. This can be done with a JSON key from a service account in GCP or using Workload Identity to authenticate based on a service account in Kubernetes.</p><p>Creating the Crossplane GCP provider in Kubernetes will also introduce all of the Custom Resource Definitions (CRDs) representing resources on GCP. Including one for managing storage buckets, `Bucket`. <a href="https://marketplace.upbound.io/providers/upbound/provider-gcp/latest/resources/storage.gcp.upbound.io/Bucket" target="_blank">The CRD</a> defines the structure of the resource and the parameters that can be configured, such as the bucket name, location, and access credentials.</p><p>A basic example looks like this:</p>

<h3>Adding the Storage Bucket to the Application</h3><p>To add the storage bucket to the Kubernetes resources managed by the Application, you need to add the `Bucket` manifest to the `guestbook` folder of the `akuity/argo-cd-crossplane` repository. Argo CD will then deploy the new `Bucket` custom resource (CR) alongside the existing Kubernetes resources, ensuring that the live state of the cluster matches the state described in the Git repository. Once Argo CD creates the CR in the cluster, Crossplane will use the provider to create the corresponding resource on GCP using all of the configurations from the `Bucket` CR.</p>

<p>This approach provides several benefits, including:</p><ul><li>creating visibility into how the bucket is configured,</li><li>reducing the likelihood of human error, and</li><li>making it easier to manage infrastructure alongside application resources using Kubernetes tooling.</li></ul><h2>Conclusion</h2><p>Using Argo CD and Crossplane together provides a powerful toolset for managing infrastructure and application dependencies in Kubernetes environments. Defining infrastructure as Kubernetes resources makes it easier to manage the entire stack using GitOps practices and Kubernetes tooling.</p><p>Resources that were once a distant and implied dependency of the application can now be managed declaratively alongside the Kubernetes-native resources. The complete configuration for the Bucket is transparent and easily reproducible. Creating a simple mechanism for deploying new instances of the application in any Kubernetes environment with Crossplane and Argo CD set up.</p>

Argo CD and Crossplane for Managing the Whole Application Stack

<p>Welcome to the latest Upbound Resource Roundup! Last month, we delved into a spectrum of tailored solutions. This month, we’re going to dive into a few more that will help you streamline platform team operations with Upbound's versatile capabilities powered by Crossplane. Here are some resources that can help you stay ahead in the ever-changing cloud-native landscape and achieve cloud native excellence.</p><h2>New Resources From Upbound</h2><p><a href="https://blog.upbound.io/upbound-announces-new-crossplane-reference-implementation-for-cnoe">Upbound Announces New Crossplane Reference Implementation for CNOE</a> – Upbound announced a practical application of Crossplane's capabilities through a new reference implementation for Cloud Native Operational Excellence (CNOE)! Now, you can start spinning up more resources within this powerful stack, customizing it to fit your specific needs.</p><p><a href="https://blog.upbound.io/k8s-is-not-the-platform-or-is-it-and-we-all-misunderstood">K8s Is Not the Platform – Or Is It and We All Misunderstood?</a> – Stefan Schimanski, a long-time Kubernetes expert, takes a look at Kubernetes and Crossplane as the core of a developer platform beyond container orchestration.</p><p><a href="https://blog.upbound.io/ai-ml?utm_medium=email&amp;utm_source=email-newsletter&amp;utm_term=blog">AI Made Easy: Your Guide to Fast and Reliable AI/ML Workload Deployment</a> – The AI and ML landscape of software, tools, and techniques is growing larger by the hour. Let’s take a look at how you can build a platform to get ahead in the AI/ML race.</p><p><a href="https://blog.upbound.io/2023-11-07-product-updates?utm_medium=email&amp;utm_source=email-newsletter&amp;utm_term=blog">New from Upbound: Dashboard Filtering &amp; Control Plane Explorer Flow Enhancements</a> – See new enhancements in the Upbound Console to share – from improvements to the user flow in the control plane explorer, to new filtering functionality in the control plane dashboard, to linked references in the resource details view to support debugging.</p><h2>Crossplane Highlights</h2><p><a href="https://blog.crossplane.io/crossplane-v1-14/" target="_blank" rel="noopener noreferrer">Crossplane 1.14 is here</a> – This month, the latest version of the Crossplane open source control plane framework was released. It is packed full of big features that will make Platform Engineers more productive and effective in building control planes to power their infrastructure.</p><p><a href="https://blog.crossplane.io/design-compositions-for-your-organizational-needs-with-crossplane/" target="_blank" rel="noopener noreferrer">Design Compositions For Your Organizational Needs With Crossplane</a> – This article is for curious individuals with specific requirements for building and operating cloud infrastructure efficiently along corporate guidelines, governance, security, and the breadth of systems that can be configured and resource lifecycle managed with Crossplane.</p><p><a href="https://blog.crossplane.io/platform-building-with-crossplane-your-questions-answered-at-control-plane-day-with-crossplane/" target="_blank" rel="noopener noreferrer">Platform Building with Crossplane: Your Questions Answered at Control Plane Day with Crossplane</a> – A few months ago Crossplane hosted Control Plane Day with Crossplane event. The event was a smashing success with hundreds of attendees and some outstanding talks from both maintainers and users. Here are some answers to all of the most-asked questions from the event.</p><h2>What’s Next?</h2><p>Interested in seeing these updates in action? Our team is on standby to walk you through it! <a href="https://www.upbound.io/request-a-demo">Request to have a live demo with them now</a>.</p>

Upbound Resource Roundup: November 2023

<p>Imagine if you could create, for people in your company, a platform that would provide them with the same experience they have when working with AWS, Google Cloud, Azure, or any other public cloud provider. Imagine if there would be a service for everything they do.</p><p>Do you need a database that works exactly as we expect it to work in this company with all the security, backup, compliance, and other policies we have?</p><p>Well...There is a service for that. Do you need to run a backend application in Kubernetes? There is a service for that as well. Do you need a Kubernetes cluster itself? There is yet another service for that.</p><p>All you have to do is define a simple manifest that contains only the things you care about and abstracts all the unavoidable complexity. From there on, all you have to do is submit that manifest, the desired state, to the control plane API and observe the actual state. Even better, you can forget about the API and just push it to Git.</p><p>Wouldn’t that be awesome?</p><p>Wouldn’t it be great if we could replicate the experience of using a public cloud provider but made specifically for our needs? Wouldn’t it be great if there were a clearly defined API and a clear separation between the tasks end-users need to perform and the tasks that are the responsibility of the platform itself?</p><h2>Crossplane: The Cloud Native Control Plane</h2><p>If that sounds like something you might need, Crossplane is the project that will get you there. It enables you with capabilities of creating control planes based on the same principles public cloud providers have. It democratizes technology that was previously reserved mostly for big cloud providers like AWS, Azure, and Google Cloud. It enables you to create internal developer platforms.</p><h2>Crossplane Implementation</h2><p>How can we get started with implementing it? You’ll need to understand these topics:</p><ol><li>Crossplane Providers</li><li>Managed resources</li><li>Composite Resource Definitions</li><li>Compositions</li><li>Configuration Packages</li><li>Composition Functions</li><li>And more…</li></ol><p>While these topics above may seem daunting, I’ve written it all out in my book, <a href="https://www.upbound.io/resources/lp/book/crossplane-cloud-native-control-plane" target="_blank">Crossplane: The Cloud Native Control Plane</a>, in an incredibly hands-on way.</p><p>To make things easy, instead of providing step-by-step setup instructions for this, I created scripts that will do that for you. You can run them and follow the instructions they present through questions, or you can choose to inspect the scripts if you prefer setting up everything manually.</p><p>This is a book that reflects the way I learn. I learn by doing. I learn by having my hands on the keyboard at all times. I learn by making mistakes and fixing them. I learn by experimenting. That’s the approach I’ll take with this book. I’ll explain theory and concepts, and I’ll show diagrams, but only through hands-on exercises. I’ll explain what we did rather than do what I explained. You can grab <a href="https://www.upbound.io/resources/lp/book/crossplane-cloud-native-control-plane" target="_blank">Crossplane: The Cloud Native Control Plane</a> for <a href="https://www.upbound.io/resources/lp/book/crossplane-cloud-native-control-plane" target="_blank">free from Upbound</a>.</p>

Why Choose Crossplane?

<p>Platform Engineering is the “new hotness.” It’s what your engineers use to build, deploy, and operate their services in production. Companies everywhere realize the value of having Internal Platforms and restructuring Infrastructure Teams into Platform Teams with the mandate to standardize and consolidate their software development lifecycle around a consistent set of services, tooling, and processes.</p><p>While the value of an Internal Platform driven by Platform Teams is ever-so-clear, best practices around how to structure a team to execute against this vision are fuzzier. Most Platform Teams have a standard set of antipatterns that slow them down and prevent them from fully achieving the Platform Dream. So here is my Top Ten List of the things that are slowing down your Platform Team and how to fix them:</p><ol><li>Not being staffed appropriately</li><li>Having to support the most critical parts of the company</li><li>Maintaining too high a percentage of toil</li><li>Doing too much context-switching</li><li>Needing to support Shadow Infrastructure</li><li>Underestimating the time to support new technology</li><li>Constantly having to support or defend migrations</li><li>Suffering from the effects of&nbsp;<a href="https://en.wikipedia.org/wiki/Conway%27s_law"><u>Conway’s Law</u></a></li><li>Constantly taking on cost-related work with no clear customer value</li><li>No consistency in how to build and operate a platform</li></ol><p>If you want a high-performing Platform Team, you must be honest about the above pitfalls. They’re anti-patterns and will slow your Platform Team down. I’d argue that 50% of the work involved in building a platform is the same across most companies. So, focus your Platform Team on providing differentiated value for the business.</p><p>Now, how to avoid these traps?</p><h2>Not being staffed appropriately</h2><p>One of companies' biggest mistakes is not adequately staffing a platform team. Most platform teams are under-resourced and don’t have critical mass in team or organization size to meet the needs of a never-decreasing product backlog. Investing in your platform has a compounding return in terms of efficiency for your engineers. By providing consistency and giving engineers a golden path to build, deploy, and operate their software in production, you remove some of the massive amounts of context required to be productive. You allow them to iterate more quickly on features needed to operate and grow the business reliably.</p><p>A healthy ratio of engineers working on customer-facing features to engineers on technology consumed by other engineers should be 1:1. For every 50 folks working on product features, 50 folks should also work on teams that empower those engineers. This can be internal platform teams, infrastructure teams, reliability teams, quality teams, and more.</p><p>One thing that is also a challenge for platform teams is&nbsp;<a href="https://blog.upbound.io/__GHOST_URL__/not-hiring-a-bunch-of-quarterbacks/"><u>ensuring that you have the right skills</u></a>. You must ensure the appropriate mix of software and {system, infrastructure, reliability, quality, security, networking, etc.} engineering skill sets across the team.</p><h2>Having to support the most critical parts of the company</h2><p>Every time I hear about a global outage at a company, I immediately think one of three things happened:</p><ul style="list-style-type:disc;"><li>Their global service discovery system failed</li><li>Some type of internal or external network routing issue</li><li>Someone shipped a change to&nbsp;<a href="https://resources.upbound.io/all-content/whats-a-platform-its-that-thing-youre-speeding-on"><u>The Monolith and it died</u></a></li></ul><p>We keep preaching the wonders of microservices and distributed systems. Still, these critical core components exist in most organizations and are always directly supported by the platform or infrastructure team. This work has no glory, but there is all hell to pay when an outage occurs. From a staffing perspective, there is also not much investment in these areas and generally not much time allocated to improving these systems. It'd be a no-brainer if you had more resources, but as the previous item suggests, you’re always resource constrained.</p><h2>Maintaining too high a percentage of toil</h2><p>The&nbsp;<a href="https://sre.google/sre-book/table-of-contents/"><u>Google SRE Book</u></a>&nbsp;gave us this fantastic description of toil: work that is tactical and doesn’t provide strategic value, work that is repeatable and done more than once, work that can be automated if taken the time, and work that is accomplished by hand and manual. &nbsp;</p><p>While Platform Teams can only partially eliminate this type of work, automating these processes is similar to spending time on paying down technical debt—it is a form of debt. You can manage it for a while until it eventually snowballs and consumes you in the form of …</p><h2>Doing too much context-switching</h2><p>This happens when your platform team is under-resourced, has to support too many critical components in the company, and thus doesn’t have time to automate processes and reduce their toil. Your platform engineers spend all their time jumping from one task to the next and doing 50 tasks in a day. These tasks are usually never related to each other, span many different contexts and domains, and incur a cost every time a human has to unload the context from the previous task and load the context of the new task.</p><h2>Needing to support Shadow Infrastructure</h2><p>Shadow Infrastructure happens when your Platform Team takes on too much work and is incapable of meeting the needs of their customers, aka the company’s engineers. Shadow Infrastructure includes work like deploying new infrastructure or technology in the organization in a way that multiple teams and other engineers can easily consume. Platform teams tend to have a global view of the world and like to provide global solutions, but feature teams, and rightly so, need to prioritize getting value into the business's customers.</p><p>They will solve this problem independently and implement new infrastructure or technology in a locally optimized way that works for the individual or team but may not be easily consumable by others. And because the priority is shipping features for the organization, a long-term feature development team has no interest in supporting and operating an underlying technology. Ultimately, ownership gets shifted over to the platform team.</p><p>Wonder why Platform teams are constantly context-switching and unable to reduce the amount of toil while supporting the most critical parts of the business? It is because the amount of things they support is constantly growing, and Shadow Infrastructure can be one of the significant, silent contributors.</p><h2>Underestimating the time to support new technology</h2><p>While the previous five items listed were a direct result of the interactions between a platform team and its customers, these next five are things that a platform team does directly to itself. &nbsp;It starts with underestimating how much time it will take to support new technology.</p><p>Engineers love to build things and learn new technologies. That isn’t always the best thing for the business and often downplays how much effort and time creating something new takes practice. Hint: It is measured in years, not months.</p><p>Engineers often need to pay more attention to how long it will take to build, operate and adopt a new service or technology in production. Ultimately this is the classic&nbsp;<a href="https://resources.upbound.io/webinars/control-planes-the-missing-ingredient-for-cloud-native-developer-platforms"><u>build versus buy</u></a>&nbsp;argument. In either case, platform teams always need to pay more attention to the time it will take to support anything, whether it was built internally or bought and integrated. &nbsp;</p><p>Building is always more expensive than buying, so platform teams need to be honest about only building where necessary and where it provides differentiated value for their internal customers and the business. It may cost more, but it is easier to buy in most situations and offload some of that support directly onto a vendor. Engineers are not cheap from a cost and time/onboarding perspective, and you’re generally better off buying and offloading some of that support to a vendor; it is always almost cheaper in the long run.</p><h2>Constantly having to support or defend migrations</h2><p>Platforms naturally evolve, and as new technology and patterns emerge (like Kubernetes), you want your platform to take advantage of these improvements. What you don’t want to do is force your customers as a platform team to do a significant amount of new work to adopt this next iteration of your platform. This often shows up in the form of migrations:</p><ul><li>Migrating from virtual machines to containers</li><li>Migrating from one cloud provider to another</li><li>Migrating from one backend service or API to another incompatible one</li></ul><p>These are all variations of the same thing and force work on your engineers. This is another area where platform teams need to consider the support burden. You want to make it easier for your engineers to adopt new technology and do the heavy lifting for them, otherwise, they and you are running in place. Work that may provide a benefit in the future but doesn’t help solve problems today.</p><p>Platform teams should limit these forced migrations as much as possible and offload as much work as possible from their engineers. This means being honest upfront about the resources and time it will take to implement, and not having too many things on the truck simultaneously.</p><p>This is also a side-effect of platform teams not engaging enough with their customers, building without understanding their needs, and creating the right abstractions, APIs, and contracts up-front. A migration should be a seamless process.</p><h2>Suffering from the effects of Conway’s Law</h2><p>Platform teams or organizations are usually a combination of infrastructure (compute, storage, networking), managed services (database, streaming, object store), other platforms (machine learning platform, application service platform), and more. Conway’s law states that companies tend to organize their internal systems or products around internal teams or communication structures.</p><p>An anti-pattern exposes your platform in a way that mirrors your organizational structure. If you have a compute team, then you have a computing platform. If you have a network or database team, you have a networking or database platform. The problem is that your engineers don’t care how you’re organized. They will usually view your platform as a singular distinct entity. It is the single thing they use to build, deploy, and operate their services. &nbsp;</p><p>When designing your platform, you should always start with the end-user perspective and abstract the underlying complexities from them—it is optional for them to know and ultimately be leaky.</p><h2>Constantly taking on cost-related work with no clear customer value</h2><p>The above statement is not necessarily fair. Whenever a platform team takes on cost-optimization work, it is usually for the Finance Organization and usually in the form of calling out the cloud spend for being too high, heh! That said, platform teams generally take on the burden of optimizing infrastructure costs and spending, and there is no direct benefit to their engineers. An engineer does not necessarily care that the platform is cheap, only that it is reliable, fast, and does what it does.</p><p>This means that factoring in costs upfront when building out your platform, not just from a vendor perspective but also from a people perspective, is extremely important. You don’t want to think about this after the fact because it can be pretty expensive and time-consuming to make decisions around optimizing your platform after it already has significant usage. This also contributes to being unable to reduce toil, doing too much context-switching, and being under-resourced.</p><h2>No consistency in how to build and operate a platform</h2><p>This is where standardizing on an open-source technology can fit in. You wouldn’t try to create your own web server or database server today unless you had a highly specialized need, but everyone tries to build their own platform! It’s exceptionally wasteful when the foundational layers of a platform are essentially the same across companies.</p><p>Crossplane is one example of such a technology. It’s not just a control plane framework but provides the building blocks you need to provide the basic foundational layer of your platform. It includes a cloud abstraction layer via its provider ecosystem, allowing you to consistently talk to anything with a public API. It also gives you the ability of composability. It enables you to define complex relationships of underlying cloud providers, managed services, and resources, and provide your engineers with a simpler abstraction or API. Combine this with a configuration format that can quickly be packaged and shared with others, and you have the basic building blocks for any platform!&nbsp;<a href="https://blog.upbound.io/__GHOST_URL__/crossplane-first-look/"><u>Learn Crossplane once and use it anywhere.</u></a></p><p>Now that is progress.</p><p>You’ve learned about what’s slowing down your Platform Team—from team misalignments, to managing toil, to managing bespoke layers of your own platform. And now you have an idea of how to fix them–invest in your team and in your foundational tooling. Want to try out Upbound for yourself? Enable SRE teams with a more efficient workflow and the building blocks of platform engineering by setting up your first cloud platform powered by Upbound’s managed control planes. &nbsp;<a href="https://upbound.io/register"><u>Sign up for your 30 day free trial here</u></a>!</p>

<h4>A little bit about me:</h4><p>I am the Head of Engineering here at&nbsp;<a href="http://upbound.io/"><u>Upbound</u></a>, the company behind the open-source project <a href="http://crossplane.io/"><u>Crossplane</u></a>. I have over two decades of experience working in the platform, infrastructure, and reliability engineering space at companies like AirBnB, Uber, and Twilio but have a non-traditional background. I started my career as a software engineer. I have taken on a variety of roles over the years from founder to network engineer, software engineer, platform engineer, system engineer, reliability engineer, engineering manager, engineering leader, and more. I started my early career in several startups as the Internet grew in popularity and quickly learned how to build and scale infrastructure, striving to strike the right balance between solving the right technical challenges while ensuring the business continued to grow. I eventually started several companies because I thought I had some excellent ideas but wanted to learn more about problem-solving from a product and customer perspective. Some great (and expensive) lessons learned are a big reason why I am heavily customer-focused today. I always have my customer hat on, whether one of business or engineering.</p><p>— Sumbry</p>

Top Ten Things Slowing Down Your Platform Team

<p><a href="https://www.crossplane.io/" target="_blank">Crossplane</a> has fundamentally changed the game in allowing organizations to converge on a single platform for all of their needs – apps, infrastructure and everything in between. If you have not heard about Crossplane before and want to learn more about its core concepts, I recommend Pete Lumbis’ blog post “<a href="https://blog.upbound.io/__GHOST_URL__/crossplane-operators-new-bff/">Crossplane, the Operator's New Best Friend</a>.”</p><p>Those who have at least dabbled in Kubernetes and have heard about Crossplane may still find there’s a learning curve between the two. It’s common to wonder how to get started on the journey to building your own platform by designing your own APIs. <a href="https://marketplace.upbound.io/" target="_blank">Upbound Marketplace</a> is an essential tool for users embarking on their Crossplane journey, and I will demonstrate why.</p><h2>A Guided Tour of the Upbound Marketplace</h2><p>The Marketplace is a critical tool for Day-Zero Crossplane users. Before a user sets out to build their API, it’s important to first outline the shape of their API, based on inbound requirements. Let’s suppose the user has these requirements:</p><ul><li>Their organization runs predominantly on Google Cloud Platform (GCP).</li><li>Their goal is to deliver a platform that other teams can use to create VM instances that have already been configured to comply with their company policy. Other app teams don’t want to hassle with knowing which properties to set or what sizes are outside of company policy. They just need a VM to host their app</li><li>They’ve decided they’ll only ask the app team to specify whether they want small, medium or large-sized VMs. They’ll also give app teams the option of selecting a region: us-east or us-west.</li></ul><p>From these requirements, the user knows they’ll need to build API abstractions on top of GCP’s Compute Engine service (VMs). So, their first objective is to check whether there is a Crossplane provider for the cloud service they want to build on. This is where Upbound Marketplace already comes into the picture.</p><h2>Provider Discovery</h2>

<p>The Marketplace promotes a set of popular providers on the frontpage, of which the GCP Crossplane provider happens to be one. Just the same, users can also use the search bar to browse a catalog of nearly 30 Crossplane providers. If the user were to search “Google,” it should return two results: one implementation that is maintained by Upbound &nbsp;(the benefits explained in this <a href="https://blog.upbound.io/__GHOST_URL__/first-official-providers/">blog post</a>) and one maintained by the Crossplane community.</p><p>The Upbound Marketplace already has popular providers like AWS or Azure, and the Crossplane community continues building more Crossplane providers with each passing week, making it that much easier to get started with Crossplane.</p><h2>API Discovery</h2><p>If the user were to click into the Upbound official provider-gcp, they’ll be taken to a page where users can browse through what the provider has to offer. This page gives users a consolidated view to peruse through and explore the API types available for users to build on top of.</p>

<p>Thinking back to the requirements mentioned earlier, if a user is building a platform on top of Compute Engine VMs, they can use the search bar in the CRDs pane to look for “instance”.&nbsp;<i>Note: you may have to try a couple search terms. The marketplace searches based on a few fields exposed by the CRD. For example, if you search for “VM,” &nbsp;you will get no results, but “instance” gets accurate results.</i></p>

<p>If the user clicks the CRD that has an “Instance” kind under the “compute.gcp.upbound.io” API group, they’ll be able to read the description provided by the CRD and see that, “Instance is the schema for the Instances API. Manages a VM instance resource within GCE.” With confidence that they’ve found the correct base CRD to build on top of, they can now familiarize themselves with the API documentation of the CRD, especially the “spec” field.</p><p>Upbound Marketplace makes it easy to learn which fields are required objects under the “forProvider” field. This tells users these fields will need to be set when they instantiate them as part of their API abstraction. For example, scanning the fields of the Instance CRD, users can see that “boot disk,” “machineType,” “networkInterface” and “zone” are required.</p>

<p>If the required fields are an “object” type, users can recursively expand the object’s fields and see the required fields for that nested object.</p><h2>API Examples</h2><p>The Marketplace is also able to provide examples for each API. These examples are functioning .yaml configs that users could submit directly to their control plane to create instances of this CRD as a managed resource. This is a useful tool that gives users a launch pad for writing .yaml config.</p>

<h2>Discovering Configurations</h2><p>A third way the Marketplace is useful for Day-Zero Crossplane users is it can be used to discover Crossplane configurations. Many of the configurations in the Marketplace are reference implementations with permissive licenses. Configurations can either be discovered by scrolling down on the main page or using the search bar. If you use the search bar, make sure to change the filter type to “Configurations.”</p>

<p>Users may want to build an API for which there is already an implementation (like a Kubernetes cluster or PostgreSQL instance). It is completely fair practice to use these as launch pads, since they already have a number of fully completed compositions. If you click into the “platform-ref-gcp” configuration, you can explore the XRDs and compositions that make up the configuration.</p>

<p>All of these examples are but a few ways that Upbound Marketplace bootstraps platform builders’ knowledge and accelerates them down the path of getting started building with Crossplane. There’s plenty more for Crossplane users to explore with the Marketplace, like publishing your own configurations and providers. If this is one of your first times hearing about Crossplane and you want to learn more, check out the&nbsp;<a href="https://docs.crossplane.io/v1.9/"><u>getting started</u></a>&nbsp;guide or join the community on the&nbsp;<a href="http://crossplane.slack.com/"><u>Crossplane slack</u></a>.</p><p>Best,<br>Craig D Wilhite</p><p>This post was originally published on&nbsp;<a href="https://thenewstack.io/crossplane-curious-marketplace-makes-onboarding-easy/"><u>The New Stack here.</u></a></p>

Crossplane Curious? Marketplace Makes Onboarding Easy

<p>Update: As of June 13, 2023, Upbound has updated its Official Providers with Provider Families. Go to the announcement to learn more:&nbsp;<a href="https://blog.upbound.io/new-provider-families"><u>https://blog.upbound.io/new-provider-families</u></a>.</p><p>&nbsp;</p><hr><p>&nbsp;</p><p>Upbound founded Crossplane to help you build cloud control planes. These control planes sit one level above the cloud providers and allow you to customize the APIs they expose according to the needs of your organization. Platform teams at leading enterprises are increasingly adopting Crossplane to offer their developers simpler, safer, self-service interfaces to the cloud.</p><p><a href="https://crossplane.io/" target="_blank">Crossplane is an open-source project</a> incubating with the Cloud Native Computing Foundation (CNCF). It’s built on Kubernetes, making it a great fit for organizations that have embraced cloud-native tools and practices.</p><p>Crossplane has the notion of providers that are the glue between Crossplane and external APIs, such as AWS and Azure. This glue takes a different shape for almost every resource in those APIs, so you need custom implementations in most integrations.</p><h2>Provider Coverage Problem</h2><p>A typical new Crossplane user starts their journey by installing Crossplane and playing with the provider they use at their company. After the initial prototyping step, folks go to the API reference of a given provider to see if it includes support for all the cloud resources they use. At this point, we see many users drop out because of the lack of coverage. Unfortunately, it’s usually zero or one — they don’t use Crossplane even if only 10% of the infrastructure of a given application is not covered because you don’t want to have two ways of orchestrating the cloud infrastructure. We call this the provider coverage problem, which is present in every infrastructure provisioning tool that has come so far, like Terraform, Ansible, etc.</p><p>The Crossplane community has been adding <a href="https://blog.upbound.io/scaling-kubernetes-to-thousands-of-crds">CustomResourceDefinitions (CRDs)</a> to cover the API surface, but the pace is unable to catch up with users who are blocked from getting into production with Crossplane due to the lack of coverage. To give you an idea, AWS has ~1000 resources and the classic AWS provider has yet to cover ~200 as of writing.</p><h2>Solution</h2><p>At Upbound, we have concluded that even though we are able to re-use many utilities across the providers, it won’t scale to implement support for every CRD one by one. We should stand on the shoulders of what has come before.</p><p>The first iteration of this conclusion was <a href="https://github.com/crossplane/terrajet" target="_blank">Terrajet</a> — a code generation framework for CRDs and a generic controller that runs Terraform under the hood. But Terraform doesn’t have all the metadata we need to satisfy the <a href="https://github.com/crossplane/crossplane/blob/master/design/one-pager-managed-resource-api-design.md" target="_blank">Crossplane Resource Model (XRM)</a> experience that we’re all used to. We came up with a configuration framework in Go that allowed providing additional metadata that can scale to many implementations.</p><p>In fact, we experimented with having just the default configs for all resources and it resulted in thousands of working CRDs which was a first in the Kubernetes community! This uncovered bugs in the Kubernetes API server that would appear only when you reach that scale. We worked closely with API Machinery team to provide patches to accommodate that scale. More on this <a href="https://blog.upbound.io/scaling-kubernetes-to-thousands-of-crds">here</a>.</p><p><a href="https://github.com/upbound/upjet" target="_blank">Upjet</a> is the second generation of this approach that is more stable, has more customization points, and includes new code generators that enable automatically generating API documentation and example YAMLs ready to be used as well as inferring metadata configurations from the Terraform documentation such as cross-resource references so that the providers get to be production-ready with much less effort and time. Provider maintainers can replace Terrajet with Upjet with minimal code changes to utilize these new features. Our first set of Upbound Official Providers — AWS, Azure, and GCP — that Upbound uses to meet SLAs to its customers are all built on Upjet, contributing to its stability.</p><h2>Introducing Official Providers</h2><p>Official Providers are production-ready Crossplane providers that are authored or maintained by Upbound. As a vendor, Upbound can provide support and SLAs around using Official Providers, giving customers peace of mind. They have the highest number of CRDs at the v1beta1 quality level and are tested continuously using real cloud provider endpoints. The first set contains the big three cloud providers (AWS, Azure, and GCP), and we are continuously expanding the program to cover more APIs and infrastructure providers.</p><h3>Coverage</h3><p>As of writing, Official Providers have the highest coverage of the big three Cloud APIs in the Crossplane ecosystem.</p>

<p>With these ever-growing numbers, you can use these Official Providers to orchestrate your most complex scenarios. In addition to production-level quality, the Upjet-based providers have extended coverage of given APIs, i.e. more fields and features are supported in the CRDs compared to other providers. This is achieved by relying on mature Terraform providers under the hood that have already implemented most, if not all, &nbsp;features of a given API.</p><h3>Quality</h3><p>Great power comes with great responsibility. With so many APIs to deal with, it’s essential that we keep the complexity under control and don’t introduce regressions. To do this at scale, we have developed a new tool called <a href="https://github.com/upbound/uptest" target="_blank">Uptest</a> that deploys the resource into a cluster, makes sure it gets to a ready state, and that the removal succeeds. Over time, we will add more scenarios to build even higher confidence levels.</p><p>With Uptest in place, all official provider CRDs are at <a href="https://kubernetes.io/docs/reference/using-api/#api-versioning" target="_blank">v1beta1 maturity</a> — meaning they are safe to use in production, and we guarantee that even if there is a breaking change, it will be handled automatically by a standard Kubernetes webhook.</p><p>One of the concerns we had with Terrajet was the cases where the underlying Terraform provider may not provide a high-quality implementation for a given resource and we may end up stuck with it till the upstream fix is accepted. In Official Providers, we can add custom implementations, in addition to generated ones, whenever the need arises. <a href="https://marketplace.upbound.io/providers/upbound/provider-aws/latest/resources/eks.aws.upbound.io/ClusterAuth/v1beta1" target="_blank">ClusterAuth</a> in AWS EKS is a good example where we added a <a href="https://github.com/upbound/provider-aws/blob/75f320d/internal/controller/eks/clusterauth/controller.go#L145" target="_blank">manual implementation</a> to cover the authentication story of AWS EKS in a way that provides ever-refreshing tokens through its connection details secret.</p><h3>Improved User Experience</h3><p>We want to provide the best experience for users getting started with Official Providers. A big step towards this goal is having examples for every resource. You will see that every CRD in our Official Providers has at least one example, and Uptest tests all examples you see continuously to make sure they are always up-to-date. See AWS VPC <a href="https://marketplace.upbound.io/providers/upbound/provider-aws/v0.18.0/resources/ec2.aws.upbound.io/VPC/v1beta1" target="_blank">here</a> with 19 examples!</p><p>In addition to examples, official providers have a high bar that all fields in the API reference of a given CRD should have as much documentation as possible. To see the rich experience, see AWS EKS Cluster API Reference <a href="https://marketplace.upbound.io/providers/upbound/provider-aws/latest/resources/eks.aws.upbound.io/Cluster/v1beta1" target="_blank">here</a>. With Upjet, this is automated, and future Official Providers will have this as well.</p><h3>Peace of Mind</h3><p>At Upbound, we talked with many customers and uncovered that while having a community around providers is great, many companies want formal support from a vendor. With Official Providers, Upbound supports its customers with timely fixes and provides patch releases immediately! All fixes we will be doing for our customers will land on the same repository.</p><h2>What’s Next</h2><p>Official Providers are released under the Apache 2.0 license, just like Crossplane. We encourage everyone to play around with them, ask questions in <a href="https://slack.crossplane.io/" target="_blank">Slack</a>, and feel free to contribute. We were delighted to see <a href="https://github.com/upbound/upjet/pull/112" target="_blank">a community contribution</a> even on the launch day!</p><p>We have a <a href="https://github.com/upbound/upjet/blob/f6b62b4/docs/manual-migration-guide-to-op.md" target="_blank">comprehensive guide</a> that will help you migrate your existing resources to official providers. In addition, we are working on a new tool that will create a migration plan, let you edit specifics when necessary and execute it; making it a breeze to migrate to official providers!</p><p>Keep an eye out for the next release since <a href="https://marketplace.upbound.io/" target="_blank">Official Providers are released every week</a>! In the coming weeks, we’ll see more CustomResourceDefinitions added, and you can expect them to reach full coverage soon, all the while providing the same high-quality level. In addition to increasing the coverage, we are getting close to the first Crossplane providers with “v1” CRDs!</p><p><i>Note: If you're interested in working on these with us, </i><a href="https://www.upbound.io/careers/working-with-us" target="_blank">view our open roles here<i>.</i></a></p>

The First Official Providers

<p>I recently joined “<a href="https://www.youtube.com/watch?v=Kgn53TfWyWE">Platform Automation Using Crossplane</a>”, a webinar hosted by <a href="https://www.infracloud.io/">InfraCloud</a>, where we talked about platform automation using Crossplane and how it enhances platform engineering. This recap is for any Platform Engineer, DevOps expert, or Cloud Engineer interested in streamlining their processes.</p><h2>Platform Engineering: A User-Centric Approach</h2><p>Platform engineering should be treated as a product. This means understanding the needs of users—developers—and designing the platform to meet those needs. By doing this, platform engineering can significantly boost developer experience and productivity. It's not about creating something that platform engineers think is cool; it's about creating something that developers find genuinely useful.</p><h2>Defining Platform Engineering</h2><p>Platform engineering can be seen as an enabler that boosts developer experience by codifying best practices into reusable and self-service capabilities. It underscores the role of platform engineering in providing a seamless interface between developers and the underlying infrastructure.</p><h2>Balancing Flexibility and Rigidity</h2><p>A critical <a href="https://blog.upbound.io/top-ten-things-slowing-down-your-platform-team">challenge in platform engineering</a> is balancing flexibility and rigidity. If the platform is too rigid, it stifles innovation; if it's too flexible, it overwhelms users. The right approach is to provide enough configurability to meet diverse needs without burdening users with unnecessary complexity.</p><h2>The API-First Approach</h2><p>In platform engineering, adopting an API-first approach is crucial. APIs provide a standardized way for developers to interact with various services and resources. By focusing on APIs first, platforms can ensure that all services are accessible programmatically, allowing for greater automation and integration. This approach also simplifies the addition of new interfaces, such as CLI or web UIs, which can be built on top of the existing APIs.</p><h2>Challenges in Learning Kubernetes</h2><p><a href="https://resources.upbound.io/kubernetes/why-kubernetes-is-inappropriate-for-platforms-and-how-to-make-it-better">Kubernetes</a> is the foundation of modern platform engineering, but its complexity can be daunting. The initial learning curve is steep, but once mastered, Kubernetes offers a consistent framework for managing infrastructure and applications. Its extensibility and unified management capabilities make it a vital tool for platform engineers.</p><h2>Crossplane: Extending Kubernetes Capabilities</h2><p>During the <a href="https://www.youtube.com/watch?v=Kgn53TfWyWE">webinar</a>, I demonstrated how <a href="https://blog.upbound.io/crossplane-intro-and-deep-dive">Crossplane</a> extends Kubernetes capabilities, enabling the management of cloud resources using Kubernetes-native APIs. Crossplane allows platform engineers to define and manage infrastructure declaratively through Kubernetes, leveraging its robust ecosystem and declarative nature.</p><h2>Key Features of Crossplane</h2><ol style="list-style-type:decimal;"><li><strong>Kubernetes-Native</strong>: Integrates seamlessly with Kubernetes, using its APIs and patterns.</li><li><strong>Resource Management</strong>: Manages cloud resources like databases, networks, and clusters declaratively.</li><li><strong>Composable Infrastructure</strong>: Allows combining multiple resources into higher-level abstractions, simplifying complex infrastructure setups.</li></ol><h2>Practical Application of Crossplane</h2><p>In the <a href="https://www.youtube.com/watch?v=Kgn53TfWyWE">demo</a>, I showed how Crossplane automates the creation and management of infrastructure. By defining resources and their dependencies in Kubernetes manifests, Crossplane ensures that the desired state is maintained, automatically reconciling any drift. This approach not only streamlines operations but also enhances reliability and consistency across deployments.</p><h2>Conclusion and Resources</h2><p>The webinar highlighted the importance of treating platforms as products, balancing flexibility with rigidity, and adopting an API-first approach. Tools like Crossplane, built on top of Kubernetes, provide powerful capabilities to automate and manage infrastructure, making platform engineering more accessible and effective.</p><p>For those interested in exploring Crossplane further, I have a book, “<a href="https://www.upbound.io/resources/lp/book/crossplane-cloud-native-control-plane">Crossplane: The Cloud Native Framework</a>”. It gives an overview and deep dive into the CNCF project to help you start streamlining your structure today.</p><p>Thank you to everyone who joined the webinar. It was a pleasure to share insights and showcase the capabilities of Crossplane. Stay tuned for more discussions and demos, and happy platform engineering.</p>

Automating Platform Engineering with Crossplane

<p>At <a href="https://www.upbound.io/"><u>Upbound</u></a> we’re building your Universal Cloud Platform. Platform teams use Upbound to build bespoke cloud APIs and cloud consoles for the application development teams they support.</p>

<h2>The Upbound Cloud Console</h2><p>At the heart of Upbound is <a href="https://crossplane.io/" target="_blank">Crossplane</a>, the open source cloud control plane we founded and <a href="https://blog.crossplane.io/crossplane-cncf-incubation/" target="_blank">donated to the CNCF</a>. Crossplane can take anything with an API (like an AWS service, an SQL server, or a Kubernetes cluster) and let you build your own declarative API for it. Perhaps as a platform engineer you want to offer your application developers access to deploy and manage their own AWS RDS instances for example, but you want a very simple, opinionated API where only storage size is configurable.</p>

<h2>Crossplane enables a simpler database provisioning API</h2><p>Upbound helps you design and deploy Crossplane-powered cloud APIs. It also offers a cloud console to help manage them. Think AWS management console, but all the services and concepts are designed by you to suit your organisation’s unique needs. We can do this in part thanks to the Crossplane Resource Model, or XRM. While Crossplane encourages you to be opinionated about the <i>content</i> of your APIs (e.g. what knobs they expose) the XRM ensures the <i>shape</i> of your APIs is always consistent. Metadata is always distinct from desired and observed state, for example, while external identity, and secret properties like credentials are always presented one way. Crossplane is built on <a href="https://kubernetes.io/" target="_blank">Kubernetes</a>, and the XRM is an extension of the <a href="https://blog.upbound.io/__GHOST_URL__/how-the-kubernetes-resource-model-enables-configuration-as-data/">Kubernetes Resource Model</a> (KRM).</p><p>A typical production API powered by Crossplane will consist of bespoke types designed by your platform team, as well as types installed by a <a href="https://crossplane.io/docs/v1.5/concepts/providers.html" target="_blank">Provider</a>. The latter are called “<a href="https://crossplane.io/docs/v1.5/concepts/managed-resources.html" target="_blank">Managed Resources</a>” (MRs) and represent external systems like an AWS service. They are the building blocks used to create the former, which we call “<a href="https://crossplane.io/docs/v1.5/concepts/composition.html" target="_blank">Composite Resources</a>” (XRs). Crossplane’s API is powered by the Kubernetes API server, which serves a <a href="https://kubernetes.io/docs/concepts/overview/kubernetes-api/" target="_blank">JSON based HTTP REST API</a>.</p>

<h2>The API Server serves a JSON REST API</h2><p>The Upbound Console’s API server queries can be expensive. Take for example a platform engineer who has defined a <code>PostgreSQLInstance</code> XR. Each <code>PostgreSQLInstance</code> is &nbsp;composed of various MRs — say an <code>RDSInstance</code>, a <code>DBSubnetGroup</code>, and a <code>SecurityGroup</code>. If there were 10 <code>PostgreSQLInstances</code> the Upbound Console would have to make 31 REST API calls to show an application developer everything there was to know about these <code>PostgreSQLInstances</code> - one call to list the 10 XRs, then three calls per XR to lookup the MRs of which they are composed. To make matters worse, the responses to these requests always include every detail about the queried resource. Even if the developer just wanted to know whether or not they were all healthy, they would be forced to retrieve the entire JSON documents. These large queries get even slower when network latency between the Upbound Console and the API Server its querying increases.</p><p>Enter <a href="https://graphql.org/" target="_blank">GraphQL</a>. GraphQL is a strongly typed query language that is frequently used to build APIs. Compared to a typical JSON REST API a GraphQL API allows the caller to define the structure of the data it requires, allowing the server to omit any irrelevant data from the response. As the name suggests, GraphQL can query a graph of related types — so in our example above it would be possible to retrieve <i>only</i> the health of our 10 <code>PostgreSQLInstances</code> and all the Managed Resources of which they are composed in a single GraphQL query. GraphQL <a href="https://graphql.org/users/" target="_blank">enjoys broad adoption</a>. Invented at Facebook a decade ago, it’s now an open standard and powers high profile APIs like those of <a href="https://docs.github.com/en/graphql" target="_blank">GitHub</a> and <a href="https://shopify.dev/api/admin-graphql#top" target="_blank">Shopify</a>. Often large organisations will build a GraphQL aggregation layer to tie their many microservices together.</p>

<h2>An example GraphQL query with xgql</h2><p>The Upbound Console team identified the benefits of GraphQL early on, but despite its broad adoption GraphQL has seen little use within the Kubernetes ecosystem. Perhaps this is due to a technology stack disconnect. While language agnostic as a standard, much of the GraphQL ecosystem favors TypeScript whereas the Kubernetes ecosystem favors Go. Nascent projects such as <a href="https://github.com/qlkube/qlkube" target="_blank">qlkube</a> and <a href="https://github.com/onelittlenightmusic/kubernetes-graphql" target="_blank">kubernetes-graphql</a> weren’t a good fit as they are primarily auto-generated wrappers around “built in” Kubernetes API types like <code>Pod</code> and <code>Deployment</code>. Upbound needed a solution to support the XRM’s more advanced type system, which has subtypes (e.g. <code>RDSInstance</code> is a subtype of Managed Resource) and dynamic types that are added and removed at runtime (e.g. by defining a new Composite Resource, or installing a Provider). We decided to build our own GraphQL representation of the XRM — a Crossplane Resource Graph.</p>

<h2>The GraphQL schema of a Crossplane Composite Resource</h2><p><a href="https://github.com/upbound/xgql/" target="_blank">xgql</a> is Upbound’s implementation of the Crossplane Resource Graph. It’s open source, and part of our <a href="https://github.com/upbound/universal-crossplane" target="_blank">Universal Crossplane</a> distribution. xgql is written in Go using <a href="https://gqlgen.com/" target="_blank">gqlgen</a>. This approach has two key benefits:</p><ol><li>gqlgen is “schema first” — it generates much of its server code from a schema.</li><li>xgql can leverage the extensive caching support of the Kubernetes Go libraries.</li></ol><p>GraphQL schemas are all about types and relationships. Designing the schema up-front was an excellent test of the XRM, forcing us to think deeply about an idiomatic GraphQL representation of concepts like Providers, Configurations, Composite Resources, Managed Resources, and the relationships between them. In fact doing so highlighted <a href="https://github.com/crossplane/crossplane/issues/2255" target="_blank">one</a> or <a href="https://github.com/crossplane/crossplane/issues/2262" target="_blank">two</a> small gaps in the XRM that we intend to address in upstream Crossplane. The resulting schema enables GraphQL queries like:</p><ul><li>“Show me the ‘Ready’ status condition for the tree of resources created by this Claim.”</li><li>“Show me all events pertaining to provider-aws’s Managed Resources”.</li><li>“Tell me whether this is a Managed Resource or a Composite Resource”.</li><li>“Fetch the definition of this Composite Resource”.</li></ul><p>Building xgql in Go allows us to use best-in-class Kubernetes client machinery. xgql maintains an in-memory “watch cache” for each unique caller, meaning it subscribes to API server updates rather than making tens or hundreds of REST API calls to resolve each GraphQL query. This often results in xgql answering queries 10x faster with a warm cache.</p><p>There’s a lot more to the Upbound Console, including another GraphQL layer that uses a novel approach to route queries to thousands of xgql instances. <a href="https://twitter.com/upbound_io" target="_blank">Let us know on Twitter</a> if you’d be interested in another post about how we leverage GraphQL at Upbound. In the meantime if you’re building on the Crossplane API do try Universal Crossplane and xgql.</p><p>Finally, we’re hiring! We’re a fully remote company, deeply invested in open source, and recently <a href="https://blog.upbound.io/__GHOST_URL__/upbound-raises-60m-in-funding-to-advance-its-universal-cloud-platform/">closed a strong Series B</a> to help us build a universal cloud platform. We have open engineering roles from frontend to distributed systems to management — <a href="https://boards.greenhouse.io/upbound" target="_blank">apply today</a>.</p>

The Crossplane Resource Graph

<p>With <a href="https://marketplace.upbound.io/providers/upbound/provider-terraform/v0.3.0" target="_blank">provider-terraform</a> we have a new Crossplane provider that runs Terraform configurations and enables platform teams to include custom Terraform code within a Crossplane powered architecture. This provider enables many integration scenarios and allows teams who have invested in Terraform to leverage a powerful control plane architecture. Please read <a href="https://blog.upbound.io/official-provider-terraform-launch/">the announcement</a> for more information. Furthermore, if you haven’t consider watching the webinar: <a href="https://resources.upbound.io/webinars/kubernetes-called-and-it-wants-your-iac-back-using-control-planes-to-modernize-your-iac-tech-stack" target="_blank">Kubernetes Called and it Wants Your IaC Back: Using Control Planes to Modernize Your IaC Tech Stack</a>.</p><p>In this blog post, we would like to explore a phased approach on how to include existing Terraform code and start migrating towards a Crossplane native solution.</p><p><strong>Example: Let developers create their own subnet</strong></p><p>Let’s say we are a platform team and are tasked to create AWS Subnets for developers. We have already spent a significant amount of time into a Terraform module, and we want to re-use that code, so developers can self-serve subnets for their application. Let’s see on how can use Crossplane to do so and port our custom code to Crossplane and possibly migrate fully to Crossplane if we'd like to.</p><h2>Overview</h2><p>We will do this in four steps:</p><ol><li>Create a common interface for developers to use</li><li>Run the full module in provider-terraform</li><li>Extract one resource (subnet) into the native AWS provider</li><li>Move the second resource into the native AWS provider</li></ol><p>The beauty of this approach is that we have a running platform after step one and can test our platform and do the subsequent steps 3 and 4 when the time is right.</p>

<p>The following steps will contain short abbreviated code examples. See the <a href="https://github.com/upbound/provider-terraform/tree/main/examples/transition" target="_blank">upbound/provider-terraform repository</a> for the complete example.</p><h3>1. Create a common interface for developers to use</h3><p>With Crossplane, a platform team offers their users a way to create a subnet by simply creating a specific Kubernetes resource. e.g. something like:</p>

<p>In order for this to materialize, we first need to create a Composite Resource Definition (or short XRD). This is Kubernetes resources that Crossplane understands. See&nbsp;<a href="https://docs.crossplane.io/v1.10/concepts/composition/"><u>Composite Resources · Docs</u></a>&nbsp;for more information. The XRD contains a spec where we can describe the interface for the developer. In this case, we want to make it easy for the developer and make him only specify the vpcName.</p>

<p>(Abbreviated. See <a href="https://github.com/upbound/provider-terraform/blob/main/examples/transition/definition.yaml" target="_blank">definition.yaml</a> for the complete version.)</p><h3>2. Run the full module in provider-terraform</h3><p>After we have created the interface, we need to create an implementation, which is called a Composition in Crossplane. In this step, we want to run the full Terraform module in this composition by using provider-terraform. As you might guess, we are creating another Kubernetes resource with the type Composition. We start by just specifying one Crossplane resource by the kind Workspace which includes the Terraform code and a patches to convert the necessary information to Kubernetes. In this example, we have in-lined all the Terraform code, but we can also specify remote resources (see <a href="https://github.com/upbound/provider-terraform/blob/main/examples/workspace-remote.yaml#L19" target="_blank">examples/workspace-remote.yaml</a> for an example.)</p>

<p><i>(Abbreviated. See&nbsp;</i><a href="https://github.com/upbound/provider-terraform/blob/main/examples/transition/01-composition-tf-only/composition.yaml"><i><u>01-composition-tf-only/composition.yaml</u></i></a><i>&nbsp;for the complete version.)</i></p><p>🎉 Success! We now have a fully running control plane with our custom Terraform logic. You can start testing it with your team.</p><h3>3. Extract one resource into the native provider</h3><p>To fully transition to Crossplane, we now want to refactor the Subnet resource to use <a href="https://marketplace.upbound.io/providers/upbound/provider-aws/v0.26.0" target="_blank">provider-aws</a>. For this, we adapt our composition to now have two resources of kind Workspace and Subnet and use patching to combine these two.</p>

<p><i>(Abbreviated. See </i><a href="https://github.com/upbound/provider-terraform/blob/main/examples/transition/02-composition-tf-and-native/composition.yaml" target="_blank"><i>02-tf-and-native/composition.yaml</i></a><i> for the complete version.)</i></p><p>You can see on how this enables us to transition to Crossplane native providers gradually. And depending on the complexity of the resources you have been developing, you can repeat this step.</p><h3>4. Move the second resource into the native provider</h3><p>In our last step, we want to remove provider-terraform from our composition and fully use the native aws provider. The outcome is for folks familiar with Crossplane straightforward. One composition that contains the two resource VPC and Subnet:</p>

<p><i>(Abbreviated. See </i><a href="https://github.com/upbound/provider-terraform/blob/main/examples/transition/03-composition-native-only/composition.yaml" target="_blank"><i>03-native-only/composition.yaml</i></a><i> for the complete version.)</i></p><h2><strong>Summary</strong></h2><p>Crossplane is a great way to build a developer platform powered by Kubernetes. Terraform has a great ecosystem and existing investments from platform teams. With provider-terraform we can combine both ecosystems and enable integration scenarios. This blogpost showed a quick get started guide on how to take a step-by-step guide behind a stable developer facing API. If you want to learn more and see the above in action, please check out the <a href="https://docs.upbound.io/providers/provider-terraform/migrate-provider-tf/" target="_blank">migration guides.</a></p>

Migrating to Crossplane With Provider-Terraform

<p>Organizations and users find <a href="https://blog.upbound.io/__GHOST_URL__/composing-cloud-services-for-fintech-apps/www.crossplane.io">Crossplane</a> in different ways. For Kelly Ferrone, DevOps and software engineer at <a href="https://blog.upbound.io/__GHOST_URL__/composing-cloud-services-for-fintech-apps/www.akirix.com">Akirix</a>, the challenge he had was that another team relied on a Kubernetes service that had been deprecated and he needed a solution - fast.</p><p>Ferrone outlined how he and his company found Crossplane during a session at the Crossplane Community Day event, in December 2020. <a href="https://blog.upbound.io/__GHOST_URL__/composing-cloud-services-for-fintech-apps/www.akirix.com">Akirix</a> is a global payment platform that allows buyers and sellers from around the world to conduct secure transactions. The development team was using a Kubernetes project called the service broker that had become deprecated. While that project went away, there was a need for another tool for Ferrone and his team to help manage the company's Amazon Web Services (AWS) infrastructure and synchronize it with all of the services in Kubernetes that Akirix uses.</p><p>"Throughout the community a number of people suggested Crossplane as an alternative," Ferrone said. "I tried it out and I haven't looked back since and it has been great."</p><h2>The Akirix Use Case</h2><p>Ferrone explained that his team is replatforming a monolithic app with Kubernetes. He noted that overall Kubernetes has made it alot easier to maintain, as did other AWS services that helped to enable the deployment.</p><p>"As we are a smaller shop, I had to maintain alot and I wanted to have a consistent interface for managing Kubernetes and AWS simultaneously and have it all in one project which I was able to do with Crossplane," Ferrone said. "Now all from Kubernetes, AWS is maintained, as is our app and it greatly simplified everything."</p><h2>Attractive Features in Crossplane</h2><p>Ferrone primarily uses Crossplane to enable setting up a type of account in Akirix he called an IRSA account—which is basically an Identity and Access Management (IAM) role for service accounts.</p><p>"We heavily use the IAM section of Crossplane and then map that out to service accounts and other common Kubernetes components like external DNS, cluster auto-scaler and various load balancers that all want permission to access AWS," Ferrone explained. "Crossplane has made that quite easy and very stateful and I can keep it all backed up as YAML files in a git repository."</p><p>Ferrone is also making use of Crossplane Compositions. He noted that his team has been using Compositions to roll out new services including Redis and MySQL data services.</p><p>As part of his session Ferrone gave a brief demo showing his use case for IAM roles using Crossplane. Among the steps he demonstrated was using a customized plug-in to inject various variables into the YAML, based on different stacks and different environments, as well as the provider configuration. The end result is that the account has YAML backed up in a git repository and it then simply takes one command to launch all of the YAML into the cluster. "Thanks to Crossplane it makes it very easy to manage, look at and view. And it makes it pretty fun too," Ferrone said.</p>

Composing Cloud Services for Fintech Apps

<p>Upbound&nbsp;<a href="https://blog.upbound.io/blog/upbound-general-availability/"><u>launched&nbsp;</u></a>this week and we’re excited to see how platform teams begin building above the clouds with&nbsp;<a href="https://www.upbound.io/product/upbound"><u>Upbound</u></a>. Users beginning their journey in the product will notice the Get Started experience only offers launching managed control planes from Crossplane configurations built on AWS or GCP.</p>

<p>Does this mean users who want to build above Azure are out of luck? Not at all! While we work on building a configuration for Azure (coming soon!), this blog demonstrates how you can use the features &amp; capabilities of Upbound to build a platform above Azure.</p><h2>Complete the Get Started experience</h2><p>Upbound’s Get Started experience must be completed before you can gain access to the Upbound Console. Once you reach the Console, you will be able to provision another managed control plane and create a GitHub-synced configuration where you will define your APIs for Azure. If you have not already completed the Get Started experience, “speed run” through it by selecting any of the configurations above, connect Upbound to your GitHub account, and give your first managed control plane a name.</p><p>Finally, when Upbound asks for parameters to connect Upbound to GCP or AWS, provide fake strings. Upbound uses these inputs to concoct a ProviderConfig for your managed control plane to connect to GCP or AWS. But we don’t care about this – we’re creating our own managed control plane for Azure!</p>

<h2>Use the Upbound Console</h2><p>Let’s get to work building a platform above Azure driven by an Upbound managed control plane. From Upbound Console start by creating a new configuration. Hover over the `Create Configuration` button and select `Start from Scratch`.</p>

<p>To create a new configuration from scratch, give your configuration a name (for example, “my-azure-platform-apis”), set the GitHub repo visibility to whichever you prefer, and click `Clone configuration to GitHub`. After the configuration is created, Upbound will take you to the overview screen for your new configuration. Confirm that Upbound created a repo for you (it should look something like “https://github.com/&lt;your-gh-account&gt;/my-azure-platform-apis.git</p><p>Navigate back to the dashboard of the Console by clicking the icon in the top navigation bar. Now, create a managed control plane by clicking `Create Control Plane`.</p>

<p>Select the configuration you created previously to install it on your new managed control plane. In the next screen, give your control plane a name (for example, “azure-control-plane”), and click `Create`.</p>

<p>You should now have a new managed control plane created and configured to install from a new “scratch” Crossplane configuration. The remaining steps you need to do are:</p><ul><li>Author a new API for Azure resources in your git repo based on <a href="https://marketplace.upbound.io/providers/upbound/provider-azure/v0.30.0" target="_blank">provider-azure</a></li><li>Deploy the new configuration with API to your managed control plane</li><li>Create a ProviderConfig for Azure on your managed control plane</li></ul><h2>Author your API</h2><p>Go to the GitHub repository Upbound created for you in the previous step. Open the `crossplane.yaml` file and replace the contents with the following (<a href="https://gist.github.com/cwilhit/9316f435b75fcf564633928cb1b823fc" target="_blank">gist reference</a>):</p>

<p>The important thing we have done above is take a dependency on provider-azure for this configuration. Next, create a folder called “apis” in the root of your repo. In the “apis” folder, create a folder called “blob”. In the blob folder, create a “definition.yaml” and “composition.yaml”. Your directory structure should look like this:</p>

<p>We are going to build an API for Azure Blob storage. To start, let’s define the shape of the API. Open the `definition.yaml` file and paste in the following (<a href="https://gist.github.com/cwilhit/046b4eec3bd9070c07444e6bb25fbf26"><u>gist reference</u></a>):</p>

<p>Close and save the file. Next, open the `composition.yaml` file and paste in the following (<a href="https://gist.github.com/cwilhit/7e132e8115b34226ead4acfc7669f3e1"><u>gist reference</u></a>):</p>

<p>Save and close the file. Commit and push the changes to your repo on GitHub.</p><h2>Deploy new configuration definition</h2><p>Back on Upbound Console, click into your managed control plane for Azure. You should see an update is available to be installed on your control plane. Initiate the update by clicking the `Update Available` text and confirm by clicking `Update Configuration`.</p>

<p>It will take about a minute or so to apply the latest configuration. You can click over to the Settings tab, scroll down to Providers, and see that provider-azure gets installed. Once the configuration has been updated, there should now be a “Blob” type card in the explorer view (you may need to refresh the page once the update is completed).</p>

<h2>Create a ProviderConfig</h2><p>The last thing you need to do is create a ProviderConfig on your managed control plane for provider-azure, so your control plane can communicate with Azure and process resource requests. Provider-azure does not support Upbound’s <a href="https://docs.upbound.io/knowledge-base/oidc/" target="_blank">OIDC integration</a> yet, so you will need to provide credentials for a service account. You can find instructions for how to do this in the <a href="https://marketplace.upbound.io/providers/upbound/provider-azure/v0.30.0/docs/quickstart" target="_blank">quickstart documentation</a> for provider-azure under “Create a Kubernetes secret”.</p><p>In order to push your ProviderConfig to your managed control plane, you can use the up CLI’s <a href="https://docs.upbound.io/cli/command-reference/#controlplane-kubeconfig-get" target="_blank">ctp kubeconfig get</a> command to set your kubeconfig. For that command, you need</p><h2>Make a claim against your API</h2><p>Now that you’ve configured your managed control plane to communicate with Azure, it’s time to create a claim for the Blob API. You can use your control plane’s <a href="https://docs.upbound.io/concepts/ctp-portal/" target="_blank">portal</a> to do this or you can submit the claim directly via CLI (<a href="https://gist.github.com/cwilhit/1352e81b2a982ff30a79772f70585e11" target="_blank">gist reference</a>).</p>

<p>After you’ve submitted the claim, you can return to the control plane explorer view and observe the resources get created. It can take a few minutes, but you should eventually see green status icons for each of the resources created.</p>

<p>Congratulations: you now have a managed control plane in Upbound configured to talk to Azure &amp; the definition of your APIs is now syncing from GitHub.</p><p>&nbsp;</p><hr><p>&nbsp;</p><p>Want to try out Upbound for yourself? Enable SRE teams with a more efficient workflow and the building blocks of platform engineering by setting up your first cloud platform powered by Upbound’s managed control planes. &nbsp;<a href="https://upbound.io/register?ref=blog.upbound.io"><u>Sign up for your 30 day free trial here</u></a>!</p>

Using Upbound to Build a Platform Above Azure

<p>Today we're proud to announce <a href="https://upbound.io/" target="_blank">Upbound</a> as a launch <a href="https://www.upbound.io/why-upbound/partners">partner for AWS</a>'s <a href="https://aws.amazon.com/blogs/opensource/introducing-amazon-eks-distro/" target="_blank">recently launched</a> EKS-D, Amazon's official upstream Kubernetes distribution that powers AWS EKS. Using EKS-D, AWS customers can create reliable and secure Kubernetes clusters on-premises, in the cloud, and at the edge.</p><p>As a launch partner, Upbound is joining the APN to give AWS customers a first-class experience with our product, <a href="https://www.upbound.io/register/a" target="_blank">Upbound Cloud</a>, the production implementation of popular open source control plane project <a href="https://crossplane.io/" target="_blank">Crossplane.</a></p><p><a href="https://www.upbound.io/product/upbound">Upbound Cloud</a> makes it easy for EKS-D users to consolidate configuration of clusters into a single, environment-agnostic repository of Crossplane configuration. Using this configuration, new clusters can then be provisioned on-premises, in the cloud, or at the edge through a single unified API interface or easy to use console interface found inside of Upbound Cloud.</p><p>Together, Upbound and AWS are not only helping customers build and manage hybrid cloud Kubernetes deployments, but also providing a standard way for the cloud-native community to deploy Kubernetes into hybrid cloud environments using our popular open source project, Crossplane.</p><p>If this is interesting to you, please reach out, we would love to <a href="https://www.upbound.io/request-a-demo">give you a demo</a> and learn more about your hybrid cloud Kubernetes use case. You can get started with Crossplane using <a href="https://www.upbound.io/register/a">Upbound Cloud</a>, and we’ll be updating Upbound Registry with Crossplane configurations for EKS-D alongside the currently available <a href="https://cloud.upbound.io/registry/upbound/platform-ref-aws" target="_blank">AWS reference configuration</a>. Continue reading to learn more about our vision for cloud agnostic cluster deployment, and what's next for our partnership.</p><h2>Kubernetes in the Hybrid Cloud</h2><p>Today, most organizations have a healthy mixture of on-premises and cloud infrastructure. Sometimes this is strategic, other times it's due to acquisitions or specific workload requirements. Regardless, it’s reality for most customers. With Kubernetes becoming the standard application runtime across cloud providers and on-premises environments, customers have more freedom to run workloads wherever they are best suited. However, with this freedom has come some unforeseen challenges.</p><p>First, setting up clusters in a uniform manner across environments is tedious and risk prone. Infrastructure-as-Code tools help with this, but configuration drift can still occur, resulting in clusters being perceived as fragile and finicky - the exact opposite of what the cloud-native community promotes with Kubernetes.</p><p>Second, another challenge organizations have is figuring out how to bring the provisioning experience of the cloud back into their data centers. With EKS, customers can easily login to a web UI and provision a cluster. With traditional on-premises Kubernetes deployments, emails are exchanged between application and operations teams, tickets created, and one-off scripts are run.</p><p>Upbound Cloud provides an easy to use provisioning and management interface for Kubernetes clusters running in any environment, including EKS-D clusters. Administrators define cluster configuration for both cloud and on-premises environments inside of a single repository of Crossplane configuration YAML, then give application teams access to Upbound Cloud where they're able to provision and manage clusters running across cloud and data center environments.</p><h2>What’s Next for Upbound and EKS-D</h2><p>These scenarios are being enabled thanks to the work the Crossplane community is doing with <a href="https://github.com/crossplane-contrib/provider-capi" target="_blank">crossplane/provider-capi</a>. Though it's still early, by using provider-capi, Crossplane users will be able to use Cluster API (CAPI) through Crossplane to manage Kubernetes clusters (such as ones running EKS-D) on-premises or in the cloud.</p><p>Upbound Reference Platforms (like <a href="https://cloud.upbound.io/registry/upbound/platform-ref-aws" target="_blank">upbound/platform-ref-aws</a>) can compose EKS-D deployments with provider-capi resources (Clusters, MachineDeployments) and configure additional logging, telemetry, and GitOps controllers using the extensive set of Crossplane Providers including provider-aws, provider-helm, and more.</p><p>While composing platform configurations on top of Cluster API is powerful by itself, Crossplane also supports defining Cluster Infrastructure underneath CAPI, allowing EKS-D users to define custom infrastructure configurations with the exact specifications they require.</p><p>Our team is excited to be partnering with AWS on this major milestone, and enthusiastic about helping customers solve challenges the cloud-native community has been dealing with for many years.</p><p>Upbound Product Manager Phil Prasek and AWS Principal Engineer Jay Pipes will be talking more about our partnership during our upcoming <a href="https://blog.upbound.io/modernizing-with-control-planes">Crossplane Community Day</a>.</p><p>&nbsp;</p><hr><p>&nbsp;</p><p><i>PS: Powering every Upbound Cloud account is the CNCF popular open source project, </i><a href="https://crossplane.io/" target="_blank"><i>Crossplane</i></a><i>. Crossplane is hosted on </i><a href="https://github.com/crossplane" target="_blank"><i>Github</i></a><i>, and we have a thriving community on </i><a href="https://slack.crossplane.io/" target="_blank"><i>Slack</i></a><i> and </i><a href="https://twitter.com/crossplane_io" target="_blank"><i>Twitter</i></a><i>. Come join the fun!</i></p>

AWS and Upbound Partner on EKS-D Launch

<p>Continuous Integration and Continuous Deployment, better known by the acronym CI/CD is the foundation of modern DevOps.</p><p>At the Crossplane Community Day Europe virtual event, Jan Willies, Platform Architect at Accenture outlined <a href="https://www.youtube.com/watch?v=KqRolAqKV_8&amp;list=PLj6h78yzYM2OFWBatWHbWgyoLNmCyAqJ0&amp;index=10" target="_blank">how to connect all the different platform pieces together for automated CI/CD</a>. Accenture is one of the world's largest consulting firms and has been actively engaged in many different cloud native efforts including Kubernetes and <a href="https://blog.upbound.io/crossplane.io">Crossplane</a>.</p><p>The first part of the CI/CD process is onboarding a new application, which can be an involved and complicated process in many organizations. Willies explained that onboarding a new application often involves lots of manual steps including finding the right settings for a Git code repository, connecting code integration systems and figuring out deployment manifests. From a platform engineering perspective, Willies said that there is a clear need and desire to automate the application onboarding process.</p><blockquote><p>"Obviously the application needs to be deployed and for the GitHub software we also need to connect the Git repo and we need to connect the execution environment," Willies said. “We can do that manually, but we can also use API's for that.”</p></blockquote><h2>Using Crossplane for Application Onboarding</h2><p>Willies explained that among Crossplane's core features is the Crossplane Provider capability that implements arbitrary API's and mirrors them into Kubernetes.&nbsp;</p><blockquote><p>"This is really helpful because we now have drivers for each of those API's and are able to configure the systems in a standardized way via Kubernetes," Willies said.<br>&nbsp;</p></blockquote><p>So for example, a YAML configuration file can be created for a given Git repository. That said, Willies noted that there is still a need to know what to put in the YAML files, since they can potentially expose everything the upstream provider based API enables. That's where Crossplane Compositions come into play.</p><blockquote><p>"A composition is a templated resource, which acts as a controller and creates a single YAML file," Willies said.</p></blockquote><p>A Crossplane Composition from a technical perspective is an XRC, a Composite Resource Claim (<a href="https://crossplane.io/docs/v0.14/getting-started/compose-infrastructure.html" target="_blank"><strong>XRC)</strong></a> which enables the composition to abstract the underlying implementation details from the assembled provider resources.</p><p>In a <a href="https://www.youtube.com/watch?v=KqRolAqKV_8&amp;list=PLj6h78yzYM2OFWBatWHbWgyoLNmCyAqJ0&amp;index=10" target="_blank">demo</a>, Willies showed how it was possible to easily onboard a new application using Crossplane. The demo involved creating a GitLab project, an Argo CD repository and an Argo CD application as well. The providers for<a href="https://github.com/crossplane-contrib/provider-gitlab" target="_blank"> <strong>GitLab</strong></a> and ArgoCD are all available in the Crossplane GitHub repository for anyone to use and create their own onboarding approach.</p><blockquote><p><span style="background-color:rgb(255,255,255);color:rgb(3,7,36);font-family:Avenir-Roman, Arial, sans-serif;">"All of this is open source," Willies said.</span></p></blockquote>

Connecting the Platform Pieces for Automated CI/CD

<p>This article was <a href="https://itnext.io/gitopsify-cloud-infrastructure-with-crossplane-and-flux-d605d3043452" target="_blank" rel="noopener noreferrer">originally posted on medium</a>. In this article, we are going to learn how to automate the provisioning of cloud resources via <a href="http://crossplane.io/" target="_blank" rel="noopener noreferrer">Crossplane</a> and combine it with GitOps practices.</p><p>You will most benefit from this blog if you are a Platform or DevOps Engineer, Infrastructure Architect or Operations Specialist.</p><p><i>Note: If you are one of the above, we are hiring for various engineering roles. </i><a href="https://grnh.se/54d95d844us" target="_blank" rel="noopener noreferrer"><i>View our open roles here</i></a><i>.</i></p><blockquote><p>If you are new to GitOps, read more about it in my blog <a href="https://itnext.io/gitops-with-kubernetes-740f37ea015b" target="_blank" rel="noopener noreferrer">GitOps with Kubernetes</a></p></blockquote><p>Let’s set the stage by imagining the following context. We are working as a part of a Platform</p><p>Team in a large organization. Our goal is to help Development Teams to onboard get up to speed with using our Cloud Infrastructure. Here are a few base requirements:</p><ul><li>Platform Team doesn’t have resources to deal with every request individually, so there must be a very high degree of <strong>automation</strong></li><li>Company policy is to adopt the <strong>principle of least privilege</strong>. We should expose the cloud resources only when needed with the lowest permissions necessary.</li><li>Developers are not interested in managing the cloud, they should only <strong>consume cloud resources</strong> without even needing to login to a cloud console.</li><li>New Teams should get <strong>their own</strong> set of cloud resources for <strong>self-managing</strong> when on-boarding to the Platform.</li><li>It should be easy to provision new cloud resources on demand.</li></ul><h2>Initial Architecture</h2><p>The requirements lead us to an initial architecture proposal with the following high-level solution strategy.</p><ul><li>create template repositories for various types of workloads (using <a href="https://backstage.io/docs/features/software-templates/software-templates-index" target="_blank" rel="noopener noreferrer">Backstage Software Templates</a> would be helpful)</li><li>once a new Team is onboarded and creates the first repository from a template, it will trigger a CI pipeline and deploy common infrastructure components by adding the repository as Source to Flux infrastructure repo</li><li>once a Team wants to create more cloud infrastructure, they can place the Crossplane claim YAMLs in the designated folder in their repository</li><li>adjustments to this process are easily implemented using Crossplane Compositions</li></ul><blockquote><p>In a real world scenario we would manage Crossplane also using Flux, but for demo purposes we are focusing only on the application level.</p></blockquote><p>The developer experience should be similar to this:</p>

<h2>Tools and Implementation</h2><p>Knowing the requirements and initial architecture, we can start selecting the tools. For our example, the tools we will use are <a href="https://fluxcd.io/" target="_blank" rel="noopener noreferrer">Flux</a> and <a href="https://crossplane.io/" target="_blank" rel="noopener noreferrer">Crossplane</a>.</p><blockquote><p>We are going to use Flux as a GitOps engine, but the same could be achieved with ArgoCD or Rancher Fleet.</p></blockquote><p>Let’s look at the architecture and use cases that both tools support.</p><h3>Flux Architecture Overview</h3><p>Flux exposes several components in the form of Kubernetes CRDs and controllers that help with expressing a workflow with the GitOps model. Short description of 3 major components. All those components have their corresponding CRDs.</p>

<p><a href="https://fluxcd.io/docs/components/source/" target="_blank" rel="noopener noreferrer">Source Controller</a> main role is to provide a standardized API to manage sources of the Kubernetes deployments; Git and Helm repositories.</p>

<p><a href="https://fluxcd.io/docs/components/kustomize/" target="_blank" rel="noopener noreferrer">Kustomize Controller</a> this is a CD part of the workflow. Where source controllers specify sources for data, this controller specifies what artifacts to run from a repository.</p>

<p><i>This controller can work with kustomization files, but also plain Kubernetes manifests</i></p><p><a href="https://fluxcd.io/docs/components/helm/" target="_blank" rel="noopener noreferrer">Helm Controller</a> this operator helps manage Helm chart releases containing Kubernetes manifests and deploy them onto a cluster.</p>

<h2>Crossplane Architecture Overview</h2><p>Let’s look at what the <a href="https://blog.upbound.io/__GHOST_URL__/scaling-kubernetes-to-thousands-of-crds/" target="_blank" rel="noopener noreferrer">Crossplane</a> component model looks like. A word of warning, if you are new to Kubernetes this might be overwhelming, but there is value in making an effort to understand it. The below diagram shows the Crossplane component model and its basic interactions.</p>

<p>Learn more about Crossplane in my blog “Infrastructure as Code: the next big shift is here”</p><h3>Demo</h3><p>If you want to follow along with the demo, clone <a href="https://github.com/Piotr1215/crossplane-gitops-model" target="_blank" rel="noopener noreferrer">this repository</a>, it contains all the scripts to run the demo code.</p><h3>Prerequisites</h3><p>In this demo, we are going to show how to use Flux and Crossplane to provision an EC2 instance directly from a new GitHub repository. This simulates a new team onboarding to our Platform.</p><p>To follow along, you will need AWS CLI configured on your local machine.</p><blockquote><p>Once you obtain credentials, configure the default profile for AWS CLI following <a href="https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-quickstart.html#cli-configure-quickstart-config" target="_blank" rel="noopener noreferrer">this tutorial</a>.</p></blockquote><p>Locally installed you will need:</p><ul><li>Docker Desktop or other container run time</li><li>WSL2 if using Windows</li><li>kubectl</li></ul><p>Run <code>make</code> in the root folder of the project, this will:</p><p>If you are running on a Mac, use <code>make setup_mac</code> instead of <code>make</code></p><ul><li>Install <a href="https://kind.sigs.k8s.io/" target="_blank" rel="noopener noreferrer">kind</a> (Kubernetes IN Docker) if not already installed</li><li>Create KIND cluster called crossplane-cluster and swap context to it</li><li>Install crossplane using helm</li><li>Install crossplane CLI if not already installed</li><li>Install flux CLI if not already installed</li><li>Install AWS provider on the cluster</li><li>Create a temporary file with AWS credentials based on the default CLI profile</li><li>Create a secret with AWS credentials in the crossplane-system namespace</li><li>Configure AWS provider to use the secret for provisioning the infrastructure</li><li>Remove the temporary file with credentials so it’s not accidentally checked in the repository</li></ul><p>Following tools need to be installed manually</p><ul><li><a href="https://github.com/cli/cli#installation" target="_blank" rel="noopener noreferrer">GitHub CLI gh</a></li></ul><blockquote><p><strong>IMPORTANT: </strong>The demo code will create a small EC2 Instance in eu-central-1 region. The instance and underlying infrastructure will be removed as part of the demo, but please make sure all the resources were successfully removed and in case of any disruptions in the demo flow, be ready to remove the resources manually.</p></blockquote><h2>Setup Flux Repository</h2><ul><li>Create a new KIND cluster with <code>make</code>, this will install Crossplane with AWS provider and configure secret to access selected AWS account</li><li>Flux CLI was installed as part of the Makefile script, but optionally you can configure shell completion for the CLI . <code>&lt;(flux completion zsh)</code></li><li>Refer to the <a href="https://fluxcd.io/docs/installation/#install-the-flux-cli" target="_blank" rel="noopener noreferrer">Flux documentation</a> page for more installation options</li><li>Create an <a href="https://github.com/settings/tokens" target="_blank" rel="noopener noreferrer">access token</a> in GitHub with full repo permissions.</li></ul>

<ul><li>export variables for your GitHub user and the newly created token</li></ul><p><code>export GITHUB_TOKEN=&lt;token copied form GitHub&gt;</code></p><p><code>export GITHUB_USER=&lt;your user name&gt;</code></p><ul><li>use flux to bootstrap a new GitHub repository so flux can manage itself and underlying infrastructure</li></ul><blockquote><p>Flux will look for GITHUB_USER and GITHUB_TOKEN variables and once found will create a private repository on GitHub where Flux infrastructure will be tracked.</p></blockquote>

<h2>Setup Crossplane EC2 Composition</h2><p>Now we will install a <a href="https://crossplane.io/docs/v1.6/concepts/composition.html#configuring-composition" target="_blank" rel="noopener noreferrer">Crossplane Composition</a> that defines what cloud resources to crate when someone asks for an EC2 claim.</p><ul><li>setup Crossplane composition and definition for creating EC2 instances</li><li><code>kubectl crossplane install configuration piotrzan/crossplane-ec2-instance:v1</code></li><li>fork repository with the EC2 claims <code>gh repo fork https://github.com/Piotr1215/crossplane-ec2</code> and answer YES when prompted whether to clone the repository</li></ul><h2>Clone Flux Infra Repository</h2><ul><li>clone the flux infra repository created in your personal repos</li><li><code>git clone git@github.com:${GITHUB_USER}/flux-infra.git</code></li><li><code>cd flux-infra</code></li></ul><h2>Add Source</h2><ul><li>add source repository to tell Flux what to observe and synchronize</li><li>Flux will register this repository and every 30 seconds check for changes.</li><li>execute below command in the flux-infra repository, it will add a Git Source</li></ul>

<ul><li>the previous command created a file in clusters/crossplane-cluster subfolder, commit the file</li></ul><p><code>git add .</code></p><p><code>git commit -m "Adding Source Repository"</code></p><p><code>git push</code></p><ul><li>execute <code>kubectl get gitrepositories.source.toolkit.fluxcd.io -A</code> to see active Git Repositories sources in Flux</li></ul><h2>Create Flux Kustomization</h2><ul><li>setup watch on the AWS managed resources, for now, there should be none</li><li><code>watch kubectl get managed</code></li><li>create Flux Kustomization to watch for a specific folder in the repository with the Crossplane EC2 claim</li></ul>

<p><code>git add .</code></p><p><code>git commit -m "Adding EC2 Instance"</code></p><p><code>git push</code></p><ul><li>after a minute or so you should see a new EC2 Instance being synchronized with Crossplane and resources in AWS</li></ul>

<h2>Recap</h2><p>Let’s take a step back and make sure we understand all the resources and repositories used.</p>

<p>The first repository we have created is what Flux uses to manage itself on the cluster as well as other repositories. To tell Flux about a repository with Crossplane EC2 claims, we have created a <code>GitSource</code> YAML file that points to the HTTPS address of the repository with the EC2 claims.</p><p>The EC2 claims repository contains a folder where plain Kubernetes manifest files are located. To tell Flux what files to observe, we have created a <code>Kustomization</code> and linked it with <code>GitSource</code> via its name. <code>Kustomization</code> points to the folder containing K8s manifests.</p><h2>Cleanup</h2><ul><li>to clean up the EC2 Instance and underlying infrastructure, remove the claim-aws.yaml demo from the crossplane-ec2 repository</li></ul><p><code>rm ec2-claim/claim-aws.yaml</code></p><p><code>git add .</code></p><p><code>git commit -m "EC2 instance removed"</code></p><ul><li>after a commit or timer lapse Flux will synchronize and Crossplane will pick up the removed artifact and delete cloud resources</li></ul><blockquote><p>the ec2-claim folder must be present in the repo after the claim yaml is removed, otherwise Flux cannot reconcile</p></blockquote><h2>Manual Cleanup</h2><p>In case you cannot use the repository, it's possible to cleanup the resources by deleting them from flux.</p><ul><li>deleting Flux kustomization <code>flux delete kustomization crossplane-demo</code> will remove all the resources from the cluster and AWS</li><li>to clean up the EC2 Instance and underlying infrastructure, remove the ec2 claim from the cluster <code>kubectl delete VirtualMachineInstance sample-ec2</code></li></ul><h2>Cluster Cleanup</h2><ul><li>wait until <code>watch kubectl get managed</code> the output doesn't contain any AWS resources</li><li>delete the cluster with <code>make cleanup</code></li><li>optionally remove the <code>flux-infra</code> repository</li></ul><h2>Summary</h2><p><a href="https://www.youtube.com/watch?v=Y7x7Nka_d_0&amp;list=PLHhKcdBlprMdIMzUZX6ho0OPTikTamLwa&amp;index=97&amp;ab_channel=ContainerDays" target="_blank" rel="noopener noreferrer">GitOps with Flux or Argo CD and Crossplane</a> offers a very powerful and flexible model for Platform builders. In this demo, we have focused on the applications side with Kubernetes clusters deployed some other way, either with Crossplane or Fleet or Cluster API etc.</p><p>What we achieved on top of using Crossplane’s Resource Model is the fact that we do not interact with kubectl directly any longer to manage resources, but rather delegate this activity to Flux. Crossplane still runs on the cluster and reconciles all resources. In other words, we’ve moved the API surface from kubectl to Git.</p>

GitOpsify Cloud Infrastructure with Crossplane and Flux

<h2>“Sumbry’s Philosophy on building a great engineering team.”</h2><p>“We’re hiring a team of rockstars!”</p><p>Every time I hear that quote, I cringe. Now while that phrase normally compares hiring at a startup to starting a band I’m going to adjust this analogy to a sports one, specifically American Football since I’m a huge College Football fan. Anytime I hear that a company wants to hire a team of rockstars, I immediately translate that to:</p><p>“We’re hiring a team of quarterbacks.”</p><p>… which sounds absolutely ludicrous! Let’s just look at what a typical football team, specifically its offense, actually looks like and all the different positions it entails:</p>

Why Upbound

Upbound Elevates Platforming with Crossplane Through New Builder Tooling and Platform Services

Upbound

Start for Free

Pricing

17 Added benefits of Upbound over OSS Crossplane

Using Upbound to Roll Out Cluster-as-a-Service to Your Developers

Resources

Events

Docs

Blog

Contact Support

Report a Vulnerability

Streamlining Platform Building with Crossplane: Working with with Upbound's New Platform Engineering Tools

Open Source Crossplane or Upbound? 17 Added Benefits of Using Upbound

Featured

Upbound Elevates Platforming with Crossplane Through New Builder Tooling and New Core Platform Services

Now Upbound Official Packages Meet The Most Demanding Enterprise Requirements

Popular topics:

Filter by year:

Subscribe to our Newsletter

Press Releases

Upbound Launches Spaces, Bringing the Power of Managed Control Planes to Fully Self-Hosted Environments

Upbound’s Platform for Platform Teams Now Available in AWS Marketplace

Upbound Contributes Control Plane Provider Technology to the Crossplane Project, Governed by the Cloud Native Computing Foundation