Streamlining Crossplane Testing: End-to-End Validation with Uptest

In our previous testing blog, we covered rendering and validating compositions. This is a great start, but we still want to be confident that our compositions will work in the cloud. Here, we have an end-to-end tool to ensure that our compositions work in this real world: Uptest. Let's take a look at what Uptest is together.

Introduction to Uptest

Uptest is a tool for testing and validating Crossplane providers and configuration packages. This tool is utilized to test the behavior of Crossplane controllers, resource management operations, and other Crossplane components.

The primary goal of Uptest is to facilitate the testing process of Crossplane resources. It integrates seamlessly with Crossplane and provides a testing infrastructure that enables users to create and run test scenarios for validating the reliability and functionality of Crossplane resources.

We will discuss how Uptest, which includes many stages during the test, does these. But first, let's take a closer look at its history.

Brief History

When the idea of Uptest was first introduced, the aim was to increase the speed of development by creating a tool that would enable quick testing of new resources and features in Upbound Official Providers. In this context, the main purpose was to create a short and compact end-to-end pipeline, which would include the creation, assertion, and deletion of Managed Resources.

In the long run, however, the tool has become more than that. It has been integrated into many crossplane providers and configuration packages. It has also become a tool for testing development or production environments not only for developers but for all users.

What are the Highlights? / Overview

Uptest stands out with its comprehensive steps and simple user interface. It is useful to draw attention to some important points:

• The user does not have to write the Go Code. Also, the user will not write any test framework-specific code. Users can start testing by preparing example manifests as input in YAML format.
• Uptest can be used for testing Crossplane Providers and Crossplane Configurations.
• Uptest automatically creates tests based on examples/inputs and templates. The user doesn’t need to write test files.
• Uptest can cover the whole lifecycle of a resource: Creation, Update, Import, and Deletion.
• Uptest has some specific hooks that can be configured via annotations.
• There are some helper functionalities:
• Injecting arbitrary values to the manifests with ${data.key}.
• Randomize metadata.name with ${Rand.RFC1123Subdomain}.
• Recently contributed to CNCF https://github.com/crossplane/uptest.

How Uptest Works

Uptest is not a test framework written from scratch. Uptest is also not a generic testing framework for every Kubernetes resource. Instead, it is a test tool built on top of a selected declarative framework that has the ability to test in the Kubernetes environment, equipped with customizations specific to the Crossplane environment.

When Uptest was first written, it used kuttl as the underlying test framework to have good declarative testing capabilities. Over time, it was realized that there are better and more compatible alternatives and a perception that kuttl isn't being actively maintained, so it was decided to evaluate an alternative underlying framework.

End to End testing with Uptest

Uptest can be easily run both locally and in GitHub workflows. You can easily integrate Uptest into your CI Pipeline and test your Compositions or MRs. This integration is done via the reusable workflow. When you call this workflow in your own GitHub repository with the proper variables, then you can trigger your Uptest runs easily with pull-request comments like the following:

Here are a few examples of integrated repos:

• provider-upjet-aws (Provider Package)
  • Makefile integration
  • GitHub Action Integration
• configuration-aws-eks (Configuration Package)
  • Makefile Integration
  • GitHub Action Integration

Now, let's look at the innovations and improvements in the v1.0.0 version of Uptest.

Uptest Improvements: What is new in the v1.0.0 Version?

Some alternatives were considered during the underlying test framework of uptest. The alternative considered is chainsaw, which provides a declarative approach to test Kubernetes operators and controllers. While chainsaw is designed for testing operators and controllers, it can declaratively test any Kubernetes objects. Chainsaw is an open-source tool initially developed to define and run Kyverno end-to-end tests.

One of the most important news from the v1.0.0 version is migrating the underlying test framework from kuttl to chainsaw.

Why Chainsaw? What Are the Advantages?

Now, let's look at why we need to change this underlying framework.

Similar Functionality - Backward Compatibility

Although we changed the underlying test framework, we did not change the source code much, providing backward compatibility and preserving the old behaviors even with the major version release.

Up-to-date - Maintenance

While kuttl meets the basic requirements for Uptest so far, chainsaw is a more up-to-date and maintainable tool. So, one of the main motivations is responsive maintenance, detailed documentation, and frequent releases for the underlying test tool.

Assertion trees

In Uptest, we make assertions like status condition checks. We want to improve and enhance the assertions by adding comprehensive field assertions like checking values of fields in status.atProvider and spec.forProvider. The assertion trees feature of the chainsaw easily and declaratively supports this requirement. An already used example:

This is from the Apply Assertion step. The goal is to check the value of the UpToDate condition. Before, we had to run an imperative kubectl wait command for this assertion. However, now we can use the chainsaw’s syntax to determine the desired condition and check its value declaratively without any imperative comment.

An iteration example from chainsaw’s documentation:

Another example:

Better UI

Chainsaw has a more descriptive, simple, and clear UI. This is helpful to follow the test logs easily.

More modular management

Chainsaw test files and framework support more modular management on the tests. In uptest, the test files are generated by using go template libraries. Chainsaw’s more modular testing strategy and configurable APIs give more flexibility in our testing design

Community/User Requests

Community members and some Uptest users have this type of request and proposal to switch to chainsaw.

Migration-related Changes and Enhancements

• Previously, the Apply, Update, Import, and Delete operations were part of a single end-to-end test in kuttl. Now, each operation is treated as a separate test in chainsaw, improving management and isolation. Each test consists of distinct steps, simplifying log tracking.
• Updated YAML templates to align with chainsaw requirements, ensuring compatibility with the framework’s features.
• Removed outdated kuttl specific templates and references, decluttering the test setup.
• Merged operation and assertion steps into single files in chainsaw, streamlining test execution.
• Enhanced condition checks and resource cleanup processes to be more robust and error-aware, promoting cleaner and safer test executions.

New Flags Introduced

• skip-import and skip-update to disable the Import and Update steps during tests. This is especially useful for the Configuration Packages tests.
• log-collect-interval to set specific intervals for log collection during tests, enhancing debugging and monitoring capabilities. Before, this was hard coded to 10 seconds, and this caused the “log pollution”.
• render-only to render test configurations without execution, useful for verifying setups without running full tests. This is also an important feature, especially for the developers and users of the uptest. Without running the tests, you can also easily see what will happen during tests via this command. This flag is very useful for local tests.

Updated Logging Practices

• Shifted from using kubectl get managed/claim to the crossplane beta trace for logging, better integrating with the crossplane ecosystem and providing clearer logs.
• The log-collect-interval now defaults to 30 seconds, allowing for flexible and effective log management during tests.

Breaking Changes

• The user interface/setup was changed for the uptest. kuttl-based setup

Before:

After:

• To consume the chainsaw installation, you must bump the crossplane/build dependency to the f67b099.
• The value type of the --timeout flag was changed to the duration for the chainsaw compatibility.

Here is an example switch for the new uptest version.

Apart from the breaking changes, the old behavior of the uptest was preserved, and there is no need for any change in the input example files for the tests.

Try Uptest / Uptest Quickstart

Now let's try Uptest together. We will use the upbound/configuration-app repository. The reason for this, is that you will be able to clone and try uptest without any external cloud dependency and standard github-hosted runners. You will be able to test the uptest in your local environment. Let's go step by step:

1. Clone Repository

2. Investigate relevant Make targets of uptest and e2e

3. Pull Crossplane build module

4. Run Test