Introducing "function-azresourcegraph" for Crossplane: Query Azure Resources With Ease

As cloud infrastructure grows in complexity, managing and querying resources across multiple environments becomes increasingly challenging. Crossplane, the open-source multicloud control plane, has been a game-changer in simplifying infrastructure management. Today, I’m excited to introduce a new addition to the Crossplane ecosystem: the function-azresourcegraph function, now available on GitHub.

This function enables you to query Azure resources using Azure Resource Graph, a powerful API that provides efficient and scalable resource exploration across your Azure environment. Whether you're managing a handful of resources or thousands, function-azresourcegraph simplifies the process of fetching and working with Azure resource data directly within Crossplane.
 

What Is function-azresourcegraph?

function-azresourcegraph is a Crossplane Composition Function that integrates Azure Resource Graph into your Crossplane workflows. It allows you to:

• Query Azure Resources: Use Azure Resource Graph's powerful query language to fetch details about your Azure infrastructure, such as VM sizes, storage account configurations, network settings, and more.
• Fetch Resource Data: Retrieve real-time information about your Azure resources, including their properties, configurations, and relationships.
• Use Query Results in Compositions: Leverage the fetched data during the instantiation of composed resources in Crossplane. For example, you can dynamically select VM sizes, configure storage accounts, or set up network rules based on the observed state of your Azure environment.

This function is particularly useful for scenarios where you need to make informed decisions based on the current state of your Azure infrastructure. For example, you can use it to:

• Dynamically query the existing infrastructure data.
• Fetch existing resource IDs or properties to use in compositions.
• Import and manage existing Azure resources within Crossplane, ensuring consistent management across your infrastructure.
   

How Does It Work?

The function-azresourcegraph function leverages Crossplane's Composition Functions framework, which allows you to extend Crossplane's capabilities with custom logic. Here's a high-level overview of how it works:

1. Define a Query: You provide an Azure Resource Graph query in your Crossplane composition.
2. Execute the Query: The function sends the query to Azure Resource Graph and retrieves the results.
3. Process the Results: The query results are returned to Crossplane, where they can be used to inform resource provisioning, updates, or other actions

Examples

There is a great library of Query examples that demonstrate the power of Azure Resource Graph API at Microsoft documentation:

• Starter query samples
• Advanced query samples
• Sample queries by category

In the rest of this section, we will focus on function-azresourcegraph usage examples in the Crossplane Composition context.

Execute Static Query and Publish Results to XR status

In our first example, we query Azure to retrieve a list of existing Virtual Machines (VMs) that are currently deployed in the environment. The query results are stored in the status of our composite resource under status.azResourceGraphQueryResult.

This example query is a static string. You predefine it within the Composition and publish the query results to Composite Resource status for further consumption by a human or a machine.
 

Keep in mind that in this case, you need to predefine azResourceGraphQueryResult status field in your XRD definition to make this work end-to-end.

Please check the function repository for a full testable example that you can run with crossplane render and eventually end-to-end.

Publish Results to Context

In a similar manner to the previous example, you can execute a static query but publish it to the Function pipeline context instead of the XR status

In this case, the query result will be accessible for the next function in the pipeline so you can use the results of the query further down in your Composition.

Check the full testable example in the function repo.

Later in this guide, we will see how we can use context data to drive the resource import.

Reference Query from the existing API field

In some cases, you want to construct your Query in a dynamic manner.
You can use one of the generic Composition Functions, such as function-patch-and-transform, function-go-templating, or function-kcl, to dynamically construct the query string according to your custom logic.

In the next pipeline step you can reference the dynamically constructed query with the queryRef notation. The queryRef field allows referencing any arbitrary field within the Pipeline Context or Status of a composite resource when executing an Azure Resource Graph query. This enables dynamic query construction based on runtime data rather than relying on a static query string. For more details, refer to the Query Reference documentation. Check the example implementation below and the full example in the repository.

Resource Import

Given all the power you get with function-azresourcegraph, you can do interesting things like dynamically importing existing resources under Crossplane explicit control.

Consider this example from the function repo.

Here we are querying the VirtualNetwork resource by a specific tag and publishing results to the Context field
 

In the next step of the pipeline, we are using function-kcl to drive the import of the resource in an Observe-only manner. We are retrieving the query result from the context and using it to propagate the crossplane.io/external-name annotation with the dynamically retrieved identifier.

In this example, we fully automated the import process from the upstream Crossplane documentation. 

Running Multiple Queries

There are no limits on the number of queries within the single Composition so you can pipe many of them and publish results in different places
 

Example Library

As you can see, by combining function-azresourcegraph with other functions in the Pipeline, you can build very powerful Compositions.

Check the example library, where I tried to collect useful representative examples that we already discussed above and some others. I plan to collect more of them as we gather experience building dynamic Compositions with function-azresourcegraph.

In the next section, we discuss the end user experience with this function in the real large-scale infrastructure.

Real-World Success: A Customer Perspective

To highlight the real-world impact of function-azresourcegraph, I’m thrilled to share insights from a Cloud Architect at one of the largest health insurance companies in the world. Their organization manages a massive, high-scale Azure infrastructure, and they’ve been actively using function-azresourcegraph in production. Here’s what they had to say:

Platform Builder Perspective 

“As a cloud architect at a large health insurance company, managing our Azure infrastructure at scale is a critical part of our operations. We handle thousands of resources across multiple regions, and ensuring that our infrastructure is accurately represented and managed in Crossplane is a top priority. When we discovered function-azresourcegraph, it immediately stood out as a tool that could help us streamline our workflows.

One of the biggest challenges we faced was retrieving accurate, up-to-date information about our Azure resources and using that data to inform our Crossplane compositions. With function-azresourcegraph, we were able to bridge this gap effectively. Here are a few ways we’re using it:

1. Retrieve Storage Account Details: We use the function to query Azure Resource Graph for details about our storage accounts, such as their names, SKUs, and configurations. This data is then imported into Crossplane, allowing us to manage these resources declaratively and ensure they align with our desired state.
2. Import VM Sizes and Configurations: We dynamically fetch available VM sizes and configurations from specific Azure regions using the function. This information is used in our Crossplane compositions to provision VMs with the most cost-effective and regionally available options.
3. Optimize Route Calculation:  We replaced the Crossplane Terraform provider, which was resource-intensive and slow for read-only operations, with a function. This new approach fetches advertised routes in seconds, providing real-time data with improved efficiency.
4. Dynamic Resource Provisioning: We use a Graph Query to fetch resource properties dynamically instead of relying on static mappings. This ensures real-time data is applied during provisioning, improving flexibility and accuracy in resource configurations.
5. Extend Existing Infrastructure: With minimal input, you can use a query to fetch configurations from your existing infrastructure, making it easier to extend and integrate new resources seamlessly.
6. Discover and Import Existing Resources: We use function-azresourcegraph to discover and import existing Azure resources, like virtual networks and load balancers, into Crossplane, ensuring consistent management within our control plane. With little effort, a query can dynamically fetch and import all resources in a resource group, enabling the migration of existing infrastructure to the control plane.

The integration with Crossplane has been seamless. By using function-azresourcegraph, we’ve been able to make our infrastructure management more dynamic and data-driven. For example, we now use it to validate resource configurations before provisioning, ensuring that new resources meet our standards from day one.

What I appreciate most about function-azresourcegraph is its ability to simplify the retrieval and import of Azure resource data into Crossplane. It has saved us countless hours and reduced the risk of human error. It’s become an indispensable tool in our cloud management toolkit.

I’m excited to see how the community adopts and builds on this function. For anyone managing Azure resources at scale, I highly recommend giving function-azresourcegraph a try. It’s a game-changer.”

Key Takeaways from End User Experience

• Data Retrieval: function-azresourcegraph simplifies the process of fetching detailed information about Azure resources, such as storage accounts and VMs.
• Resource Import: The function enables seamless import of existing Azure resources into Crossplane, ensuring consistent management across your infrastructure.
• Dynamic Compositions: By integrating query results into Crossplane compositions, you can make resource provisioning and management more intelligent and data-driven.

Why I Built This Function

function-azresourcegraph was born out of a customer request to seamlessly integrate Azure Resource Graph with Crossplane. The goal was to enable dynamic resource discovery, configuration, and management using real-time Azure resource data. From day one, the function was designed to address real-world challenges faced by organizations managing large-scale Azure infrastructure.

This function is a testament to the flexibility and extensibility of Crossplane. By combining Crossplane's declarative approach with Azure Resource Graph's querying capabilities, we can create more intelligent, dynamic, and efficient infrastructure management workflows.

Join the Community

function-azresourcegraph is open-source and available to everyone. I encourage you to try it out, share your feedback, and contribute to its development. Whether you're a Crossplane enthusiast, an Azure expert, or just getting started with cloud infrastructure, your input is valuable.

• Upbound Marketplace: - https://marketplace.upbound.io/functions/upbound/function-azresourcegraph 
• GitHub Repository: https://github.com/upbound/function-azresourcegraph/
• Issues and Feedback: If you encounter any issues or have suggestions for improvement, please open an issue on GitHub.
• Contributions: Pull requests are welcome! Let’s work together to make this function even better.