Skip to content

2023

How to Perform Efficient E2E Testing with Chainsaw

chainsaw

Introduction

Chainsaw is a powerful tool that enables efficient end-to-end (E2E) testing, making it an essential asset for software developers. Built to be stronger and more cohesive than KUTTL, Chainsaw offers a superior suite of features for Kubernetes (k8s) operators and objects. In this blog post, we will explore how to leverage Chainsaw for efficient E2E testing and enhance your overall testing workflow. Let's dive in!

Table of Contents

Section 1: Installing Chainsaw

1.1 Downloading Chainsaw

To get started with Chainsaw, you'll first need to download the appropriate release for your operating system and architecture. The releases are available on the official GitHub page at the following link: Release

Here's a breakdown of the available releases:

  • For macOS (Darwin) Users:
  • Intel-based Macs: chainsaw_darwin_amd64.tar.gz

  • Apple Silicon Macs: chainsaw_darwin_arm64.tar.gz

  • For Linux Users:

  • 32-bit: chainsaw_linux_386.tar.gz
  • 64-bit (AMD): chainsaw_linux_amd64.tar.gz
  • 64-bit (ARM): chainsaw_linux_arm64.tar.gz

Each release also comes with associated .pem, .sbom, .sbom.pem, and .sig files. These are used for verification and security purposes. If you're interested in verifying the integrity of your download, you can use these files. However, for the purpose of this guide, we'll focus on the main .tar.gz files.

1.2 Installing Chainsaw

Once you've downloaded the appropriate .tar.gz file for your system, you can proceed with the installation. Here's a step-by-step guide:

  • Extract the Tarball:
tar -xzf chainsaw_<your_version_and_architecture>.tar.gz
  • Move the Extracted Binary:
sudo mv chainsaw /usr/local/bin/
  • Verify the Installation:
chainsaw --version

Section 2: Writing Configurations

2.1 Understanding Chainsaw Configurations

Chainsaw is a robust tool for end-to-end (e2e) testing in Kubernetes environments. It offers a flexible configuration system that allows users to define their testing parameters and conditions. The configuration can be set up in two primary ways:

  • Configuration File: This is a structured file where you can define all your testing parameters.
  • Command-line Flags: These are options you can pass directly when running Chainsaw commands. If both a configuration file and command-line flags are provided, the flags will override the settings in the configuration file.

Chainsaw follows a hierarchy in loading its configurations:

  1. User-specified Configuration: If you provide a configuration file explicitly using a command-line flag.
  2. Default Configuration File: If no configuration is specified, Chainsaw will search for a default file named .chainsaw.yaml in the current directory.
  3. Internal Default Configuration: If neither of the above is available, Chainsaw will resort to a default configuration embedded within its binary.

To specify a custom configuration, use the following command:

chainsaw test --config path/to/your/config.yaml
2.2 Writing Chainsaw Configurations

When writing a configuration for Chainsaw, you'll be working with a YAML file. Here's a basic example of what this might look like:

apiVersion: chainsaw.kyverno.io/v1alpha1
kind: Configuration
metadata:
  name: custom-config
spec:
  timeout: 45s
  skipDelete: false
  failFast: true
  parallel: 4
  # ... other configurations

The full structure of the configuration file, including all possible fields and their descriptions, can be found in the Configuration API reference.

If you don't provide a custom configuration, Chainsaw will use its default:

apiVersion: chainsaw.kyverno.io/v1alpha1
kind: Configuration
metadata:
  name: default
spec: {}

You can also override specific configurations using command-line flags, even after loading a configuration file. For instance:

chainsaw test --config path/to/your/config.yaml --timeout 45s --parallel 4

In this example, even if the configuration file specifies different values for timeout and parallel, the command-line flags will take precedence.

For a comprehensive list of all supported flags and their descriptions, refer to the Chainsaw test command reference.

Section 3: Creating Tests

3.1 Understanding Chainsaw Tests

In Chainsaw, a test is essentially an ordered sequence of test steps. These steps are executed sequentially, and if any step fails, the entire test is considered failed. Each test step can consist of multiple operations, such as creating, updating, or deleting resources in a Kubernetes cluster, or asserting that certain conditions are met.

Chainsaw supports three primary test definition mechanisms:

  • Manifests based syntax: This is a straightforward method where you provide Kubernetes resource manifests. Chainsaw uses these manifests to create, update, or assert expectations against a cluster.
  • TestSteps based syntax: A more verbose method that offers flexibility in defining each test step. It allows for additional configurations and can be combined with the Manifests based syntax.
  • Test based syntax: The most explicit method that provides a comprehensive overview of the test. It doesn't rely on file naming conventions and offers flexibility at both the test and test step levels.
3.2 Creating Chainsaw Tests

Creating tests in Chainsaw requires a clear understanding of the test definition mechanisms. Here's a detailed guide for each syntax:

  1. Using Manifests based syntax

File Naming: Files should follow the convention <step index>-<name|assert|error>.yaml. For instance, 00-configmap.yaml for resource creation, 01-assert.yaml for assertions, and 02-error.yaml for expected errors.

Example:

# 00-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: chainsaw-example
data:
  key: value

  1. Using TestSteps based syntax

    Define a TestStep: Create a YAML file that defines a TestStep resource. This resource should specify operations like Apply, Assert, Delete, and Error.

Example:

# 01-test-step.yaml
apiVersion: chainsaw.kyverno.io/v1alpha1
kind: TestStep
metadata:
  name: apply-configmap-step
spec:
  apply:
  - file: /resources/configmap.yaml

You can combine TestStep resources with raw Kubernetes manifests. For instance, a TestStep might apply a resource, while a separate manifest file makes assertions about that resource.

Example:

# 02-assert.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: chainsaw-example
data:
  key: value
  1. Using Test based syntax

Define a Test: Start by creating a YAML file that defines a Test resource. This resource will have a spec section that outlines the entire test.

Example:

# chainsaw-test.yaml
apiVersion: chainsaw.kyverno.io/v1alpha1
kind: Test
metadata:
  name: full-test-example
spec:
  timeout: 10s
  steps:
  - spec:
      apply:
      - file: /resources/configmap.yaml
  - spec:
      assert:
      - file: /resources/configmap-assert.yaml

Chainsaw processes the test by executing each step in sequence, ensuring that your Kubernetes environment meets the defined conditions.

Section 4: Running Tests

4.1 Preparing for Test Execution

Before executing your tests, ensure that:

  1. Kubernetes Cluster: You have access to a Kubernetes cluster where the tests will run. This could be a local cluster (like Minikube or kind) or a remote one.
  2. Kubeconfig: Your kubeconfig is correctly set up to point to the desired cluster. Chainsaw will use the current context from your kubeconfig to interact with the cluster.
  3. Test Files: All your test files, whether they are written using Manifests based, TestSteps based, or Test based syntax, are organized and accessible.
4.2 Executing Tests in Chainsaw

Once you're set up, running tests in Chainsaw is straightforward:

  1. Navigate to the Test Directory:
cd path/to/your/test/directory
  1. Run the Tests:
chainsaw test

This command will execute all tests in the current directory.

  1. View the Results: Chainsaw will display the results in the terminal. It provides a summary of passed, failed, and skipped tests. For any failed tests, Chainsaw will display detailed error messages to help diagnose the issue.

The output will resemble:

Loading default configuration...
...
Running tests...
=== RUN   quick-start
...
--- PASS: quick-start (5.22s)
PASS
Tests Summary...
- Passed  tests: 3
- Failed  tests: 1
- Skipped tests: 2
Done.

Conclusion

Chainsaw provides a robust and flexible framework for end-to-end testing in Kubernetes environments. By understanding its configuration and test definition mechanisms, developers can create comprehensive test suites that ensure their Kubernetes applications and configurations are functioning as expected.