Overview of Test Driven Infrastructure with Chef
// Chef Blog
This post is all about test driven infrastructure with Chef: an overview in testing Chef cookbooks for the current landscape. This post is focused on tools included in ChefDK. Some other tools and projects are mentioned for completeness or historical purposes.
This post serves as general overview of the various components and tools that are currently used, or have been used in the past to do test driven infrastructure with Chef. For a full book on the topic, see Test Driven Infrastructure with Chef, by Stephen Nelson-Smith. For training materials, see this learnchef repository.
In this post, I want to discuss what and why for each section, but not how. This is not because the how is unimportant, it's because I don't want to get bogged down in implementation details for each tool. This post is light on examples for that reason.
Before we talk about the individual tools, let's cover what the various phases of Chef development are, and what the types of testing are.
Chef Development PhasesIn the context of this discussion there are three phases of development: pre-convergence, convergence, and post-convergence.
Pre-convergencePre-convergence is the phase before a node is converged by Chef, and is the phase when testing that doesn't require a node to run Chef happens. Syntax checking, lint checking, or unit tests are performed during pre-convergence. Automated testing in CI usually does pre-convergence tests, such as GitHub repositories configured with Travis CI.
ConvergenceConvergence is the phase when Chef runs and makes changes to the system to "converge" it to be in the desired state. This is where resources are tested for current state, and repaired (action taken by providers) if they're not correct.
Post-convergencePost-convergence is the phase after a node finishes running Chef. Testing in this phase verifies that the node is in the desired state. This can include checks that a particular port is listening, that a configuration file contains the correct content, or that a service is running with the proper configuration.
Types of TestingIt is outside the scope of this post to cover the various types of testing in great detail. I'll provide a general overview of two types of testing as they pertain to testing Chef cookbooks: Unit and Integration testing. It is left as a reader exercise to further research the types of testing. This blog post by Seth Vargo is a good reference to start.
Unit TestingThe intent of unit testing is to confirm that given specific input, the recipe yields the expected output. Unit tests are meant to execute fast, and happen without converging the node, so they are done in the pre-convergence phase. Unit testing Chef cookbooks is done with ChefSpec. It is very important that one does not fall into the trap of testing that Chef itself works. For example, given a recipe that has the following resource:
It may be tempting to write a unit test like this:
package "httpd"do action :install end
The thing is, we know that the recipe will install the "httpd" package, because that is exactly how Chef works! However, depending on the node's platform, the package name might be different. We can use unit tests to verify that on given platforms, specific package names will be installed ("httpd" on RHEL-family, "apache2″ on Debian-family, using this example). For more detailed discussion about testing conditionals in ChefSpec, see my blog post on the subject.
it "installs the httpd package"do expect(chefrun).to installpackage("httpd") end
Integration TestingThe intent of integration testing is to verify that the end state of the system is in fact what we wanted after Chef converges the node so we can have a higher degree of confidence that our code is doing what we need.
Integration testing Chef cookbooks is often performed in Test Kitchen through "bussers" for various supported frameworks like Serverspec, BATS, or Minitest. See the respective sections below for specifics.
Common Chef Testing ToolsThis section includes the tools commonly used with Chef development.
What and WhyRubocop is a Ruby command-line tool that performs lint and style checks based on the community driven Ruby Style Guide. It performs static analysis of any Ruby code, which includes Chef recipes, resources, library helpers, and so forth. Rubocop can be configured via .rubocop.yml to exclude certain rules, and it can be run with "–lint" to perform only lint checking, excluding all style checks.
Rubocop is used in the Chef community in cookbooks to make contributions more consistent and easier to manage.
Phase of Chef DevelopmentRubocop is run in pre-convergence, or in CI systems to ensure that changes are consistent with the desired style preferences in a cookbook. It comes with rake tasks that can be used in any Rakefile.
How to Get ItRubocop is included in ChefDK. It can be invoked via its command-line tool, rubocop, or it can be included in a Rakefile with
What and WhyFoodcritic is a Ruby command-line tool to perform lint checking against Chef cookbooks. It makes it easier to flag problems in cookbooks that can cause Chef to raise an exception during convergence. Cookbook authors use Foodcritic to help enforce good patterns and create higher quality cookbooks. Foodcritic includes a lot of rules, and users can write their own rules. It can also be configured to ignore certain rules through a .foodcritic config file, inline comments, or command-line flags.
Phase of Chef DevelopmentFoodcritic is run as in pre-convergence, or in CI systems such as Travis CI.
How to Get ItFoodcritic is included in ChefDK. It can be invoked via its command-line tool, foodcritic. Like RuboCop, it includes rake tasks that can be included in a Rakefile with
What and WhyChefSpec is a unit testing framework for Chef cookbooks. It runs tests locally without making changes to the system. ChefSpec executes quickly, as it doesn't have to perform the normal test-and-repair actions to converge resources. It also uses Fauxhai (discussed later) data so it can be used to conditionally test against different platforms easily. ChefSpec is strongly recommended for cookbook authors as it helps protect against regressions.
High-value tests in ChefSpec include:
- Multiple platforms
- Conditional branching
- Behavior driven by attributes or data bags
- Search results
- Helper methods
- Custom Resources and Providers
Phase of Chef DevelopmentChefSpec is used in the pre-convergence phase of testing cookbooks. It is primarily used to validate that given specific inputs, especially from variable data like attributes, Chef will have an expected resource collection as an output.
How to Get ItChefSpec is included in ChefDK. It is invoked using rspec, when chefspec is required in spec files.
What and WhyServerspec is an "outside-in" integration test framework. It is platform and tool agnostic, and is used by other configuration management systems to verify systems are configured as desired. It checks the actual state of the target node by executing commands locally, via SSH, via WinRM, or other remote transports. Serverspec is implemented in RSpec, and uses RSpec test syntax.
Phase of Chef DevelopmentServerspec is used in the post-convergence phase. It tests that the actual outcome of having run Chef on a node resulted in the desired state.
How to Get ItServerspec is a dependency of Chef 12.1.0+ for audit mode, so it is included in ChefDK.
When used by Test Kitchen, it is installed with busser on the node being tested. To use it with Test Kitchen, create test/integration/SUITE/serverspec directories, and busser will handle the rest.
Chef's audit mode
What and WhyChef audit mode is a feature introduced in version 12.1.0. It is a mode of running chef-client that allows tests (audit controls) written in recipes to run. Audit results are sent back to the Chef Server and then picked up by Chef Analytics (if installed) for further analysis and processing. Audit mode is implemented as a subset of Serverspec, and thus RSpec.
Audit mode can be run as part of a chef-client run that converges resources on the node. For example, a recipe can have package, template, and service resources that get configured to their desired state, and then audit mode will validate that everything is as it should be and compliant with policy. Audit mode can also be run without converging resources. That means that same recipe that contains package, template, or service resources won't modify the state of those, but the audit controls will be run.
The main benefit of using audit mode is in combination with Chef Analytics, to perform analysis of the results of a run, and send notifications about the results, e.g., through email. The language used is very specific and oriented toward auditing for compliance purposes, such as PCI, SOX, or individual organization policies. The output reports are useful from an audit documentation perspective, and that the controls are written in the recipe, we have both the documentation of intent (controls to validate), and the remediation step (resources to converge for that intent).
The main way that audit mode differs from normal Serverspec is intent. With audit mode, our intent is to verify that the node is compliant with policy. With Serverspec, our intent is to verify that the state of a node after running the recipes is correct.
Phase of Chef DevelopmentAudit mode is run as part of the convergence phase. It runs after the node has converged all the resources in the resource collection when it is set to enabled. It can be considered "post-convergence" if it is set to auditonly.
How to Get ItAudit mode is included with Chef 12.1.0 and higher. It is not supported in earlier versions of Chef.
What and Why"Test Kitchen is an integration tool for developing and testing infrastructure code and software on isolated target platforms." It creates test machines, converges them, and runs post-convergence tests against them to verify their state. Test Kitchen is written in Ruby. It has a plugin system for supporting machine creation through a variety of virtual machine technologies such as vagrant, EC2, docker, and several others. Test Kitchen makes it easy for Chef developers to test cookbooks on a variety of platforms. It uses busser to install post-convergence integration test tools such as Serverspec or BATS that actually perform the tests.
Phase of Chef DevelopmentTest Kitchen is used to actually converge machines and run post-convergence tests automatically. It is invoked manually, or through a CI system.
How to Get ItTest Kitchen is included in ChefDK. To get started, create a .kitchen.yml file in a cookbook (this can be done with kitchen init), and then use the kitchen commands – usually kitchen test, kitchen converge, and kitchen verify to work with the configured machines.
Supporting Tools and DependenciesThis section includes the tools that support the common tools, or are their dependencies. They are included for completeness
Rake and Thor
What and WhyRake and Thor are command-line task running tools, similar to make. Tasks are written in a Rakefile or Thorfile, respectively. They're both implemented in Ruby, albeit slightly differently internally. They're listed here together because they're often interchangeable. That is, some people prefer one over the other. Rake or Thor have tasks for running test suites, or build and release cookbooks, depending on any individual project. Some of the other tools mentioned have Rake and Thor integration, such as Foodcritic and ChefSpec.
Phase of Chef DevelopmentRake and Thor are used during pre-convergence to perform tasks such as running ChefSpec/RSpec, or lint checks with Rubocop and Foodcritic. Rubygem projects usually have tasks for releasing the gem to rubygems.org. Cookbook projects sometimes have tasks for releasing the cookbook to Supermarket, or uploading it to a Chef Server.
Many projects use Travis CI to automatically run certain tasks such as unit/spec tests and lint/style checks on every commit.
How to Get ThemBoth Rake and Thor are included in the ChefDK. To use Rake, you need a Rakefile. To use Thor, you need a Thorfile. Run "rake -T" or "thor -T" to get a list of tasks in a project.
- Example Rakefile in a cookbook
- Example Thorfile in a cookbook
- Example rake use in Travis CI
What and WhyGuard is a Ruby command-line tool that handles events from filesystem modifications. It is an "auto-runner" for other tools that are described here. Most commonly this means it will run a test suite when .rb files are changed. Guard is configured with a Guardfile that tells it which directories and files to monitor, and what command to run when they change.
Phase of Chef DevelopmentGuard is used during pre-convergence to run syntax checks with "knife cookbook test", lint checks with "rubocop" and "foodcritic", and unit tests with "rspec" (ChefSpec). It also can be used to run "kitchen converge" or "kitchen verify" when recipes change to automatically run Test Kitchen.
How to Get ItGuard is included in the ChefDK. For specific integration with Test Kitchen, use the "guard-kitchen" gem.
What and WhyRSpec is a Test Driven Development (TDD) or Behavior Driven Development (BDD) framework. It is used widely in the Ruby community for testing, and it is extensible for creating new testing frameworks. As such, ChefSpec and Serverspec (and thus Chef audit mode) are based on RSpec. Chef Software, Inc. uses RSpec for testing chef itself, and other Ruby-based tools and libraries we develop. RSpec is also the basis of the "pedant" testing framework that validates the server products – Chef Server, Chef Analytics, Chef Delivery.
Phase of Chef DevelopmentRSpec is used by ChefSpec during pre-convergence. It is used by Audit Mode (via Serverspec) in convergence. It is also used by Serverspec in post-convergence.
How to Get ItRSpec is a dependency of ChefSpec and Serverspec, so it is included in ChefDK. To include it in other non-cookbook projects, add it to a Gemfile for installation with bundler. Most of the Ruby-based projects related to Chef use RSpec for their unit tests.
What and WhyFauxhai is a tool for mocking out ohai data to provide node attributes for testing. Fauxhai is used within ChefSpec.
Most cookbook developers don't use Fauxhai directly, unless they're contributing support for a new platform to it.
How to Get ItFauxhai is a dependency of ChefSpec, so it is included in ChefDK.
What and WhyFrom the project README: "Busser is a test setup and execution framework designed to work on remote nodes whose system dependencies cannot be relied upon." Busser is used by Test Kitchen to support running post-convergence tests. It has a plugin-based architecture that allows different test frameworks to be used.
Phase of Chef DevelopmentBusser is used during post-convergence automatically by Test Kitchen during the kitchen verify step.
How to Get ItBusser is automatically installed by Test Kitchen.
Uncommon or Deprecated ToolsThis section includes the tools that are either uncommonly used, or may be considered deprecated in favor of newer tools described above.
What and WhyCucumber is a "plain text" based DSL (domain specific language) for writing user stories that get executed against specifications written in a language called Gherkin. It is commonly used in BDD for Ruby on Rails projects, but also has been used for Chef cookbook development.
Phase of Chef DevelopmentCucumber is not commonly used in Chef cookbook development. It was previously used for post-convergence acceptance or integration testing, but most people are using Serverspec or BATS for this functionality.
How to Get ItCucumber is included in the ChefDK.
What and WhyBATS is a Bourne-again shell (bash) test framework. It is simple to write tests because users can implement the exact commands they would run after logging into a system to verify it. However, it is platform specific to Unix/Linux systems that have bash installed, so it does not work on Windows systems. This makes it more difficult to test cross-platform cookbooks, so many users prefer Serverspec. As Serverspec is also the basis of Chef audit mode, it is more widely used, and we recommend Serverspec over BATS for post-convergence testing.
Phase of Chef DevelopmentBATS is used in the post-convergence phase. It tests that the actual outcome of running Chef on a node resulted in the desired state.
How to Get ItBATS is not included in ChefDK. It is installed by Test Kitchen with Busser. See the home page for more information on how to use it outside of Test Kitchen.
What and Whyminitest is a Ruby test framework included in the Ruby standard library. It is designed to be fast and easy to use. However, RSpec is more commonly used within the Chef community for non-cookbook Ruby projects (like Chef itself). It is included here because users may encounter it via the minitest-chef project present in older cookbooks. Some users may prefer it for Test Kitchen post-convergence testing.
Phase of Chef Developmentminitest is used during the post-convergence phase. Historically, it was only used as part of the minitest-chef-handler as part of a Chef run report handler. It can be used as a busser in kitchen verify.
How to Get Itminitest is included in the Ruby Standard Library, so it's present on any system with Ruby installed, including within the embedded Ruby included by ChefDK's omnibus install.
When using minitest-chef-handler it is invoked through a Chef report handler.
When used by Test Kitchen, it is installed with busser on the node being tested. To do this, create test/integration/SUITE/minitest directories and busser will handle the rest.
What and WhyThe minitest-chef-handler is the original post-convergence test framework for Chef. It runs as a report handler after a successful Chef run to verify the state of the system using minitest.
The handler is considered deprecated in favor of testing frameworks that are run in Test Kitchen with Busser.
Phase of Chef Developmentminitest-chef-handler is run in the post-convergence phase. It actually executes as part of the Chef run by a report handler.
How to Get ItThe minitest-handler cookbook is used to automatically install minitest-chef-handler and register it as a report handler for the run.
Shared via my feedly reader
Sent from my iPhone