Tuesday, April 21, 2015

Overview of Test Driven Infrastructure with Chef [feedly]



----
Overview of Test Driven Infrastructure with Chef
// Chef Blog

This post is all about test driven infrastructure with Chef: an overview in testing Chef cookbooks for the current landscape. This post is focused on tools included in ChefDK. Some other tools and projects are mentioned for completeness or historical purposes.

This post serves as general overview of the various components and tools that are currently used, or have been used in the past to do test driven infrastructure with Chef. For a full book on the topic, see Test Driven Infrastructure with Chef, by Stephen Nelson-Smith. For training materials, see this learnchef repository.

In this post, I want to discuss what and why for each section, but not how. This is not because the how is unimportant, it's because I don't want to get bogged down in implementation details for each tool. This post is light on examples for that reason.

Before we talk about the individual tools, let's cover what the various phases of Chef development are, and what the types of testing are.

Chef Development Phases

In the context of this discussion there are three phases of development: pre-convergence, convergence, and post-convergence.

Pre-convergence

Pre-convergence is the phase before a node is converged by Chef, and is the phase when testing that doesn't require a node to run Chef happens. Syntax checking, lint checking, or unit tests are performed during pre-convergence. Automated testing in CI usually does pre-convergence tests, such as GitHub repositories configured with Travis CI.

Convergence

Convergence is the phase when Chef runs and makes changes to the system to "converge" it to be in the desired state. This is where resources are tested for current state, and repaired (action taken by providers) if they're not correct.

Post-convergence

Post-convergence is the phase after a node finishes running Chef. Testing in this phase verifies that the node is in the desired state. This can include checks that a particular port is listening, that a configuration file contains the correct content, or that a service is running with the proper configuration.

Types of Testing

It is outside the scope of this post to cover the various types of testing in great detail. I'll provide a general overview of two types of testing as they pertain to testing Chef cookbooks: Unit and Integration testing. It is left as a reader exercise to further research the types of testing. This blog post by Seth Vargo is a good reference to start.

Unit Testing

The intent of unit testing is to confirm that given specific input, the recipe yields the expected output. Unit tests are meant to execute fast, and happen without converging the node, so they are done in the pre-convergence phase. Unit testing Chef cookbooks is done with ChefSpec. It is very important that one does not fall into the trap of testing that Chef itself works. For example, given a recipe that has the following resource: 
package "httpd"do    action :install  end
It may be tempting to write a unit test like this: 
it "installs the httpd package"do    expect(chefrun).to installpackage("httpd")  end
The thing is, we know that the recipe will install the "httpd" package, because that is exactly how Chef works! However, depending on the node's platform, the package name might be different. We can use unit tests to verify that on given platforms, specific package names will be installed ("httpd" on RHEL-family, "apache2″ on Debian-family, using this example). For more detailed discussion about testing conditionals in ChefSpec, see my blog post on the subject.

Integration Testing

The intent of integration testing is to verify that the end state of the system is in fact what we wanted after Chef converges the node so we can have a higher degree of confidence that our code is doing what we need.

Integration testing Chef cookbooks is often performed in Test Kitchen through "bussers" for various supported frameworks like Serverspec, BATS, or Minitest. See the respective sections below for specifics.

Common Chef Testing Tools

This section includes the tools commonly used with Chef development.

RuboCop

What and Why
Rubocop is a Ruby command-line tool that performs lint and style checks based on the community driven Ruby Style Guide. It performs static analysis of any Ruby code, which includes Chef recipes, resources, library helpers, and so forth. Rubocop can be configured via .rubocop.yml to exclude certain rules, and it can be run with "–lint" to perform only lint checking, excluding all style checks.

Rubocop is used in the Chef community in cookbooks to make contributions more consistent and easier to manage.

Phase of Chef Development
Rubocop is run in pre-convergence, or in CI systems to ensure that changes are consistent with the desired style preferences in a cookbook. It comes with rake tasks that can be used in any Rakefile.
How to Get It
Rubocop is included in ChefDK. It can be invoked via its command-line tool, rubocop, or it can be included in a Rakefile with RuboCop::RakeTask.new(:ruby).
More Information

Foodcritic

What and Why
Foodcritic is a Ruby command-line tool to perform lint checking against Chef cookbooks. It makes it easier to flag problems in cookbooks that can cause Chef to raise an exception during convergence. Cookbook authors use Foodcritic to help enforce good patterns and create higher quality cookbooks. Foodcritic includes a lot of rules, and users can write their own rules. It can also be configured to ignore certain rules through a .foodcritic config file, inline comments, or command-line flags.
Phase of Chef Development
Foodcritic is run as in pre-convergence, or in CI systems such as Travis CI.
How to Get It
Foodcritic is included in ChefDK. It can be invoked via its command-line tool, foodcritic. Like RuboCop, it includes rake tasks that can be included in a Rakefile with FoodCritic::Rake::LintTask.new.
More Information

ChefSpec

What and Why
ChefSpec is a unit testing framework for Chef cookbooks. It runs tests locally without making changes to the system. ChefSpec executes quickly, as it doesn't have to perform the normal test-and-repair actions to converge resources. It also uses Fauxhai (discussed later) data so it can be used to conditionally test against different platforms easily. ChefSpec is strongly recommended for cookbook authors as it helps protect against regressions.

High-value tests in ChefSpec include:

  • Multiple platforms
  • Conditional branching
  • Behavior driven by attributes or data bags
  • Search results
  • Helper methods
  • Custom Resources and Providers
Phase of Chef Development
ChefSpec is used in the pre-convergence phase of testing cookbooks. It is primarily used to validate that given specific inputs, especially from variable data like attributes, Chef will have an expected resource collection as an output.
How to Get It
ChefSpec is included in ChefDK. It is invoked using rspec, when chefspec is required in spec files.
More Information

Serverspec

What and Why
Serverspec is an "outside-in" integration test framework. It is platform and tool agnostic, and is used by other configuration management systems to verify systems are configured as desired. It checks the actual state of the target node by executing commands locally, via SSH, via WinRM, or other remote transports. Serverspec is implemented in RSpec, and uses RSpec test syntax.
Phase of Chef Development
Serverspec is used in the post-convergence phase. It tests that the actual outcome of having run Chef on a node resulted in the desired state.
How to Get It
Serverspec is a dependency of Chef 12.1.0+ for audit mode, so it is included in ChefDK.

When used by Test Kitchen, it is installed with busser on the node being tested. To use it with Test Kitchen, create test/integration/SUITE/serverspec directories, and busser will handle the rest.

More Information

Chef's audit mode

What and Why
Chef audit mode is a feature introduced in version 12.1.0. It is a mode of running chef-client that allows tests (audit controls) written in recipes to run. Audit results are sent back to the Chef Server and then picked up by Chef Analytics (if installed) for further analysis and processing. Audit mode is implemented as a subset of Serverspec, and thus RSpec.

Audit mode can be run as part of a chef-client run that converges resources on the node. For example, a recipe can have package, template, and service resources that get configured to their desired state, and then audit mode will validate that everything is as it should be and compliant with policy. Audit mode can also be run without converging resources. That means that same recipe that contains package, template, or service resources won't modify the state of those, but the audit controls will be run.

The main benefit of using audit mode is in combination with Chef Analytics, to perform analysis of the results of a run, and send notifications about the results, e.g., through email. The language used is very specific and oriented toward auditing for compliance purposes, such as PCI, SOX, or individual organization policies. The output reports are useful from an audit documentation perspective, and that the controls are written in the recipe, we have both the documentation of intent (controls to validate), and the remediation step (resources to converge for that intent).

The main way that audit mode differs from normal Serverspec is intent. With audit mode, our intent is to verify that the node is compliant with policy. With Serverspec, our intent is to verify that the state of a node after running the recipes is correct.

Phase of Chef Development
Audit mode is run as part of the convergence phase. It runs after the node has converged all the resources in the resource collection when it is set to enabled. It can be considered "post-convergence" if it is set to auditonly.
How to Get It
Audit mode is included with Chef 12.1.0 and higher. It is not supported in earlier versions of Chef.
More Information

Test Kitchen

What and Why
"Test Kitchen is an integration tool for developing and testing infrastructure code and software on isolated target platforms." It creates test machines, converges them, and runs post-convergence tests against them to verify their state. Test Kitchen is written in Ruby. It has a plugin system for supporting machine creation through a variety of virtual machine technologies such as vagrant, EC2, docker, and several others. Test Kitchen makes it easy for Chef developers to test cookbooks on a variety of platforms. It uses busser to install post-convergence integration test tools such as Serverspec or BATS that actually perform the tests.
Phase of Chef Development
Test Kitchen is used to actually converge machines and run post-convergence tests automatically. It is invoked manually, or through a CI system.
How to Get It
Test Kitchen is included in ChefDK. To get started, create a .kitchen.yml file in a cookbook (this can be done with kitchen init), and then use the kitchen commands – usually kitchen test, kitchen converge, and kitchen verify to work with the configured machines.
More Information

Supporting Tools and Dependencies

This section includes the tools that support the common tools, or are their dependencies. They are included for completeness

Rake and Thor

What and Why
Rake and Thor are command-line task running tools, similar to make. Tasks are written in a Rakefile or Thorfile, respectively. They're both implemented in Ruby, albeit slightly differently internally. They're listed here together because they're often interchangeable. That is, some people prefer one over the other. Rake or Thor have tasks for running test suites, or build and release cookbooks, depending on any individual project. Some of the other tools mentioned have Rake and Thor integration, such as Foodcritic and ChefSpec.
Phase of Chef Development
Rake and Thor are used during pre-convergence to perform tasks such as running ChefSpec/RSpec, or lint checks with Rubocop and Foodcritic. Rubygem projects usually have tasks for releasing the gem to rubygems.org. Cookbook projects sometimes have tasks for releasing the cookbook to Supermarket, or uploading it to a Chef Server.

Many projects use Travis CI to automatically run certain tasks such as unit/spec tests and lint/style checks on every commit.

How to Get Them
Both Rake and Thor are included in the ChefDK. To use Rake, you need a Rakefile. To use Thor, you need a Thorfile. Run "rake -T" or "thor -T" to get a list of tasks in a project.
More Information

Guard

What and Why
Guard is a Ruby command-line tool that handles events from filesystem modifications. It is an "auto-runner" for other tools that are described here. Most commonly this means it will run a test suite when .rb files are changed. Guard is configured with a Guardfile that tells it which directories and files to monitor, and what command to run when they change.
Phase of Chef Development
Guard is used during pre-convergence to run syntax checks with "knife cookbook test", lint checks with "rubocop" and "foodcritic", and unit tests with "rspec" (ChefSpec). It also can be used to run "kitchen converge" or "kitchen verify" when recipes change to automatically run Test Kitchen.
How to Get It
Guard is included in the ChefDK. For specific integration with Test Kitchen, use the "guard-kitchen" gem.
More Information

RSpec

What and Why
RSpec is a Test Driven Development (TDD) or Behavior Driven Development (BDD) framework. It is used widely in the Ruby community for testing, and it is extensible for creating new testing frameworks. As such, ChefSpec and Serverspec (and thus Chef audit mode) are based on RSpec. Chef Software, Inc. uses RSpec for testing chef itself, and other Ruby-based tools and libraries we develop. RSpec is also the basis of the "pedant" testing framework that validates the server products – Chef Server, Chef Analytics, Chef Delivery.
Phase of Chef Development
RSpec is used by ChefSpec during pre-convergence. It is used by Audit Mode (via Serverspec) in convergence. It is also used by Serverspec in post-convergence.
How to Get It
RSpec is a dependency of ChefSpec and Serverspec, so it is included in ChefDK. To include it in other non-cookbook projects, add it to a Gemfile for installation with bundler. Most of the Ruby-based projects related to Chef use RSpec for their unit tests.
More Information

Fauxhai

What and Why
Fauxhai is a tool for mocking out ohai data to provide node attributes for testing. Fauxhai is used within ChefSpec.

Most cookbook developers don't use Fauxhai directly, unless they're contributing support for a new platform to it.

How to Get It
Fauxhai is a dependency of ChefSpec, so it is included in ChefDK.
More Information

Busser

What and Why
From the project README: "Busser is a test setup and execution framework designed to work on remote nodes whose system dependencies cannot be relied upon." Busser is used by Test Kitchen to support running post-convergence tests. It has a plugin-based architecture that allows different test frameworks to be used.
Phase of Chef Development
Busser is used during post-convergence automatically by Test Kitchen during the kitchen verify step.
How to Get It
Busser is automatically installed by Test Kitchen.
More Information

Uncommon or Deprecated Tools

This section includes the tools that are either uncommonly used, or may be considered deprecated in favor of newer tools described above.

Cucumber

What and Why
Cucumber is a "plain text" based DSL (domain specific language) for writing user stories that get executed against specifications written in a language called Gherkin. It is commonly used in BDD for Ruby on Rails projects, but also has been used for Chef cookbook development.
Phase of Chef Development
Cucumber is not commonly used in Chef cookbook development. It was previously used for post-convergence acceptance or integration testing, but most people are using Serverspec or BATS for this functionality.
How to Get It
Cucumber is included in the ChefDK.
More Information

BATS

What and Why
BATS is a Bourne-again shell (bash) test framework. It is simple to write tests because users can implement the exact commands they would run after logging into a system to verify it. However, it is platform specific to Unix/Linux systems that have bash installed, so it does not work on Windows systems. This makes it more difficult to test cross-platform cookbooks, so many users prefer Serverspec. As Serverspec is also the basis of Chef audit mode, it is more widely used, and we recommend Serverspec over BATS for post-convergence testing.
Phase of Chef Development
BATS is used in the post-convergence phase. It tests that the actual outcome of running Chef on a node resulted in the desired state.
How to Get It
BATS is not included in ChefDK. It is installed by Test Kitchen with Busser. See the home page for more information on how to use it outside of Test Kitchen.
More Information

minitest

What and Why
minitest is a Ruby test framework included in the Ruby standard library. It is designed to be fast and easy to use. However, RSpec is more commonly used within the Chef community for non-cookbook Ruby projects (like Chef itself). It is included here because users may encounter it via the minitest-chef project present in older cookbooks. Some users may prefer it for Test Kitchen post-convergence testing.
Phase of Chef Development
minitest is used during the post-convergence phase. Historically, it was only used as part of the minitest-chef-handler as part of a Chef run report handler. It can be used as a busser in kitchen verify.
How to Get It
minitest is included in the Ruby Standard Library, so it's present on any system with Ruby installed, including within the embedded Ruby included by ChefDK's omnibus install.

When using minitest-chef-handler it is invoked through a Chef report handler.

When used by Test Kitchen, it is installed with busser on the node being tested. To do this, create test/integration/SUITE/minitest directories and busser will handle the rest.

More Information

minitest-chef-handler

What and Why
The minitest-chef-handler is the original post-convergence test framework for Chef. It runs as a report handler after a successful Chef run to verify the state of the system using minitest.

The handler is considered deprecated in favor of testing frameworks that are run in Test Kitchen with Busser.

Phase of Chef Development
minitest-chef-handler is run in the post-convergence phase. It actually executes as part of the Chef run by a report handler.
How to Get It
The minitest-handler cookbook is used to automatically install minitest-chef-handler and register it as a report handler for the run.
More Information
----

Shared via my feedly reader


Sent from my iPhone
Posted by at

No comments:

Post a Comment

Subscribe to: Post Comments (Atom)