Debug Recipes, Chef Infra Client Runs

Elements of good approaches to building cookbooks and recipes that are reliable include:

A consistent syntax pattern when constructing recipes
Using the same patterns in Ruby
Using resources included in Chef Infra Client or community cookbooks before creating custom ones

Ideally, the best way to debug a recipe is to not have to debug it in the first place. That said, the following sections discuss various approaches to debugging recipes and failed Chef Infra Client runs.

Basic

Some simple ways to quickly identify common issues that can trigger recipe and/or Chef Infra Client run failures include:

Using an empty run-list
Using verbose logging with knife
Using logging with Chef Infra Client
Using the log resource in a recipe to define custom logging

Empty Run-lists

Use an empty run-list to determine if a failed Chef Infra Client run has anything to do with the recipes that are defined within that run-list. This is a quick way to discover if the underlying cause of a Chef Infra Client run failure is a configuration issue. If a failure persists even if the run-list is empty, check the following:

Configuration settings in the config.rb file
Permissions for the user to both the Chef Infra Server and to the node on which a Chef Infra Client run is to take place

Knife

Use the verbose logging that is built into knife:

-V, --verbose: Set for more verbose outputs. Use -VV for much more verbose outputs. Use -VVV for maximum verbosity, which may provide more information than is actually helpful.

Note

Plugins do not always support verbose logging.

Chef Infra Client

Use the verbose logging that is built into Chef Infra Client:

-l LEVEL, --log_level LEVEL: The level of logging to be stored in a log file. Possible levels: auto (default), debug, error, fatal, info, trace, or warn. Default value: warn (when a terminal is available) or info (when a terminal is not available).
-L LOGLOCATION, --logfile c: The location of the log file. This is recommended when starting any executable as a daemon. Default value: STDOUT.

log Resource

Use the log resource to create log entries. The log resource behaves like any other resource: built into the resource collection during the compile phase, and then run during the execution phase. (To create a log entry that is not built into the resource collection, use Chef::Log instead of the log resource.)

Note

By default, every log resource that executes will count as an updated resource in the updated resource count at the end of a Chef run. You can disable this behavior by adding count_log_resource_updates false to your Chef client.rb configuration file.

New in 12.0, -o RUN_LIST_ITEM. Changed in 12.0 -f no longer allows unforked intervals, -i SECONDS is applied before a Chef Infra Client run.

Syntax

A log resource block adds messages to the log file based on events that occur during a Chef Infra Client run:

log 'message' do
  message 'A message add to the log.'
  level :info
end

The full syntax for all of the properties that are available to the log resource is:

log 'name' do
  level        Symbol # default value: :info
  message      String # default value: 'name' unless specified
  action       Symbol # defaults to :write if not specified
end

where:

log is the resource.
name is the name given to the resource block.
action identifies which steps Chef Infra Client will take to bring the node into the desired state.
level and message are the properties available to this resource.

Actions

The log resource has the following actions:

:nothing: This resource block does not act unless notified by another resource to take action. Once notified, this resource block either runs immediately or is queued up to run at the end of a Chef Infra Client run.
:write: Default. Write to log.

Properties

The log resource has the following properties:

level

Ruby Type: Symbol | Default Value: :info

The logging level for displaying this message. Options (in order of priority): :debug, :info, :warn, :error, and :fatal.

message

Ruby Type: String | Default Value: The resource block's name

The message to be added to a log file. Default value: the name of the resource block. See “Syntax” section above for more information.

Examples

The following examples demonstrate various approaches for using resources in recipes:

Specify a log entry

log 'a string to log'

Set debug logging level

log 'a debug string' do
  level :debug
end

Create log entry when the contents of a data bag are used

log 'a debug string' do
  level :debug
end

Add a message to a log file

log 'message' do
  message 'This is the message that will be added to the log.'
  level :info
end

Advanced

Some more complex ways to debug issues with a Chef Infra Client run include:

Using the chef_handler cookbook
Using the chef-shell and the breakpoint resource to add breakpoints to recipes, and to then step through the recipes using the breakpoints
Using the debug_value method from chef-shell to identify the location(s) from which attribute values are being set
Using the ignore_failure method in a recipe to force Chef Infra Client to move past an error to see what else is going on in the recipe, outside of a known failure
Using chef-solo to run targeted Chef Infra Client runs for specific scenarios

chef_handler

Use a handler to identify situations that arise during a Chef Infra Client run, and then tell Chef Infra Client how to handle these situations when they occur.

There are three types of handlers:

Handler	Description
exception	An exception handler is used to identify situations that have caused a Chef Infra Client run to fail. An exception handler can be loaded at the start of a Chef Infra Client run by adding a recipe that contains the chef_handler resource to a node's run-list. An exception handler runs when the `failed?` property for the `run_status` object returns `true`.
report	A report handler is used when a Chef Infra Client run succeeds and reports back on certain details about that Chef Infra Client run. A report handler can be loaded at the start of a Chef Infra Client run by adding a recipe that contains the chef_handler resource to a node's run-list. A report handler runs when the `success?` property for the `run_status` object returns `true`.
start	A start handler is used to run events at the beginning of a Chef Infra Client run. A start handler can be loaded at the start of a Chef Infra Client run by adding the start handler to the `start_handlers` setting in the client.rb file or by installing the gem that contains the start handler by using the chef_gem resource in a recipe in the chef-client cookbook. (A start handler may not be loaded using the `chef_handler` resource.)

chef-shell

chef-shell is a recipe debugging tool that allows the use of breakpoints within recipes. chef-shell runs as an Interactive Ruby (IRb) session. chef-shell supports both recipe and attribute file syntax, as well as interactive debugging features.

chef-shell is tool that is run using an Interactive Ruby (IRb) session. chef-shell currently supports recipe and attribute file syntax, as well as interactive debugging features. chef-shell has three run modes:

Mode	Description
Standalone	Default. No cookbooks are loaded, and the run-list is empty.
Solo	chef-shell acts as a Chef Solo Client. It attempts to load the chef-solo configuration file at `~/.chef/config.rb` and any JSON attributes passed. If the JSON attributes set a run-list, it will be honored. Cookbooks will be loaded in the same way that chef-solo loads them. chef-solo mode is activated with the `-s` or `--solo` command line option, and JSON attributes are specified in the same way as for chef-solo, with `-j /path/to/chef-solo.json`.
Client	chef-shell acts as a Chef Infra Client. During startup, it reads the Chef Infra Client configuration file from `~/.chef/client.rb` and contacts the Chef Infra Server to get the node's run_list, attributes, and cookbooks. Chef Infra Client mode is activated with the `-z` or `--client` options. You can also specify the configuration file with `-c CONFIG` and the server URL with `-S SERVER_URL`.

Configure

chef-shell determines which configuration file to load based on the following:

If a configuration file is specified using the -c option, chef-shell will use the specified configuration file
If a NAMED_CONF is given, chef-shell will load ~/.chef/NAMED_CONF/chef_shell.rb
If no NAMED_CONF is given chef-shell will load ~/.chef/chef_shell.rb if it exists
If no chef_shell.rb can be found, chef-shell falls back to load:
- /etc/chef/client.rb if -z option is given.
- /etc/chef/solo.rb if –solo-legacy-mode option is given.
- .chef/config.rb if -s option is given.
- .chef/knife.rb if -s option is given.

chef-shell.rb

The chef-shell.rb file can be used to configure chef-shell in the same way as the client.rb file is used to configure Chef Infra Client. For example, to configure chef-shell to authenticate to the Chef Infra Server, copy the node_name, client_key, and chef_server_url settings from the config.rb file:

node_name                'your-knife-clientname'
client_key               File.expand_path('~/.chef/my-client.pem')
chef_server_url          'https://api.opscode.com/organizations/myorg'

and then add them to the chef-shell.rb file. Other configuration possibilities include disabling Ohai plugins (which will speed up the chef-shell boot process) or including arbitrary Ruby code in the chef-shell.rb file.

Run as a Chef Infra Client

By default, chef-shell loads in standalone mode and does not connect to the Chef Infra Server. The chef-shell can be run as a Chef Infra Client to verify functionality that is only available when Chef Infra Client connects to the Chef Infra Server, such as search functionality or accessing data stored in data bags.

chef-shell can use the same credentials as knife when connecting to a Chef Infra Server. Make sure that the settings in chef-shell.rb are the same as those in config.rb, and then use the -z option as part of the command. For example:

chef-shell -z

Manage

When chef-shell is configured to access a Chef Infra Server, chef-shell can list, show, search for and edit cookbooks, clients, nodes, roles, environments, policyfiles, and data bags.

The syntax for managing objects on the Chef Infra Server is as follows:

chef-shell -z named_configuration

Where:

named_configuration is an existing configuration file in ~/.chef/named_configuration/chef_shell.rb, such as production, staging, or test.

Once in chef-shell, commands can be run against objects as follows:

chef (preprod) > items.command

Where:

items is the type of item to search for: cookbooks, clients, nodes, roles, environments or a data bag.
command is the command: list, show, find, or edit.

For example, to list all of the nodes in a configuration named “preprod”, enter:

chef (preprod) > nodes.list

Which will return something similar to:

=> [node[i-f09a939b], node[i-049a936f], node[i-eaaaa581], node[i-9154b1fb],
    node[i-6a213101], node[i-c2687aa9], node[i-7abeaa11], node[i-4eb8ac25],
    node[i-9a2030f1], node[i-a06875cb], node[i-145f457f], node[i-e032398b],
    node[i-dc8c98b7], node[i-6afdf401], node[i-f49b119c], node[i-5abfab31],
    node[i-78b8ac13], node[i-d99678b3], node[i-02322269], node[i-feb4a695],
    node[i-9e2232f5], node[i-6e213105], node[i-cdde3ba7], node[i-e8bfb083],
    node[i-743c2c1f], node[i-2eaca345], node[i-aa7f74c1], node[i-72fdf419],
    node[i-140e1e7f], node[i-f9d43193], node[i-bd2dc8d7], node[i-8e7f70e5],
    node[i-78f2e213], node[i-962232fd], node[i-4c322227], node[i-922232f9],
    node[i-c02728ab], node[i-f06c7b9b]]

The list command can take a code block, which will applied (but not saved), to each object that is returned from the server. For example:

chef (preprod) > nodes.list {|n| puts "#{n.name}: #{n.run_list}" }