Building Chef Cookbooks with GitLab (Part 1)
Foreword
Want a TL;DR? - Go to the GitLab CI section, for the snippet you'll need to add to your .gitlab-ci.yml file to add integration test support.
The repository for this article can be found at jamietanna/user-cookbook.
This tutorial expects you have the Chef Development Kit (ChefDK) and Docker Command Line tools installed, have an account on GitLab.com with a repo created.
Note: This tutorial is using master as the primary branch for development. This is not the method in which I normally work, which I will expand on in the next part of the series.
Bootstrapping
We'll start by creating a new cookbook, by running chef exec generate cookbook user-cookbook. This is going to be a pretty boring cookbook which will create a user and optionally create a file in their home directory.
Let's start by pushing the code up to GitLab, i.e. git remote add origin git@gitlab.com:jamietanna/user-cookbook.git && git push -u origin master.
Creating a Recipe
Now we have our empty cookbook available, let's start adding some functionality:
recipes/default.rb:
user 'create user jamie' do
username 'jamie'
end
spec/unit/recipes/default_spec.rb:
require 'spec_helper'
describe 'user-cookbook::default' do
context 'When all attributes are default, on an Ubuntu 16.04' do
let(:chef_run) do
# for a complete list of available platforms and versions see:
# https://github.com/customink/fauxhai/blob/master/PLATFORMS.md
runner = ChefSpec::ServerRunner.new(platform: 'ubuntu', version: '16.04')
runner.converge(described_recipe)
end
it 'converges successfully' do
expect { chef_run }.to_not raise_error
end
it 'creates the jamie user' do
expect(chef_run).to create_user('create user jamie')
.with(username: 'jamie')
end
end
end
Now let's push this to GitLab.
Initial CI Setup
As we've not configured anything in GitLab CI to run, we won't actually have any automated means of determining whether the code we're pushing is correct or not.
In an ideal world, we'd want to at least have our pipeline set up to run our unit tests whenever we pushed a commit.
The easiest route we can go is to run on a Debian image, and install the ChefDK on top via the handy .deb package. At the time of writing, the latest version is 1.3.43, which is done as follows:
.gitlab-ci.yml:
image: debian:jessie
test:
script:
- apt update && apt install -yq curl
- curl https://packages.chef.io/files/stable/chefdk/1.3.43/debian/8/chefdk_1.3.43-1_amd64.deb -o /tmp/chefdk.deb
- dpkg -i /tmp/chefdk.deb
- chef exec rspec
Which now means that when we push to GitLab, our CI process runs our unit tests against the code.
Making Our Recipe More Useful
Having a configurable user
Now, having a cookbook that only ever creates a single, hardcoded, user isn't actually very useful. So let's make it possible to configure it via our cookbook's attributes (CI):
attributes/default.rb:
node.default['user'] = 'jamie'
recipes/default.rb:
user "create user #{node['user']}" do
username node['user']
end
spec/unit/recipes/default_spec.rb:
require 'spec_helper'
describe 'user-cookbook::default' do
context 'When all attributes are default, on an Ubuntu 16.04' do
# ...
it 'creates the jamie user' do
expect(chef_run).to create_user('create user jamie')
.with(username: 'jamie')
end
end
context 'When the user attribute is set' do
let(:chef_run) do
runner = ChefSpec::ServerRunner.new(platform: 'ubuntu', version: '16.04') do |node|
node.automatic['user'] = 'test'
end
runner.converge(described_recipe)
end
it 'converges successfully' do
expect { chef_run }.to_not raise_error
end
it 'creates the test user' do
expect(chef_run).to create_user('create user test')
.with(username: 'test')
end
end
end
Having a configurable group
So what if we want to specify the group of the user (CI)?
recipes/default.rb:
user "create user #{node['user']}" do
username node['user']
group node['group']
end
spec/unit/recipes/default_spec.rb:
require 'spec_helper'
describe 'user-cookbook::default' do
context 'When all attributes are default, on an Ubuntu 16.04' do
# ...
it 'creates the jamie user' do
expect(chef_run).to create_user('create user jamie')
.with(username: 'jamie')
.with(group: nil)
end
end
context 'When the user attribute is set' do
# ...
it 'creates the test user' do
expect(chef_run).to create_user('create user test')
.with(username: 'test')
.with(group: nil)
end
end
context 'When the user and group attributes are set' do
let(:chef_run) do
runner = ChefSpec::ServerRunner.new(platform: 'ubuntu', version: '16.04') do |node|
node.automatic['user'] = 'test'
node.automatic['group'] = 'users'
end
runner.converge(described_recipe)
end
it 'converges successfully' do
expect { chef_run }.to_not raise_error
end
it 'creates the test user' do
expect(chef_run).to create_user('create user test')
.with(username: 'test')
.with(group: 'users')
end
end
end
Create a file for the user
Next, we will create a file, owned by the user, in their own home directory, which is done as follows (CI):
recipes/default.rb:
file 'creates the hello.txt file' do
path "~#{node['user']}/hello.txt"
content "hello #{node['user']}"
mode '0600'
owner node['user']
only_if { node['hello'] }
end
spec/unit/recipes/default_spec.rb:
require 'spec_helper'
describe 'user-cookbook::default' do
context 'When all attributes are default, on an Ubuntu 16.04' do
# ...
it 'doesn\'t create the hello.txt file' do
expect(chef_run).to_not create_file('creates the hello.txt file')
end
end
context 'When the user attribute is set' do
# ...
it 'doesn\'t create the hello.txt file' do
expect(chef_run).to_not create_file('creates the hello.txt file')
end
end
context 'When the user and group attributes are set' do
# ...
it 'doesn\'t create the hello.txt file' do
expect(chef_run).to_not create_file('creates the hello.txt file')
end
end
context 'When the hello attribute is set' do
let(:chef_run) do
runner = ChefSpec::ServerRunner.new(platform: 'ubuntu', version: '16.04') do |node|
node.automatic['user'] = 'jamie'
node.automatic['hello'] = true
end
runner.converge(described_recipe)
end
it 'converges successfully' do
expect { chef_run }.to_not raise_error
end
it 'creates the hello.txt file' do
expect(chef_run).to create_file('creates the hello.txt file')
.with(path: '~jamie/hello.txt')
.with(content: 'hello jamie')
.with(mode: '0600')
.with(owner: 'jamie')
end
end
end
Integration Testing
As well as writing unit tests to ensure that at the component level we have a fully tested set of recipes, we also need to ensure that once the recipes are used in conjunction, everything still works. This is where we can bring in our integration tests.
Now, it's not often worth running integration tests against all combinations of machines you're going to run against, every time you commit. I prefer to run them when it gets to develop, or as it is on its way to master. However, we'll cover this workflow in the next part of the series, and for now, we'll run it on every commit.
Local Testing
The most common method of integration testing cookbooks is by using Vagrant. However, I've found that can be a little slow, as it has the overhead of requiring a full Virtual Machine. We can instead speed up our testing by using Docker (which conveniently means that we can use the same method of integration testing both locally and as part of our pipelines.
We can do this by using the kitchen-docker driver for Test Kitchen, which provides the same goodness that we can expect from Test Kitchen, but with the perk of it being run against a Docker image.
The first Docker-related changes we need to make in our .kitchen.yml:
---
driver:
name: docker
# make kitchen detect the Docker CLI tool correctly, via
# https://github.com/test-kitchen/kitchen-docker/issues/54#issuecomment-203248997
use_sudo: false
privileged: true
This ensures that we're using the kitchen-docker driver, and that we ensure that it can correctly hook into the docker CLI tools. Note that this process requires you to have set yourself up with the Manage Docker as a non-root user steps.
Next, we need to tell kitchen-docker what platforms we want to be running against:
platforms:
- name: debian
driver_config:
image: debian:jessie
This will specify that we want to test against Debian Jessie. Adding another platform to test against is straightforward and won't be expanded on until the next post.
Next, we define our test suite to run against.
suites:
- name: default
run_list:
- recipe[user-cookbook::default]
verifier:
inspec_tests:
- test/integration/default
attributes:
user: 'tester'
This lets us specify the recipes we want to run (our run_list) as well as any attributes we want to pass in to configure the node. For now, let's ignore the verifier section, which is covered later.
These steps in full can be found in this commit (CI).
To test this, we'll run kitchen converge. This will create our image if it's not already created, and then will run the cookbook on the new node.
Now that it works with basic settings, let's add some more integration tests (CI) to cover some more combinations:
.kitchen.yml:
suites:
- name: default
# ...
- name: custom-group
run_list:
- recipe[user-cookbook::default]
verifier:
inspec_tests:
- test/integration/custom-group
attributes:
group: 'test'
- name: hello
run_list:
- recipe[user-cookbook::default]
verifier:
inspec_tests:
- test/integration/hello
attributes:
hello: true
After running another kitchen converge, it turns out that actually things aren't quite working!
Fixing integration test issues
When we look at the errors returned by Chef, we can see a couple of glaring issues in the test suites.
custom_group test suite
It looks like it's trying to add jamie to the test group, which is what we expected. But what we didn't know is that the group needs to be created before we can add it to the group. This is the reason we do integration tests!
recipes/default.rb:
group 'create the group' do
group_name node['group']
not_if { node['group'].nil? }
end
# ...
spec/unit/recipes/default_spec.rb:
require 'spec_helper'
describe 'user-cookbook::default' do
context 'When all attributes are default, on an Ubuntu 16.04' do
# ...
it 'doesn\'t create the users group' do
expect(chef_run).to_not create_group('create the group')
end
# ...
end
context 'When the user attribute is set' do
# ...
it 'doesn\'t create the users group' do
expect(chef_run).to_not create_group('create the group')
end
# ...
end
context 'When the user and group attributes are set' do
# ...
it 'creates the users group' do
expect(chef_run).to create_group('create the group')
.with(group_name: 'users')
end
# ...
end
context 'When the hello attribute is set' do
# ...
it 'doesn\'t create the users group' do
expect(chef_run).to_not create_group('create the group')
end
# ...
end
end
hello test suite
This is a problem due to the expansion of the string ~jamie not working, due to Chef not interpolating the ~ character as a special marker to denote a user's home directory.
The easiest (but not nicest) way of doing this, is to update the home directory path (CI) to /home/#{node['user']}, which expands out to i.e. /home/jamie:
recipes/default.rb:
# ...
file 'creates the hello.txt file' do
path "/home/#{node['user']}/hello.txt"
content "hello #{node['user']}"
mode '0600'
owner node['user']
only_if { node['hello'] }
end
spec/unit/recipes/default_spec.rb:
require 'spec_helper'
describe 'user-cookbook::default' do
context 'When all attributes are default, on an Ubuntu 16.04' do
# ...
end
context 'When the user attribute is set' do
# ...
end
context 'When the user and group attributes are set' do
# ...
end
context 'When the hello attribute is set' do
# ...
it 'creates the hello.txt file' do
expect(chef_run).to create_file('creates the hello.txt file')
.with(path: '/home/jamie/hello.txt')
.with(content: 'hello jamie')
.with(mode: '0600')
.with(owner: 'jamie')
end
end
end
However, that still doesn't quite work. Chef by default doesn't actually 'manage' the home directory. This means that we don't actually have the directory created until we explicitly set manage_home true (CI) when creating the user:
recipes/default.rb:
# ...
user "create user #{node['user']}" do
username node['user']
group node['group']
manage_home true
end
# ...
spec/unit/recipes/default_spec.rb:
require 'spec_helper'
describe 'user-cookbook::default' do
context 'When all attributes are default, on an Ubuntu 16.04' do
# ...
it 'creates the jamie user' do
expect(chef_run).to create_user('create user jamie')
.with(username: 'jamie')
.with(group: nil)
.with(manage_home: true)
end
# ...
end
context 'When the user attribute is set' do
# ...
it 'creates the test user' do
expect(chef_run).to create_user('create user test')
.with(username: 'test')
.with(group: nil)
.with(manage_home: true)
end
# ...
end
context 'When the user and group attributes are set' do
# ...
it 'creates the test user' do
expect(chef_run).to create_user('create user test')
.with(username: 'test')
.with(group: 'users')
.with(manage_home: true)
end
# ...
end
context 'When the hello attribute is set' do
# ...
it 'creates the hello.txt file' do
expect(chef_run).to create_file('creates the hello.txt file')
.with(path: '/home/jamie/hello.txt')
.with(content: 'hello jamie')
.with(mode: '0600')
.with(owner: 'jamie')
end
end
end
GitLab CI
Now we have it working locally, let's add our setup to test this when we're pushing up to GitLab, too:
.gitlab-ci.yml:
# ...
integration_test:
image: docker:latest
services:
- docker:dind
script:
- 'echo gem: --no-document > $HOME/.gemrc'
- apk update
- apk add build-base git libffi-dev ruby-dev ruby-bundler
- gem install kitchen-docker kitchen-inspec berkshelf
- kitchen test
So there are a few things new here. Firstly, we're now using the docker image as our base. This is so we get access to the docker CLI tools, which are required by kitchen-docker. Next, we use the dind, or Docker in Docker, service which allows us to build and run another Docker image within our docker image.
We then need to install some dependencies such as the gems we need so we can actually run our tests, after which, we can run kitchen test to perform a full run of all our test suites.
And now, looking at our pipelines, we can see that this commit has run the integration tests! But, the job is still failing...
So it converged, now what?
You may notice that when running kitchen test, we actually fail. This is due to Inspec, a system verification tool, not finding the correct integration tests in the specified directories in our .kitchen.yml:
---
driver:
name: docker
# make kitchen detect the Docker CLI tool correctly, via
# https://github.com/test-kitchen/kitchen-docker/issues/54#issuecomment-203248997
use_sudo: false
privileged: true
provisioner:
name: chef_zero
# You may wish to disable always updating cookbooks in CI or other testing environments.
# For example:
# always_update_cookbooks: <%= !ENV['CI'] %>
always_update_cookbooks: true
require_chef_omnibus: 12.19.36
verifier:
name: inspec
platforms:
- name: debian
driver_config:
image: debian:jessie
suites:
- name: default
# ...
verifier:
inspec_tests:
- test/integration/default # <----
- name: custom-group
# ...
verifier:
inspec_tests:
- test/integration/custom-group # <----
# ...
- name: hello
run_list:
- recipe[user-cookbook::default]
verifier:
inspec_tests:
- test/integration/hello # <----
attributes:
# ...
Before we get to this, though, we notice as we're looking through the test suites that we've not actually got any cases where there is a different user, just group. Let's tack it on with the hello case (CI), and run a quick kitchen converge to ensure that the cookbook still converges.
.kitchen.yml:
---
# ...
suites:
# ...
- name: hello
run_list:
- recipe[user-cookbook::default]
verifier:
inspec_tests:
- test/integration/hello
attributes:
user: 'everybody'
hello: true
Now that's resolved, let's write some quick integration tests (CI):
test/integration/custom-group/default.rb:
describe user('jamie') do
it { should exist }
its('groups') { should eq ['test'] }
end
describe group('test') do
it { should exist }
end
describe directory('/home/jamie') do
it { should exist }
end
test/integration/default/default.rb:
describe user('jamie') do
it { should exist }
its('groups') { should eq ['jamie'] }
end
describe group('jamie') do
it { should exist }
end
describe directory('/home/jamie') do
it { should exist }
end
test/integration/hello/default.rb:
describe user('everybody') do
it { should exist }
its('groups') { should eq ['everybody'] }
end
describe group('everybody') do
it { should exist }
end
describe directory('/home/everybody') do
it { should exist }
end
describe file('/home/everybody/hello.txt') do
it { should exist }
its('mode') { should cmp '0600' }
its('owner') { should eq 'everybody' }
its('content') { should eq 'hello everybody' }
end
Conclusion
So we've seen how to build a basic cookbook from the ground up, taking care to unit test first, then work on integration tests after the functionality is complete.
Once our integration tests have worked locally, we've configured our GitLab CI pipelines to perform the same tests, so we know that when pushing code, we can ensure that it will have full integration coverage, too.
