A month ago, Amazon released Amazon Linux 2015.03, their “continuous release” Linux distribution. Widespread sadness ensued, and I tweeted:

While factually correct, that’s a bad tweet which I quickly regretted.

At ScaleSummit last week, I mentioned that I had some unreleased patches to Ansible’s Cloudformation module to implement a dry-run mode. This piqued some interest, so I’ve cleaned it up a little and am sharing it here.

First, a disclaimer: I’ve spent precious little time with Python in the last decade, and my primary goal was to write code that worked. I wouldn’t say I’m happy with it, but it works well enough for me. Feel free to take it and run with it.

If you just want the code, you’ll find it on this branch; otherwise read on for details…

By default, Jenkins is configured to run one “build executor” for each core in the server. This is a reasonable default for CPU-bound workloads (like most compile/test jobs), but it’s a bit pessimistic in my case – CI pipelines for my Chef cookbooks.

Fortunately, the executor count can be increased with a small Groovy script – this thread on StackOverflow told me what I needed to know.

If you’re already using the Jenkins cookbook to manage your Jenkins server, you can run two executors per core like so:

1
2
3
4
5
6
7
8
9
10
11

jenkins_script 'set number of executors' do
  command <<-EOF
import hudson.model.*

// Set executors to twice the number of CPU cores
Hudson hudson = Hudson.getInstance()
hudson.setNumExecutors(#{node['cpu']['total'] * 2})
hudson.setNodes(hudson.getNodes())
hudson.save()
  EOF
end

Seven years ago, brian d foy proposed a question for would-be language advocates: “What do you hate most about your language?”. His point was that if you’re not familiar enough with something to hate parts of it, you don’t know it well enough to be an effective advocate. At the time, it inspired some great posts in a few programming language communities. More recently, I was reminded of the question after reading one too many posts praising some new config management tool that went something like this:

I’ve been using X for two weeks, and it’s awesome!
I used Chef/Puppet before and it took me weeks to understand it.
X was so easy to use, I did [some trivial task] in my first half hour!
I thoroughly recommend it.

Such advocacy is profoundly uninteresting. Beyond the fact that you only get started with a tool once, the consequences of a tool’s design decisions often take time to make themselves apparent. The magic that made your first hour easy as a solo user might make life difficult working in a team; or the elegant simplicity in your pilot project might require a pile of hacks six months later at scale. One person’s minor irritations are another’s dealbreakers, so I love hearing what folks think of their preferred tools after they’ve had to live with them for a while.

I’ve been using Chef for the last few years, and it’s currently my favourite tool, so here’s five things I hate about it:

In my writeup of Scale Summit, I referred to a talk that Simon Croome would be giving at Puppet Camp London, and promised to link to it when it was online. The slides and video are now online, so check out “Increasing Agility by Understanding Risk”.

The back half of the presentation introduces Gonzo (github), a tool Simon wrote to identify and review pending changes in his Puppet manifests, and it’s this that piqued my interest at Scale Summit.

Whether or not they’re using a configuration management tool, “What was changed?” is one of the first questions people ask when something goes wrong. The desire to have prompt answers to that question is one of the drivers behind change management processes, and Chef and Puppet both offering reporting features to help with this. If our only desire was to respond more quickly when things went wrong, that would be enough.

However, “move fast and break things” notwithstanding, most businesses would prefer to avoid incidents caused by planned changes. This is usually the primary motivation for introducing change control processes, and involves (at the very least) identification of the proposed changes, an assessment of the associated risk, and review of these before deciding whether to proceed.

Humans aren’t great at assessing risk, particularly when making an off-the-cuff assessment of something they’re about to do. As sysadmins, we’re often over-confident about changes we’re going to make by hand – after all, we’re smart! If something goes wrong in the middle of the change, we’ll notice it and react with our cat-like reflexes.

When moving to “fly-by-wire system administration” using tools like Chef and Puppet, this attitude often shifts dramatically in the other direction. Misplaced confidence in our own control of the situation is replaced by fear. We admit that we aren’t really certain we know what effect a change will have – and we’re sure the system will inflict any mistake at scale, without a second thought.¹

Gonzo is the first tool I’ve seen that helps users of a modern configuration management tool to identify and review pending changes across their entire infrastructure. Even in environments without formal change control processes, this visibility can help to avoid unexpected changes and improve confidence and comfort with the tools being used.

No-op/why-run modes are necessarily imperfect, but they’re not worthless. As a Chef user, Gonzo gives me another reason to be jealous of Puppet and I hope its ideas make their way into the Chef ecosystem before long.

So, Scale Summit has now been and gone. As the spiritual descendant of UK Scale Camp (which remains my unconference benchmark) I’d been looking forward to this one since I heard about it. An unconference, if you haven’t been to one, puts the content in the hands of the participants. The general idea is to gather a bunch of folks with a shared interest, keep them fed and watered, and give them space to talk. On the day, participants propose topics to discuss, set a schedule, then make it happen.

As you might expect, this is a bit of a gamble – sometimes magic happens, sometimes you get a damp squib.

One of the great beauties of ChefSpec (and of course Chef) is that at
its heart it’s just Ruby. This means that at almost any point you can
reach into the standard Ruby development toolkit for your testing or
infrastructure development needs. Hopefully this little example will
be helpful to you.

Although you don’t need to know Ruby to be get started with Chef, a willingness to learn will be rewarded. This is especially true when it comes to testing. This post by Stephen Nelson-Smith provides a gentle introduction to an essential tool in the testing toolkit – stubs – that should be accessible to most Chef users.

Last week I went to Gent for cfgmgmtcamp.eu, a two day conference bringing together various config management communities. Around 300 people attended, representing (at least) Puppet, Chef, CFEngine, Ansible, Saltstack, and Juju. It was a great conference, with loads of interesting talks (no video, unfortunately) and a fantastic hallway track.

There were a bunch of good sessions in the Chef community room, and that (plus the hallway conversations) got in the way of spending any time in the other community rooms. I’m a little disappointed about that, though I’m sure the same would be true if I’d missed out on the Chef sessions.

I presented a talk about Orchestrating Chef with MCollective. It’s been a long time since I spoke, and I wasn’t particularly happy with how it went – this was the one session where I was glad to have no video! I personally learned a lot in the process, and would like to give a few (shorter) talks this year to work on this skill. Hopefully the folks who sat through it learned a thing or two as well.

Common themes in presentations and discussion were continuous delivery, testing (and TDD) for infracode, and orchestration. In “Be More Pushy”, Michael Ducy talked about Opscode’s orchestration system, which I’m looking forward to using. Ulf Mansson and George Miranda both talking about building continuous delivery pipelines, and Andrew Crump explained how Foodcritic works (and how to extend it).

There were no talks scheduled in the Chef room for the final few hours, so this became a space for hacking and lightning talks. Highlights were Andrew Crump showing me how to fix a couple of bugs in Foodcritic, Adam Jacob demonstrating test-driven development, and a lightning talk (I didn’t catch the presenter’s name, sorry) about community cookbook governance, which spurred a much longer conversation on the subject.

I departed the conference wishing it had been longer, and looking forward to the next one. Now that I’m home, I’m wishing I had more time to apply the various things I learnt while I was there. All in all, highly recommended.

The recent 1.0 release of test-kitchen (and the existence of kitchen-docker) gave me the push I needed to start using Jenkins to test my Chef cookbooks. Although I’ve heard plenty of people talk about doing this, I struggled to find much in the way of guidance or detailed descriptions.

Finding nothing to crib from, I decided to first build a spike to better understand what was involved. The result is Cooking With Jenkins, a cookbook and Vagrantfile that installs Docker and Jenkins, configured to run tests on a few cookbooks. If you’re new to these tools, I hope it will be a useful example to help you get started. If you’re already using them, your suggestions for improvement will be rewarded with thanks and beverages.

So, Vagrant is a great tool, which costs you nothing to use if you pair it with Virtualbox. Unfortunately, Virtualbox is somewhat less awesome in heavy use. VMware does a better job of things, and Vagrant added support for it with a paid-for plugin. As the first commercial product of Hashicorp (Vagrant’s developer), this is a nice way to say thankyou for Vagrant while getting a better user experience too.

In theory.