Cryptocracy

Amazon Linux, And The Perils Of Continuous Release

2015-04-26T23:35:00+01:00

A month ago, Amazon released Amazon Linux 2015.03, their "continuous release" Linux distribution. Widespread sadness ensued, and I tweeted:

Amazon Linux users! If you didn't want surprise updates, you should have locked your yum repo_releasever. See release notes for details.

— Zac Stevens (@zts) March 25, 2015

While factually correct, that's a bad tweet which I quickly regretted. It's bad for being counterfactual, and displaying no trace of empathy.

"If you didn't want surprise updates" - who does want surprise updates?
"You should have locked repo_releasever" - I realise that now, thanks a lot asshole.
"See release notes for details" - ...aaaand you're making me Google it.

It's clearly directed at the person whose day sucked because of this issue, and it's kicking them while they're down.

Worse still, it wasn't really without empathy; I had a bad day following the 2014.03 update a year ago, so I know full well what it feels like. I guess I'm still kicking myself for my lack of foresight, and made a frustrated tweet aimed in the wrong direction. Sorry, people of Twitter.

The Constructive Part

Amazon says:

"Beginning with the 2011.09 release of the Amazon Linux AMI, the repository structure is configured to deliver a continuous flow of updates that allow you to roll from one version of the Amazon Linux AMI to the next. ... In other words, Amazon Linux AMIs are treated as snapshots in time, with a repository and update structure that gives you the latest packages that we have built and pushed into the repository."

— "Amazon Linux FAQ", http://aws.amazon.com/amazon-linux-ami/faqs/

This means, by default, that an instance built using the Amazon Linux 2014.09 AMI will now upgrade packages to those in 2015.03. Some of these upgrades will be harmless, while others will break your stuff. Since computers lack a "don't break my stuff" option, you either live with that risk or disable these updates.

The way to disable these updates is by pinning a Yum variable called repo_releasever. If you want to do this on a running instance, the file to edit is /etc/yum.conf. You can set this when starting new instances by providing this userdata:

#cloud-config
repo_releasever: 2015.03

See "How do I lock my AMI to a specific version?" in the Amazon Linux FAQs.

The Critical Part

Amazon employs loads of smart people, and I'm sure there was a robust debate when considering the appropriate default to this setting. But since I can't find a public justification for this decision, I will be blunt: they screwed this one up. It's not hard to think of ways this default will end up ruining your day, and I struggle to find even tenuous arguments in support of it.

I don't recall the specifics of my issues with 2014.03 - the default ruby changed from 1.8 to 2.0, and that definitely screwed up the mcollective packages I was using. I also recall that the mysql gem I compiled for use with Chef broke, perhaps because the glibc version changed.

2015.03 took greater care when changing the default python version; running systems updated from 2014.09 would install 2.7 alongside 2.6, but would not change the default. This caused problems for anyone invoking the default python and expecting to interact with the yum repository (as all the system tooling was updated to use 2.7).

The idea of "immutable infrastructure" is something of a fiction at the best of times, but I don't imagine anyone bakes an AMI expecting it will boot up with a certain set of packages on one day, and a different set of packages on another. Similarly, I don't imagine anyone expects a CM run on their system to upgrade glibc or language runtimes unless they've explicitly configured it to do so.

tl;dr

The Amazon Linux AMIs ship with a default upgrade policy that thoroughly violates the principle of least surprise.
This can be easily corrected by setting appropriate userdata (for new instances) or updating yum.conf (for running instances).
I'll be linking to this post in August to avoid looking like an asshole again in September.

Dry Run CloudFormation Updates Using Ansible

2015-04-03T00:15:00+01:00

At ScaleSummit last week, I mentioned that I had some unreleased patches to Ansible's Cloudformation module to implement a dry-run mode. This piqued some interest, so I've cleaned it up a little and am sharing it here.

First, a disclaimer: I've spent precious little time with Python in the last decade, and my primary goal was to write code that worked. I wouldn't say I'm happy with it, but it works well enough for me. Feel free to take it and run with it.

If you just want the code, you'll find it on this branch; otherwise read on for details...

Dry-Run Mode, You Say?

CloudFormation (the web service) doesn't have a dry-run mode - you can't pass it a set of inputs and find out what it would do. Still, we can build a useful approximation by taking inspiration from Chef's naming of this feature, "Why-Run?" CloudFormation records the inputs (template and parameters) used for each update, and we can compare these to the inputs we plan to use. The changes we find are the reasons why CloudFormation might do something during an update.

What this can't tell us is what resources will be updated¹, whether those updates will involve replacement of the resource, whether our template is nonsense², etc.

Will this be useful to you? If your templates are simple enough that a list of modified inputs is enough for you to reason about the outcome, you'll find this useful.

What Does It Do?

When run in check mode (ansible-playbook --check), a CloudFormation task will be output like so:

TASK: [create example-test cfn stack] **************************************
changed: [example] => {"changed": true, "output": "Stack would be updated, changes: [{'Template': {u'Resources': {u'DummyWaitHandle2': 'Added', u'DummyWaitHandle': 'Removed'}}}, {u'SnapshotID': {'to': u'test-20150329', 'from': u'test-20150330'}}]", "stack_outputs": {"DBInstanceId": "rdz5lp9i8ovmog", "FrontendELB": "example-Frontend-1VIHBAZL8WUA1", "FrontendHostname": "example-Frontend-1VIHBAZL8WUA1-1582281064.eu-west-1.elb.amazonaws.com", "SecurityGroup": "example-test-SecurityGroup-16PP0KV5SCAS9", "SecurityGroup2": "example-test-SecurityGroup2-I2ENEANIVZPF"}}

This is not particularly readable when wrapped in a terminal window - still, it's an improvement on nothing. I couldn't readily find any way of producing nicely formatted output. I originally wanted to show a unified diff of the template changes, but that was completely unreadable. I did find a --diff option in Ansible, but this apparently only works for template changes and I couldn't figure out how to use that here.

I've also added a described state, which just exposes a stack's parameters. I believe the idiomatic solution to this in Ansible would be a lookup plugin³, but I prefer this way.

What (Else) Does It Do Badly?

When I mentioned this at ScaleSummit, I didn't include a diff of the template - only an indication that the template had changed. Verifying parameter changes was valuable to me, but ScaleSummit prodded me into taking another look at template comparison.

So, the template diff mindlessly compares two json documents - it doesn't understand anything about the semantics of CloudFormation templates. It descends recursively into dicts to find changes, ~~but arrays are shown as the stringified before/after copies. This makes changes to some resource types (eg, security groups) very noisy~~.

Room For Improvement

~~Show only added/removed/changed array elements~~
Limit template diff to a certain depth
Show unified diff via --diff option (if possible)
Validate local template
~~Check mode for stack creation should show the parameters which will be used~~
~~Fake up outputs that will be created by a stack~~

EDIT: I found the motivation to fix a few of those warts.

What's Next?

I mostly improve this as something annoys me when I happen to have time for a diversion. While Ansible has made my use of CloudFormation somewhat less vexing, it's not a huge step forward and I'm hoping to replace this with Terraform in the near future.

My gut feeling is that this won't be accepted upstream as-is, but I haven't tried. Leave a comment if you want to encourage me to upstream it, or feel free to tidy it up and open a PR yourself.

We can be confident that any resources with template changes will update, but we don't know what others will be updated in consequence. ie, we suffer from false negatives. ↩
Though we could use the validate_template function for this. ↩
See this post by Dean Wilson. ↩

Increasing Jenkins Executors With Chef

2014-09-20T16:03:00+01:00

By default, Jenkins is configured to run one "build executor" for each core in the server. This is a reasonable default for CPU-bound workloads (like most compile/test jobs), but it's a bit pessimistic in my case - CI pipelines for my Chef cookbooks.

Fortunately, the executor count can be increased with a small Groovy script - this thread on StackOverflow told me what I needed to know.

If you're already using the Jenkins cookbook to manage your Jenkins server, you can run two executors per core like so:

jenkins_script 'set number of executors' do
  command <<-EOF
import hudson.model.*

// Set executors to twice the number of CPU cores
Hudson hudson = Hudson.getInstance()
hudson.setNumExecutors(#{node['cpu']['total'] * 2})
hudson.setNodes(hudson.getNodes())
hudson.save()
  EOF
end

Five Things I Hate About Chef

2014-04-29T14:00:00+01:00

Seven years ago, brian d foy proposed a question for would-be language advocates: "What do you hate most about your language?". His point was that if you're not familiar enough with something to hate parts of it, you don't know it well enough to be an effective advocate. At the time, it inspired some great posts in a few programming language communities. More recently, I was reminded of the question after reading one too many posts praising some new config management tool that went something like this:

I've been using X for two weeks, and it's awesome! I used Chef/Puppet before and it took me weeks to understand it. X was so easy to use, I did [some trivial task] in my first half hour! I thoroughly recommend it.

Such advocacy is profoundly uninteresting. Beyond the fact that you only get started with a tool once, the consequences of a tool's design decisions often take time to make themselves apparent. The magic that made your first hour easy as a solo user might make life difficult working in a team; or the elegant simplicity in your pilot project might require a pile of hacks six months later at scale. One person's minor irritations are another's dealbreakers, so I love hearing what folks think of their preferred tools after they've had to live with them for a while.

I've been using Chef for the last few years, and it's currently my favourite tool, so here's five things I hate about it:

1. No partial updates to node data in the Chef server

Chef doesn't provide any way to update a subset of a node's data - each update replaces the copy on the server, and the last writer wins. This means that if you update the server while a node is in the middle of a Chef run, you may find your changes are reverted when chef-client saves the node at the end of the run.

Related to this, knife node from file should have a big disclaimer attached to it. Some folks want to keep their basic node data in version control, and use that subcommand to upload changes. This erases almost everything about the node from the server, so any searches relying on computed or saved attributes will fail until the next chef-client run saves the full node again.

This is not an easy problem to solve. Fortunately, you're allowed to hate things that can't be easily fixed.

2. No standard solution for secure attributes

The standard way of parameterising recipes is attributes. The standard way of securing secrets for use by Chef is encrypted databags. Databags and attributes are entirely separate mechanisms, and different methods are used to load encrypted and regular databags. This means that the only cookbooks that load secrets from encrypted databags only do that (they can implement their own fallback through these mechanisms, but Chef doesn't offer any help).

But wait, wrapper cookbooks! You could write a wrapper to load your secrets from your encrypted databag, override the appropriate node attributes, then include the original recipe. This works, but now your secrets are recorded in node attributes and will be saved to the Chef server in the clear.

But wait, Chef handlers! You could write a Chef handler to remove the sensitive attributes from the node before it's saved! Maybe - I forget whether report handlers are actually called before the node is saved at the run - and anyway, recipes can (and do) save the node at any point.

I guess you could monkey-patch node.save (or something), but the point is you're on your own.

3. Underdevelopment of the Recipe DSL

The Recipe DSL hasn't changed much in the last couple of years, barring the addition of some helpers for Windows platforms.

The most glaring example is the absence of a helper for Encrypted Data Bags. Here's how you'd load a regular, unencrypted databag:

item = data_bag_item('users', 'zts')

Here's how you'd load the same databag if it was encrypted with the default secret:

item = Chef::EncryptedDataBagItem.load('users', 'zts')

Alright, it's only a small thing - but it sticks out like a sore thumb, and I know the glaring rubyism rubs some people up the wrong way. Seth Vargo's excellent Chef Sugar gem provides a load of enhancements to the recipe DSL (including an encrypted_data_bag_item method), many of which should be part of core Chef. Dumping every last idea into the recipe DSL might be worse than never adding anything at all, but the lack of love shown to such a core part of the Chef user experience makes me sad.

4. No local storage

Why would I need local storage? Loads of reasons. A big one is that I do a lot of work with chef-solo, which can't save any state at all (chef-client -z might fix this for me, but it's not quite a drop-in replacement).

Still, node-local storage would be useful for chef-client too. One application would be storage of secrets generated by Chef for services it installs and manages (eg, a mysql service). Another would be to enable chef-client to detect when node attributes have been changed on the server - or something like the immutable attributes, Dan Carley prototyped for Puppet some time ago.

5. Two-Phase Runs and the Resource Queue

In principle, a Chef run has two phases. In the first phase, recipes are "compiled", a process which pushes resources into a queue. In the second phase, the compiled resources are "converged" one by one. As resources are queued in the order they're declared, the order of actions is easy to predict. This is simple, elegant, and - in practice - a great big lie.

Cracks first start to appear in the two-phase ideal when you want to extend Chef - say, adding resources to manage mysql databases. To manage mysql using Ruby, we'll want to install the mysql gem. Before we can do that, we will first need to install the mysql libraries and a toolchain to compile the gem. Chef can do all these things, but it won't do any of them until it gets to the converge phase - and that means, we won't be able to use our shiny new mysql resource until the next time we run Chef.

Fortunately(?), it's possible to converge a specific resource during the compile phase using resource.run_action() and that's the go-to solution to this problem. That breaks the expectation that resources will be converged in the order they're declared, but at least the syntax makes it obvious - except in the case of chef_gem, which does converges in the compile phase without any visual indication in your recipe.

The upshot is that extending Chef with gems is a monumental pain. Seth Vargo's post "Using Gems With Chef" post explores some more options.

The waters are further muddied by LWRPs using inline compile mode. This creates additional, disconnected run_contexts which are compiled and converged when the LWRP is converged. This makes LWRPs work better, at the cost of obscuring the Chef resources they used to do their thing - run_context.resource_collection.all_resources won't contain those nested resources. (I suspect you can write a custom event subscriber to collect this information for yourself.)

Once again, I don't know what the solution looks like here (though I believe people are talking about doing something for Chef 12).

What about you?

So, that's five things I hate about Chef. I hope a few of you will be motivated to share five things you hate about your favourite tool. I'm not working in a large team at the moment, so I've mostly been thinking about technical niggles - but I'd love to hear perspectives on the human side of tool use, too.

Gonzo: Increasing Agility by Understanding Risk

2014-04-26T12:45:00+01:00

In my writeup of Scale Summit, I referred to a talk that Simon Croome would be giving at Puppet Camp London, and promised to link to it when it was online. The slides and video are now online, so check out "Increasing Agility by Understanding Risk".

The back half of the presentation introduces Gonzo (github), a tool Simon wrote to identify and review pending changes in his Puppet manifests, and it's this that piqued my interest at Scale Summit.

Whether or not they're using a configuration management tool, "What was changed?" is one of the first questions people ask when something goes wrong. The desire to have prompt answers to that question is one of the drivers behind change management processes, and Chef and Puppet both offering reporting features to help with this. If our only desire was to respond more quickly when things went wrong, that would be enough.

However, "move fast and break things" notwithstanding, most businesses would prefer to avoid incidents caused by planned changes. This is usually the primary motivation for introducing change control processes, and involves (at the very least) identification of the proposed changes, an assessment of the associated risk, and review of these before deciding whether to proceed.

Humans aren't great at assessing risk, particularly when making an off-the-cuff assessment of something they're about to do. As sysadmins, we're often over-confident about changes we're going to make by hand - after all, we're smart! If something goes wrong in the middle of the change, we'll notice it and react with our cat-like reflexes.

When moving to "fly-by-wire system administration" using tools like Chef and Puppet, this attitude often shifts dramatically in the other direction. Misplaced confidence in our own control of the situation is replaced by fear. We admit that we aren't really certain we know what effect a change will have - and we're sure the system will inflict any mistake at scale, without a second thought.¹

Gonzo is the first tool I've seen that helps users of a modern configuration management tool to identify and review pending changes across their entire infrastructure. Even in environments without formal change control processes, this visibility can help to avoid unexpected changes and improve confidence and comfort with the tools being used.

No-op/why-run modes are necessarily imperfect, but they're not worthless. As a Chef user, Gonzo gives me another reason to be jealous of Puppet and I hope its ideas make their way into the Chef ecosystem before long.

The unthinking application of changes is itself a problem - we need autonomation, not automation. But that's a subject for another post. ↩

Scale Summit 2014

2014-03-23T01:00:00+00:00

So, Scale Summit has now been and gone. As the spiritual descendant of UK Scale Camp (which remains my unconference benchmark) I'd been looking forward to this one since I heard about it. An unconference, if you haven't been to one, puts the content in the hands of the participants. The general idea is to gather a bunch of folks with a shared interest, keep them fed and watered, and give them space to talk. On the day, participants propose topics to discuss, set a schedule, then make it happen.

As you might expect, this is a bit of a gamble - sometimes magic happens, sometimes you get a damp squib.

What Was Good?

The venue was easy to get to, the catering was top notch, and wifi worked everywhere (which was just as well, as cellphone reception was abysmal). I also appreciated the wifi having a single password for all attendees.

I liked that the Chatham House rule applied by default, and that a code of conduct was circulated ahead of the event (and called out in the introduction). For that matter, communication ahead of the event did a good job explaining what to expect.

The event was about the right size (~130 people), if perhaps a little too large.

What Could Improve?

Introductions

After Michael Brunton-Spall's welcomed everyone to the conference, a microphone was passed around the room for everyone to introduce themselves in a "What's your name? What do you want to learn? What are you good at?" format. Despite providing some moments of hilarity (eg, "I think I'm at the wrong conference, I came to learn about scaling mountains.") this didn't really work for me.

One benefit of introductions (beyond their value as a basic icebreaker) is to help foster conversations through the rest of the day - if I say I'm good at Chef, and you say you're interested in learning about Chef, perhaps we'll have a chat during the day. Perhaps I was insufficiently caffeinated, but with so many people in attendance (and poor visibility, as we remained seated for the introductions), I didn't really manage to keep track of the folks I thought I might like to speak to.

I'm not sure how to improve this. One idea might be to do the introductions online ahead of the event - this would avoid putting people on the spot, and foster opportunities for people to connect in some way before they get into the room.

Another thought is to ask people to come up and introduce themselves on the morning, but only if they want to encourage other participants to start conversations with them during the day. This could be seen as an odd perversion of mst's Hallway++ idea, and I haven't particularly thought it through. Perhaps I really just want to see Hallway++ spread to more conferences.

Topics and Scheduling

The session board filled up fairly quickly, it wasn't necessarily clear what folks had in mind for the proposed sessions, and there wasn't any useful way to have conversations about it (due to lack of time, and lack of space around the board). It didn't feel like the schedule was a collaborative effort of the participants, so much as a bunch of individuals throwing thoughts at a wall.

Funnily enough, the introductions segment I've just complained about was provides some input to the scheduling process - eg, if several people mention AWS CloudFormation in their introductions, it's probably a good topic to propose a session around. (Several people did, and apparently the "Cloud Orchestration" session was an interesting one). On the flip side, the events schedule originally gave half an hour to "Collaboration over the sessions grid for the day", so it would seem the time spent on introductions came at the expense of something more focused on producing a good schedule.

At London DevOpsDays, the open spaces schedule was built in several phases. First, folks proposing topics got up and told the rest of the participants what they had in mind. Second, participants noted which topics they were interested in. Finally, topics were assigned to rooms suitable for the level of interest. I'm unfamiliar with other approaches to solving this problem, though I daresay they exist. This is the area I'd most like to see improve for the next Scale Summit.

Room Sizes and Engagement

The available rooms were relatively large - 3 for 60 people and 2 for 30 people, although another small room (~8 people) was employed when splinter groups decided to talk in the lobby. This lead to some large sessions with half a dozen people speaking and ~40 listening, which didn't feel ideal. Conversely, Scale Camp at the Guardian offices had a greater number (I seem to recall) of much smaller rooms, and one big one. This resulted in serious overcrowding in many of the sessions, which wasn't ideal either.

I suspect that any arrangement of rooms will have some kind of drawback (particularly for an unconference), so perhaps the answer is a little more structure or facilitation of the sessions. Michael did suggest some techniques in the day's introduction, though I didn't witness any of them being used. I'd like to have heard from more voices during the day - even if the majority have come to a session to learn, they could still be asking questions or providing prompts for the folks bringing the experience; conversely, if more folks would like to participate, the structure of the session should facilitate that.

Takeaways

Dean Wilson has previous tweeted about using Ansible to provision AWS resources, but the lightbulb hadn't turned on for me - I need to explore that idea as an alternative to CloudFormation.

On that note, I regret not joining the "Cloud Orchestration" session, but it was scheduled against a session I proposed on the subject of Scaling Config Management (testing, processes, teams, etc). The latter was well attended, so I hope people got some value out of it. I think I could have done a better job of introducing it, but I'll reflect on that another time.

Another lightbulb moment came when Simon Croome gave me a sneak preview of the work he'll be presenting at Puppet Camp London in his session "Increasing Agility By Understanding Risk". I'll be linking to that as soon as it's online (and I plan to incorporate some of his ideas in my Chef CD pipeline).

I went into the day wanting some more suggestions for host monitoring in elastic environments, or at least reassurance that Sensu was the best thing to play with. Apparently it is. A few folks who had spent a little time with Riemann were enthusiastic about it too, so that's now higher up my list than it was at the start of the day.

If you ever need to goad people into giving lightning talks, Jonty is remarkably effective at doing so.

Here's three good posts with far more notes about the topics discussed.

Conclusion

I really enjoyed the day, got plenty of value out of it, and I'm already looking forward to the next one. There's room for improvement but (for me, at least) that could only elevate the day from "really good" to "utterly awesome".

Special thanks to organisers Jon Topper (and the rest of the crew at The Scale Factory), Michael Brunton-Spall, and Sarah Hakewill, and to all the sponsors who helped keep the price affordable.

cfgmgmtcamp.eu 2014

2014-02-11T18:30:00+00:00

Last week I went to Gent for cfgmgmtcamp.eu, a two day conference bringing together various config management communities. Around 300 people attended, representing (at least) Puppet, Chef, CFEngine, Ansible, Saltstack, and Juju. It was a great conference, with loads of interesting talks (no video, unfortunately) and a fantastic hallway track.

There were a bunch of good sessions in the Chef community room, and that (plus the hallway conversations) got in the way of spending any time in the other community rooms. I'm a little disappointed about that, though I'm sure the same would be true if I'd missed out on the Chef sessions.

I presented a talk about Orchestrating Chef with MCollective. It's been a long time since I spoke, and I wasn't particularly happy with how it went - this was the one session where I was glad to have no video! I personally learned a lot in the process, and would like to give a few (shorter) talks this year to work on this skill. Hopefully the folks who sat through it learned a thing or two as well.

Common themes in presentations and discussion were continuous delivery, testing (and TDD) for infracode, and orchestration. In "Be More Pushy", Michael Ducy talked about Opscode's orchestration system, which I'm looking forward to using. Ulf Mansson and George Miranda both talking about building continuous delivery pipelines, and Andrew Crump explained how Foodcritic works (and how to extend it).

There were no talks scheduled in the Chef room for the final few hours, so this became a space for hacking and lightning talks. Highlights were Andrew Crump showing me how to fix a couple of bugs in Foodcritic, Adam Jacob demonstrating test-driven development, and a lightning talk (I didn't catch the presenter's name, sorry) about community cookbook governance, which spurred a much longer conversation on the subject.

I departed the conference wishing it had been longer, and looking forward to the next one. Now that I'm home, I'm wishing I had more time to apply the various things I learnt while I was there. All in all, highly recommended.

Cooking With Jenkins (and Test-Kitchen, and Docker)

2014-01-03T01:00:00+00:00

The recent 1.0 release of test-kitchen (and the existence of kitchen-docker) gave me the push I needed to start using Jenkins to test my Chef cookbooks. Although I've heard plenty of people talk about doing this, I struggled to find much in the way of guidance or detailed descriptions.

Finding nothing to crib from, I decided to first build a spike to better understand what was involved. The result is Cooking With Jenkins, a cookbook and Vagrantfile that installs Docker and Jenkins, configured to run tests on a few cookbooks. If you're new to these tools, I hope it will be a useful example to help you get started. If you're already using them, your suggestions for improvement will be rewarded with thanks and beverages.

What's In The Box?

The base system is Ubuntu 13.04 (from a Bento box). Jenkins and Docker are installed via the jenkins and docker cookbooks, and ruby1.9.3 comes from a system package.

The cookbook installs and configures various Jenkins plugins A recipe installs and configures various Jenkins plugins, including git, warnings (to parse Foodcritic output), ansicolor (to colourise console output), and config-file-provider (to add static files to the job workspace before testing).

A Chef definition wraps up the details of Jenkins job creation, and it assumes that the cookbook repo provides a Gemfile and Rakefile similar to those in mlafeldt's skeleton-cookbook. You can see the definition in use in this recipe.

Questions To Consider

You'll find a lot of questions when building a Jenkins system for cookbook testing, and most have more than one right answer - you'll need to consider the trade-offs, and choose what's best for you.

The Jenkins server I'm building will only be used for testing a private set of Chef cookbooks deployed in a homogeneous environment. This is as simple as it gets, so I've avoided some complexity that might be essential in other environments.

How should I install Ruby?

If you need to test your cookbook with multiple rubies, you'll probably want to use something like rbenv or rvm. I only need to worry about 1.9, which I installed with a system package.

Where should I install gems?

The tools we'll use to test our cookbooks are packaged as gems, and they need to be installed somewhere before we can use them. The generally recommended option is to include a Gemfile in each of our cookbooks, and use Bundler to install a local copy of the gems for each Jenkins job. Another option is to install all the tools once, in your system gem location.

If you use system gems, you'll save some time and disk space (they only need to be installed once), and you won't have to manage a Gemfile in each of your cookbook repositories. On the other hand, you can't use different gems (or versions) on a case-by-case basis.

Using Bundler means more boilerplate in your cookbook repositories, but it has the advantage of being explicit about your requirements. This makes life easier for other people developing your cookbooks, and it's absolutely essential if you'll also be testing them on something like Travis CI.

Where should I define the test commands?

When we define jobs in Jenkins to test our cookbooks, we'll need to specify the commands used to run the tests. An obvious approach is to use the command-line tools (eg, foodcritic) directly, but many people wrap the tasks up in a Rakefile and call everything through rake.

The main advantage of using the tools directly is that you don't need to maintain a (possibly identical) Rakefile in each of your cookbook repositories. On the other hand, you'll still want to run tests in your development environment, and a good Rakefile saves you from remembering the precise options you need for each command.

How should I map cookbook tests to Jenkins jobs?

A Jenkins "job" involves a workspace (a directory containing the revision of code you're testing), one or more build commands (for us, that's test commands), and zero or more actions to run following the build. Jenkins also supports chaining jobs, so that one job finishing schedules a second job. This allows you to build pipelines of activities, contingent on the success or failure of earlier jobs. Various plugins exist to extend the core functionality, such as the Build Pipeline Plugin and the Multijob Plugin.

That's a lot of complexity, which I opted to ignore by creating a single job (for each cookbook) that runs all of the required tests. I will certainly revisit this in the next iteration.

Installation Instructions

The Vagrantfile specifies a "Bento" basebox, which doesn't include a provisioner - so you'll need the "vagrant-omnibus" plugin installed if you want to try it out. It should work fine under Virtualbox, but I've been using VMware as it seems to cope better under load.

git clone https://github.com/zts/cooking-with-jenkins
cd cooking-with-jenkins
vagrant up

When Vagrant finishes provisioning the machine, Jenkins will be available on http://localhost:8080/.

Known Issue: The machine creates test jobs for three cookbooks, and the first test run for each (during provisioning) fails, so when you first load Jenkins you'll see something like this. I suspect this would be fixed by restarting Jenkins after adding plugins (the "configure-jenkins" recipe) and before defining the jobs, but it didn't bother me enough to add the necessary hackery.

Click "Build Now" on each job and you'll see them work correctly.

What Does It Look Like?

Cookbook overview As this cookbook has a single foodcritic warning, and a single "pending" chefspec test, the graphs are uninteresting yellow rectangles. Clicking the Foodcritic link on the left takes you to...

Foodcritic report Again, this would look more interesting if a bunch of problems were being reported, instead of just one example.

Colourised job console output This shows the bottom of the console output for a job that ran test-kitchen. It's not the easiest on the eyes, but I think it's better than raw ANSI colour codes in your browser. Also, note that the time to test this (admittedly simple) kitchen instance was just under 43 seconds. Test Kitchen with kitchen-docker is awesome.

What's Missing? What's Next?

In this demo, you always need to click "Build Now" to run tests for a cookbook - Jenkins doesn't even poll github to see if the repo has changed. A better solution would be to use commit hooks to notify the Jenkins API that a new revision is available for testing.
test-kitchen instances are currently tested sequentially, but using docker lowers the overhead enough that a few could be run in parallel.
Nothing happens after the tests pass. I plan to add another step that uses Berkshelf to publish the cookbook to a Chef server. Tagging the git repository might also be a nice touch.
Instead of running all the tests in a single job, a more sophisticated pipeline would first run ruby/erb syntax checks, then foodcritic and chefspec in parallel, then test-kitchen if both of those passed, before finally uploading the cookbook using Berkshelf.
When test-kitchen fails, you'll need to look at the console output to see why - it can't currently export test results the way that chefspec can. Keep an eye on this pull request
For my use case (running identical tests for internal cookbooks), I'm not convinced about maintaing a Gemfile+Rakefile in each cookbook. This feels like a violation of the DRY principle.

Other References

Although I didn't really find anything offering a detailed, end-to-end walk through setting up Jenkins for cookbook testing, I did find some clues in these posts:

http://blog.bscott.me/2012/12/08/Chef-Testing-Part-2.html
http://blog.safariflow.com/2013/06/26/continuous-integration-for-chef-with-vagrant-jenkins-and-gitlab/
http://jtimberman.housepub.org/blog/2013/05/08/test-kitchen-and-jenkins

Vagrant, VMware, and Berkshelf

2013-12-27T16:00:00+00:00

So, Vagrant is a great tool, which costs you nothing to use if you pair it with Virtualbox. Unfortunately, Virtualbox is somewhat less awesome in heavy use. VMware does a better job of things, and Vagrant added support for it with a paid-for plugin. As the first commercial product of Hashicorp (Vagrant's developer), this is a nice way to say thankyou for Vagrant while getting a better user experience too.

In theory.

While Vagrant generally does a great job of hiding the differences between the various "providers" (virtualisation systems) it supports, some still leak through. I ran into the following problem when running "vagrant provision" to re-run Chef in the guest system after making some changes:

[default] Running provisioner: chef_solo...
Shared folders that Chef requires are missing on the virtual machine.
This is usually due to configuration changing after already booting the
machine. The fix is to run a `vagrant reload` so that the proper shared
folders will be prepared and mounted on the VM.

Contrary to the message, vagrant reload didn't help.¹

When I looked at the guest system, I found that the directory that should have contained my Chef cookbooks (managed by Berkshelf) no longer appeared - the mountpoint was still listed, but it no longer worked. Eventually, that lead me to issue #88 on the vagrant-berkshelf repo, which explains the cause - that directory is deleted and recreated on every run, which breaks the "vmware" and "nfs" methods of sharing folders under Vagrant.

In lieu of a proper fix, I bodged one together that relies on rsync being available on your path. If you're having the same problem and want to give it a try, check out this branch on github.

It turns out that this would have worked on Vagrant releases prior to 1.3.0. After that, vagrant reload --provision would have worked. In both cases, this workaround is very slow as it restarts the VM. ↩

Dick Jokes and DevOps, Redux

2013-12-02T11:25:00+00:00

Last week, I wrote "Dick Jokes and DevOps" after someone was called out for making a dick joke on a devops mailing list. Although the reaction was largely positive, some valid criticisms highlighted flaws in the piece that I hope to correct in revisiting the topic.

The first suggestion that I may have done something wrong was a comment on Twitter:

I really think THAT is sexist. "we must protect the delicate womens!"

— @amyhoy, https://twitter.com/amyhoy/status/404270049332322304

She wasn't alone. Comments from others (male and female) complained that "crude" was being equated with "sexist", and that it shouldn't be assumed that all dick jokes were offensive to all women.

I felt a bit bad about this, as it wasn't where I was coming from when I wrote the post. Nonetheless, you see what I wrote, not what I was thinking when I did it, so I figured the problem must be somewhere on the page. Sure enough, when I re-read it with that interpretation in mind, I saw that I did draw the issue along gender lines - fairly explicitly at the top of the post, still implicitly towards the end. Mea culpa.

Another criticism concerned my choice not to link to the original mailing list thread. While that choice was motivated by a desire to avoid reviving a thread that had run its course, the comments it provoked were interesting. People were unhappy that they couldn't look at the joke and make their own judgement about whether or not it was offensive.

Maybe it's a stretch, but I see a certain symmetry in these two criticisms. The first group was concerned that I was judging what they would be offended by, while the second group was concerned that I was denying them the power to judge what other people should be offended by. I'll blame my framing for both these concerns, and hope that this restatement brings a few more people on-side.

It's Not A Gender Thing

I don't think that all women want or need to be protected from seeing dick jokes. Beyond fundamental biological needs, like eating and breathing, I don't think there's much that is true of all women, or all men, or all people.

I know women who are comfortable hearing and making obscene jokes anywhere, any time. I know men who feel the same way.

I know men who are uncomfortable hearing crude jokes anywhere, any time. I know women who feel the same way.

I know people of all stripes who are comfortable discussing some things in a close group of friends, but uncomfortable having the exact same discussions in other contexts.

To the best of my knowledge, none of these positions has any bearing upon one's ability to operate a computer.

It's Not About Protecting Another Group

Everyone gets to own their own offence - it's condescending and disempowering to behave as though they need you to step in on their behalf. Beyond that, if they weren't actually offended in the first place you will have created a problem where none existed. Don't be offended for somebody else.

I opened the original post by describing the dick joke as "pretty benign". Doesn't that mean I was being offended on behalf of some unidentified others?

No.

Although I wasn't offended by the content of the joke, I was offended by the context - making the joke on a professional mailing list of nearly 800 people who have come together "to discuss technology, process and enabling harmonious DevOps."¹

A Gender-Neutral Frame

Here's two gender-neutral reasons why dick jokes have no place in a devops discussion:

1) They're off-topic

Dick jokes are not intrinsic to building or maintaining software, infrastructures, or teams. This is not a value judgement, and it says nothing about whether dick jokes are good, bad, or otherwise.

2) They alienate part of the audience

Some people on the list don't appreciate dick jokes. There are a range of reasons for this, none of which should matter to you. They joined the list to discuss devops - you did, too. The barrier to participation should be no higher than a desire to talk about that topic, and enabling the discussion is the most important thing we can do.

That's it.

Note that people who love dick jokes are just as welcome in the discussion as people who loathe them, and people without a strong opinion either way. When you leave the dick jokes out of the conversation, nobody needs to know or care where you stand.

Tell Me I'm Wrong

Telling dick jokes doesn't make you sexist, but insisting on your right to tell dick jokes when you know they're alienating your audience is the act of an asshole. Don't act like an asshole. You're better than that.

I struggle to see why this issue is contentious, and yet it seems to be. If you think I'm wrong, I will appreciate your good-faith effort to explain why.

https://groups.google.com/forum/#!aboutgroup/devops ↩

Premature Publication, and Other Problems

2013-11-28T10:45:00+00:00

It happened some time during primary school, I'm not sure exactly when. I was old enough to be given homework, but not so old that failing to do it had real consequences. I was already a bit of a goody two-shoes, and getting good marks was pretty important to me.

So, I'd been given some homework, and I got into trouble for not doing it. The truth was that I had done it, but I didn't think I'd done it very well - so I lied, and said I hadn't done it at all. I figured it was better to look like I hadn't even tried, than to have given it a good shot and come up short.

I shared that anecdote with a few people over drinks at DevOpsDays, the outward expression of a train of thought concerning my failure to blog. It even occurred to me as a neat framing device for a post reflecting on that failure, considered as part of an ongoing struggle with perfectionism. Neat, but disingenuous.

Over the years, I've produced unexceptional work for various reasons - sometimes I've been focussed on some other subject, other times I just haven't been particularly interested in the task at hand. When neither of those has played a part, procrastination has often been to blame. When facing a deadline, pedestrian effort in hand, you can choose between turning in something that may reflect badly on you, or turning in nothing (that definitely will). Both perfectionism and procrastination are counterproductive traits, and their battle for my mind has brought about a grudging acceptance of adequacy.

The real reason I fail to publish is not that I'm striving for excellence, but simply that I fear embarrassing myself. Without deadlines, or anyone calling me to account, the easiest way to avoid embarrassing myself through my writing is not to do any. Sometimes, I'll write 90% of an article before deciding the whole idea was stupid, other times it's no more than an outline. I've thrown out some drafts for failing to address every possible objection, and others for saying too much.

So, I decided that I needed to tackle this by making a conscious effort to publish things that I thought might be a little flawed - a vague sense that I might do better wouldn't be reason enough to hold back. That seemed like a fair compromise. I'll generally have mulled an idea over a little before I start writing, and I've never been able to resist editing and reworking as I go. After getting the whole idea out, and re-reading it a couple of times, I'll generally know whether I'm uncomfortable with the way I've presented, or with the idea of sharing it. Release early, release often! It's easy to write a follow-up if you need to.

You might recognise this as a false dichotomy. While "publish prematurely" is an improvement over "publish never", they are not the only strategies I could have adopted.

I'm not particularly ashamed of my last post, but it has some serious flaws. The conclusion is fine (and is the idea I wanted to share), but the build up to it was a bit sloppy - indeed, it could have supported a different argument entirely. This invited some readers to miss the point I was trying to make - and that's my fault, not theirs. The first few comments I read were enough to show me what I'd done wrong, and I started outlining a follow-up. Around that time, Matt S Trout saw the original and immediately pointed out every issue I'd belatedly noticed. I'd have saved a lot of time (and a little personal embarrassment) if I'd asked for some other people to review it in the first place.

So, lesson learned. In future, I'll be soliciting feedback before publishing material that could be misunderstood, or risks preaching to the client. Although my honest motivation will be to avoid looking stupid in public, I expect this will also help me to communicate more effectively. Seems like a win-win to me.

Dick Jokes and DevOps

2013-11-23T15:00:00+00:00

Update: This post has flaws that are addressed in a follow-up article.

So, someone made a dick joke on a devops mailing list.¹

It was pretty benign, as such things go - an allusion to penis enlargement in the context of a discussion about managing spam. I'd have made the same joke in a group of friends, though perhaps not in front of my mother. Certainly not in front of a nun, or on a global mailing list largely populated by people who I do not know, and who do not know me.

Mark Imbriaco politely called this out with a reply that could be gold standard for such things:

I just wanted to point out that your dick joke, which I'm quite certain isn't intended to be malicious in any way, doesn't do anything whatsoever to help the gender diversity problem that we have in technology, and most especially in technical operations.

I've been guilty of making jokes like that in the past myself, so I understand that they don't come from a desire to exclude anyone from the group. That said, the only way we make things better is by being aware that they do result in making some people feel excluded. And the only way we improve the situation is by eliminating that kind of thing from our discourse. And by having our friends remind us when we miss the mark, as I'm quite certain I will in the future.

— Mark Imbriaco

Note that Mark explained why the comment was bad without accusing the author of being bad. Conscious malice isn't required to say something exclusionary - thoughtlessness is often enough.

Matt Trout's advice shares the same spirit, while being somewhat more direct:

"Tell them they sound like an asshole. Don't accuse them of being one, because your enemy never believes he's an evil man. Just tell them they sound like one."

— Matt S Trout, On Being Part Of The Solution

In a perfect world, the joke's author would have responded the way Anil Dash did when he was recently called out on Twitter. In our actual world, he instead replied thus:

"Yeah, I'm not really into subjecting myself to your morality."

Alas.

Discussion Ensues

When a predominantly male mailing list dives into this kind of discussion, its membership can appear to separate into three groups.

1) "Help! Help! I'm being repressed!"

This group doesn't see anything wrong with whatever was said, and therefore nobody else should either. "The world is a cruel and unforgiving place, and delicate flowers need to toughen up a little. Anyway, this is censorship, and censorship is evil."

2) "How can you think this is appropriate?"

This group might not have been personally offended by the comment, but they don't believe it has any place on the list. Maybe the wider world is cruel and unforgiving, but that doesn't mean we have to be. "Be the change you wish to see in the world."

3) "Not another one of these threads. Can't we just get back on topic?"

Some in this group think the comment shouldn't have been made, others think it was no big deal, and the rest don't particularly care either way. They might feel that the discussion is uncomfortable, or just plain tedious, but wholly off-topic either way. "You've all shared your opinions, now shut up and let's move on."

While the third group aren't actively causing the problem, they're certainly not part of the solution. They also offer a small illustration of privilege: if you can feel just as comfortable ignoring the problem as fixing it, you're probably part of the privileged group.

Despite actively contributing to the problem, folks in the first group often don't see any misogyny in their position. "I don't hate women. I'd say the same things to my female friends, and they wouldn't have a problem with it." Perhaps that's fair. I don't know whether dick jokes are inherently misogynistic, but I do know that this line of reasoning entirely misses the point.

The Point

The DevOps movement is inclusive. It recognises that we suffer in our own lives when we scorn the people whose professional choices differ from our own, and that we benefit from putting a little effort into getting along.

If you can build bridges with someone working in a different job, how can you insist that people with a different gender (or sexual preference, or skin colour, or religion) are the ones who need to change if we are going work together? Those personal attributes are far less relevant than the job they do. If they are impediments to someone's involvement in a professional group, the group has made it so.

"tl;dr - Don't do it, not because you are being censored, but because it makes part of your audience uncomfortable, and if your job is getting people on the same page, that's pretty much the worst thing you can do."

— Eric Shamow

Be mindful of your audience, try not to exclude anyone, and listen when someone suggests you've slipped up. If you see someone slip up, let them know that their comment falls short of the aspirations of the DevOps movement.

It isn't asking much, and we all benefit from the inclusive community that results.

I haven't linked to it, because I don't think giving it more publicity is going to help anything. Am I wrong? Let me know. ↩

Errata for Test-Driven Infrastructure with Chef (2nd Ed)

2013-11-18T21:30:00+00:00

Update: I'm very happy to update this post to note that Test Kitchen 1.0 has been released. See the announcement on the Opscode blog, and check out the KitchenCI home page.

A couple of years ago, Stephen Nelson-Smith wrote a little book called Test-Driven Infrastructure with Chef. Somewhat ahead of its time, it gave a brief introduction to the idea of outside-in, test-driven development, in the context of infrastructure code. Since then, the rest of the world has started to catch up, and a variety of tools and community practices have grown up around the idea.

Last month, the greatly revised and expanded second edition of the book was released. I mean to write a full review when I've spent some more time with the book, but here's the short version: if you use Chef, buy it.

That having been said, nothing in life is perfect, and that's true here. The book optimistically assumes that Opscode (or whoever) would have managed to make a 1.0 release of Test-Kitchen by now. They haven't, so you'll currently have issues following some of the instructions when the recommended toolchain is introduced in Chapter 7.

Berkshelf and Test Kitchen Installation

The book assumes that Berkshelf (installed with gem install berkshelf) will also install the test-kitchen gem. As of Berkshelf 2.0.10, that's not the case - and if you don't install it yourself, Berkshelf won't generate the Gemfile you'll see next.

You can fix this by installing test-kitchen yourself, but be warned - if you just run gem install test-kitchen, you'll get a prehistoric version. You can tell gem to install the latest pre-release version by running gem install test-kitchen --pre. This won't be required after the final 1.0 release is made.

Berkshelf - Gemfile

The book looks at the Gemfile created by berks init, which adds dependencies on test-kitchen and kitchen-vagrant. If you follow the output's instructions to run bundle install, you'll get a failure:

    Bundler could not find compatible versions for gem "test-kitchen":
      In Gemfile:
        kitchen-vagrant (>= 0) ruby depends on
          test-kitchen (~> 1.0.0.alpha.0) ruby

        test-kitchen (0.5.0)

Again, this won't be a problem after test-kitchen gets a a 1.0 release. Today, you'll need to manually update the Gemfile to add a version constraint like so:

source 'https://rubygems.org'

gem 'berkshelf'

# gem 'test-kitchen', :group => :integration
gem 'test-kitchen', '~> 1.0.0.beta.4', :group => :integration
gem 'kitchen-vagrant', :group => :integration

With that change, bundle install will run to completion.

Parting Thought

It's a fact of life that books about software tools go out of date as those tools are improved, so it's only a mild irritation that this book looks forward to a release that has yet to arrive.

Test Kitchen was discussed at Opscode's "Chef Summit" last week, and the notes of that session suggest that a release is now waiting for documentation. That's a well-intentioned reason, but a lousy excuse. A 1.0 release with documentation would be vastly superior to a 1.0 release without, but the current situation impedes the community's ability to move forward and help each other out. This is particularly frustrating when I keep hearing that the software itself is awesome, stable, and generally ready for use.

I look forward to updating this post to note that it should be ignored in its entirety.

DevOpsDays London - November 2013

2013-11-16T21:30:00+00:00

Last week saw the second London DevOpsDays conference for 2013. BMC sponsored video of the plenary sessions, all of which are now on Vimeo. I think the videos look better than the sessions did in person (the lighting in the venue wasn't great). Read on for my thoughts...

Three Recommended Videos

Mark Burgess introduced his talk, "What Science Tells Us About Information Infrastructure", as attempting to condense the essence of his new book into a brief lecture. The start is a little rough owing to technical hiccoughs, but it's worth sticking with it. While practical development is often more art than science, operations still struggles to grow beyond finger-painting. On the one hand, it's great that mugs like me can produce useful operationally-focussed work without learning the lessons of the development profession; on the other, it's depressing that this is par for the course.

Mark stands out from the crowd by bringing a reasoned, academic approach to his work in operations over the last two decades. "In Search of Certainty" is cheap enough that you should probably just buy it, but watch the video if you'd like a taster.

Ben Hughes gave us "That's Mr. The Plague to you! Security and this Devops Thing". A fun talk, peppered with geek cultural references, this session took a look at the current reality of advanced security threats, and showed how Etsy's security folks work to manage risk. If you believe security means saying "no" to everything, watch this to hear about a cooperative, pragmatic approach.

Jeffrey Fredrick described the title for his talk as "possibly the worst he'd ever come up with". "Crossing the Uncanny Valley of Culture Through Mutual Learning" is a bit of a mouthful, but this was my favourite session of the conference. As the title suggests, it isn't a technical talk - instead, it introduces a variety of cultural and psychological ideas that have yet to really make it into the DevOps echo chamber. If you've appreciated John Allspaw's writing about human error investigation, this one's for you.

However, the breadth of ideas (and limited time) precluded depth in some areas. In particular, the section on Chris Argyris' work deserves (at least) a session of its own. I hope to see more of that at a future DevOpsDays. Until then, I'd recommend reading Benjamin Mitchell's post about "The Ladder of Inference", and William Noonan's book, "Discussing the Undiscussable".

Not So Awesome

There was an Ignite session likening Game of Thrones to IT stereotypes, or something. I haven't read or watched GoT so this was never going to grab me, but I was disappointed that one of the final slides was a picture of an comely woman from the show that the speaker openly described as "gratuitous". While this was on screen, he explained that he had delivered the deck without that picture at Velocity NYC, and the feedback was that it needed a picture of her.

Honestly, what the fuck?

How much attention does this kind of shit need to get before people cut it out? Even if you personally can't see anything wrong with it, haven't the reactions to similarly gratuitous conference presentations over the last couple of years done enough to make you realise that maybe you should reassess your opinions?

Shame on whoever suggested the addition of a gratuitous slide, and shame on the speaker for listening to them. And shame on me for failing to call it out at the time.

Open Spaces

Both afternoons at DevOpsDays are open spaces, where the attendees propose topics of discussion, gather together, follow things where they lead. I have to confess that I can't review this part of the conference, as I eschewed it in favour of off-site sessions (cough) at the Euston Tap.

I can report that the Euston Tap offered a broad selection of delicious beers, and provided a suitable forum for extended conversations with friends both old and new. I had a great time, but feel more than a little guilty for wasting the opportunity afforded by the conference.

If at any time during our time together you find yourself in any situation where you are neither learning nor contributing, use your two feet, go someplace else.

— Harrison Owen, The Law of Two Feet

I did go to the open spaces at DevOpsDays in March, and was left a little underwhelmed. This isn't a knock against open spaces in general; Scale Camp embraced the format, and blew my mind. I'm not entirely sure why the DevOpsDays didn't work for me in March (though I have a few ideas), but the Law of Two Feet didn't really help me. I might argue that I simply applied it more thoroughly this time around, but that's a bit of a cop-out.

Having spoken to some other attendees, I suspect I would have been disappointed again this time around - but I can't know that, and I regret not going to find out. In the worst case, I might have developed a better understanding of what wasn't working for me, in service of more constructive suggestions.

On the flip side, other folks I spoke to clearly got value out of the open space sessions, and I hope that most people fell into that camp.

Final Thoughts

Next time, I will aim to be more involved with the conference. If my opinions represent a wider group, then constructive contribution serves a greater good. If I'm alone in my opinions, perhaps greater engagement will lead me to some kind of enlightenment.

In any case, DevOpsDays remains worthy of a recommendation. "DevOps" still means a lot of different things to a lot of different people, and few conferences bring so many of them together and give them the opportunity to work it out.

(Dis)Proof of Concept: MCollective SQS/SNS Connector

2013-11-16T03:30:00+00:00

Yesterday, someone joined the #mcollective IRC channel to ask how to connect MCollective to Amazon's Simple Queue Service. I explored that idea earlier this year, and decided not to pursue it, but it seems I didn't get around to sharing the results of the experiment. Until now.

Context

I was setting up a small infrastructure in EC2, but I wanted to have MCollective available. I couldn't justify the cost of the extra capacity required to run ActiveMQ - I didn't want the extra hassle, either. On the face of it, SQS seemed like a good place to start my investigation.

Amazon Simple Queue Service (SQS) is a fast, reliable, scalable, fully managed message queuing service. SQS makes it simple and cost-effective to decouple the components of a cloud application.

— Amazon, SQS Overview

As the name implies, SQS just provides queues - to have something analogous to pub-sub topics, we're pointed towards the Simple Notification Service.

When combined with Amazon Simple Notification Service (SNS), developers can 'fanout' identical messages to multiple SQS queues in parallel. When developers want to process the messages in multiple passes, fanout helps complete this more quickly, and with fewer delays due to bottlenecks at any one stage. Fanout also makes it easier to record duplicate copies of your messages, for example in different databases.

— Amazon, SQS Overview

Fast, reliable, simple - what's not to like?

Experiment

Though I maintain that I'm not much of a programmer, I thought I could take a shot at writing a suitable MCollective connector plugin. There were enough examples of working connectors for me to refer to, and it only had to be good enough to prove a point - I figured I could get someone to help tidy it up if it worked.

You can find the code on my Github, but don't expect to use it. It's state is somewhere between "doesn't work very well" and "doesn't work at all". To be fair, most of that is down to my code - but I came to the conclusion that I would not be able to make it work well enough that I'd want to use it.

Unhelpful characteristics of SQS:

messages may not be delivered in FIFO order
messages may be delivered more than once
deleted messages may be redelivered under some circumstances
the standard polling behaviour might not return all available message - and if there's fewer than 1000 messages in the queue, it may return nothing at all!
"long polling" guarantees that if messages are in the queue, at least 1 will be returned - but the definition of this behaviour is pretty vague.
"long polling" can wait a maximum of 20 seconds for a message.
SQS is charged per-request - polling (and receiving nothing) is chargeable.

I also ran into trouble automating subscriptions of SQS queues to SNS topics - while it worked, it was possible to end up with duplicate subscriptions that eventually resulted in silent failures.

None of the issues is a show-stopper as long as you can accept mcollective latency being measured in seconds instead of milliseconds.

Future Possibilities?

I'm not planning to spend any more time on this. If I was, I'd look at the following:

Add a thread to handle long polling messages from SQS into a buffer
Dedupe received messages to ensure each is delivered to mco only once
Potentially fiddle order of received messages (whether or not this really matters will depend on your use)
Carefully groom SNS topics and subscriptions

Alternatives

I found a project on Github called ec2_collective, which aims to build something like MCollective on top of SQS. It claims arbitrary command execution as a feature, and I suspect it underestimates usage charges, but it's worth a look.

As for me, I'm looking at MCollective's redis connector. That involves its own tradeoffs, which I'll save for another post.

Summary

You can certainly build solid, scalable applications around SNS and SQS - but they're not a straight replacement for a traditional STOMP or AMQP message broker. Fire-and-forget and loosely-coupled asynchronous messaging patterns should work out alright - orchestration, not so much.

MCollective Chef cookbook updated to 0.11.0

2013-04-21T23:00:00+01:00

The recent release of an x509 security provider for MCollective has motivated me to do some more work on the mcollective cookbook. Although it worked well enough to play around with, the configuration was not especially flexible and the cookbook did not lend itself to wrapping.

MCollective 1.2 was the current release when I originally wrote the cookbook, and the configuration reflected that. New features introduced in later releases of MCollective weren't enabled - it worked, but not at its best. The configuration defaults now reflect MCollective 2.2.

Configuration is almost entirely parameterised. If you have need to override the cookbook's templates to implement your configuration, I'd like to improve this further.

The MCollective "identity" is now configurable via an attribute, but continues to default to node['fqdn']. When your node name and your identity match, you can use the chef-server discovery plugin - so this default may change in the next major release.

Version 0.11.0 of the cookbook is now available on the community site and on Github. If you find it doesn't do what you need, or you have suggestions to improve it, please open issues on github.

(Very) Simple x509 PKI with Chef

2013-04-20T21:00:00+01:00

Last year, Venda released a project to create and manage a simple x509 PKI using Chef and Chris Andrews introduced it with his blog post, "Deploying a PKI With Chef".

A few people tried out it out after the initial release (and submitted patches or bug reports - thankyou!), and it has since been renamed to become the x509 cookbook, which you can find on the community site or on github.

I've found it useful of late, so let's take another look.

What's The Problem?

You've decided to SSL-enable one of your internal services, and that means you need an x509 certificate. The cheapest and easiest option is to generate a self-signed certificate, but this option is not without drawbacks.

When you connect to a service using a self-signed certificate, you can be confident that your communication is encrypted, but you can't be sure who you're communicating with. You are protected from attackers "sniffing" data from an insecure network, but not from attackers creating a fake service in front of the one you expect to connect to (a man-in-the-middle attack).

It's also annoying to users, as most software will (rightly!) warn you that self-signed certificates are not to be trusted.

A better option is to run an internal Certificate Authority, and use that to sign the certificates for your SSL-enabled services. You can import your CA's certificate into your browser (or OS), which will then trust services using certificates that it has signed.

It's not hard to make your own CA, but getting a signed certificate for your service necessarily involves a number of steps:

On the host, generate a secret key and a certificate signing request (CSR)
Get the CSR to the internal CA
Create a signed certificate using the CSR and the internal CA
Get the signed certificate to the host
Install the signed certificate

Venda wanted to automate this process and the result is the x509 cookbook, and the chef-ssl-client gem.

1. Installation

Note: these tools work with chef-client and the Chef server - chef-solo is not supported.

Certificate Authority

To create and manage a CA, we will be using the chef-ssl utility. It loads your knife configuration to determine how to connect to the Chef server, so you'll have to install it somewhere you can already use knife.

The chef-ssl utility is provided by the chef-ssl-client gem.

$ gem install chef-ssl-client --no-rdoc --no-ri
Fetching: chef-ssl-client-1.0.5.gem (100%)
Successfully installed chef-ssl-client-1.0.5

$ chef-ssl --help
  chef-ssl

  Chef-automated SSL certificate signing tool

  Commands:
    autosign             Search for CSRs and sign them with the given CA
    help                 Display global or [command] help documentation.
    issue                Issue an ad hoc certificate
    makeca               Creates a new CA
    search               Searches for outstanding CSRs
    sign                 Search for the given CSR by name and provide a signed certificate

  Global Options:
    -h, --help           Display help documentation
    -v, --version        Display version information
    -t, --trace          Display backtrace when an error occurs

Chef

To create certificates, we'll need the x509 cookbook on our Chef server. Some of this cookbook's functionality relies on the vt-gpg cookbook, which is listed as a dependency. We won't be using that functionality today, but Chef will insist that we upload both.

Both cookbooks are available on the community site, so downloading and installing them using Knife might look like this:

$ knife cookbook site install x509
Installing x509 to /Users/zts/code/chef-repo/cookbooks
Checking out the master branch.
Creating pristine copy branch chef-vendor-x509
...
Cookbook x509 version 1.0.3 successfully installed
Installing vt-gpg to /Users/zts/code/chef-repo/cookbooks
Checking out the master branch.
Creating pristine copy branch chef-vendor-vt-gpg
...
Cookbook vt-gpg version 1.0.0 successfully installed
$ knife cookbook upload x509 vt-gpg
Uploading vt-gpg         [1.0.0]
Uploading x509           [1.0.3]
Uploaded 2 cookbooks.

2. Usage

Creating a Certificate Authority

To sign certificates, we're going to need a CA. This is accomplished using the chef-ssl makeca command on the host where you installed chef-ssl-client. The required arguments are --dn (the Distinguished Name of the CA), and --ca-path, the place where the CA's files will be stored. The DN will be visible in the CA certificate, and any certificates it signs.

 $ chef-ssl makeca --dn '/CN=My Test CA' --ca-path ./testCA
New CA DN: /CN=My Test CA
Enter new CA passphrase:
Re-enter new CA passphrase:

Creating new CA: done

Defining a certificate

To create a certificate on one of our Chef nodes, we use the x509_certificate LWRP provided by the x509 cookbook. This LWRP has some dependencies, which are installed by the default recipe - but this means that recipe[x509::default] needs to be applied. It doesn't matter whether you put it on the run_list or load it using include_recipe, but listing it as a dependency in another cookbook's metadata.rb is not enough.

In this example, we're creating a server certificate for "www.example.com", and indicating that we'd like it to be signed by "testCA".

include_recipe "x509::default"

x509_certificate "www.example.com" do
  certificate  "/etc/ssl/www.example.com.cert"
  key          "/etc/ssl/www.example.com.key"
  ca           "testCA"
  type         "server"
  bits         2048
  days         365
end

The first time we run chef-client with this recipe, we'll see logs similar to this:

[zts@test-centos6]~% sudo chef-client --force-logger
[2013-04-20T19:14:35+00:00] INFO: *** Chef 11.4.0 ***
[2013-04-20T19:14:36+00:00] INFO: Run List is [recipe[x509::example]]
[2013-04-20T19:14:36+00:00] INFO: Run List expands to [x509::example]
[2013-04-20T19:14:36+00:00] INFO: Starting Chef Run for test-centos6.nat0.cryptocracy.com
[2013-04-20T19:14:36+00:00] INFO: Running start handlers
[2013-04-20T19:14:36+00:00] INFO: Start handlers complete.
[2013-04-20T19:14:36+00:00] INFO: Loading cookbooks [vt-gpg, x509]
[2013-04-20T19:14:36+00:00] WARN: X509 library dependency 'eassl' not loaded: cannot load such file -- eassl
[2013-04-20T19:14:36+00:00] INFO: Processing chef_gem[eassl2] action nothing (x509::default line 25)
[2013-04-20T19:14:36+00:00] INFO: Processing chef_gem[eassl2] action upgrade (x509::default line 25)
[2013-04-20T19:16:21+00:00] INFO: chef_gem[eassl2] upgraded from uninstalled to 2.0.1
[2013-04-20T19:16:21+00:00] INFO: Processing chef_gem[eassl2] action nothing (x509::default line 25)
[2013-04-20T19:16:21+00:00] INFO: Processing x509_certificate[www.example.com] action create (x509::example line 3)
[2013-04-20T19:16:21+00:00] INFO: Processing file[/etc/ssl/www.example.com.cert] action create (/var/cache/chef/cookbooks/x509/providers/certificate.rb line 5)
[2013-04-20T19:16:21+00:00] INFO: entered create
[2013-04-20T19:16:21+00:00] INFO: file[/etc/ssl/www.example.com.cert] owner changed to 0
[2013-04-20T19:16:21+00:00] INFO: file[/etc/ssl/www.example.com.cert] group changed to 0
[2013-04-20T19:16:21+00:00] INFO: file[/etc/ssl/www.example.com.cert] mode changed to 644
[2013-04-20T19:16:21+00:00] INFO: file[/etc/ssl/www.example.com.cert] created file /etc/ssl/www.example.com.cert
[2013-04-20T19:16:21+00:00] INFO: Processing file[/etc/ssl/www.example.com.key] action create (/var/cache/chef/cookbooks/x509/providers/certificate.rb line 11)
[2013-04-20T19:16:21+00:00] INFO: entered create
[2013-04-20T19:16:21+00:00] INFO: file[/etc/ssl/www.example.com.key] owner changed to 0
[2013-04-20T19:16:21+00:00] INFO: file[/etc/ssl/www.example.com.key] group changed to 0
[2013-04-20T19:16:21+00:00] INFO: file[/etc/ssl/www.example.com.key] mode changed to 600
[2013-04-20T19:16:21+00:00] INFO: file[/etc/ssl/www.example.com.key] created file /etc/ssl/www.example.com.key
[2013-04-20T19:16:22+00:00] INFO: Chef Run complete in 106.180117863 seconds
[2013-04-20T19:16:22+00:00] INFO: Running report handlers
[2013-04-20T19:16:22+00:00] INFO: Report handlers complete

Three things have happened during this run.

The eassl gem is installed, required by the x509_certificate LWRP.
A self-signed certificate is generated and installed.
A Certificate Signing Request (CSR) is created and saved in the csr_outbox attribute on the node.

We can check the last action using knife node show:

 $ knife node show test-centos6.nat0.cryptocracy.com -a csr_outbox
test-centos6.nat0.cryptocracy.com:
  csr_outbox:
    www.example.com:
      ca:   testCA
      csr:  -----BEGIN CERTIFICATE REQUEST-----
      MIIC7DCCAdQCAQAwgaYxCzAJBgNVBAYTAkdCMQ8wDQYDVQQIEwZMb25kb24xDzAN
      BgNVBAcTBkxvbmRvbjEUMBIGA1UEChMLRXhhbXBsZSBMdGQxHzAdBgNVBAsTFkNl
      cnRpZmljYXRlIEF1dG9tYXRpb24xGDAWBgNVBAMTD3d3dy5leGFtcGxlLmNvbTEk
      MCIGCSqGSIb3DQEJARMVeDUwOS1hdXRvQGV4YW1wbGUuY29tMIIBIjANBgkqhkiG
      9w0BAQEFAAOCAQ8AMIIBCgKCAQEAv6N6WZIiwwuOcFIni9b7lEcbd/Djpe00TZ7B
      mB8dJa3pFZM3Le5MsUtswwIMe9GpFyDNBImYb0ebTsOwGykqWg4qjG0/tNu0DARt
      9JX3SKeC31h9ShUt2XOL5ZpGqMlDWQY2qdKZnA0ge+vmV1UW2fyb1e0Wwyi+CgUK
      fZd6A3JWMd1TykVHZQJwt3xuTYiu/6wZbOj9OLlNqOFjEtnIstXgKTBhI3qs3Zuj
      lQU6FEWdqghzwaPSXG1/tNQn6Ql1wKsnbflONlzQbjLdaItDRnYzj5KlYroXChjH
      tE4Y4f6KC/BhQlENfytSgGOC9W+KWtaAfIIg/E+YLHFa8QmqLwIDAQABoAAwDQYJ
      KoZIhvcNAQEFBQADggEBAIW3og6ARIJFs4Ipj7AEebN0E8YYMsgBhqtAlg7DkLZt
      myQeCpL8Gn04tkCYYkvh+BoaE0PZ7wvFRkdphQ6Fvxji4CaDoeItSppM+C79nqo/
      gfi1xmufb1d0RKwDua8nLN428SddWeTMGw65X7GfXkl9JQcEpmA4ecKmVMDUUgNo
      dNRRJpSKT+a+OPT+9PFXY9ApSKENOSkJn/XtF8Dnx1xI5WBL7eYzRyM1es44RcuN
      OtKdA1u/qtFAX1gRrvTvlqmVbevC1kI5Wx7tAmvNCkQ2RrhI9vdwFS3pGpiNpKRx
      eENUk/N/rhOPXO9/v0JHZX455UZh5EqINoZ7eGrY4hE=
      -----END CERTIFICATE REQUEST-----

      date: 2013-04-20 19:16:21 +0000
      days: 365
      id:   80fc0fb9266db7b83f85850fa0e6548b6d70ee68c8b5b412f1deea6ebdef0404
      key:
      type: server

Signing a CSR (chef-ssl)

After our node has created a signing request, we return to the chef-ssl utility to process it. The chef-ssl autosign command finds pending CSRs for a given CA by matching the --ca-name argument against the ca parameter used in the x509_certificate resource. You'll be asked whether you'd like to sign each request it finds.

 $ chef-ssl autosign --ca-name="testCA" --ca-path=./testCA
Enter CA passphrase:
Search CA: testCA
     Node Hostname: test-centos6.nat0.cryptocracy.com
  Certificate Type: server
    Certificate DN: /C=GB/ST=London/L=London/O=Example Ltd/OU=Certificate Automation/CN=www.example.com/emailAddress=x509-auto@example.com
      Requested CA: testCA
Requested Validity: 365 days
-----BEGIN CERTIFICATE REQUEST-----
MIIC7DCCAdQCAQAwgaYxCzAJBgNVBAYTAkdCMQ8wDQYDVQQIEwZMb25kb24xDzAN
BgNVBAcTBkxvbmRvbjEUMBIGA1UEChMLRXhhbXBsZSBMdGQxHzAdBgNVBAsTFkNl
cnRpZmljYXRlIEF1dG9tYXRpb24xGDAWBgNVBAMTD3d3dy5leGFtcGxlLmNvbTEk
MCIGCSqGSIb3DQEJARMVeDUwOS1hdXRvQGV4YW1wbGUuY29tMIIBIjANBgkqhkiG
9w0BAQEFAAOCAQ8AMIIBCgKCAQEAv6N6WZIiwwuOcFIni9b7lEcbd/Djpe00TZ7B
mB8dJa3pFZM3Le5MsUtswwIMe9GpFyDNBImYb0ebTsOwGykqWg4qjG0/tNu0DARt
9JX3SKeC31h9ShUt2XOL5ZpGqMlDWQY2qdKZnA0ge+vmV1UW2fyb1e0Wwyi+CgUK
fZd6A3JWMd1TykVHZQJwt3xuTYiu/6wZbOj9OLlNqOFjEtnIstXgKTBhI3qs3Zuj
lQU6FEWdqghzwaPSXG1/tNQn6Ql1wKsnbflONlzQbjLdaItDRnYzj5KlYroXChjH
tE4Y4f6KC/BhQlENfytSgGOC9W+KWtaAfIIg/E+YLHFa8QmqLwIDAQABoAAwDQYJ
KoZIhvcNAQEFBQADggEBAIW3og6ARIJFs4Ipj7AEebN0E8YYMsgBhqtAlg7DkLZt
myQeCpL8Gn04tkCYYkvh+BoaE0PZ7wvFRkdphQ6Fvxji4CaDoeItSppM+C79nqo/
gfi1xmufb1d0RKwDua8nLN428SddWeTMGw65X7GfXkl9JQcEpmA4ecKmVMDUUgNo
dNRRJpSKT+a+OPT+9PFXY9ApSKENOSkJn/XtF8Dnx1xI5WBL7eYzRyM1es44RcuN
OtKdA1u/qtFAX1gRrvTvlqmVbevC1kI5Wx7tAmvNCkQ2RrhI9vdwFS3pGpiNpKRx
eENUk/N/rhOPXO9/v0JHZX455UZh5EqINoZ7eGrY4hE=
-----END CERTIFICATE REQUEST-----

Sign with: /CN=My Test CA
Sign this? (yes or no)
yes
Signed: SHA1 Fingerprint=67:39:44:79:31:74:50:E5:D0:54:4F:4E:6B:E5:E5:DB:2A:E8:7F:10
-----BEGIN CERTIFICATE-----
MIIDfjCCAuegAwIBAgIBBDANBgkqhkiG9w0BAQUFADAVMRMwEQYDVQQDDApNeSBU
ZXN0IENBMB4XDTEzMDQyMDE5MjAxMFoXDTE0MDQyMDE5MjAxMFowgaYxCzAJBgNV
BAYTAkdCMQ8wDQYDVQQIEwZMb25kb24xDzANBgNVBAcTBkxvbmRvbjEUMBIGA1UE
ChMLRXhhbXBsZSBMdGQxHzAdBgNVBAsTFkNlcnRpZmljYXRlIEF1dG9tYXRpb24x
GDAWBgNVBAMTD3d3dy5leGFtcGxlLmNvbTEkMCIGCSqGSIb3DQEJARMVeDUwOS1h
dXRvQGV4YW1wbGUuY29tMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA
v6N6WZIiwwuOcFIni9b7lEcbd/Djpe00TZ7BmB8dJa3pFZM3Le5MsUtswwIMe9Gp
FyDNBImYb0ebTsOwGykqWg4qjG0/tNu0DARt9JX3SKeC31h9ShUt2XOL5ZpGqMlD
WQY2qdKZnA0ge+vmV1UW2fyb1e0Wwyi+CgUKfZd6A3JWMd1TykVHZQJwt3xuTYiu
/6wZbOj9OLlNqOFjEtnIstXgKTBhI3qs3ZujlQU6FEWdqghzwaPSXG1/tNQn6Ql1
wKsnbflONlzQbjLdaItDRnYzj5KlYroXChjHtE4Y4f6KC/BhQlENfytSgGOC9W+K
WtaAfIIg/E+YLHFa8QmqLwIDAQABo4HHMIHEMAkGA1UdEwQCMAAwHQYDVR0OBBYE
FOuq7ztithiBI1Al05/OrAliIVV7MDcGCWCGSAGG+EIBDQQqFihSdWJ5L09wZW5T
U0wvRWFTU0wgR2VuZXJhdGVkIENlcnRpZmljYXRlMD0GA1UdIwQ2MDSAFA6/VF54
PUt7pkF5z9ZYUOOS7iDtoRmkFzAVMRMwEQYDVQQDDApNeSBUZXN0IENBggEAMAsG
A1UdDwQEAwIFoDATBgNVHSUEDDAKBggrBgEFBQcDATANBgkqhkiG9w0BAQUFAAOB
gQA73eaeESHnYeWjk8uOg5qq4VL+rLk+M/MB3rON4g6lUzBVBUzTEULU+ziGEnWw
1f8EO99GVPf1hAywcfycj2GskkPbg4JgBHFQ3EeQVYEBzQcKCLqboRaUsgXqpDlg
OGd5BpT33Jh2E8u/5G81WGhCScIVHpH1JwGyAkwvsnyokw==
-----END CERTIFICATE-----

Saved OK
All CSRs processed.

Signed certificates are saved to the "certificates" data bag on the Chef server.

 $ knife search certificates "host:test-centos6.nat0.cryptocracy.com" -a dn
1 items found

data_bag_item_certificates_80fc0fb9266db7b83f85850fa0e6548b6d70ee68c8b5b412f1deea6ebdef0404:
  dn: /C=GB/ST=London/L=London/O=Example Ltd/OU=Certificate Automation/CN=www.example.com/emailAddress=x509-auto@example.com

Installing the signed certificate

To update the node with our newly-issued certificate, we run chef-client once again.

[zts@test-centos6]~% sudo chef-client --force-logger
[2013-04-20T19:36:45+00:00] INFO: *** Chef 11.4.0 ***
[2013-04-20T19:36:45+00:00] INFO: Run List is [recipe[x509::example]]
[2013-04-20T19:36:45+00:00] INFO: Run List expands to [x509::example]
[2013-04-20T19:36:45+00:00] INFO: Starting Chef Run for test-centos6.nat0.cryptocracy.com
[2013-04-20T19:36:45+00:00] INFO: Running start handlers
[2013-04-20T19:36:45+00:00] INFO: Start handlers complete.
[2013-04-20T19:36:45+00:00] INFO: Loading cookbooks [vt-gpg, x509]
[2013-04-20T19:36:45+00:00] INFO: Processing chef_gem[eassl2] action nothing (x509::default line 25)
[2013-04-20T19:36:45+00:00] INFO: Processing chef_gem[eassl2] action upgrade (x509::default line 25)
[2013-04-20T19:36:45+00:00] INFO: Processing chef_gem[eassl2] action nothing (x509::default line 25)
[2013-04-20T19:36:45+00:00] INFO: Processing x509_certificate[www.example.com] action create (x509::example line 3)
[2013-04-20T19:36:45+00:00] INFO: installing certificate www.example.com (id 80fc0fb9266db7b83f85850fa0e6548b6d70ee68c8b5b412f1deea6ebdef0404)
[2013-04-20T19:36:45+00:00] INFO: Processing file[/etc/ssl/www.example.com.cert] action create (/var/cache/chef/cookbooks/x509/providers/certificate.rb line 5)
[2013-04-20T19:36:45+00:00] INFO: file[/etc/ssl/www.example.com.cert] backed up to /var/lib/chef/etc/ssl/www.example.com.cert.chef-20130420193645
[2013-04-20T19:36:45+00:00] INFO: file[/etc/ssl/www.example.com.cert] contents updated
v[2013-04-20T19:36:45+00:00] INFO: Processing file[/etc/ssl/www.example.com.key] action nothing (/var/cache/chef/cookbooks/x509/providers/certificate.rb line 11)
[2013-04-20T19:36:45+00:00] INFO: Chef Run complete in 0.351435563 seconds
[2013-04-20T19:36:45+00:00] INFO: Running report handlers
[2013-04-20T19:36:45+00:00] INFO: Report handlers complete

We can use openssl to confirm that the certicate was signed by the CA we requested:

[zts@test-centos6]~% openssl x509 -in /etc/ssl/www.example.com.cert -issuer | head -1
issuer= /CN=My Test CA

Conclusion

The x509 cookbook takes some of the pain out of generating SSL certificates, signing them with a private CA, and installing them on your nodes.

There's room for improvement, but I already find it useful. Bug reports, feature requests, and PRs will all be warmly received.

HOWTO perlbrew a 32-bit Perl on a 64-bit Centos

2012-09-16T22:00:00+01:00

I recently needed to run 32-bit Perl 5.8 on a 64-bit Centos 6 system. Initial research suggested that perlbrew would be the easiest way of achieving this, but I wasn't able to find a walkthrough.

Here's what worked for me...

1. Install perlbrew

You'll need to install perlbrew from the CPAN, and it has a load of dependencies. The wonderful App::cpanminus makes this experience as painless as possible, so I installed it before moving onto perlbrew itself.

$ sudo yum install perl-CPAN
$ sudo cpan App::cpanminus
$ sudo cpanm install App::perlbrew

1. Initialise perlbrew

Next, get perlbrew ready for use. Pay attention to the output of the init step - it will direct you to make a change to your shell configuration.

$ perlbrew init
$ perlbrew install-patchperl

2. Install 32-bit Libraries

Installing these two packages was enough to build a 32-bit perl core. If you're building additional XS modules against the 32-bit perl, they may require other 32-bit libraries to be installed.

$ sudo yum install glibc-devel.i686 libgcc.i686

3. Build A Perl

$ perlbrew install 5.8.9 -Accflags="-m32 -march=i686" -Aldflags="-m32 -march=i686" -Alddlflags="-shared -m32 -march=i686"
Fetching perl-5.8.9 as /home/zts/perl5/perlbrew/dists/perl-5.8.9.tar.bz2
Installing /home/zts/perl5/perlbrew/build/perl-5.8.9 into ~/perl5/perlbrew/perls/perl-5.8.9

This could take a while. You can run the following command on another shell to track the status:

  tail -f ~/perl5/perlbrew/build.perl-5.8.9.log

perl-5.8.9 is successfully installed.

That's all there is to it, though the result isn't quite perfect. While the above invocation builds a 32-bit perl, it doesn't override the system's archname - so the resulting @INC looks like this:

  @INC:
    /home/zts/perl5/perlbrew/perls/perl-5.8.9/lib/5.8.9/x86_64-linux
    /home/zts/perl5/perlbrew/perls/perl-5.8.9/lib/5.8.9
    /home/zts/perl5/perlbrew/perls/perl-5.8.9/lib/site_perl/5.8.9/x86_64-linux
    /home/zts/perl5/perlbrew/perls/perl-5.8.9/lib/site_perl/5.8.9
    .

For my purposes, this is simply an aesthetic issue - the x86_64-linux directories contain 32-bit shared objects - and I chose not to spend any more time perfecting it. If you happen to know which option(s) I'm missing, please leave a comment below.

Cookbook Reusability In Practice

2012-06-09T22:35:00+01:00

I recently attempted to use Chef to configure several VMs with software I wanted to play with. My goal only went as far as provisioning a minimally useful instance of each application - it didn't need to be production-ready, but it did need to start.

The applications were:

Jenkins, a CI tool written in Java,
Graphite, graphing tool written in Python, and,
Sensu, a monitoring tool written in Ruby.

Additionally, Sensu requires:

RabbitMQ, a message broker written in Erlang, and,
Redis, a key-value store written in C.

None of the dependencies are particular esoteric in a modern environment, though the variety seen here underscores the need for good configuration management. The guest OS was Ubuntu 12.04, as many community-contributed Chef cookbooks prefer or require Ubuntu and I wanted things to proceed as smoothly as possible.

Results

Jenkins

My experience installing Jenkins was not without issues, but it came closest to meeting my expectations. Adding recipe[jenkins] to the run_list will also pull in recipe[java::default], which will install openjdk unless configured otherwise using node attributes. I used the defaults.

The first chef-client run failed with the following unhelpful error in the Java cookbook:

[Sat, 09 Jun 2012 17:27:41 +0000] ERROR: TypeError: ruby_block[update-java-alternatives] (java::openjdk line 43) had an error: TypeError: can't convert nil into String

Without fixing or changing anything, I ran chef-client again - this second time, it completed without error, leaving Jenkins available on port 8080.

Graphite

For the next VM, I added recipe[graphite] to my run_list without overriding any attributes - the README suggested that an override was required on Ubuntu 11.10, but I decided to try my luck with the defaults on 12.04.

The graphite cookbook pulls in recipe[python::default], which is currently broken on Chef 0.10.10 and later. I had to patch my local copy of the cookbook (see this pull request for the fix) before re-running chef-client.

The second run completed without error, leaving Graphite running on port 80. But for the bug in the Python cookbook, this would have worked on the first go, without any manual tweaks.

Sensu

Bringing up a VM to try out Sensu was a yak shaving saga. The Sensu cookbook tries to make the process easy by shipping example roles, which show how to install and configure the tool in various ways. To simply install of the dependencies (Redis and RabbitMQ), recipes are provided which wrap their respective cookbooks, configuring the software for use by Sensu.

The cookbook's docs specify where to get each of its cookbook dependencies from, and even includes a Cheffile (for use with librarian-chef) that will take care of it for you. I downloaded the dependencies by hand, and got the wrong version of the Redis cookbook in the process.

That's how I learned that the Redis cookbook on the community site depends on metachef, which only works on Ruby 1.9 and is therefore off limits to most people running Chef on the Ruby shipped with their OS. That's pretty weak. (The redis cookbook referred to by the documentation does not depend on metachef)

After that, I was blocked installing RabbitMQ as their website was down, and stayed down for most of the four-day weekend the UK had for the Queen's Silver Jubilee.

Eventually, everything installed successfully and Sensu started, but didn't appear to work as intended. I'm still working on this, and have yet to determine whether it's an error on my part.

Credit Where Due

Despite the problems described above, I want to sincerely thank everyone who shared the code and cookbooks I'm using. It sounds ungrateful to complain when things don't quite work, given just how much did work. I'm bothered by this state of affairs, because I know I'm not immune to it - the cookbooks I share work for me, but I'd like them to work for everyone else. I doubt they do, and I have no way of knowing.

Re-use in the large is a hard problem, and one that we need to solve as a community. We have not yet done so. Until we do, individual cookbook authors can only do so much.

A possible future

The perl community does a lot of work to support code re-use via CPAN, the Comprehensive Perl Archive Network. CPAN is more than a package index - mirrors and testers are key features.

Like Chef, perl runs on many platforms, and many different releases can be found in the wild. When something goes wrong, it's valuable to know whether it's only going wrong for you, or whether a given module is known not to work on your class of system.

CPAN Testers collates results of tests for every module across a broad set of platforms and perl versions. Here are two examples for DBD::Pg:

As a user, I can quickly find out whether a module is likely to work on my system. As a module author, I get rapid feedback when I screw something up for a user on a platform I don't test on myself.

We have little hope of promoting widespread use of shared cookbooks until we can show that the work we share functions correctly across the set of platforms we intend to support - and continues to do so as both platforms and cookbooks evolve.

IPsec/L2TP VPN server with Ubuntu 12.04

2012-05-13T17:30:00+01:00

In an earlier post describing my PPTP VPN configuration, one reason I gave for my use of the (relatively) insecure protocol was that the IPsec alternative appeared to require building updated versions of the software. I'm delighted to report that that's no longer the case.

The versions of openswan and xl2tpd shipping with Ubuntu 12.04 (precise) are more recent than those mentioned in the elderly walkthroughs I'd found, and have worked for the OS X clients I'm using.

Simple Configuration

If you need instructions for creating a VPN using pre-shared keys, this post by Riobard Zhan is good despite overlooking the required firewall configuration.

IPsec traffic can be permitted using the ufw tool:

$ sudo ufw allow proto udp from any to any port 500
Rule added
$ sudo ufw allow proto udp from any to any port 4500
Rule added

L2TP is a little more tricky. We only want to allow L2TP traffic that has been secured by IPsec, which isn't a scenario that ufw(1) supports. The solution is to add a rule to the ufw-before-input chain by adding the following lines to /etc/ufw/before.rules.

# Allow L2TP only over IPSEC
-A ufw-before-input -m policy --dir in --pol ipsec -p udp --dport 1701 -j ACCEPT

Riobard's instructions disable ICMP redirects through the proc filesystem, and do this on boot from rc.local. I chose to use sysctl instead:

$ echo "net.ipv4.conf.all.accept_redirects = 0" | sudo tee -a /etc/sysctl.conf
$ echo "net.ipv4.conf.all.send_redirects = 0" | sudo tee -a /etc/sysctl.conf
$ sudo sysctl -p

Advanced Configuration

Setting Up an IPsec L2TP VPN Server on Ubuntu gives a very thorough walkthrough for those looking to support Windows clients with certificates for IPsec and user authentication against Active Directory. Each facet of the configuration is discussed separately, so it's useful even if (like me) you aren't looking to use AD.

OS X and Split DNS

It turns out that OS X configures routing for IPsec/L2TP VPNs in the same way as PPTP, so I'm still unable to change my DNS resolver configuration - but not my default route - when the VPN connection is established. I've done a little more investigation, but it looks as though this "split DNS" configuration is only available for Cisco-flavoured IPsec VPNs.

Fortunately, the OS X resolver can be configured on a per-domain basis. A near-enough solution changes my domain so that the VPN's nameserver is tried initially, before falling back to my local resolver after a shortened request timeout.

options timeout:1
nameserver 192.168.42.1
nameserver 192.168.1.1

When the VPN is disconnected, this configuration occasionally delays lookups for records in my public-facing zone. I could live with the inconvenience, but it's an "infrastructure smell" that I'd rather not have.

A better solution is to use a different domain for the hosts that will only be accessible when I'm on the VPN.

Bootstrapping Chef with ubuntu-vm-builder

2012-05-12T20:00:00+01:00

It's very easy to configure a machine as a KVM host these days. Modern releases of Ubuntu and Centos both offer simple installation methods that provide a useful system with minimal configuration.

Creating guests is a different story. A number of good tools are available, supporting a huge variety of installation scenarios. While it's reassuring to know you won't be limited by your tools, it can be a frustrating experience when you're just getting started.

This post shows how ubuntu-vm-builder can be used to go from zero to fully-installed Chef client in under 2 minutes.

Installing KVM

Configuring the server as a KVM host is pretty straightforward. Ubuntu's KVM documentation is well-written, and a good place to start.

$ sudo apt-get install qemu-kvm libvirt-bin ubuntu-vm-builder

By default, libvirt will create a host-only NAT network, with DHCP service, for guests to use. All my guests will be using a network I've created manually, so I've removed the default one.

$ virsh net-destroy default
Network default destroyed

$ virsh net-undefine default
Network default has been undefined

(Optional) Install apt-cacher-ng

Using apt-cacher-ng can speed up guest VM creation by keeping required packages on the host. This forum post provides a good walkthrough.

$ sudo apt-get install apt-cacher-ng
$ echo 'Acquire::http::Proxy "http://localhost:3142";' | sudo tee /etc/apt/apt.conf.d/01proxy
$ sudo ufw allow proto tcp from 192.168.42.0/24 to 192.168.42.1 port 3142

Configuring ubuntu-vm-builder

The easiest way of creating Ubuntu guests appears to be the ubuntu-vm-builder tool. The wiki documentation is decent, and the command-line help is comprehensive - see ubuntu-vm-builder kvm --help.

My ~/.vmbuilder.cfg sets the options that will be used for most (or all) of my Ubuntu guests.

[DEFAULT]
domain    = cryptocracy.com
gw        = 192.168.42.1
dns       = 192.168.42.1
firstboot = /home/zts/.vmbuilder/first-boot.sh
copy      = /home/zts/.vmbuilder/copy-files

[ubuntu]
proxy           = http://gateway.cryptocracy.com:3142
mirror          = http://mirror.hetzner.de/ubuntu/packages
security-mirror = http://mirror.hetzner.de/ubuntu/security
suite           = precise
components      = main, universe
addpkg          = openssh-server, acpid, chef

[kvm]
libvirt = qemu:///system
bridge  = nat0
mem     = 1024
cpus    = 2

The initial build process happens in a RAM disk, which means that package installation is much faster than inside the guest.

As my guests will all be running Chef, I want to install the required packages during the initial build. This requires enabling an additional package repository, which can be achieved by customising the APT sources template used by vmbuilder.

deb $mirror $suite #slurp
#echo ' '.join($components)

deb $updates_mirror $suite-updates #slurp
#echo ' '.join($components)

deb $security_mirror $suite-security #slurp
#echo ' '.join($components)

#if $ppa
#for $p in $ppa
deb http://ppa.launchpad.net/$p/ubuntu $suite main

#end for
#end if

# The Opscode repository, for Chef
deb http://apt.opscode.com/ precise-0.10 main

For the new VM to register as a Chef client, it needs my validation key. The copy option in .vmbuilder.cfg lists files on the host to install in the guest.

/home/zts/.chef/validation.pem /etc/chef/validation.pem

Finally, we need a script to run when the guest first boots.

# Fix locale
echo 'LANGUAGE="en_US.UTF-8"' >> /etc/default/locale
echo 'LC_ALL="en_US.UTF-8"' >> /etc/default/locale

# Configure guest to use the host's apt-cacher-ng
echo 'Acquire::http::Proxy "http://gateway.cryptocracy.com:3142";' > /etc/apt/apt.conf.d/01proxy

# Configure and start chef-client
echo "chef chef/chef_server_url string http://chef.cryptocracy.com:4000" | debconf-set-selections
dpkg-reconfigure -u chef

Creating a Guest

At this point, we're ready to create guests. The required steps are:

Create a volume for the guest's disk.
Run ubuntu-vm-builder, passing a hostname and IP address along with the path to the volume.
Start the guest.

$ sudo lvcreate -L 5G -n guest0 vg0
  Logical volume "guest0" created
$ sudo ubuntu-vm-builder kvm -o --tmpfs - \
         --raw /dev/mapper/vg0-guest0     \
         --hostname guest0                \
         --ip 192.168.42.3
2012-05-12 18:08:33,777 INFO    : Mounting tmpfs under /tmp/tmpLRgUKMtmpfs
2012-05-12 18:08:33,810 INFO    : Calling hook: preflight_check
2012-05-12 18:08:33,814 INFO    : Calling hook: set_defaults
2012-05-12 18:08:33,814 INFO    : Calling hook: bootstrap
2012-05-12 18:08:55,343 INFO    : Calling hook: configure_os
2012-05-12 18:09:00,374 INFO    : W: GPG error: http://apt.opscode.com precise-0.10 Release: The following signatures couldn't be verified because the p
ublic key is not available: NO_PUBKEY 2940ABA983EF826A
Extracting templates from packages: 100%
.
. output elided
.
2012-05-12 18:09:59,940 INFO    : Calling hook: post_install
2012-05-12 18:09:59,941 INFO    : Copying files specified by copy in: /home/zts/.vmbuilder/copy-files
2012-05-12 18:09:59,948 INFO    : Calling hook: unmount_partitions
2012-05-12 18:09:59,948 INFO    : Unmounting target filesystem
2012-05-12 18:10:03,713 INFO    : Calling hook: convert
2012-05-12 18:10:03,714 INFO    : Calling hook: fix_ownership
2012-05-12 18:10:03,715 INFO    : Calling hook: deploy
2012-05-12 18:10:04,140 INFO    : Unmounting tmpfs from /tmp/tmpLRgUKMtmpfs
$ virsh start guest0
Domain guest0 started

After waiting a moment for the guest to boot, we can check that it has registered itself with the Chef server:

$ knife node show guest0.cryptocracy.com
Node Name:   guest0.cryptocracy.com
Environment: _default
FQDN:        guest0.cryptocracy.com
IP:          192.168.42.3
Run List:
Roles:
Recipes:
Platform:    ubuntu 12.04

Voila!

Setting up a PPTP VPN

2012-05-07T23:30:00+01:00

First things first: PPTP, as a protocol, has a number of serious security issues. As a general rule, don't use it. I'm aware of the risks, and am accepting them for this implementation.

PPTP's main advantage is that it's very easy to configure. I briefly investigated the state of IPsec/L2TP, and found a lot of advice to build things from source because the distro packaged versions are too old. That's fine advice, but I wanted to get something started quickly. Once I've built a build box, I'll be revisiting this decision.

dnsmasq

Dnsmasq is a forwarding resolver that also answers queries based on the contents of /etc/hosts. This is a simple way of providing resolvable names for VM guests, and it seemed appropriate to use it for the VPN.

The default configuration is a good starting point.

# apt-get install dnsmasq
# ufw allow from 192.168.42.0/24 to 192.168.42.1 port 53

pptpd

The default configuration installed by pptpd only needs a couple of customisations, and the creation of a password file.

Install the package:

apt-get install pptpd

Update /etc/pptpd.conf to specify local and remote addresses.

localip 192.168.42.2
remoteip 192.168.42.250

The localip will be assigned to the ppp0 interface created by pptpd. The remoteip value can specify a range of addresses; as I'm only supporting a single client, I've only listed one.

(Optionally) update /etc/ppp/pptpd-options to return the address where dnsmasq is listening.

ms-dns 192.168.42.1

Use a long, random string for the password. Specifying the IP address is optional - an asterisk would have the same effect here.

# Secrets for authentication using CHAP
# client    server    secret      IP addresses
username    pptpd     password    192.168.42.250

Open the firewall and start the service:

ufw allow 1723
service pptpd start

I read several HOWTO's when setting this up, but none was quite what I needed. I imagine that others will have the same problem with this one, so I'll recommend the pptpd manpage.

OS X VPN client

At that point, everything was ready to connect the client. Not coincidentally, that's also the point where frustrations emerged.

The basic OS X configuration is simple - add a new VPN connection in Network Preferences, enter address and authentication details, and hit Connect. I did that, the connection came up, and I was able to ping other hosts on the NAT network (192.168.42.0/24).

However, the resolver configuration hadn't been updated with the nameserver specified by PPP. It appears that this is governed by the order of the connections in Network Preferences - the resolvers specified by the first active connection will be used.

After re-ordering the connections to place the VPN first, my client started resolving via the dnsmasq server. This was good.

Unfortunately, it now also changed my default route to point over the VPN. This was less good.

Beyond the fact that it wasn't necessary (and slowed down my web browsing), hitting Google via the VPN gave me the Russian language page.

The Advanced settings for the VPN connection have a checkbox, "Send all traffic over VPN connection". It turns out that OS X will create a default route to the connection whether or not this is set - it merely increases the routes priority. ie, if the VPN is first in the list of connections, it is already at the highest priority and the checkbox has no practical effect.

The Current, Grudging Solution

A post on SuperUser offered an AppleScript to connect a VPN, then adjust routing. My slightly tweaked version looks like this:

tell application "System Events"
    tell network preferences
        tell current location
            tell service "My VPN"
                connect
                tell current configuration
                    repeat until get connected = true
                        delay 1
                    end repeat
                end tell
            end tell
        end tell
    end tell
end tell

do shell script "route change default 192.168.1.1" with administrator privileges

Using this to bring up the connection, I end up with a resolver pointing to dnsmasq, and a default route via my home router.

It's not ideal. If I forget to use the script and instead connect the "normal" way, my routing is broken. If I do use the script, I'm confronted with a dialog box requesting an an administrator password.

Adding insult to injury, the dialog box doesn't have focus when displayed.

Next Steps

Giving up the use of dnsmasq for the VPN connection would neatly sidestep the issues above, and I'm tempted to do just that. Manually managing /etc/hosts wouldn't be too onerous in this case.

As noted above, I'm planning to implement IPsec/L2TP before long, and I'm holding out hope that OS X will deal with default routes differently for those connections. Not a lot of hope, but it's there.

A Dedicated Server

2012-05-06T22:30:00+01:00

We've been building out a new development environment at work - virtualised using KVM, and managed with Chef and MCollective. It has made it so easy to try out new things that I found myself wishing I had the same facilities for my own projects.

An article on Hacker News had me taking another look at Hetzner, which was followed a few days later by an order for a dedicated server with a quad-core i7, 32GB memory, and a pair of 3TB SATA disks. Hetzner gets mixed reviews, but the negatives weren't enough to put me off - the prices are good, and I'm not planning to host any critical services. I'll try not to complain when I get what I pay for.

The rest of this post describes the initial configuration of the server for use as a KVM host.

Initial (Re)Installation

Hetzner's default partitioning is a little questionable - with mirrored 3TB disks, I was given a system with a 1TB root and 1.7TB /home. Fortunately, this is easy to customise using installimage after booting into the Hetzner rescue image.

The following installimage configuration allocates most of the space to LVM, and creates a 10GB volume for the root filesystem. Ubuntu 11.10 is the most recent version currently supported by Hetzner.

DRIVE1 /dev/sda
DRIVE2 /dev/sdb
SWRAID 1
SWRAIDLEVEL 1
BOOTLOADER grub
HOSTNAME illuminati.cryptocracy.com
PART /boot ext3 512M
PART lvm vg0 all
LV vg0   root   /        ext4         10G
LV vg0   swap   swap     swap          4G
IMAGE /root/.oldroot/nfs/install/../images/Ubuntu-1110-oneiric-64-minimal.tar.gz

Upon booting into the newly provisioned machine, I found that the firewall wasn't enabled. That is easily fixed:

# ufw allow ssh
# ufw enable

I then used do-release-upgrade to bring the system up to 12.04.

Creating a NAT network

Although libvirt can be used to manage a NAT network for guests (and does this out of the box), its simplicity comes at a cost. I want to create a VPN that gives my workstation an address on the same network as the guests, and that requires custom iptables rules. I couldn't figure out a clean way of doing this with libvirt managing the interface, so I set it up manually instead.

Define a detached bridge by appending to /etc/network/interfaces:

# detached bridge for VMs and VPN
auto nat0
iface nat0 inet static
 address 192.168.42.1
 netmask 255.255.255.0
 pre-up brctl addbr nat0
 post-down brctl delbr nat0

Install brctl, and bring it up.

# apt-get install bridge-utils
# ifup nat0
# brctl show
bridge name   bridge id           STP enabled       interfaces
nat0          8000.fe5400f3c41d   no

Enabling IP masquerading (NAT)

To complete the setup of our new network, we need enable IP forwarding and configure the firewall. I followed the instructions in the Ubuntu firewall documentation and they worked as written.

Conclusion

At this point, we have Ubuntu 12.04 installed under LVM, with a manually created network ready for use by our VMs and VPN. In the next post, we'll configure a VPN and confirm that the nat0 network works as intended.

Chef LWRP - managed_directory

2012-04-07T22:37:00+01:00

A common configuration management pattern is to generate snippets of configuration into a ".d" directory, which are subsequently compiled into a complete configuration - either by a script, or through support from the application itself.

This pattern works well when you're maintaining a set of snippets, or adding to it, but removing snippets from the set can present a challenge. If you need to specifically remove a snippet from a node, you can always use the File resource with action :delete to do so. But what do you do if you don't know which snippets need to be removed?

The managed_directory LWRP (Github Repo) inspects the node's resource collection to identify the Chef-managed files in a given directory. It then compares this to the list of files actually present in the directory and removes the difference.

Walking the resource collection is pretty easy, although I had trouble finding examples when I sat down to try it. Hence this simple one:

  managed_files = run_context.resource_collection.all_resources.map { |r|
    r.name if r.name.start_with?("#{new_resource.path}/")
  }.compact

(Follow the "full source" link to see it in context.)

I haven't uploaded this cookbook to the community site yet, as the list of assumptions and caveats is almost as long as this post. If there's sufficient interest, I'll fix a few of those and put it up.

Suggestions and pull requests are welcome.

Starting over with Emacs

2012-04-04T12:13:00+01:00

Starting

I switched to Emacs sometime in 2009. At the time, I was using TextMate most of the time (though diving frequently into vi, as befitting my background as a system administrator). There wasn't any singular event that drove me to try out Emacs, but I do remember being frustrated by TextMate's performance with a particularly large project at work. I'd noticed Emacs-using colleagues having a much better time of things, and some stories about Emacs on Hacker News had piqued my interest.

The learning curve was initially quite steep, which presented something of a challenge - I needed to use the tool to get work done, and couldn't afford to significantly slow myself down for an unknown period. I bought PeepCode's "Meet Emacs" screencast to see a quick introduction to the essential features, and get a handle on my ignorance. It's important to know what you don't know, and the screencast helped me reach "conscious competence" quickly enough to stick with it.

Following the advice in "Meet Emacs", I used the Emacs Starter Kit for my initial configuration. It was awesome. More awesome than I could comprehend; despite spending time poking around the contents of my new ~/.emacs.d, only the most superficial aspects of the configuration made much sense. I had an editor that was powerful and feature-rich, but the much-vaunted malleability of the Emacs environment lay beyond my grasp.

Fortunately, my earlier switch to OS X had brought with it a certain serenity in the face of technology. If an opinionated application (or OS) does what you want, most of the time, it's worth going along with it in the places you disagree. That this is largely antithetical to the Emacs view of the world is not lost on me.

So, I spent my first couple of years with Emacs using a barely-customised ESK configuration, and I was pretty happy with it. While missing some of vi's functionality, I choose not to install any vi-compatibility features. The lynch-pin of my commitment turned out to be magit, the excellent git mode for Emacs. Having become accustomed to such seamless interaction with my VCS, I can no longer imagine working without it.

Restarting

Though I was largely happy with the end result, the amount of magic (or sufficiently advanced technology) bothered me. Nor was this entirely academic - I had been putting up with a bunch of quirks I was sure could be fixed, if only I knew how. Last week, I came across the article that helped me start down that road.

"Emacs Configuration Rewrite" documents Steven Danna's experience rebuilding his Emacs configuration from scratch. The result is positively minimalist alongside ESK, and lacks many features and customisations the latter delivers "out of the box". I decided that probably didn't matter to me - after all, if I missed something enough to add it back in, that would simply drive me to learn how to do that. Above all else, Steven's configuration showed me that I could start out with something I could comprehend completely, and expand from there. So that's what I did.

The initial commit to my emacs configuration repo doesn't tweak anything - it simply introduces the skeleton that everything else hangs off. I've added one thing at a time as I've needed it, and understood it. That's not to say it's entirely (or even mostly) original work - I've picked things up from various places, but they only go into my config when I understand what they do. No blind copy-and-paste, no cargo cults.

Emacs makes this exercise pretty easy. The built-in help can give you the value and meaning of any variable (C-h v), and the documentation for any function (C-h f). For anything implemented in elisp (and that's nearly everything), you can even jump straight to the source. This was way too much to deal with when I first started using Emacs, but I regret leaving it so long to come back to it.

The Future

I don't like unnecessary wheel reinvention. My goal is to understand how things fit together, that I might change them according to my needs - I don't feel that meeting that goal demands personally entering every single sexp into my Emacs configuration.

The Emacs Starter Kit has evolved considerably since I made a personal fork in 2009, and I made no attempt to evolve my forked configuration with it. It contains lots of good stuff, and I've started to look more closely at exactly how it's put together. As my understanding grows, I expect to be able to reuse larger pieces of elisp (from ESK and elsewhere) without feeling that my configuration is outgrowing my competence.

I may move to an ESK-based configuration again in the future, and I'm glad I used one in the past. In the present, I'm appreciating a greater understanding of a smaller set of features.

Migrating to Octopress

2012-03-20T08:16:00+00:00

As may be obvious from the new look, this site is now running on Octopress. Migrating from Posterous wasn't especially traumatic, but the paucity of content helped in that regard.

Importing from Posterous

Jekyll ships with an importer for pulling content out of Posterous, but it no longer works after the latter changed their API. There's a fix, but wasn't in a released version when I needed it.

With the update, the importer appeared to do a reasonable job of importing my posts (although it failed to set published: false on my drafts). One caveat: the date used is the creation date, not the publish date, of the source posts.

The importer didn't grab any of the "pages" on my Posterous site, but I chose to recreate the "About Me" page in markdown rather than determine whether this was intended behaviour.

I manually converted my posts and drafts into markdown, so let me know if you notice any errors.

Publishing

The site is now hosted on GitHub pages, backed by a private repository. This works well enough, although it doesn't appear to be possible to assign two custom hostnames for the same site. I'd like to migrate from "www.cryptocracy.com" to "cryptocracy.com", so a further hosting change is on the cards.

Chef MCollective cookbook, v0.9.0

2011-09-25T23:29:00+01:00

After far too long, I've released an updated cookbook for installing MCollective which addresses the three issues called out in the original announcement:

Directly invoking Ohai as a fact source is pretty slow.
The full set of Ohai facts can be pretty large.
Chef doesn't normalise 'default' recipe names in the expanded run list.

Ohai facts via YAML

Prior to this release, this cookbook configured MCollective to use the opscodeohai_facts plugin from the mcollective-plugins repository. While this continues to be supported, the default configuration now has MCollective loading facts from a YAML file written by a Chef report handler.

A new attribute has been added to control this - set node.mcollective.factsource to ohai to restore the old behaviour.

Configurable fact list

Out of the box, Ohai produces a lot of facts – not all of which are especially useful as MCollective Facts. The report handler responsible for building the YAML fact file looks at an attribute (node.mcollective.fact_whitelist) to filter the full list of facts available to Chef.

Default recipe name

The original responsibility of the report handler was to produce a list of "classes", the roles and recipes Chef used on this node. Unfortunately, Chef offers two ways of referring to a cookbook's default recipe - either with an explicit recipe name, recipe[cookbook::default], or without, recipe[cookbook].

This isn't spectacularly helpful, as the result is the same no matter which name was used. With this release, default recipes in the classlist always use their full name.

Other Changes

Site Plugins

A separate directory is now created to hold third-party plugins. Its path is stored as an attribute, node.mcollective.site_plugins.

Recipe for Puppetlabs repo installation

The cookbook previously assumed that the user would want to install the MCollective packages from the Puppetlabs apt/yum repository. While this simplifies installation for internet-connected machines, it's not the best approach for larger sites. If the nodes you're configuring don't have access to the internet, it obviously won't work at all.

This version introduces recipe[mcollective::puppetlabs-repo], which is included by mcollective::default. If you wish to use a local package mirror for installation, make sure it has been installed before using mcollective::server and mcollective::client.

Futures

What's left before a 1.0 release?

Paths to the generated facts/classes files should be controlled by a node attribute.
Optionally include non-automatic attributes in the fact file.
Support for using security plugins other than the default 'PSK'.
Tests (cucumber-chef, I'm looking at you).

Where can I get it?

You can find the MCollective Cookbook on the Opscode Community site, install it using Knife (knife cookbook site install mcollective), or check out the source on github.

Using MCollective with Chef

2011-08-21T13:27:00+01:00

MCollective is a Ruby framework "to build server orchestration or parallel job execution systems". Using messaging middleware (currently STOMP) for communication, a server running on each node can announce its existence, receive requests, and respond with structured data. Writing clients and agents is a straightforward exercise - while you'll need to know enough Ruby to implement your desired functionality, the framework itself adds little complexity. A number of non-core plugins are available on github.

Although it often seems that MCollective goes hand-in-hand with Puppet, there's no technical limitation behind this. It can quite readily be used with any other configuration management tool, or none at all. The plugins supporting integration with Puppet may be a little more mature today, but users of Chef won't be missing out on anything.

Installation

To satisfy MCollective's requirement for a STOMP server, it's easiest to use ActiveMQ. For Ubuntu users, this can be easily installed using a cookbook from the Opscode site. After installation, we need to configure the service for use by MCollective - for simplicity, we'll use the sample configuration from the Getting Started documentation.

See this gist for the necessary changes to the ActiveMQ cookbook.

The community site didn't have a cookbook to install MCollective, so I've been working on one to fill the gap. I've released it on the community site, but be warned - this is my first attempt at writing a public cookbook, and your mileage may vary. You can find the source repository on github.

Demonstration

We can use Vagrant to make a simple demonstration using a single node. In practice, there's not much point to running MCollective on just a single node - fortunately, using these cookbooks on a group of nodes (with Chef Server) is similarly straightforward. The repository for this demonstration is on github - the Vagrantfile specifies a "natty64" base box, but anything from "lucid" onwards should work.

Our first test that everything has worked is to use "mco ping". This is an MCollective client command which broadcasts an echo request, and presents the replies from any mcollective servers which happen to be listening. As we're running on a single node, we expect a single response:

Chef Integration

The cookbook incorporates the Chef integration code found in the MCollective documentation (link), with one small improvement - the code in the "Class filters" section has been implemented as a Chef handler, to avoid needing to pin a recipe at the end of the run list. The resulting inventory output looks something like this:

Issues

While relatively minor, there are several issues with the integration that I'm planning to work on. First, Ohai is a not especially fast. Although MCollective will cache its output, this means that requests might occasionally take several seconds to return (vs the typical tens-of-milliseconds latency). This is merely an annoyance during interactive use, but I expect it to be a problem for more sophisticated applications. An alternative approach used in the Puppet integration is to dump the facts into a YAML file, which MCollective then loads. Second, out of the box, Ohai appears to expose considerably more facts than Facter. I'm not convinced this is especially useful to MCollective, and I'm considering ways to filter the list down to facts I want to select or efficiently report on. Returning the data from only a specified list of Ohai plugins is my short-term plan. That said, I may yet change my mind about it being a problem - undesirably verbose "mco inventory" output is my only real complaint. Third, the list of Chef recipes reported to MCollective needs to be normalised. Depending upon how they made it into the expanded run list, a default recipe may be reported as either "recipe.cookbook" or "recipe.cookbook::default". I'm not sure whether Chef should be taking care of this, or whether I should just fix it in the report handler.