Chef and CloudFormation

The ephemeral nature that comes along with cloudy virtual machines means that we need to be able to go down and come up at any time without flinching. What good is hardware scriptability if the app itself doesn’t lend itself to automated management? For years, we’ve been using custom sets of homebrew scripts, checked-in alongside both the app code and any virtual hardware scripting. It was cumbersome, but it worked beautifully. Chef promised to make the cumbersome elegant. Did it?

Yes, it did. The learning curve was steep at first. It seemed to take a lot of knowledge just to get set up. We started with CloudFormation (CFN), for which AWS provides sample templates of both a Chef server and a sample app as a Chef client.

After getting them up and running, we replaced the Chef server stack with Hosted Chef. For PCI compliance, we later built a private chef server. The results were spectacular. One button launches an AWS stack in any region, as before, but now the only thing that CFN has its machines do is bootstrap chef. Have the standard AMIs include it already, and all that has to happen is a passing of Chef Server location and initial creds. Chef handles the rest! What’s more, you can configure chef-client to register a nodename corresponding to IP address, stack name, region, whatever you like (see below).

In case it helps, it took about two full-time weeks to go from homebrew stack to chef stack, with no previous knowledge of Chef. This makes the dev/ops approach much easier to sell to all concerned parties. For example, the notorious case of application configurations have a new potential place. One client wisely wrote an app which first looked to a sharding table to figure out where its DB was. The problem: the sharding table was in a DB, and self-referencing. Moving such a typically static, simple thing to Chef attributes makes perfect sense, as it does for a whole range of app configurations as well as ops configurations.

Note: If you ever wonder where ohai’s ec2 attributes are, you may have come across a known ohai bug within the VPC. The solution is simply “With Ohai 6.4.0, create /etc/chef/ohai/hints/ec2.json to enable EC2 attribute collection,” as done below using cfn-init.

Note: These examples use CloudFormation helper scripts, but there is nothing you can’t do here with simple scripting. Below lies a sample LaunchConfig. All that remains is to put the node config into one var.

Update: We recently had to retrofit my old chef stacks to work in particular subnets within a VPC and AWS’s Chef Server template, which works right out of the box. But now, when we need to tweak stack json, we modify the LaunchConfig, cfn-update-stack, and as-terminate-instance-in-auto-scaling-group, quickly iterating and saving oodles of time. Once the chef handoff is made, we can get to recipe-writing and role-assigning.

        "FrontEndLC" : {
            "Type" : "AWS::AutoScaling::LaunchConfiguration",
            "Metadata" : {
                "AWS::CloudFormation::Init" : {
                    "config" : {
                        "packages" : {
                            "rubygems" : {
                                "chef" : [],
                                "ruby-shadow" : [],
                                "ohai" : [],
                                "json" : []
                            },
                            "yum" : {
                                "ruby19"            : [],
                                "ruby19-devel"        : [],
                                "ruby19-irb" : [],
                                "ruby19-libs"            : [],
                                "rubygem19-io-console"              : [],
                                "rubygem19-json"             : [],
                                "rubygem19-rake" : [],
                                "wget"            : [],
                                "rubygem19-rdoc"        : [],
                                "rubygems19"        : [],
                                "rubygems19-devel"        : [],
                                "gcc"        : [],
                                "gcc-c++"        : [],
                                "automake"        : [],
                                "autoconf"        : [],
                                "make"        : [],
                                "curl"        : [],
                                "dmidecode"        : []
                            }
                        },
                        "files" : {
                            "/etc/chef/client.rb" : {
                                "content" : { "Fn::Join" : ["", [
                                    "log_level        :info\n",
                                    "log_location     STDOUT\n",
                                    "ssl_verify_mode  :verify_none\n",
                                    "chef_server_url  '", { "Ref" : "ChefServerURL" }, "'\n",
                                    "environment      '", { "Ref" : "Environment" }, "'\n",
                                    "validation_client_name 'chef-validator'\n"
                                ]]},
                                "mode"  : "000644",
                                "owner" : "root",
                                "group" : "root"
                            },
                            "/etc/chef/roles.json" : {
                                "content" : {
                                    "run_list": [ "role[frontend]" ],
                                    "chef_role": "frontend",
                                    "stack_name": { "Ref" : "AWS::StackName" },
                                    "aws_region": { "Ref" : "AWS::Region" },
                                    "deploy_user": { "Ref" : "DeployUser" },
                                    "deploy_pass": { "Ref" : "DeployPass" },
                                    "deploy_bucket": { "Ref" : "DeployBucket" },
                                    "warning_sns_arn": { "Ref" : "WarningTopic" },
                                    "critical_sns_arn": { "Ref" : "CriticalTopic" },
                                    "iam_access_key": { "Ref" : "IAMAccessKey" },
                                    "iam_secret_key": { "Fn::GetAtt" : ["IAMAccessKey", "SecretAccessKey"] },
                                    "frontend_endpoint": { "Fn::GetAtt" : [ "FrontEndELB", "DNSName" ] },
                                    "s3_bucket": { "Ref" : "S3Bucket" }
                                },
                                "mode"  : "000644",
                                "owner" : "root",
                                "group" : "root"
                            },
                            "/etc/chef/ohai/hints/ec2.json" : {
                                "content" : "{}",
                                "mode"   : "000644",
                                "owner"  : "root",
                                "group"  : "root"
                            }
            "Properties" : {
                "KeyName" : { "Ref" : "KeyName" },
                "SecurityGroups" : [ { "Ref" : "FrontEndSG" } ],
                "InstanceType" : { "Ref" : "FrontEndInstanceType" },
                "ImageId": { "Fn::FindInMap": [ "AWSRegionArch2AMIEBS", { "Ref": "AWS::Region" }, { "Fn::FindInMap": [ "AWSInstanceType2Arch", { "Ref": "FrontEndInstanceType" }, "Arch" ] } ] },
                "UserData" : { "Fn::Base64" :
                               { "Fn::Join" : [ "", [
                                   "#!/bin/bash\n\n",

                                   "/opt/aws/bin/cfn-init -v --region ", { "Ref" : "AWS::Region" },
                                   " -s ", { "Ref" : "AWS::StackName" }, " -r FrontEndLC ",
                                   " --access-key ", { "Ref" : "DeployUser" },
                                   " --secret-key ", { "Ref" : "DeployPass" }, "\n",

                                   "LOCAL_IP=`curl -s http://169.254.169.254/latest/meta-data/local-ipv4`\n",
                                   "IID=`curl -s http://169.254.169.254/latest/meta-data/instance-id`\n",
                                   "echo \"node_name        \\\"", { "Ref" : "AWS::StackName" }, "-frontend-$LOCAL_IP-$IID\\\"\" >> /etc/chef/client.rb\n",
                                   "/usr/local/bin/chef-client -N ", { "Ref" : "AWS::StackName" }, "-frontend-$LOCAL_IP-$IID -j /etc/chef/roles.json", "\n"
                               ]]}
                             }
            }
        },