In a previous post (Cooking up deployments on stacked clouds) I looked at using UrbanCode Deploy with OpenStack and Chef. I showed one way of creating a stack from a Heat template, injecting the UCD agent and Chef into one of the newly provisioned servers, cooking a Chef recipe and deploying an application on it. This time round I look at how to do almost the same thing with VMware vSphere.
The general implementation goes something like this:
1. The application and deployment artifacts are pushed/pulled into UrbanCode Deploy components from an SCM system (RTC, Git etc..).
2. A “blueprint” for the virtual machines the application requires is also pushed/pulled into a UrbanCode Deploy “infrastructure” component, along with the installation packages for the UCD agent and Chef.
3. A UCD generic process is used to :
- create a new UCD Environment
- run a PowerCLI script to provision each of the VMs specified in the blueprint from (2) and install the UCD agent and Chef into each new VM
- execute UCD component deployment processes on the respective nodes
Before diving into the implementation a word on my development environment for this post. I don’t have spare servers with enough grunt lying around the house to setup a proper VSphere environment. However, as I mentioned in that Openstack post, my trusty old Lenovo W520 workhorse has a decent chunk of memory and a couple of fast SSDs in it. vCenter Server 5.5 comes as an appliance which I was able to get going in VirtualBox and in the same vein I downloaded and installed ESXi 5.5 into another VirtualBox VM and got them talking to each other. Installing PowerCLI into my UCD VM and sticking an Ubuntu 14.04 VM onto ESXi to use as a template for the application servers completed the setup. Should be obvious that running VMs in a VM in a VM is less than ideal for performance but if it means having to get up and stretch my legs a bit while waiting for a deployment from scratch, so be it.
Which reminds me. There’s a lot of cool stuff happening around IBM BlueMix and DevOps Services. If you have a real need for speed head on over to https://ace.ng.bluemix.net/ or https://hub.jazz.net/ : you won’t have time to get up let alone stretch your legs in the time it takes for apps to go from development to deployment.
But I digress. Back to the task at hand.
1. The Application
My “JKE” application has just two tiers – database and web – the source for which is held in RTC. An RTC build pushes deployment artifact versions into the appropriate components in UCD. (See this post and this post for more on RTC + UCD).
Each of the application components has a Deployment process that first installs a Chef node (lays down middleware for example) and then does some other component setup.
Nothing very special about these components.
A single Application process “Deploy to Environment” is used to deploy/re-deploy the components to environments. In this case I use the “Install Multiple Components” step to just call the “Deploy” process on each component.
2. The VM “blueprint” and the jke-infrastructure component
This bit is important so pay attention:-)
UCD already has the concept of Resource Templates which are used to define Application Blueprints which can be used as the basis for creating Environments. Here is my 2-tier resource template. Each of the Agent Prototypes in the template also specifies an Agent Name Pattern which is used to automatically place Agents in the Agent Prototype’s location.
Here is the Application Blueprint for JKE based on the above resource template.
What I now need is a “VM blueprint”, something equivalent to a Heat template that defines the stack of virtual machines that need to be provisioned. I went with a simple XML specification for the purposes of this exercise.
01 <?xml version="1.0"?> 02 <virtualmachines> 03 <VM> 04 <vmname>ubuntu_web</vmname> 05 <vmhost>192.168.56.102</vmhost> 06 <clonevm>ubuntu</clonevm> 07 <customization>UbuntuSpec01</customization> 08 <tierprefix>web</tierprefix> 09 <GuestUser>ubuntu</GuestUser> 10 <GuestPassword>ubuntu</GuestPassword> 11 </VM> 12 <VM> 13 <vmname>ubuntu_db</vmname> 14 <vmhost>192.168.56.102</vmhost> 15 <clonevm>ubuntu</clonevm> 16 <customization>UbuntuSpec01</customization> 17 <tierprefix>db</tierprefix> 18 <GuestUser>ubuntu</GuestUser> 19 <GuestPassword>ubuntu</GuestPassword> 20 </VM> 21 </virtualmachines>
In the “blueprint” each VM has a name, the ESXi host to provision on, the existing base VM to clone, the customization specification to apply, a “tierprefix” which should match the first part of the Agent Name Pattern in the Resource Template, and credentials.
This specification is SCM-controlled, versions are pushed/pulled into the “jke-infrastructure” component and used by a little PowerCLI script described next.
3. Provisioning VMs with PowerCLI
My UCD server runs on Windows 2008 so it was easy enough to install PowerCLI and cook up a basic script to read and use my VM blueprint. Here are the relevant snippets from the script with some command options truncated for brevity.
01 #powershell.exe -NoLogo -NonInteractive -File .\CreateVM.ps1
02 param( [String] $vcenterHost, [String] $appName, [String] $envName, [String] $ucdhost, [String] $ucdport )
03
04 $ErrorActionPreference = "Stop"
05 Add-PSSnapin VMware.VimAutomation.Core
06 Connect-VIServer $vcenterHost -User $vcU -Password $vcP
09
10 #Read blueprint
11 [xml]$s = Get-Content .\provisioning\jke\vm.xml
12 $nodes = $s.SelectNodes("//virtualmachines/VM")
15
16 #Create VM, start, and install UCD agent
17 foreach ($node in $nodes) {
18 $vm = New-VM -VMHost $node.vmhost -Name $node.vmname -VM $node.clonevm -OSCustomizationSpec (Get-OSCustomizationSpec -Name $node.customization)
20
21 Start-VM $vm | Wait-Tools
22
25 $agentname = $($node.tierprefix)+"_"+$appName+"_"+$envName
26
27 $target = "/tmp"
28 $source = ".\provisioning\packages\ibm-ucd-agent.zip"
29
30 Copy-VMGuestFile -Force -Source $source -Destination $target -vm $vm -LocalToGuest
31 $source = ".\provisioning\packages\chef_11.12.4-1_i386.deb"
32
33 Copy-VMGuestFile -Force -Source $source -Destination $target -vm $vm -LocalToGuest
34
36 Invoke-VMScript -VM $vm -ScriptText "sudo -i unzip -qo /tmp/ibm-ucd-agent.zip -d /tmp"
37
38 Invoke-VMScript -VM $vm -ScriptText "sudo -i chmod +x /tmp/ibm-ucd-agent-install/*.sh"
39
40 Invoke-VMScript -VM $vm -ScriptText "sudo -i sed -i -e 's/ibm-ucdagent/$agentname/g' -e 's/localhost/$uchdhost/g' -e 's/7918/$ucdport/g' /tmp/ibm-ucd-agent-install/example.agent.install.properties"
41
43 Invoke-VMScript -VM $vm -ScriptText "sudo -i /tmp/ibm-ucd-agent-install/install-agent-from-file.sh /tmp/ibm-ucd-agent-install/example.agent.install.properties"
46
47 Invoke-VMScript -VM $vm -ScriptText "sudo -i /opt/urbancode/$agentname/bin/agent start"
48
50 Invoke-VMScript -VM $vm -ScriptText "sudo -i dpkg -i /tmp/chef_11.12.4-1_i386.deb"
51
52 Write-Host "Created:$($node.vmname):$($vm.Guest.IPAddress[0]):$agentname"
53 }
The VM blueprint is read (lines 11-12) and used to:
- create a new vm, start it and wait for it to initialize (18-21)
- copy the UCD agent and Chef installation packages into the VM (27-33)
- extract the UCD agent files, set execute permissions on the install script, replace the values for the agent name, UCD server host and port and start the agent (36-47). The agent name is constructed to match the Agent Name Pattern in the UCD Resource Template.
- install Chef (50)
As each VM is created the Agents come online and register with the UCD host.
4. The Generic Process to kick off the whole thing
The Generic process to get the whole thing moving looks like this.
Step 1: Download Provisioning artifacts
Downloads the VM blueprint XML, the PowerCLI script and the UCD agent and Chef client packages from the “jke-insfrastructure” component.
Step 2: Create Environment from BP
Given an Application Name, Environment name and Blueprint Name generates a new,unique environment under the specified Base Resource Path.
Step 3: Create VMs and install agents via PowerCLI
This is a Shell step that invokes via PowerShell the PowerCLI script passing in values for the vCenter host, Application Name, newly generated environment name and UCD host and port.
If the script fails for any reason the new Environment, attached Resources and Agents are cleaned up by the alternate path in the process.
Step 4: Run Application Deployment Processes
If step 4 succeeds, this step requests the “Deploy to Environment” application process to be executed on the new environment’s resources, specifying a version for each component.
Step 5. Create Snapshot Of Environment
If all goes according to plan a snapshot of the new environment is created.
The process is executed on an agent on the Windows box with PowerCLI installed and, when executed from the UCD GUI, will prompt for the various parameter values.
As the process executes the various bits start popping into existence – the new environment with place holders for the agents, the new VMs in VCenter. Once the VMs are fully provisioned and the agents online the Application deployment process runs.
And of course a new environment with the application deployed is now available.
Now for some closing thoughts.
1. It might seem a bit strange to have the “jke-infrastructure” component mapped to both the tiers when there’s really nothing in it for them. There are no artifacts in it for either tier and my “Deploy” process for this component doesn’t actually do anything:
Including the component and its process this way ties the application components to the stuff used to provision the environment and resources they were deployed to. It lets me look at an environment and immediately tell exactly what was used to provision the environment and its resources – the VM blueprint version, the provisioning script.
This also lets me manage “everything as code”. Well, almost. The missing bits of “code” are the UCD Resource Template and Application Blueprint.
Of course the “infrastructure” process could actually be used do something useful and the component could be used to hold (for example) the Chef cookbooks.
2. Using PowerCLI is one fairly simple way to get the VMs up and running. Since Chef is already used here, another option would be to use “knife vsphere vm clone” to create the VMs, bootstrap them with Chef and apply roles/run-lists on them. (I’ll save that for another short post maybe.)
3. My “VM blueprint” + PowerCLI solution is admittedly a bit simplistic but served the purpose here. I haven’t addressed how updates to the resource requirements would be handled for example. For “serious” cloud portable, full-stack environment management and deployment that integrates with UCD, take a look at the newly announced “UrbanCode Deploy with Patterns“. The graphical full-stack environment designer is just the tip of a very cool iceberg of capabilities.
Freddy on DevOps and Cloudy Stuff 