Building vCAC Multi Machine Workflows

An important part of provisioning machines with vCAC is creating multi machine blueprints. As such, It is important to understand the flow of these kinds of blueprints when inside a custom workflow. With a regular workflow, things are pretty easy. The “virtualmachineid” parameter is passed into the workflow you are running at the moment (e.g workflow stubs, or any other custom workflows) and vualla everything is accessible with it, and the standard vCAC Designer actions will probably meet your needs.

When we’re running a workflow on a multi machine blueprint, things are a bit different, and some questions arise. Where does the workflow run? when? and how many times? how can I know which machines are inside the multi machine blueprint so I can interact with each separately during my workflow? Well, lets dive in and start answering some of these questions.

First of all lets get some must knowledge on MultiMachine blueprints / vCAC workflows:
– When you apply a custom property to a Multi Machine blueprint, it applies to all component virtual machines as well (Remember that when you apply a custom property or a stub that launches a workflow… yes, the workflow will run the number of N component VMs you have + 1 for the multi machine blueprint object)
– The mgmtContext parameter is in fact a link to the database  at the moment of the workflow. Use LINQ queries to interact with it and manipulate it.

So after this quick preview of things lets get to it.

The first thing you will want to do when running a multi machine workflow, is decide whither you want to run it from a master machine perspective, or for each machine separately. For this decision process we will use a simple “if” flow object. In order to determine what is exactly the machine type. We will start by getting the virtual machine object from the databaase, using a common LINQ query used in vCAC workflows:

/*This is a long line, please notice*/
mgmtContext.VirtualMachines.Expand(Function(v) v.VirtualMachineProperties).Where(Function(v) v.VirtualMachineID = VirtualMachineId).FirstOrDefault()


This query will get us the current VM according to its ‘virtualmachineid’ which we get as a parameter into the workflow. The ‘MasterMachine’ variable is defined of type “DynamicOps.ManagementModel.VirtualMachine”  which is the object type definition for a VM in vCAC. Applying this LINQ query to the ‘MasterMachine’ variable is done with a simple ‘Assign’ workflow action:

Getting vCAC Machine Object

Now, with our Machine variable populated, we can check the following ‘If’ statement:

If Machine is Not a Component

This will essentially tell us whether the VirtualMachineId we’re working on is an actual VM in the MultiMachine BluePrint, or the “Master Machine” itself. meaning, the MultiMachine Blueprint Object itself.

Once we have the MultiMachine object, we can access all of it’s component VMs in a few more steps!

First, we will use a second LINQ query, to get a list of all Component Machines for our MultiMachine Blueprint.

mgmtContext.AppServiceComponents.Expand(Function(AppServiceComponents) AppServiceComponents.VirtualMachine.VirtualMachineProperties).Where(Function(v) v.AppServiceVirtualMachine.VirtualMachineID=VirtualMachineId).toList()

We will create an ‘Assign’ workflow object to Assign this LINQ query to a variable we’ll create, with the name AllChildMachines, of type ‘System.Collections.GenericList’ This is a .Net type for a linked list. Each of the list’s component type should be ‘Dynamicops.ManagementModel.AppServiceComponent’

Gell All child Machines

AllChildMachines Variable Type

Next, we will need to create a ‘Foreach’ workflow object, that is also of type: ‘Dynamicops.ManagementModel.AppServiceComponent’. The Foreach loop will run on each of are associated vCAC VMs with the Master VM, and in at that part of the workflow, we will populate a VirtualMachine variable for that particular VM.

Foreach VM in Master Blueprint

The  LINQ Query used here, is similar to our first query, and is:

mgmtContext.VirtualMachines.Expand(Function(v) v.VirtualMachineProperties).Where(Function(v) v.VirtualMachineID = item.virtualmachine.virtualmachineid).FirstOrDefault()

If your sharp, you’ll notice the logic change here, this time the LINQ Query is grabbing the virtual machine with the same ID as our current ‘item’ in the foreach loop, thus running on each and each VM.

If you’re using the regular vCAC Designer objects, such as “GetVirtualMachineName” you’ll use them like:


From this point of the workflow you can run whatever you want on each ‘childmachine’. For example, apply a certain logic to look at all VMs , read properties, and decide what to do. Act upon each machine separately, or whatever else you might need to do. I recommend using the ‘Flowchart’ object for maximum flexibility (and not the sequence object shown here for easier reading purposes).

Don’t forget to set our last variable, “ChildMachine” of type ‘DynamicOps.ManagementModel.VirtualMachine’!

Enjoy, and leave your comments below!

Dynamically Assigning Property Dictionary Values

What makes vCAC such a dynamic product is definitely the use of the Property Dictionary.
If you’re still not familiar with it, you can read about it here (or here)
When using the property dictionary, you can unleash the full potential of the product. When you are able to update the property dictionary values dynamically, the possibilities are endless.

One example of using this scenario of dynamically updated property dictionary is allowing the user to set which vSphere portgroup (or VLAN) the machine will have when created. When a new portgroup is created in vsphere, the vCAC machine request page will be updated automatically as well.

For this special (and really important) use case , I’ve written a powershell module for vCAC Property dictionary (currently is @ version v0.9) with the help of some REST/JSON Functions written by Tom O’rourke . You can download the powershell module at the bottom of the page as well. Now, with the help of powerCLI & and the vCACPSModule we can grab all of vSphere’s portgroups, and dynamically add them to a vCAC property. Lets see how!

First, we need to open a powershell console on our vCAC Module Manager web server , and type in the following commands:

– This will create a new Property Dictionary for us to use, we can do this through the UI as well.

Import-Module vCACPSModule
New-vCACProperty -server -PropertyName "VirtualMachine.Netowrk.Choice" -PropertyType DropdownList -IsRequired True

– Our second stage is to create and schedule a script with the following code, in order to get vsphere portgroups and dynamically insert them into the vCAC Property dictionary with the help of the vCAC Powershell Module (Adjust the logic for getting the networks you actually need by cluster / name etc / use different network choices for different blueprints etc)

# GrabvSphereNetworks.ps1
Import-Module vCACPSModule
Add-PSSnapIn vmware.vimautomation.core
connect-viserver $vCenter -username $user -password $Password
# Edit to retrieve portgroups according to any logic you want
$Networks = Get-virtualportgroup | where-object Name -match "dvp" | Select Name -unique
# Changes the network list to something in format of a comma delimited string: "net1,net2,net3"
$Networks = ($networks | select -expand name ) -join ","
# Updates the vCAC property with the new values
update-vcacpropertyattribute -server -PropertyName "VirtualMachine.Network.Choice" -AttributeName Networks -AttributeType ValueList -AttributeValue $Networks
disconnect-viserver -confirm:$false

– Third stage, is to add the “VirtualMachine.Network.Choice” property we’ve created to our blue print, and create a workflow for the machine to set “VirtualMahchine.Network0.Name” property, with the network we want it to have. Here, we’ll use the building machine stub workflow in order to do so.

Setting our blueprint with the relevant properties:

Adding the Blueprint with Custom Property we created

When requesting the machine we will be able to select a Network from our property:

Now lets create the workflow for actually setting the machine’s network property.

Open the vCAC Designer, and create a workflow for changing the VM network. First, load the “ExternalWFStubBuildingMachine” workflow, and edit it like so:


Don’t forget to create a Variable named “NetworkName” (or any other name) with a String Type.

Also, you need to set the red marked properties for each activity. On GetMachineProperty we’ll set

  • MachineID -> virtualmachineid
  • PropertyName – > “VirtualMachine.Network.Choice” (dont forget ” sign)
  • PropertyValue -> NetworkName (Our created string variable)

At the SetMachineProperty, we’ll just set

  • MachineID -> virtualmachineid
  • PropertyName -> “VirtualMachine.Network0.Name”
  • PropertyValue -> NetworkName (Our created string variable)

Now we’ll Load the workflow back to the database.

In order for this to work , technically the networks that are in the custom property should be checked at the vCAC reservation level. you can simply check all of the networks when you create the reservation, and re-select each newly added portgroup to the vSphere environment, while I figure out how to get this automatically done via the API’s as well :)

Also, you need to schedule the GrabvSphereNetwork.ps1 script , for it to update the dictionary property on a permanent basis.

But more importantly, this PS Module can be of use to plenty of other use cases, for example:

  • Building machines in specific vSphere Directories per property choice
  • Choosing Machine’s Service classification from CMDB (simply sync the dictionary properties with services existing in CMDB using powershell)

If you think of any other scenarios – Let me Know!!! Comment section is below :)

vCAC PS Module Download

You can see example usage of the module’s functions at the file’s bottom section! There are some really cool CSV capabilities!

Troubleshooting “The Cloud” ?

One of the main characteristics of cloud environments as a whole, and vCloud director as a private, is acting as a software layer on top of an infrastructure/other software layer.

As such, troubleshooting can be a real pain, and getting to a definite root cause is not easy, as all of the layers need to be taken into account.

Some days ago I got to encounter a weird malfunction at one of my customers which got me thinking on how troubleshooting could get complicated when engaging a cloud environment.

So lets describe the symptoms that we saw while troubleshooting:

Specs: vCD 5.1.1 / vSphere 5.1

  1.  vCloud Director is unable to instantiate new VMs, with a vague “Internal Error” message.
  2.  vSphere on the other hand responds well, all looks OK cluster wise, vApp directories are  created when a vApp starts instantiation process, then nothing happens. No errors what so    ever.
  3. vCloud / vCenter connection is well, reconnecting / refresh  – everything is O.K
  4. Restarting vCenter Services solves issue for an unknown amount of time.

After putting some thought into this (and checking logs without any success for some interesting clues) I decided to start imitating the vCD process from bottom up.

So I created a new VM in the Cloud Resource cluster , with the standard vSphere C# wizard.

Reaching the Datastore cluster, to choose where to put my VMs vmdk disks, the wizard suddenly halted on a “loading” state, generating an error. hmmmmm…

Running the error online i’ve reached this KB 2021361 which seemed to explain my customers case,   and better yet, the suggested fix restores vCloud’s functionality.

This got me thinking about the entire process of troubleshooting cloud environments, where there are a lot of products / stacks involved, you might have to thoroughly run each platform to check for errors. This requires an administrator to be more thorough than ever, and moreover, know each platforms malfunction junctions.

Its amazing to think that once very long time ago, we were working with a troubleshooting methodology of:

Application – > OS (resource load) -> Server Hardware (fault).

them, troubleshooting stack became:

Application -> Guest OS (resource load /configuration) -> HyperVisor OS (contention) -> Server Hardware (fault)

What will happen when we dive deeper into more complex cloud stacks, that orchestrate, manage, and even try to heal our workloads?
I guess only time will tell.

Making the vCAC Description (and potentially any other) Field Mandatory

How many times have you provisioned a VM in your virtual enviroment, only to find out that you have no idea what its for a few weeks later?

One of the main difficulties in a self-serviced environment , is that this scenario is now a lot more common, as people can create their own VMs, giving them creative names like “MyVM1” for example.

Here’s a little customization tip for you to be able to enforce the description field on your happy vCAC Selfservice portal users.

Customization Steps:

1. Navigate to vCAC selfservice web files directory on your vCAC Server: %programfiles(x86)%VMwarevCACSelf-Service PortalViewsRequests

2. Back up (!!!) your original “NewRequest.cshtml” file and “AppService.cshtml” if your using vCloud vApps

3. Open the file, preferably with  notepad++ or something cshtml/tag aware

3.1 Search for the description label section looking like this:


Edit it in order to look like this (added red code):

<label><span class="required">*</span>@Labels.Description</label>

3.2 Edit the next line by adding required attribute, in order to look like this (add red code):

@Html.textareafor(m=>m.MachineRequest.Description, new{ cols=40, rows=4 , id= "descriptionText", TabIndex="1", @class = "k-input", required=""})

4. Do the same for AppService.cshtml , instead there span is a bit different:

<label><span class="required-mark">*</span>@Labels.Description</label>

adding required=”” is exactly the same.

Now, next time someone will try to submit a machine request without a description, he will get:

Enjoy :)

Note: Use this customization at your own risk! not I nor VMware inc. should be held responsible for any changes made in your environment / product

vCloud Automation Center 5.2 is now GA!

Some new features included in this release:

vCloud Director Improved integration

  • Ability to reconfigure vAPPs in vCloud Director.
  • Deployment on PAYG Org VDCs and partially reserving Org vDCs
  • vCNS networking is now detected by vCAC
  • Manage each vApp’s virtual machine separately

KVM Support

  • RHEL virtualization platform is now a supported endpoint in vCAC

Site Recovery Manager support

  • Support for failing over VMs (vCAC would now recognize DR site place holder machines)

Reclamation request customization

  • You can now customize your reclamation requests workflows within vCAC thus creating better reclamation and higher overall resource utilization

Over 230 bugs solved!

Full documentation here