It’s vBlog Voting Season!

It’s voting season in the blog-sphere, and you get to influence! For the past ±2 years i’ve been blogging about vRealize Automation (formerly vCAC ) ever since version 5.1 came out :) it only seems like yesterday but I guess its been a while ain’t it?
Since my role shift to the vRealize Air team i’ve been VERY busy, but thats a positive thing! I’m now starting to set up a new lab, and will also blog a whole lot about vRealize Air Automation (vRAA) which is our currently in Beta vRealize Automation as a Service, hosted in vCloud Air. So I expect a LOT of great blogposts in the upcoming year, just for you, my readers! Its very worthwhile to stay tuned via RSS, and even more worth while to help my blog get to the Top 100 mark in the vSphere-Land vBlog contest!

If you’ve enjoyed my work, used my site, asked a question and was replied promptly (I do try my best) please take 5 minutes to vote for my blog, as it would mean a lot to me, and will let me know i’m doing this blogging thing right :)
You can cast your vote here 


Time to Take On a New Role!

I’ve been around customers & large enterprise engagements for about 3.5 years now (2 of them, currently with VMware, 1.5 with EMC) , deploying, architecting cloud & automation solutions for my customers to help them reach cloud nirvana. It’s been a wild ride, and now its time for a new path. Something different, with new possibilities ahead, and some very interesting skill sets.

I’m very excited to announce that as of today, I will be joining the Cloud Solutions Engineering TeamVMware, where i’ll be titled an Integration Engineering Architect.

In this new role, I will actually join the group developing VMware’s vRealize Automation Air (#vRAA, Hurrah!)  SaaS platform, helping them code some of the SaaS platform, and helping with the vCAC architecture required for this solution. In this role, I will also help building official vCAC reference architectures, and further help with solutions created for vCAC, whether they’re On-Prem, or SaaS based.

This team, works in some very interesting development models, including end-to-end DevOps continuos delivery, Cloud Circles development methodology and much more.

I must say, I find this role extremely challenging, yet also extremely fun! I think you should always pursue a great deal of personal development. Never stay in your comfort zone, always push for new territories. I know this is kind of a cliche, yet in these times of cloud infrastructures, it has never been more accurate. With things like vCO automation & network virtualization starting to pick up fast, you must always stay two steps ahead of the curve.

I couldn’t agree more with Pat Gelsinger at the VMworld key note, saying that the business should be fluid. “Liquid Business” he called it. I strongly believe in maintaining a “Liquid Career”. Always mix things up, change roles & skill sets according to environment variables, and never stay in one place. Don’t hesitate to learn and engage something you’ve never done before, it’ll help you learn new skills along the way!

I will keep maintaining this blog, and it will still revolve heavily around vCAC, personally I want to thank you from the bottom of my heart for reading!

Yours truly,

vCAC 6 Edit VM Failing

Recently, I came across a strange issue  with vCloud Automation Center. And although the issue was strange, (Error: vCAC Error code: 42000) and the solution even weirder,  during the resolution I came across a secret url (well, at least an undocumented one) in the system that proved to be very useful!
When I arrived at my customer’s site, I got a grim look and the issue description – VMs cannot be edited by users from vCAC 6. Every user trying to edit a VM, gets an immediate failure for the error task, with a not-very-helpful error description:

[code]Exception during request callback with id <uuid> for item <uuid>. Error Message: [Error code: 42000 ] – [Error Msg: Infrastructure service provider error][/code]

A quick run through the IaaS server logs revealed nothing, but a short good old google query did come up with this community thread. For you lazies, the community post talks about this EXACT error code and problem description, and marks a clear solution – the IaaS server’s local isn’t set to united states. This, causes the exception on the IaaS provider, that the error code is talking about.

Great! looks like a simple solution, just change the local back to united states setting, and everything will be fixed. This must be our issue / solution.
A quick check on our IaaS servers showed that non of the IaaS server components had a non-english local. Meaning, everything was supposedly set OK on the servers. This begs the question – why the hell are we still experiencing our issue??? After some discussions and thoughts, I still held my opinion that this was probably an IaaS OS configuration issue, if not specific related to the OS local.

We opened a GSS ticket, and asked for engineering to give some input since this was a production environment. Engineering, pointed us to the same locale configuration, and directed us to fix it since this was a known bug and will be fixed in the next version. BUT the great thing engineering also revealed on that call, is a hidden url that is used to detect issues on the IaaS API end. Let’s take a step back to understand how calls are being made from the vCAC VA to the IaaS server.

See, the iaas-provider (iaas servers) , gets the jobs requested by the user via an interface called WAPI or Web-API. This is a web-app installed via the Web Model Manager installer. If we go to a url –
https://<iaas web server fqdn>/WAPI you would be able to see some help information and the methods exposed via WAPI. Although you won’t be able to utilize them, since you’ll need an SSO token, it’s a nice url to know and explore.

But this is not the secret url yet. So in order to figure out what’s wrong with our IaaS servers, we got redirected by engineering to this url:

(localhost = iaas web server)

But what is elmah you ask? elmah is a debugging tool for handling ASP.Net web-app exceptions. So after wondering what could be the issue with our IaaS servers, we invoked the edit operation again (which failed as expected) and paid a visit to the elmah tool built into vCAC IaaS. What we saw was this page and message:

elmah2This indicated instantly that our issue is indeed related to IaaS not being able to interpret some form of date being sent to it. It was a way better, descriptive, and informative error, since it came from the IaaS side, and not the vCAC-VA Web side as we previously saw in the request!
Getting to a solution from there was a matter of wondering what else could change the date format on the IaaS servers other then regional settings? First, I got to a an IIS .NET globalization setting that can be set using IIS Manager.

After validating that all of the settings there were correct, I found another place that could have an affect on the submitted edit request date. A registry key:

When inspecting the keys there, I’ve found that the value for sShortDate , is different from the U.S format of : M/d/yyyy
Since this looked strange, and apperantly, this key determines which type of format users will default at login time, I’ve decided to change it to M/d/yyyy as it should be. A quick reboot to the IaaS servers (a service restart would have been enough as well) and the edit problem, was solved! Edit requests were now being accepted and executed by IaaS as you would expect.

VMworld 2014 Voting is Open! Vote for my Sessions!

As you probably know, for the past year i’ve been doing a TON of work around vCloud Automation Center & vCO. Ever since day one, i’ve been constantly thinking and wondering on how to get the maximum out of the product, the use cases it can answer, as well as other nifty things around customizing it.
This year, I’ve decided to get 4 of these ideas into VMworld as breakout sessions. I’d be honored if you could take the time to support me with this initiative, and cast your vote for my sessions!
3 of these sessions are subject to public voting, and the fourth, to internal vmware voting.

My VMworld 2014 Sessions

 #1609 Automate DevOps Continuos Delivery of Quality Assurance Testing Platforms, using vCloud Automation Center 6.x, vCenter Orchestrator, Puppet&Jenkins

This session, will thoroughly and deeply go through a blog post I had wrote recently called Enabling DevOps Using vCAC 6 I will go through the use case, the technical deep dive on how I implemented it, and last but not least, the secret day 2 operation hidden at the youtube video attached to the post.

 #1812 Delivering IT as a Service with vCloud Automation Center and a Software-Defined Strategy
This session, will discuss a concept I presented at an Israeli VMUG earlier in Febuary this year, regarding IT as a Service, and how can it be achieved using the right methodologies and tools, in order to make IT processes a lot faster!

 #1537 Building 3rd Party Integrations for vCloud Automation Center Through vCenter Orchestrator

In this session, I will present best practices and tips & tricks, with everything related to vCO workflows in vCloud Automation Center. Also, a real world scenario for a simple integration with a third party system will be showcased, discussing the benefits of integrating 3rd party system with vCloud Automation Center in order to create a single holistic single pane for the organizational services.

 #(internal) Autoscaling Applications with vCloud Automation Center, vCloud Suite, and The Application Driven Cloud
In my opinion, this is by far the most interesting session, and also includes some not yet GA features of some products. I’ll be discussing how to easily achieve autoscaling of applications, with vCAC and the vCloud suite, simply by re-thinking the entire concept of the organizational cloud, and the way you design around it. Unfortunately, for this session to get into VMworld, you will have to hold your fingers for me :) if your a vmware employee, try to pitch in and vote internally!The easiest way to look for my sessions at the website, is simply by filtering ‘Omer Kushmaro’ < – Hey, That’s Me!
Also, in some of my sessions would be presenting with great & respected co-workers such as:
– Zackary Kiliech (
– Jad El-Zein (
– Tom Findling (vCops Product owner)
Of course, you can read through the full abstract for the sessions at the web site, so come on! lets get to voting!

vCAC 6.0 is GA

Carful cloud users, because vCloud Automation Center 6.0 is GA! Grab the bits while its hot! I believe that this version (and the next to come ) will be a major game changer for our customers, helping them drive some cloud automation within their organizations, and make their life EASY! Notice that this iteration of vCAC is a bit different, and is actually using an additional new engine. This is noticeable right from the get go of installing the product, and from the UI first glimpse.

vCAC Core Enhancements

vCAC 6.0 supports these cloud infrastructures out of the box:
Amazon AWS , vCloud Powered Clouds

And these hypervisors:
vSphere , Hyper-V , KVM, Openstack Grizzly

Check out the documentation section for further detail

Networking: The 6.0 release can now be used to create new networks for your deployments , using vCNS , or NSX for vSphere ! You can create isolated networks, routed networks and NAT networks. You are now able to set multi machine blueprint networking at the blueprint level, specifying load balancer settings, to which new multi machine services will be provisioned. With NSX, even more can be done with firewalls LB’s , and network provisioning.

Storage: SDRS is now supported as a reservation, allowing the administrator to utilize the auto-placement mechanism. This is important especially for large VM blueprints with multiple disks / variant disk sizes.

BCP/DR: vCAC 6.0 extends the support of SRM managed VMs, on several deployments scenarios such as managing both original & DR site, DR testing scenarios (ignoring test machines on test site) and much more!

Applications: vCAC 6.0 now supports Application Director to some extent, being able to publish application deployed by AppD as items in the new catalog. Using this functionality in congestion with vCAC 6.0, brings some powerfull options in deploying multi tier complex environments on demand. Something that was a bit lacking with the 5.2 release.

Action Items: The new “Resource Action” allows you to perform pretty much any action you’d like on the resource / items you have in your catalog, mainly the ability to run vCO workflows easily, also , the system supports the vCO action prompts , more on this functionality in upcoming posts.

A nice item to notice here is VMRC connection to vSphere VMs (this was limited to vCD VMs in 5.2)

Approvals: Approvals can now be set with a lot of granularity, allowing you to determine the approval phases, how many people will approve, whether only one of them needs to approve , or all of them. Also, a very useful feature here is the ability to set which fields can be edited (amount of vCPUs / Memory perhaps). This approval systems redefines governance for IT admins in the enterprise!

Advanced Catalog Management

The new vCAC catalog contains two new objects: Services, and Catalog items. “Services” is the left pane menu that represents the kind of services we’re providing for our users. Whether its Application Services, Fresh OS builds, or XaaS items. Also, what’s nice about the new catalog system is that you can actually set some items as “New and note worthy” this will in turn pop them up in the nice port-lets the users get when they login to the system.

I believe that this specific feature is insanely powerful, as it turns vCAC into an IT management and operation platform. Just imagine the following:
Storage Services :
Catalog items: Request a LUN / RDM / Storage Increase

Network Services:
Catalog items: Request a Firewall port change / provision a network / provision a firewall

and so on…
This feature, with the help of vCAC’s governance , will help greatly in transforming organisations and datacenters to being truly Software Defined Data Centers!

Terminology Changes

This version of vCAC we’re introducing some terminology changes, and also includes some new user roles:

vCAC Administrator – > IaaS Admin:
This role will be given to a user on a creation of a new tenant, when logging in with the “Tenant zero” account of “administrator@vsphere.local”

Enterprise Groups – > Fabric Groups:
These are the the groups of users in charge of the cloud fabrics, meaning different types of infrastructures and compute resources.
This role enables you to:
– Create infrastructure reservations.
– Create and assign business groups / business groups managers (read below)
– Configure approvals.

Provisioning Groups – > Business Groups:
What used to be provisioning groups, groups of users who can provision IaaS to a certain fabric, is now know as a business group. This directs the user a bit more in who he should correlate these types of users with.
This role enables you to:
– Configure and publish catalog items, and their entitlements.
– Configure Blueprints.
– Create/Configure actions for blueprints

Architecture Changes

vCloud Automation Center 6.0 now features two new components , vCAC Appliance, and a vCAC SSO Appliance. At the moment, vCAC 6.0 doesn’t support vCenter SSO, but i’m hoping it will in a nearby timeframe. The appliance is using a PostgreSQL DB to store all of its data, and also contains a built in vCO appliance, to deliver the XaaS feature.

Still, the IaaS component of the system is based on the windows components used in the 5.2 version, only now the blueprints can be published in the new catalog!