Recently, I came across a strange issue with vCloud Automation Center. And although the issue was strange, (Error: vCAC Error code: 42000) and the solution even weirder, during the resolution I came across a secret url (well, at least an undocumented one) in the system that proved to be very useful!
When I arrived at my customer’s site, I got a grim look and the issue description – VMs cannot be edited by users from vCAC 6. Every user trying to edit a VM, gets an immediate failure for the error task, with a not-very-helpful error description:
[code]Exception during request callback with id <uuid> for item <uuid>. Error Message: [Error code: 42000 ] – [Error Msg: Infrastructure service provider error][/code]
A quick run through the IaaS server logs revealed nothing, but a short good old google query did come up with this community thread. For you lazies, the community post talks about this EXACT error code and problem description, and marks a clear solution – the IaaS server’s local isn’t set to united states. This, causes the exception on the IaaS provider, that the error code is talking about.
Great! looks like a simple solution, just change the local back to united states setting, and everything will be fixed. This must be our issue / solution.
A quick check on our IaaS servers showed that non of the IaaS server components had a non-english local. Meaning, everything was supposedly set OK on the servers. This begs the question – why the hell are we still experiencing our issue??? After some discussions and thoughts, I still held my opinion that this was probably an IaaS OS configuration issue, if not specific related to the OS local.
We opened a GSS ticket, and asked for engineering to give some input since this was a production environment. Engineering, pointed us to the same locale configuration, and directed us to fix it since this was a known bug and will be fixed in the next version. BUT the great thing engineering also revealed on that call, is a hidden url that is used to detect issues on the IaaS API end. Let’s take a step back to understand how calls are being made from the vCAC VA to the IaaS server.
See, the iaas-provider (iaas servers) , gets the jobs requested by the user via an interface called WAPI or Web-API. This is a web-app installed via the Web Model Manager installer. If we go to a url –
https://<iaas web server fqdn>/WAPI you would be able to see some help information and the methods exposed via WAPI. Although you won’t be able to utilize them, since you’ll need an SSO token, it’s a nice url to know and explore.
(localhost = iaas web server)
But what is elmah you ask? elmah is a debugging tool for handling ASP.Net web-app exceptions. So after wondering what could be the issue with our IaaS servers, we invoked the edit operation again (which failed as expected) and paid a visit to the elmah tool built into vCAC IaaS. What we saw was this page and message:
This indicated instantly that our issue is indeed related to IaaS not being able to interpret some form of date being sent to it. It was a way better, descriptive, and informative error, since it came from the IaaS side, and not the vCAC-VA Web side as we previously saw in the request!
Getting to a solution from there was a matter of wondering what else could change the date format on the IaaS servers other then regional settings? First, I got to a an IIS .NET globalization setting that can be set using IIS Manager.
After validating that all of the settings there were correct, I found another place that could have an affect on the submitted edit request date. A registry key:
When inspecting the keys there, I’ve found that the value for sShortDate , is different from the U.S format of : M/d/yyyy
Since this looked strange, and apperantly, this key determines which type of format users will default at login time, I’ve decided to change it to M/d/yyyy as it should be. A quick reboot to the IaaS servers (a service restart would have been enough as well) and the edit problem, was solved! Edit requests were now being accepted and executed by IaaS as you would expect.