Lately, I encountered some issues related to VMware vSAN in my Lab environment. The error message that was popping up all the time was “PBM error occurred during PreCloneCheckCallback“.
So how did the problem occur? First, we start with some background information. My Lab environment is powered-on when needed and powered-off when not needed. This is, of course, a little bit different than a production 24×7 environment that you have in your datacenters worldwide.
The environment was booted successfully at first glance. We are talking about Domain Controllers, vCenter Server, VMware NSX-V, nested ESXi Hosts and vRealize Automation. When I started deploying virtual machines with a vRealize Automation (vRA) based on blueprints with vSphere Templates issues started to occur.
vRealize Automation was failing on the provisioning task and was cleaning up the deployment because of the failed state (default behavior). So it was time to dig into the underlying infrastructure.
When the issue occurred the following software versions were used in my lab environment:
VMware vCenter 6.5 Update 2B
VMware vRealize Automation 7.3.1
VMware ESXi 6.5 Update 2
VMware vSAN 6.6
Here is all the information that can be found in various locations surrounding the issue.
Error message: Screenshots
Here are the screenshots, the first one is from VMware vCenter and the second one is from vRealize Automation. As you can see there is clearly a problem.
Error message: vRealize Automation
Here is the vRealize Automation log entry related to the VMware vSAN issue:
Error in Execute DynamicOps.Common.Client.HtmlResponseException: Service Unavailable (503)
Error message: vCenter Server
Here is the VMware vCenter log entry related to the VMware vSAN issue:
A general system error occurred - PBM error occurred during PreCloneCheckCallback (2118557)
The solution is quick but is more like a quick fix because it comes back every time I start up my lab environment.
Open a web browser.
Navigate to your vCenter Server URL (https://%vc%/vsphere-client).
Login with a user that has administrator credentials (email@example.com).
Navigate to Hosts & Clusters > Select the vCenter Object.
Click on the Configure tab.
Click on the Storage Providers.
Click on the following two buttons:
Synchronizes all Storage Providers with the current state of the environment.
Rescan the storage provider for new storage systems and storage capabilities.
After pressing the buttons, you don’t see any tasks running on the vCenter Server (expected behavior). After 5 seconds everything should be working and provisioning should be possible.
In this blog post, we are going to deploy VMware vCenter 6.7 Update 1 in my Lab environment. The deployment is fully covered with all the additional notes required to perform a successful installation, migration or upgrade. I also added some guidelines for designing your environment.
If you are familiar with the VMware vCenter 6.5 graphical deployment it has been improved in VMware vCenter 6.7. In the past it was a web-based wizard, with 6.7 it is a binary executable. This means a lot faster and better-responding interface and it removes the browser dependency and browser plug-in on your workstation.
The checklist items can be verified, days or hours before the initial deployment. If you don’t have a plan before installing, migrating or upgrading things will turn out ugly…
With the checklist, you can determine if your environment is ready for vSphere 6.7 Update 1. It’s about checking and validation your current software and hardware and talking to your vendors about compatibility.
I have also added some design decision ideas. Because you can choose to install, upgrade or migrate without looking at your current architecture but maybe it is time to update your current architecture (design).
Make sure that all connected/used VMware products are compatible like (vRealize Automation / vRealize Orchestrator / vRealize Operations Manager / VMware Horizon and the list goes on). This can be verified on the VMware Product Interoperability Matrices page.
Make sure that all third-party products are compatible like (Backup & Replication software / Storage vendor software).
Determine the correct sizing for your environment. How many virtual machines and ESXi Hosts are going to be running underneath this vCenter Server. These figures determine your vCenter Server size.
Download the latest release from the VMware website.
Create firewall rules for your new vCenter Server.
Create forward and reverse DNS records in your DNS Server.
Register your IP information in your IPAM system.
Save your passwords in your Password Management system (Appliance password / SSO password).
Have a workstation ready to perform the deployment with sufficient network access and administrative rights.
Let’s start the deployment of VMware vCenter 6.7 Update 1. I have chosen for a clean installation of VMware vCenter 6.7 Update 1. I have chosen for an embedded Platform Services Controller (PSC). Based on my total amount of virtual machines and ESXi Hosts I have selected a “Small” installation footprint.
The new deployment process for vCenter Server 6.7 Update 1 consists out of two stages, one is the deployment stage and one is the setup stage.
The first part is mainly responsible for delivering the full appliance with the operation system, network settings, and installation application bundles. The second part is configuring the applications that are running on the vCenter Server. A total installation takes about 45 minutes to complete.
Mount the vCenter Server media (iso file).
Navigate to the following path “X:\vcsa-ui-installer\win32\” (X stands for the CD-ROM drive label).
Run the following application “installer.exe“.
Follow the wizard, I have uploadedall screenshots for reference.
Stage 01 – Deployment
Here are the images of the first stage of the deployment of VMware vCenter 6.7 Update 1. I have no issues to report everything was working fine on the first try!
Stage 02 – Setup
Here are the images of the second stage of the deployment of VMware vCenter 6.7 Update 1. This part was also bug-free, so it was a good deployment.
After a successful deployment of the VCSA appliance, you need to configure at least some items to get vCenter Server production ready. The items listed below are a basic set of the most common items I see in the field:
Install the vCenter Server License.
Assign rights & permissions.
Generate and installation of SSL Certificates.
Connect the required VMware products and third-party systems.
VMware vCenter 6.5.0 Update 2. The target for the vCenter 6.7.0 Update 1 deployment.
VMware ESXi 6.5.0 Update 2 in the 24×7 environment. Known as the production cluster.
VMware ESXi 6.5.0 Update 2 in the Lab environment. Known as the lab cluster.
You might ask… why don’t you upgrade the current vCenter Server? Good question! The machine has been converted/upgraded multiple times. It started out in life as a VMware vCenter 5.5 machine, that was on the Windows Server 2012 platform. So it was a good moment to start clean after this many years.
At a customer I came across the following problem, the customer was not able to remove a Content Library from vCenter Server. They just created a Content Library and after that, they wanted to remove the item. When they tried to remove the content library it failed. We started troubleshooting the log files and tried to remove the Content Library in different ways with the vSphere Web Client, PowerShell and REST API but all ended with the same error. The error messages are listed below.
To add some more background information: the customer was running the environment with an external platform services controller and a vCenter Server (VCSA). The version that was being used was VCSA 6.5 Update 1e.
Content Library – Error messages
Cannot Remove Content Libary – 01
Cannot Remove Content Libary – 02
Cannot Remove Content Libary – 03
We ended up calling VMware Global Support Services (GSS) to resolve the issues. They were very helpful and fixed it within a couple of minutes. The knowledge base article listed below is only available for internal VMware personal.
The internal knowledge base article related to the issue: – https://ikb.vmware.com/s/article/50121825 – Unable to delete the stale entry for the content library from the web client
Recently somebody asked me a question about VMware vCenter running on a Windows Server. The Windows Server was running VMware vCenter 6.5 and in case of a datacenter related problem, they wanted to get access to the vSphere Web Client (Flash) on the system locally.
It sounds easy right…? Just open the browser on the Windows Server and navigate to the vSphere Web Client page but that didn’t appear to be the case, because the system is missing the browser plugins required to open the vSphere Web Client.
So let’s dive into the problem.
Microsoft Browsers: They are running Windows Server 2016 and you might expect it to have two browsers: Internet Explorer 11 and Microsoft Edge. That does not seem to be the case. Windows Server 2016 is only shipped with Internet Explorer 11. Why? Windows Server 2016 is marked as an LTSB (Long Time Service Branch) so this means no Microsoft Edge and it is also not available for manual installation.
Microsoft: “The Long-Term Servicing Branch (LTSB) versions of Windows, including Windows Server 2016, don’t include Microsoft Edge or many other Universal Windows Platform (UWP) apps. These apps and their services are frequently updated with new functionality, and can’t be supported on systems running the LTSB operating systems.”
Third-party browsers: The company who was asking had a security policy that does not allow an installation of third-party browsers like Mozilla Firefox or Google Chrome. Alright, so this is not an option. Don’t have to look at that further.
Adobe Flash: So let’s try Internet Explorer 11. It appears to be missing Adobe Flash and you can not download and install it from Adobe Website.
At this point, I was stuck and there did not seem to be a simple solution.
vSphere Web Client missing Adobe Flash
Adobe Flash not available for installation on Windows Server 2016
After searching for a solution for about an hour. I came across a Microsoft Blog article listed below. This article is talking about installing Adobe Flash on Windows Server 2016. It appears that all the software is already on the system but just needs to be installed.
– Step 01: Close all browsers
– Step 02: Start a PowerShell session with elevated rights.
– Step 03: Run the following command: dism /online /add-package /packagepath:"C:\Windows\servicing\Packages\Adobe-Flash-For-Windows-Package~31bf3856ad364e35~amd64~~10.0.14393.0.mum"
– Step 04: Wait for the installation to complete.
– Step 05: Open a browser and navigate to the vSphere Web Client.
– Step 06: Everything should be working now.
Note: In the Microsoft Blog article they are talking about a reboot required in my case it was not required. Just a browser restart was enough.
Installing Adobe Flash on Windows Server 2016
vSphere Web Client with Adobe Flash
It sounded like an easy problem at first but it took some more time than I expected. The problem is solved with a simple one-liner and the customer is happy. I personally think that there might be other solution to the problem. If you know them please add a comment below.
Lately, I discovered an annoying feature in combination with VMware vCenter and VMware Workstation. When installing VMware Workstation on your management computer it becomes the default Remote Console viewer. To be honest, I like the VMware Remote Console (VMRC) very much. The application has all the features and is quick and light. This compared to starting VMware Workstation to open a Remote Console.
What is VMware Remote Console: “The VMware Remote Console (VMRC) is a standalone console application for Windows. VMware Remote Console provides console access and client device connection to VMs on a remote host. You will need to download this installer before you can launch the external VMRC application directly from a VMware vSphere or vRealize Automation web client.”
In October 2017, I already fixed my problem on my management computer… but after a recent VMware Workstation update, it changed the Remote Console back to VMware Workstation. Currently, there is no option in the GUI to change the default Remote Console. Ok, but how do we get VMRC back?
When I was comparing the Windows Registry, I found out that the following registry keys were different between machines. To speed up to process I created some PowerShell one-liners to fix the problem.
When you change the registry keys, the settings are direct in effect. No Operating System reboot or browser restart is required. The change is instant. I hope the blog post helps some vSphere Administrators that also prefer VMRC above VMware Workstation for viewing Remote Consoles.
@VMware: I would like to have an option to control the behaviour without changing registry keys by hand… 🙂 Thanks!
The issues occurred with the following combination of software:
VMware vCenter Server 6.5 (Update 1e)
VMware VMRC (10.0.2-7096020)
VMware Workstation (12.5.9 build-7535481)
Management Workstation: Windows 10 X64
Some screenshots that display the changes when opening the Remote Console of a Virtual Machine in VMware vCenter.