Lately, I encountered some issues related to VMware vSAN in my Lab environment. The error message that was popping up all the time was “PBM error occurred during PreCloneCheckCallback“.
So how did the problem occur? First, we start with some background information. My Lab environment is powered-on when needed and powered-off when not needed. This is, of course, a little bit different than a production 24×7 environment that you have in your datacenters worldwide.
The environment was booted successfully at first glance. We are talking about Domain Controllers, vCenter Server, VMware NSX-V, nested ESXi Hosts and vRealize Automation. When I started deploying virtual machines with a vRealize Automation (vRA) based on blueprints with vSphere Templates issues started to occur.
vRealize Automation was failing on the provisioning task and was cleaning up the deployment because of the failed state (default behavior). So it was time to dig into the underlying infrastructure.
When the issue occurred the following software versions were used in my lab environment:
VMware vCenter 6.5 Update 2B
VMware vRealize Automation 7.3.1
VMware ESXi 6.5 Update 2
VMware vSAN 6.6
Here is all the information that can be found in various locations surrounding the issue.
Error message: Screenshots
Here are the screenshots, the first one is from VMware vCenter and the second one is from vRealize Automation. As you can see there is clearly a problem.
Error message: vRealize Automation
Here is the vRealize Automation log entry related to the VMware vSAN issue:
Error in Execute DynamicOps.Common.Client.HtmlResponseException: Service Unavailable (503)
Error message: vCenter Server
Here is the VMware vCenter log entry related to the VMware vSAN issue:
A general system error occurred - PBM error occurred during PreCloneCheckCallback (2118557)
The solution is quick but is more like a quick fix because it comes back every time I start up my lab environment.
Open a web browser.
Navigate to your vCenter Server URL (https://%vc%/vsphere-client).
Login with a user that has administrator credentials (firstname.lastname@example.org).
Navigate to Hosts & Clusters > Select the vCenter Object.
Click on the Configure tab.
Click on the Storage Providers.
Click on the following two buttons:
Synchronizes all Storage Providers with the current state of the environment.
Rescan the storage provider for new storage systems and storage capabilities.
After pressing the buttons, you don’t see any tasks running on the vCenter Server (expected behavior). After 5 seconds everything should be working and provisioning should be possible.
In this blog post, we are going to deploy VMware vCenter 6.7 Update 1 in my Lab environment. The deployment is fully covered with all the additional notes required to perform a successful installation, migration or upgrade. I also added some guidelines for designing your environment.
If you are familiar with the VMware vCenter 6.5 graphical deployment it has been improved in VMware vCenter 6.7. In the past it was a web-based wizard, with 6.7 it is a binary executable. This means a lot faster and better-responding interface and it removes the browser dependency and browser plug-in on your workstation.
The checklist items can be verified, days or hours before the initial deployment. If you don’t have a plan before installing, migrating or upgrading things will turn out ugly…
With the checklist, you can determine if your environment is ready for vSphere 6.7 Update 1. It’s about checking and validation your current software and hardware and talking to your vendors about compatibility.
I have also added some design decision ideas. Because you can choose to install, upgrade or migrate without looking at your current architecture but maybe it is time to update your current architecture (design).
Make sure that all connected/used VMware products are compatible like (vRealize Automation / vRealize Orchestrator / vRealize Operations Manager / VMware Horizon and the list goes on). This can be verified on the VMware Product Interoperability Matrices page.
Make sure that all third-party products are compatible like (Backup & Replication software / Storage vendor software).
Determine the correct sizing for your environment. How many virtual machines and ESXi Hosts are going to be running underneath this vCenter Server. These figures determine your vCenter Server size.
Download the latest release from the VMware website.
Create firewall rules for your new vCenter Server.
Create forward and reverse DNS records in your DNS Server.
Register your IP information in your IPAM system.
Save your passwords in your Password Management system (Appliance password / SSO password).
Have a workstation ready to perform the deployment with sufficient network access and administrative rights.
Let’s start the deployment of VMware vCenter 6.7 Update 1. I have chosen for a clean installation of VMware vCenter 6.7 Update 1. I have chosen for an embedded Platform Services Controller (PSC). Based on my total amount of virtual machines and ESXi Hosts I have selected a “Small” installation footprint.
The new deployment process for vCenter Server 6.7 Update 1 consists out of two stages, one is the deployment stage and one is the setup stage.
The first part is mainly responsible for delivering the full appliance with the operation system, network settings, and installation application bundles. The second part is configuring the applications that are running on the vCenter Server. A total installation takes about 45 minutes to complete.
Mount the vCenter Server media (iso file).
Navigate to the following path “X:\vcsa-ui-installer\win32\” (X stands for the CD-ROM drive label).
Run the following application “installer.exe“.
Follow the wizard, I have uploadedall screenshots for reference.
Stage 01 – Deployment
Here are the images of the first stage of the deployment of VMware vCenter 6.7 Update 1. I have no issues to report everything was working fine on the first try!
Stage 02 – Setup
Here are the images of the second stage of the deployment of VMware vCenter 6.7 Update 1. This part was also bug-free, so it was a good deployment.
After a successful deployment of the VCSA appliance, you need to configure at least some items to get vCenter Server production ready. The items listed below are a basic set of the most common items I see in the field:
Install the vCenter Server License.
Assign rights & permissions.
Generate and installation of SSL Certificates.
Connect the required VMware products and third-party systems.
VMware vCenter 6.5.0 Update 2. The target for the vCenter 6.7.0 Update 1 deployment.
VMware ESXi 6.5.0 Update 2 in the 24×7 environment. Known as the production cluster.
VMware ESXi 6.5.0 Update 2 in the Lab environment. Known as the lab cluster.
You might ask… why don’t you upgrade the current vCenter Server? Good question! The machine has been converted/upgraded multiple times. It started out in life as a VMware vCenter 5.5 machine, that was on the Windows Server 2012 platform. So it was a good moment to start clean after this many years.
At a customer I came across the following problem, the customer was not able to remove a Content Library from vCenter Server. They just created a Content Library and after that, they wanted to remove the item. When they tried to remove the content library it failed. We started troubleshooting the log files and tried to remove the Content Library in different ways with the vSphere Web Client, PowerShell and REST API but all ended with the same error. The error messages are listed below.
To add some more background information: the customer was running the environment with an external platform services controller and a vCenter Server (VCSA). The version that was being used was VCSA 6.5 Update 1e.
Content Library – Error messages
Cannot Remove Content Libary – 01
Cannot Remove Content Libary – 02
Cannot Remove Content Libary – 03
We ended up calling VMware Global Support Services (GSS) to resolve the issues. They were very helpful and fixed it within a couple of minutes. The knowledge base article listed below is only available for internal VMware personal.
The internal knowledge base article related to the issue: – https://ikb.vmware.com/s/article/50121825 – Unable to delete the stale entry for the content library from the web client
Today I was facing a VMware Content Library issue. I was removing a newly created Content Library in the vSphere Web Client but that resulted in a Java Runtime error.
The conclusion was that there was no way to remove the Content Library item.
Today I was planning a NSX manager deployment in my Home Lab… But that turn out to be a problem, because I could not upload an OVF file in the vSphere Client and HTML5 Web Client. When looking in my Home Lab notes I realized the last time I deployed an OVF was when the VCSA was running 6.5 without update 1. I think something went wrong with updating to VCSA 6.5 update 1.
Both webpages display the problem in a different way.
With the vSphere Client the following pop-up appears when trying to deploy an OVF file: “This version of vCenter Server does not support Deploy OVF Template using this version of vSphere Web Client. To Deploy OVF Template, login with version 126.96.36.199 of vSphere Web Client”
HTML5 Web Client:
The HTML5 Web Client does not display any error at all. It just disables the option to deploy an OVF file.
After some googling I found the following VMware KB article 2151085 (link). This turned out to be the solution.
1. Connect to the vCenter Server Appliance with an SSH session and root credentials.
2. Run this command to enable access the Bash shell:
shell.set –enabled true
3. Type shell and press Enter.
4. Navigate to /etc/vmware-content-library/config/ with this command:
5. Create a backup of the ts-config.properties and ts-config.properties.rpmnew file with these commands:
cp ts-config.properties ts-config.properties.orig
cp ts-config.properties.rpmnew ts-config.properties.rpmnew.orig
6. Rename ts-config.properties.rpmnew to ts-config.properties.
mv ts-config.properties.rpmnew ts-config.properties
7. Restart the Content Library service:
service-control –stop vmware-content-library
service-control –start vmware-content-library
8. Refresh or close your browser and connect with one of the web interfaces.
VCSA Restarting Content Library
vSphere Client – working again
HTML5 Web Client – Working again
Be-Virtual.net - Cookies