Lately, I encountered some issues related to VMware vSAN in my Lab environment. The error message that was popping up all the time was “PBM error occurred during PreCloneCheckCallback“.
So how did the problem occur? First, we start with some background information. My Lab environment is powered-on when needed and powered-off when not needed. This is, of course, a little bit different than a production 24×7 environment that you have in your datacenters worldwide.
The environment was booted successfully at first glance. We are talking about Domain Controllers, vCenter Server, VMware NSX-V, nested ESXi Hosts and vRealize Automation. When I started deploying virtual machines with a vRealize Automation (vRA) based on blueprints with vSphere Templates issues started to occur.
vRealize Automation was failing on the provisioning task and was cleaning up the deployment because of the failed state (default behavior). So it was time to dig into the underlying infrastructure.
When the issue occurred the following software versions were used in my lab environment:
VMware vCenter 6.5 Update 2B
VMware vRealize Automation 7.3.1
VMware ESXi 6.5 Update 2
VMware vSAN 6.6
Here is all the information that can be found in various locations surrounding the issue.
Error message: Screenshots
Here are the screenshots, the first one is from VMware vCenter and the second one is from vRealize Automation. As you can see there is clearly a problem.
Error message: vRealize Automation
Here is the vRealize Automation log entry related to the VMware vSAN issue:
Error in Execute DynamicOps.Common.Client.HtmlResponseException: Service Unavailable (503)
Error message: vCenter Server
Here is the VMware vCenter log entry related to the VMware vSAN issue:
A general system error occurred - PBM error occurred during PreCloneCheckCallback (2118557)
The solution is quick but is more like a quick fix because it comes back every time I start up my lab environment.
Open a web browser.
Navigate to your vCenter Server URL (https://%vc%/vsphere-client).
Login with a user that has administrator credentials (email@example.com).
Navigate to Hosts & Clusters > Select the vCenter Object.
Click on the Configure tab.
Click on the Storage Providers.
Click on the following two buttons:
Synchronizes all Storage Providers with the current state of the environment.
Rescan the storage provider for new storage systems and storage capabilities.
After pressing the buttons, you don’t see any tasks running on the vCenter Server (expected behavior). After 5 seconds everything should be working and provisioning should be possible.
This time I decided to do a blog post about the HPE Smart Array RAID controllers with their wonderful ssacli tool. The tooling of HPE is very powerful because you can online manage a VMware ESXi host and migrate for example from a RAID 1 volume to a RAID 10 without downtime or change the read and write cache ratio.
So far as I know I haven’t seen an identical tool yet from the other server hardware vendors like Cisco, Dell EMC, IBM, and Supermicro. The main difference has always been that the HPE tool can perform the operation live without downtime.
So far as I can remember it has been there for ages. It was already available for VMware ESX 4.0 and is still available in VMware ESXi 6.7. So thumbs-up for HPE :).
Let’s talk about controller support. The tool supports the most HPE SmartArray controllers over the last 10 to 15 years, for example, the Smart Array P400 was released in 2005 and is still working fine today.
Here is an overview of supported controllers:
HPE Smart Array P2XX
HPE Smart Array P4XX
HPE Smart Array P7XX
HPE Smart Array P8XX
HPE SSACLI – Location
In case you are using the HPE VMware ESXi custom images. The tool is already pre-installed when installing ESXi. The tool is installed as a VIB (vSphere Installable Bundle). This means it can also be updated with vSphere Update Manager.
Over the years the name of the HPE Storage Controller Tool has been changed and so has the location. Here is a list of locations that have been used for the last ten years for VMware ESXi:
I have collected some screenshots over the years. Screenshots were taken by doing maintenance on VMware ESXi servers. The give you an idea what valuable information can be shown.
HPE SSACLI – Abréviation
All commands have a short name to reduce the length of the total input provided to the ssacli tool:
- chassisname = ch
- controller = ctrl
- logicaldrive = ld
- physicaldrive = pd
- drivewritecache = dwc
- licensekey = lk
### Specify drives:
- A range of drives (one to three): 1E:1:1-1E:1:3
- Drives that are unassigned: allunassigned
HPE SSACLI – Status
To view the status of the controller, disks or volumes you can run all sorts of commands to get information about what is going on in your VMware ESXi server. The extensive detail is very useful for troubleshooting and gathering information about the system.
# Show - Controller Slot 1 Basic configuration
./ssacli ctrl slot=1 show config
# Show - Controller Slot 1 Detailed configuration
./ssacli ctrl slot=1 show config detail
# Show - Controller Slot 1 Status
./ssacli ctrl slot=1 show status
# Show - All Controllers Configuration
./ssacli ctrl all show config
# Show - Controller slot 1 logical drive 1 status
./ssacli ctrl slot=1 ld 1 show status
# Show - Basic Physical Disks status
./ssacli ctrl slot=1 pd all show status
# Show - Detailed Physical Disk status
./ssacli ctrl slot=1 pd all show status
HPE SSACLI – Creating
Creating a new logical drive can be done online with the HPE Smart Array controllers. I have displayed some basic examples.
# Create - New single disk volume
./ssacli ctrl slot=1 create type=ld drives=2I:0:8 raid=0 forced
# Create - New spare disk (two defined)
./ssacli ctrl slot=1 array all add spares=2I:1:6,2I:1:7
# Create - New RAID 1 volume
./ssacli ctrl slot=1 create type=ld drives=1I:0:1,1I:0:2 raid=1 forced
# Create - New RAID 5 volume
./ssacli ctrl slot=1 create type=ld drives=1I:0:1,1I:0:2,1I:0:3 raid=5 forced
HPE SSACLI – Adding drives to logical drive
Adding drives to an already created logical drive is possible with the following commands. You need to perform two actions: adding the drive(s) and expanding the logical drive. Keep in mind: make a backup before performing the procedure.
# Add - All unassigned drives to logical drive 1
./ssacli ctrl slot=1 ld 1 add drives=allunassigned
# Modify - Extend logical drive 2 size to maximum (must be run with the "forced" flag)
./ssacli ctrl slot=1 ld 2 modify size=max forced
HPE SSACLI – Rescan controller
To issue a controller rescan, you can run the following command. This can be interesting for when you add new drives in hot swap bays.
### Rescan all controllers
HPE SSACLI – Drive Led Status
The LED status of the drives can also be controlled by the ssacli utility. An example is displayed below how to enable and disable a LED.
# Led - Activate LEDs on logical drive 2 disks
./ssacli ctrl slot=1 ld 2 modify led=on
# Led - Deactivate LEDs on logical drive 2 disks
./ssacli ctrl slot=1 ld 2 modify led=off
# Led - Activate LED on physical drive
ctrl slot=0 pd 1I:0:1 modify led=on
# Led - Deactivate LED on physical drive
ctrl slot=0 pd 1I:0:1 modify led=off
HPE SSACLI – Modify Cache Ratio
Modify the cache ratio on a running system can be interesting for troubleshooting and performance beanchmarking.
# Show - Cache Ratio Status
./ssacli ctrl slot=1 modify cacheratio=?
# Modify - Cache Ratio read: 50% / write: 50%
./ssacli ctrl slot=1 modify cacheratio=50/50
# Modify - Cache Ratio read: 0% / Write: 100%
./ssacli ctrl slot=1 modify cacheratio=0/100
HPE SSACLI – Modify Write Cache
Changing the write cache settings on the storage controller can be done with the following commands:
Viewing or changing the rebuild priority can be done on the fly. Even when the rebuild is already active. Used it myself a couple of times to lower the impact on production.
# Show - Rebuild Priority Status
./ssacli ctrl slot=1 modify rp=?
# Modify - Set rebuildpriority to Low
./ssacli ctrl slot=1 modify rebuildpriority=low
# Modify - Set rebuildpriority to Medium
./ssacli ctrl slot=1 modify rebuildpriority=medium
# Modify - Set rebuildpriority to High
./ssacli ctrl slot=1 modify rebuildpriority=high
HPE SSACLI – Modify SSD Smart Path
You can modify the HPE SDD Smart Path feature by disabling or enabling. To make clear what the HPE SDD Smart Path includes, here is a officialstatement by HPE:
“HP SmartCache feature is a controller-based read and write caching solution that caches the most frequently accessed data (“hot” data) onto lower latency SSDs to dynamically accelerate application workloads. This can be implemented on direct-attached storage and SAN storage.”
In this post, we are going to change the Virtual Storage Controller from LSI Logic Parallel to VMware Paravirtual for a CentOS 7 based Virtual Machine that is running on VMware vSphere. This blog post will contain step by step guidance for performing the operation.
In my case the virtual machine was build in VMware Workstation and after some time migrated to VMware ESXi. The VMware Paravirtual Storage Controller is not supported in VMware Workstation. That is why the virtual machine came over with the “wrong” storage controller.
My 24×7 Lab environment is running shared iSCSI based storage and all virtual machines are thin provisioned. The Virtual Machine that came over from VMware Workstation is installed with CentOS 7.
Why VMware Paravirtual?
Why should you want to migrate from an LSI Logic Parallel to a VMware Paravirtual SCSI Controller? Two simple reasons and it are two good ones:
Lower CPU utilization
Personally, I have a third reason to add… compliance. All my virtual machines should be compliant with the VMware Best Practice and my personal Home Lab standard. In my Lab environment, this means using the VMware Paravirtual where ever possible/supported.
Assign disks to the new storage controller and remove the old storage controller (screenshot 03).
Power-on the virtual machine.
Validate that everything is working and disks are mounted (screenshot 04).
Remove the virtual machine snapshot or backup after you are done.
At this point, I have swapped out three virtual machines from the LSI controller to the VMware Paravirtual SCSI Controller. The machines have been running now for about two weeks without any problems. So everything is compliant again ;).
If you encounter any problems or have any question about this subject please feel free to contact me on Twitter or the Reply option below.
Here are some interesting related articles that I used for creating this blog post:
At a customer I came across the following problem, the customer was not able to remove a Content Library from vCenter Server. They just created a Content Library and after that, they wanted to remove the item. When they tried to remove the content library it failed. We started troubleshooting the log files and tried to remove the Content Library in different ways with the vSphere Web Client, PowerShell and REST API but all ended with the same error. The error messages are listed below.
To add some more background information: the customer was running the environment with an external platform services controller and a vCenter Server (VCSA). The version that was being used was VCSA 6.5 Update 1e.
Content Library – Error messages
Cannot Remove Content Libary – 01
Cannot Remove Content Libary – 02
Cannot Remove Content Libary – 03
We ended up calling VMware Global Support Services (GSS) to resolve the issues. They were very helpful and fixed it within a couple of minutes. The knowledge base article listed below is only available for internal VMware personal.
The internal knowledge base article related to the issue: – https://ikb.vmware.com/s/article/50121825 – Unable to delete the stale entry for the content library from the web client
Recently somebody asked me a question about VMware vCenter running on a Windows Server. The Windows Server was running VMware vCenter 6.5 and in case of a datacenter related problem, they wanted to get access to the vSphere Web Client (Flash) on the system locally.
It sounds easy right…? Just open the browser on the Windows Server and navigate to the vSphere Web Client page but that didn’t appear to be the case, because the system is missing the browser plugins required to open the vSphere Web Client.
So let’s dive into the problem.
Microsoft Browsers: They are running Windows Server 2016 and you might expect it to have two browsers: Internet Explorer 11 and Microsoft Edge. That does not seem to be the case. Windows Server 2016 is only shipped with Internet Explorer 11. Why? Windows Server 2016 is marked as an LTSB (Long Time Service Branch) so this means no Microsoft Edge and it is also not available for manual installation.
Microsoft: “The Long-Term Servicing Branch (LTSB) versions of Windows, including Windows Server 2016, don’t include Microsoft Edge or many other Universal Windows Platform (UWP) apps. These apps and their services are frequently updated with new functionality, and can’t be supported on systems running the LTSB operating systems.”
Third-party browsers: The company who was asking had a security policy that does not allow an installation of third-party browsers like Mozilla Firefox or Google Chrome. Alright, so this is not an option. Don’t have to look at that further.
Adobe Flash: So let’s try Internet Explorer 11. It appears to be missing Adobe Flash and you can not download and install it from Adobe Website.
At this point, I was stuck and there did not seem to be a simple solution.
vSphere Web Client missing Adobe Flash
Adobe Flash not available for installation on Windows Server 2016
After searching for a solution for about an hour. I came across a Microsoft Blog article listed below. This article is talking about installing Adobe Flash on Windows Server 2016. It appears that all the software is already on the system but just needs to be installed.
– Step 01: Close all browsers
– Step 02: Start a PowerShell session with elevated rights.
– Step 03: Run the following command: dism /online /add-package /packagepath:"C:\Windows\servicing\Packages\Adobe-Flash-For-Windows-Package~31bf3856ad364e35~amd64~~10.0.14393.0.mum"
– Step 04: Wait for the installation to complete.
– Step 05: Open a browser and navigate to the vSphere Web Client.
– Step 06: Everything should be working now.
Note: In the Microsoft Blog article they are talking about a reboot required in my case it was not required. Just a browser restart was enough.
Installing Adobe Flash on Windows Server 2016
vSphere Web Client with Adobe Flash
It sounded like an easy problem at first but it took some more time than I expected. The problem is solved with a simple one-liner and the customer is happy. I personally think that there might be other solution to the problem. If you know them please add a comment below.
Lately, I discovered an annoying feature in combination with VMware vCenter and VMware Workstation. When installing VMware Workstation on your management computer it becomes the default Remote Console viewer. To be honest, I like the VMware Remote Console (VMRC) very much. The application has all the features and is quick and light. This compared to starting VMware Workstation to open a Remote Console.
What is VMware Remote Console: “The VMware Remote Console (VMRC) is a standalone console application for Windows. VMware Remote Console provides console access and client device connection to VMs on a remote host. You will need to download this installer before you can launch the external VMRC application directly from a VMware vSphere or vRealize Automation web client.”
In October 2017, I already fixed my problem on my management computer… but after a recent VMware Workstation update, it changed the Remote Console back to VMware Workstation. Currently, there is no option in the GUI to change the default Remote Console. Ok, but how do we get VMRC back?
When I was comparing the Windows Registry, I found out that the following registry keys were different between machines. To speed up to process I created some PowerShell one-liners to fix the problem.
When you change the registry keys, the settings are direct in effect. No Operating System reboot or browser restart is required. The change is instant. I hope the blog post helps some vSphere Administrators that also prefer VMRC above VMware Workstation for viewing Remote Consoles.
@VMware: I would like to have an option to control the behaviour without changing registry keys by hand… 🙂 Thanks!
The issues occurred with the following combination of software:
VMware vCenter Server 6.5 (Update 1e)
VMware VMRC (10.0.2-7096020)
VMware Workstation (12.5.9 build-7535481)
Management Workstation: Windows 10 X64
Some screenshots that display the changes when opening the Remote Console of a Virtual Machine in VMware vCenter.
Today I was facing a VMware Content Library issue. I was removing a newly created Content Library in the vSphere Web Client but that resulted in a Java Runtime error.
The conclusion was that there was no way to remove the Content Library item.
Today I was planning a NSX manager deployment in my Home Lab… But that turn out to be a problem, because I could not upload an OVF file in the vSphere Client and HTML5 Web Client. When looking in my Home Lab notes I realized the last time I deployed an OVF was when the VCSA was running 6.5 without update 1. I think something went wrong with updating to VCSA 6.5 update 1.
Both webpages display the problem in a different way.
With the vSphere Client the following pop-up appears when trying to deploy an OVF file: “This version of vCenter Server does not support Deploy OVF Template using this version of vSphere Web Client. To Deploy OVF Template, login with version 126.96.36.199 of vSphere Web Client”
HTML5 Web Client:
The HTML5 Web Client does not display any error at all. It just disables the option to deploy an OVF file.
After some googling I found the following VMware KB article 2151085 (link). This turned out to be the solution.
1. Connect to the vCenter Server Appliance with an SSH session and root credentials.
2. Run this command to enable access the Bash shell:
shell.set –enabled true
3. Type shell and press Enter.
4. Navigate to /etc/vmware-content-library/config/ with this command:
5. Create a backup of the ts-config.properties and ts-config.properties.rpmnew file with these commands:
cp ts-config.properties ts-config.properties.orig
cp ts-config.properties.rpmnew ts-config.properties.rpmnew.orig
6. Rename ts-config.properties.rpmnew to ts-config.properties.
mv ts-config.properties.rpmnew ts-config.properties
7. Restart the Content Library service:
service-control –stop vmware-content-library
service-control –start vmware-content-library
8. Refresh or close your browser and connect with one of the web interfaces.
In this blog post, we are going to automate the installation of VMware ESXi 5.5, 6.0 and 6.5. This can be done with a so-called “kickstart” configuration file which is officially supported by VMware. The file contains the configuration for a VMware ESXi Host to configure settings like IP address, subnet mask, hostname, license key, datastore, etc.
The kickstart configuration file can be made available in the following locations:
USB flash drive
Personally, I prefer to use the HTTP protocol.
You might ask yourself, why should I install an ESXi Host with a kickstart file? Some of the use cases I identified over the years are:
The very first ESXi Hosts for your SDDC environment (before VMware vCenter is deployed or vSphere Auto Deploy is configured).
A standalone ESXi Host for a small environment.
A Home Lab environment to install nested VMware ESXi Hosts.
Setup a web server
To make the kickstart configuration file available for the ESXi host we need a web server. Basically, every web server available on the market can serve this file. Here is a list of web server products that I have used: Apache, Microsoft IIS and NGINX.
In this environment/example I used a Microsoft IIS server on a Windows 10 Client. Do not forget to add the cfg extension to the MIME types.
Now it’s time to create a text file with your favourite text editor. The text file in this example is called (ks.cfg). I have added two configuration files as samples, one with the minimum settings and one I normally use for my Lab environment.
Configuration file – Simple (ks.cfg)
This is a default ks.cfg configuration file with just the minimum of settings required.
# Sample scripted installation file
# Accept the VMware End User License Agreement
# Set the root password for the DCUI and Tech Support Mode
# The install media is in the CD-ROM drive
install --firstdisk --overwritevmfs
# Set the network to DHCP on the first network adapter
network --bootproto=dhcp --device=vmnic0
# A sample post-install script
%post --interpreter=python --ignorefailure=true
stampFile = open('/finished.stamp', mode='w')
stampFile.write( time.asctime() )
Configuration file – Advanced (ks.cfg)
This is the more advanced version of the configuration file that also configures a lot of other settings like NTP servers, search domain, CEIP and a static IP address for the management interface.
### ESXi Installation Script
### Hostname: LAB-ESXi01A
### Author: M. Buijs
### Date: 2017-08-11
### Tested with: ESXi 6.0 and ESXi 6.5
##### Stage 01 - Pre installation:
### Accept the VMware End User License Agreement
### Set the root password for the DCUI and Tech Support Mode
### The install media (priority: local / remote / USB)
install --firstdisk=local --overwritevmfs --novmfsondisk
### Set the network to DHCP on the first network adapter
network --bootproto=static --device=vmnic0 --ip=192.168.151.101 --netmask=255.255.255.0 --gateway=192.168.151.254 --nameserver=192.168.126.21,192.168.151.254 --hostname=LAB-ESXi01A.lab.local --addvmportgroup=0
### Reboot ESXi Host
##### Stage 02 - Post installation:
### Open busybox and launch commands
### Set Search Domain
esxcli network ip dns search add --domain=lab.local
### Add second NIC to vSwitch0
esxcli network vswitch standard uplink add --uplink-name=vmnic1 --vswitch-name=vSwitch0
### Disable IPv6 support (reboot is required)
esxcli network ip set --ipv6-enabled=false
### Add NTP Server addresses
echo "server 192.168.126.21" >> /etc/ntp.conf;
echo "server 192.168.151.254" >> /etc/ntp.conf;
### Allow NTP through firewall
esxcfg-firewall -e ntpClient
### Enable NTP autostartup
/sbin/chkconfig ntpd on;
### Rename local datastore (currently disabled because of --novmfsondisk)
#vim-cmd hostsvc/datastore/rename datastore1 "DAS - $(hostname -s)"
### Disable CEIP
esxcli system settings advanced set -o /UserVars/HostClientCEIPOptIn -i 2
### Enable maintaince mode
esxcli system maintenanceMode set -e true
esxcli system shutdown reboot -d 15 -r "rebooting after ESXi host configuration"
Installing an ESXi Host with Kickstart file
The following procedure needs to be performed to boot from a kickstart file:
Boot the ESXi host with a VMware ESXi ISO (ISO file can be obtained from the VMware download page).
Press the key combination “shift + o” at boot.
Enter one of the following lines after runweasel:
For an HTTP share: ks=http://%IP_or_FQDN%/kg.cfg
For an HTTPs share: ks=https://%IP_or_FQDN%/kg.cfg
For a NFS share: ks=nfs://%IP_or_FQDN%/ks.cfg
The installation will start and use the kickstart configuration file (ks.cfg).
After the installation is complete the ESXi Host will reboot.
2018-10-04 – This article has been updated.
2018-11-16 – Code blocks were not displaying correctly.
This week VMware released vSphere Integrated Containers (VIC) version 1.1. Below are the product highlights and a small introduction to the new product.
Information about VMware Integrated Containers
vSphere Integrated Containers comprises three components:
VMware vSphere Integrated Containers Engine, a container runtime for vSphere that allows developers who are familiar with Docker to develop in containers and deploy them alongside traditional VM-based workloads on vSphere clusters. vSphere administrators can manage these workloads by using vSphere in a way that is familiar.
VMware vSphere Integrated Containers Registry, an enterprise-class container registry server that stores and distributes container images. vSphere Integrated Containers Registry extends the Docker Distribution open source project by adding the functionalities that an enterprise requires, such as security, identity and management.
VMware vSphere Integrated Containers Management Portal, a container management portal that provides a UI for DevOps teams to provision and manage containers, including retrieving stats and info about container instances. Cloud administrators can manage container hosts and apply governance to their usage, including capacity quotas and approval workflows. When integrated with vRealize Automation, more advanced capabilities become available, such as deployment blueprints and enterprise-grade Containers-as-a-Service.
With these capabilities, vSphere Integrated Containers enables VMware customers to deliver a production-ready container solution to their developers and DevOps teams. By leveraging their existing SDDC, customers can run container-based applications alongside existing virtual machine based workloads in production without having to build out a separate, specialized container infrastructure stack. As an added benefit for customers and partners, vSphere Integrated Containers is modular. So, for example, if your organization already has a container registry in production, you can use that registry with vSphere Integrated Containers Engine and vSphere Integrated Containers Management Portal.
A unified OVA installer for all three components
Upgrade from version 1.0
Official support for vSphere Integrated Containers Management Portal
A unified UI for vSphere Integrated Containers Registry and vSphere Integrated Containers Management Portal
A plug-in for the HTML5 vSphere Client
Support for Docker Client 1.13 and Docker API version 1.25
Support for using Notary with vSphere Integrated Containers Registry
Support for additional Docker commands. For the list of Docker commands that this release supports, see Supported Docker Commands in Developing Container Applications with vSphere Integrated Containers.