Home » ITQ

Tag: ITQ

vSphere 6.7 Convergence Tool: Failed to get vecs users and permissions

Last week I was converting a vSphere 6.7 Update 1 environment from external PSC to embedded PSC. After a couple of seconds running the conversion, it ended in an error message (Failed to get vecs users and permissions).

The customer was using the latest available vCenter 6.7 update 1 release available at this point vCenter Appliance 6.7 U1b (11727113). The environment consists of one Platform Services Controller (PSC) and one vCenter Server (VC) and a couple of VMware ESXi 6.7 Update 1 hosts.

Error Message

The error message in my PowerShell window displayed the following error message. Not really the best message (possible resolution is []) but it pointed me in the right direction.

### PowerShell output from vcsa-util.exe
2019-05-07 11:07:58,538 [loggable.py:102]: ================ [FAILED] Task: MonitorPSCDeployTask: Running MonitorPSCDeployTask execution failed at 11:07:58 ================
2019-05-07 11:07:58,553 [loggable.py:102]: Task 'MonitorPSCDeployTask: Running MonitorPSCDeployTask' execution failed because [ERROR: Converge Process Failed!], possible resolution is []
2019-05-07 11:07:58,553 [loggable.py:102]: ================================================================================
2019-05-07 11:07:58,631 [taskflow.py:943]: <MonitorPSCDeployTask - com.vmware.vcsa.installer.converge.monitor_psc_deploy(FAILED)> in <ConvergeTaskFlow - converge(FAILED)> status changed to: FAILED
2019-05-07 11:07:58,694 [taskflow.py:641]: Execution attempt 1 for Task <MonitorPSCDeployTask - com.vmware.vcsa.installer.converge.monitor_psc_deploy(FAILED)> FAILED with exception: ERROR: Converge Process Failed!
2019-05-07 11:07:58,694 [taskflow.py:672]: Finished executing <MonitorPSCDeployTask - com.vmware.vcsa.installer.converge.monitor_psc_deploy(FAILED)> and its status is FAILED
2019-05-07 11:07:58,694 [taskflow.py:675]: <ConvergeTaskFlow - converge(FAILED)> overall status is now FAILED

Inside the “converge_mgmt.log” logfile the following error was displayed see output below. The log file can be found on the following location on your local system: “C:\Users\User\AppData\Local\Temp\vcsaCliInstaller-2019-05-07-11-25-6pn5b67r\workflow_1557228307149\converge\converge_mgmt.log“. Keep in mind, the file path is dynamic and I was using Microsoft Windows.

2019-05-07T11:07:46.688Z ERROR converge Failed to get vecs users and permissions. Error: {
    "componentKey": null,
    "problemId": null,
    "detail": [
            "id": "install.ciscommon.command.errinvoke",
            "localized": "An error occurred while invoking external command : 'Command: ['/usr/lib/vmware-vmafd/bin/vecs-cli', 'entry', 'getcert', '--store', 'APPLMGMT_PASSWORD', '--alias', 'location_password_default', '--output', '/root/velma/old_certs/APPLMGMT_PASSWORD.crt']\nStderr: Error: No certificates were found for entry [location_password_default] of type [Secret Key].\nvecs-cli failed. Error 87: Operation failed with error ERROR_INVALID_PARAMETER (87) \n'",
            "translatable": "An error occurred while invoking external command : '%(0)s'",
            "args": [
                "Command: ['/usr/lib/vmware-vmafd/bin/vecs-cli', 'entry', 'getcert', '--store', 'APPLMGMT_PASSWORD', '--alias', 'location_password_default', '--output', '/root/velma/old_certs/APPLMGMT_PASSWORD.crt']\nStderr: Error: No certificates were found for entry [location_password_default] of type [Secret Key].\nvecs-cli failed. Error 87: Operation failed with error ERROR_INVALID_PARAMETER (87) \n"
    "resolution": null
2019-05-07T11:07:46.706Z INFO converge Cleanup successful with partial flag = True.

Solving the issue

After searching on Google on the string “ERROR converge Failed to get vecs users and permissions“. I got a hit on a VMware KB article. The VMware article can be found below and explained what was going wrong.

The solution is very simple… remove the vCenter Backup Schedule in the VAMI (VMware Appliance Management Interface):


  1. Log into the vCenter Server Appliance Management Interface (https://%vcenter-fqdn%:5480)
  2. Login with the root account.
  3. Navigate to the Backup view
  4. Next to Backup Schedule, click the Delete button to delete the current backup schedule
  5. Attempt the convergence process again!
  6. Once the convergence is complete, re-create the backup schedule. See Schedule a File-Based Backup for more information on creating a backup schedule.

Community Feedback

I got the following feedback on this article after publishing:

  • Update 08-04-2019: David Stamen reached out to me on Twitter with the response: This was fixed in #vSphere67U2.


The following websites were very usefull for troubleshooting this issue:

Share this:

No Workflow Output in vRealize Orchestrator (vRO) 7.6

About a year ago I wrote a blog about the following issue: No Workflow Output in vRealize Orchestrator (vRO) 7.4. Apparently, after the vRealize Orchestrator (vRO) 7.6 upgrade the issue has returned.

I will not go into full detail in this blog post. Just a basic instruction on how to resolve the issue in vRealize Orchestrator 7.6. More background information can be found over here with additional screenshots.

Note: vRealize Orchestrator 7.6 was released on 11-04-2019 and can be downloaded over here. The vRealize Orchestrator 7.6 release notes can be found over here.

No Workflow Output

Let’s start with a small introduction to the issue. After an upgrade from vRealize Orchestrator 7.5 to vRealize Orchestrator 7.6 the (legacy client) is not able to show any workflow logs after executing a workflow run.

To make it perfectly clear… the workflow is executed and is working fine. The logs are just not displayed in the vRealize Orchestrator Client.

Here is an example, the workflow has been executed and it should output information but the logs tab is empty. Keep in mind: this image is from a vRO 7.4 instance but looks identical to vRO 7.6.

vRealize Orchestrator 7.4 - No log output
vRealize Orchestrator 7.6 – No Workflow Logs

Restore the workflow output

Here are the commands for resolving the issue in vRealize Orchestrator ( vRO) 7.6 . The fix can be applied in under 10 minutes by a system administrator.

Before removing the files it is good practice to make sure you have a backup or virtual machine snapshot of your vRO appliance.

### Step 01: Start an SSH session with the vRealize Orchestrator Appliance (use for example Putty).

### Step 02: Login with root credentials

### Step 03: When you run the following command multiple files will be shown:
ls -l /var/log/vco/app-server/scripting.log_lucene*

### Step 04: Stop the Orchestrator service
service vco-server stop

### Step 05: Remove the log files
rm -rf /var/log/vco/app-server/scripting.log_lucene* 

### Step 06: Start the Orchestrator service
service vco-server start

### Step 07: Open the vRealize Orchestrator Client

### Step 08: Execute a workflow and logging should be working again.

I have tested it so far on two vRealize Orchestrator 7.6 appliances that were upgraded from vRealize Orchestrator 7.5. In both cases the upgrade was successful but the workflow output was not working.

There might be some vRO 7.5 to vRO 7.6 upgrades that will work without issues… like an Orchestrator that is just sitting idle or maybe a clean install that is directly upgraded from 7.5 to 7.6?

If you got any comments or tips please respond below!

Share this:

PowervRA: Invoke-RestMethod The underlying connection was closed.

At a customer, I encountered the following issue when trying to connect with PowervRA to vRealize Automation. The error message that appeared was: Invoke-RestMethod : The underlying connection was closed: An unexpected error occurred on a send.

Let go one step back: So what is PowervRA you might ask? PowervRA is a PowerShell Toolkit to manage VMware vRealize Automation (vRA). With PowervRA you can configure and manage your vRealize Automation environment, for example, create a new tenant, assigning permissions or viewing the user’s requests.

The problem

The problem started by connecting with PowervRA to vRealize Automation (vRA). There was no way to get a successful connection. I tried using the IP addresses, hostname and FQDN also different credentials didn’t make any difference. The error that returned in all cases was identical.

The customer was using the latest version of PowervRA. At this moment it was PowervRA 3.5.0. The vRealize Automation version they were using was 7.4.0.

Here is the screenshot of the error message:

PowervRA - Invoke-RestMethod : The underlying connection was closed
PowervRA – Invoke-RestMethod : The underlying connection was closed – Issue

Here is the full error message in plain text from the PowerShell Console:

Error message:
Invoke-RestMethod : The underlying connection was closed: An unexpected error occurred on a send.
At C:\Program Files\WindowsPowerShell\Modules\PowervRA\3.5.0\PowervRA.psm1:510 char:21
+         $Response = Invoke-RestMethod @Params
+                     ~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-RestMethod], WebException
    + FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeRestMethodCommand

To validate the issue further I tried the same thing in my Lab environment. The strange thing was that everything was working fine with the identical versions.

The solution

Until this moment I am not really sure why it is working in one environment and not in the other… I suspect it has something to do with Windows Updates or Domain Security Policies? To address the issue there is only one way: force PowerShell/PowervRA to use TLS 1.2 when connecting with vRealize Automation (vRA).


  1. Open the PowerShell command-prompt as administrator.
  2. Run the following command before connecting to vRealize Automation. The command is listed below. No output is expected after running this command.
  3. Run the Connect-vRAServer PowerShell command to start a session with vRealize Automation. Everything should be working and authentication should be possible.

PowerShell code

Copy and paste the code into your PowerShell console before connecting to vRealize Automation:

[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12


Here is a screenshot after the TLS 1.2 has been forced:

PowervRA - Invoke-RestMethod : The underlying connection was closed
PowervRA – Invoke-RestMethod : The underlying connection was closed – Solved


Here is the official GitHub page related to PowervRA:

Share this:

vRealize Automation – Failed to convert external resource %VM-Name%.

I ran into an error message today with vRealize Automation (vRA). The error message that came up was: Failed to convert external resource Prod-Fin-00012. The issue occurred in vRA version 7.3.1.

Inside the vRealize Automation portal, I tried to upgrade virtual machine hardware but it failed directly when issuing the request. Strange thing was it was working a couple of day ago. After some investigating the error also came back on other day-2 tasks. So it was time to dive deeper into the issue.

Here is a screenshot of the issue:

The Cause

So let us think about what vRealize Automation is performing, it is executing a task on a virtual machine. To perform this it needs to talk to vCenter Server and to talk to vCenter Server it uses vRealize Orchestrator.

Here is a simple overview of the communication that happens in this case. vRealize Automation is communicating to vRealize Orchestrator and vRealize Orchestrator is communicating to vCenter Server.

VMware vRealize Automation - vSphere Endpoint Communication
VMware vRealize Automation – vSphere Endpoint Communication

Error messages

The following error messages were found on the following systems:

vRealize Automation error message:

Error message: Failed to convert external resource Prod-Fin-00012.
Script action com.vmware.vcac.asd.mappings/mapToVCVM failed.

vRealize Orchestrator error message:

https://LAB-VC-A.Lab.local:443/sdk (unusable: java.lang.ClassCastException: com.vmware.vcac.authentication.http.spring.oauth2.OAuthToken cannot be cast to com.vmware.vim.sso.client.SamlToken)

As you can see here vRealize Orchestrator has communication issues with VMware vCenter Server. This issue needs to be addressed for vRealize Automation.


The Solution

After finding the vRealize Orchestrator vSphere endpoints in an error state it was clear that this was the issue. vRealize Orchestrator is not successfully communicating with vCenter Server so this needs to be addressed.


  1. Open the vRealize Orchestrator Client (https://%vro-node-fqdn%).
  2. Login with administrative credentials (example: administrator@vsphere.local).
  3. Navigate to the following location “Library > vCenter > Configuration“.
  4. Run the following workflow “Remove a vCenter Server instance” (screenshot 01 & screenshot 02).
  5. Run the following workflow “Add a vCenter Server instance” (screenshot 03 & screenshot 04).
  6. Validate the vRealize Orchestrator Endpoint Status (screenshots 05).
  7. Run the item in vRealize Automation again.
  8. Everything should be working again.


Share this:

vSAN: PBM error occurred during PreCloneCheckCallback

Lately, I encountered some issues related to VMware vSAN in my Lab environment. The error message that was popping up all the time was “PBM error occurred during PreCloneCheckCallback“.

So how did the problem occur? First, we start with some background information. My Lab environment is powered-on when needed and powered-off when not needed. This is, of course, a little bit different than a production 24×7 environment that you have in your datacenters worldwide.

The environment was booted successfully at first glance. We are talking about Domain Controllers, vCenter Server, VMware NSX-V, nested ESXi Hosts and vRealize Automation. When I started deploying virtual machines with a vRealize Automation (vRA) based on blueprints with vSphere Templates issues started to occur.

vRealize Automation was failing on the provisioning task and was cleaning up the deployment because of the failed state (default behavior). So it was time to dig into the underlying infrastructure.


When the issue occurred the following software versions were used in my lab environment:

  • VMware vCenter 6.5 Update 2B
  • VMware vRealize Automation 7.3.1
  • VMware ESXi 6.5 Update 2
  • VMware vSAN 6.6

Error message(s)

Here is all the information that can be found in various locations surrounding the issue.

Error message: Screenshots

Here are the screenshots, the first one is from VMware vCenter and the second one is from vRealize Automation. As you can see there is clearly a problem.

Error message: vRealize Automation

Here is the vRealize Automation log entry related to the VMware vSAN issue:

Error in Execute DynamicOps.Common.Client.HtmlResponseException: Service Unavailable (503)

Error message: vCenter Server

Here is the VMware vCenter log entry related to the VMware vSAN issue:

A general system error occurred - PBM error occurred during PreCloneCheckCallback (2118557)


The solution is quick but is more like a quick fix because it comes back every time I start up my lab environment.


  • Open a web browser.
  • Navigate to your vCenter Server URL (https://%vc%/vsphere-client).
  • Login with a user that has administrator credentials (administrator@vsphere.local).
  • Navigate to Hosts & Clusters > Select the vCenter Object.
  • Click on the Configure tab.
  • Click on the Storage Providers.
  • Click on the following two buttons:
    • Synchronizes all Storage Providers with the current state of the environment.
    • Rescan the storage provider for new storage systems and storage capabilities.
  • After pressing the buttons, you don’t see any tasks running on the vCenter Server (expected behavior). After 5 seconds everything should be working and provisioning should be possible.

Share this:

HPE Storage Controller Management (ssacli)

This time I decided to do a blog post about the HPE Smart Array RAID controllers with their wonderful ssacli tool. The tooling of HPE is very powerful because you can online manage a VMware ESXi host and migrate for example from a RAID 1 volume to a RAID 10 without downtime or change the read and write cache ratio.

So far as I know I haven’t seen an identical tool yet from the other server hardware vendors like Cisco, Dell EMC, IBM, and Supermicro. The main difference has always been that the HPE tool can perform the operation live without downtime. 

So far as I can remember it has been there for ages. It was already available for VMware ESX 4.0 and is still available in VMware ESXi 6.7. So thumbs-up for HPE :).

Let’s talk about controller support. The tool supports the most HPE SmartArray controllers over the last 10 to 15 years, for example, the Smart Array P400 was released in 2005 and is still working fine today.

Here is an overview of supported controllers:

  • HPE Smart Array P2XX
  • HPE Smart Array P4XX
  • HPE Smart Array P7XX
  • HPE Smart Array P8XX

HPE SSACLI – Location

In case you are using the HPE VMware ESXi custom images. The tool is already pre-installed when installing ESXi. The tool is installed as a VIB (vSphere Installable Bundle). This means it can also be updated with vSphere Update Manager.

Over the years the name of the HPE Storage Controller Tool has been changed and so has the location. Here is a list of locations that have been used for the last ten years for VMware ESXi:

# Location VMware ESXi 4.0/4.1/5.0

# Location VMware ESXi 5.1/5.5/6.0

# Location VMware ESXi 6.5/6.7

HPE SSACLI – Examples

I have collected some screenshots over the years. Screenshots were taken by doing maintenance on VMware ESXi servers. The give you an idea what valuable information can be shown.

HPE SSACLI – Abréviation

All commands have a short name to reduce the length of the total input provided to the ssacli tool:

### Shortnames:
- chassisname = ch
- controller = ctrl 
- logicaldrive = ld
- physicaldrive = pd 
- drivewritecache = dwc
- licensekey = lk

### Specify drives:
- A range of drives (one to three): 1E:1:1-1E:1:3
- Drives that are unassigned: allunassigned


To view the status of the controller, disks or volumes you can run all sorts of commands to get information about what is going on in your VMware ESXi server. The extensive detail is very useful for troubleshooting and gathering information about the system.

# Show - Controller Slot 1 Basic configuration
./ssacli ctrl slot=1 show config

# Show - Controller Slot 1 Detailed configuration
./ssacli ctrl slot=1 show config detail

# Show - Controller Slot 1 Status
./ssacli ctrl slot=1 show status

# Show - All Controllers Configuration
./ssacli ctrl all show config

# Show - Controller slot 1 logical drive 1 status
./ssacli ctrl slot=1 ld 1 show status

# Show - Basic Physical Disks status
./ssacli ctrl slot=1 pd all show status

# Show - Detailed Physical Disk status
./ssacli ctrl slot=1 pd all show status

HPE SSACLI – Creating

Creating a new logical drive can be done online with the HPE Smart Array controllers. I have displayed some basic examples.

# Create - New single disk volume
./ssacli ctrl slot=1 create type=ld drives=2I:0:8 raid=0 forced

# Create - New spare disk (two defined)
./ssacli ctrl slot=1 array all add spares=2I:1:6,2I:1:7

# Create - New RAID 1 volume
./ssacli ctrl slot=1 create type=ld drives=1I:0:1,1I:0:2 raid=1 forced

# Create - New RAID 5 volume
./ssacli ctrl slot=1 create type=ld drives=1I:0:1,1I:0:2,1I:0:3 raid=5 forced

HPE SSACLI – Adding drives to logical drive

Adding drives to an already created logical drive is possible with the following commands. You need to perform two actions: adding the drive(s) and expanding the logical drive. Keep in mind: make a backup before performing the procedure.

# Add - All unassigned drives to logical drive 1
./ssacli ctrl slot=1 ld 1 add drives=allunassigned

# Modify - Extend logical drive 2 size to maximum (must be run with the "forced" flag)
./ssacli ctrl slot=1 ld 2 modify size=max forced

HPE SSACLI – Rescan controller

To issue a controller rescan, you can run the following command. This can be interesting for when you add new drives in hot swap bays.

### Rescan all controllers
./ssacli rescan

HPE SSACLI – Drive Led Status

The LED status of the drives can also be controlled by the ssacli utility. An example is displayed below how to enable and disable a LED.

# Led - Activate LEDs on logical drive 2 disks
./ssacli ctrl slot=1 ld 2 modify led=on

# Led - Deactivate LEDs on logical drive 2 disks
./ssacli ctrl slot=1 ld 2 modify led=off

# Led - Activate LED on physical drive
ctrl slot=0 pd 1I:0:1 modify led=on

# Led - Deactivate LED on physical drive
ctrl slot=0 pd 1I:0:1 modify led=off

HPE SSACLI – Modify Cache Ratio

Modify the cache ratio on a running system can be interesting for troubleshooting and performance beanchmarking.

# Show - Cache Ratio Status
./ssacli ctrl slot=1 modify cacheratio=?

# Modify - Cache Ratio read: 50% / write: 50%
./ssacli ctrl slot=1 modify cacheratio=50/50

# Modify - Cache Ratio read: 0% / Write: 100%
./ssacli ctrl slot=1 modify cacheratio=0/100

HPE SSACLI – Modify Write Cache

Changing the write cache settings on the storage controller can be done with the following commands:

# Show - Write Cache Status
./ssacli ctrl slot=1 modify dwc=?

# Modify - Enable Write Cache on controller
./ssacli ctrl slot=1 modify dwc=enable forced

# Modify - Disable Write Cache on controller
./ssacli ctrl slot=1 modify dwc=disable forced

# Show - Write Cache Logicaldrive Status
./ssacli ctrl slot=1 logicaldrive 1 modify aa=?

# Modify - Enable Write Cache on Logicaldrive 1
./ssacli ctrl slot=1 logicaldrive 1 modify aa=enable

# Modify - Disable Write Cache on Logicaldrive 1
./ssacli ctrl slot=1 logicaldrive 1 modify aa=disable

HPE SSACLI – Modify Rebuild Priority

Viewing or changing the rebuild priority can be done on the fly. Even when the rebuild is already active. Used it myself a couple of times to lower the impact on production.

# Show - Rebuild Priority Status
./ssacli ctrl slot=1 modify rp=?

# Modify - Set rebuildpriority to Low
./ssacli ctrl slot=1 modify rebuildpriority=low

# Modify - Set rebuildpriority to Medium
./ssacli ctrl slot=1 modify rebuildpriority=medium

# Modify - Set rebuildpriority to High
./ssacli ctrl slot=1 modify rebuildpriority=high

HPE SSACLI – Modify SSD Smart Path

You can modify the HPE SDD Smart Path feature by disabling or enabling. To make clear what the HPE SDD Smart Path includes, here is a official statement by HPE: 

“HP SmartCache feature is a controller-based read and write caching solution that caches the most frequently accessed data (“hot” data) onto lower latency SSDs to dynamically accelerate application workloads. This can be implemented on direct-attached storage and SAN storage.”

For example, when running VMware vSAN SSD Smart Path must be disabled for better performance. In some cases worse the entire vSAN disk group fails.

# Modify - Enable SSD Smart Path
./ssacli ctrl slot=1 array a modify ssdsmartpath=enable

# Modify - Disable SSD Smart Path
./ssacli ctrl slot=1 array a modify ssdsmartpath=disable

HPE SSACLI – Delete Logical Drive

Deleting a logical drive on the HPE Smart Array controller can be done with the following commands.

# Delete - Logical Drive 1
./ssacli ctrl slot=1 ld 1 delete

# Delete - Logical Drive 2
./ssacli ctrl slot=1 ld 2 delete

HPE SSACLI – Erasing Physical Drives

In some cases, you need to erase a physical drive. This can be performed with multiple erasing options. Also, you can stop the process.

Erase patterns available:

  • Default
  • Zero
  • Random_zero
  • Random_random_zero
# Erase physical drive with default erasepattern
./ssacli ctrl slot=1 pd 2I:1:1 modify erase

# Erase physical drive with zero erasepattern
./ssacli ctrl slot=1 pd 2I:1:1 modify erase erasepattern=zero

# Erase physical drive with random zero erasepattern
./ssacli ctrl slot=1 pd 1E:1:1-1E:1:3 modify erase erasepattern=random_zero

# Erase physical drive with random random zero erasepattern
./ssacli ctrl slot=1 pd 1E:1:1-1E:1:3 modify erase erasepattern=random_random_zero

# Stop the erasing process on phsyical drive 1E:1:1
./ssacli ctrl slot=1 pd 1E:1:1 modify stoperase

HPE SSACLI – License key

In some cases a licence key needs to be installed on the SmartArray storage controller to enable the advanced features. This can be done with the following command:

# License key installation
./ssacli ctrl slot=1 licensekey XXXXX-XXXXX-XXXXX-XXXXX-XXXXX

# License key removal
./ssacli ctrl slot=5 lk XXXXXXXXXXXXXXXXXXXXXXXXX delete 

Related sources

A couple of interesting links related to the HPE Storage Controller tool for VMware ESXi:

Share this:

Changing VMware Storage Controller to Paravirtual for CentOS 7

In this post, we are going to change the Virtual Storage Controller from LSI Logic Parallel to VMware Paravirtual for a CentOS 7 based Virtual Machine that is running on VMware vSphere. This blog post will contain step by step guidance for performing the operation.

In my case the virtual machine was build in VMware Workstation and after some time migrated to VMware ESXi. The VMware Paravirtual Storage Controller is not supported in VMware Workstation. That is why the virtual machine came over with the “wrong” storage controller.

My 24×7 Lab environment is running shared iSCSI based storage and all virtual machines are thin provisioned. The Virtual Machine that came over from VMware Workstation is installed with CentOS 7.

Why VMware Paravirtual?

Why should you want to migrate from an LSI Logic Parallel to a VMware Paravirtual SCSI Controller? Two simple reasons and it are two good ones:

  • Lower CPU utilization
  • Higher Throughput

Personally, I have a third reason to add… compliance. All my virtual machines should be compliant with the VMware Best Practice and my personal Home Lab standard. In my Lab environment, this means using the VMware Paravirtual where ever possible/supported.

VMware Official Statement 1:

PVSCSI adapters are high-performance storage adapters that can result in greater throughput and lower CPU utilization. PVSCSI adapters are best for environments, especially SAN environments, where hardware or applications drive a very high amount of I/O throughput. The VMware PVSCSI adapter driver is also compatible with the Windows Storport storage driver. PVSCSI adapters are not suitable for DAS environments.VMware Paravirtual SCSI adapters are high-performance storage adapters that can result in greater throughput and lower CPU utilization.

VMware Official Statement 2:

The PVSCSI adapter offers a significant reduction in CPU utilization as well as potentially increased throughput compared to the default virtual storage adapters, and is thus the best choice for environments with very I/O-intensive guest applications.


The most important step in the process is to make sure you have a valid backup! After that, it is just following the steps described below:

  • Create a virtual machine snapshot or backup before you begin.
  • Power-off the virtual machine.
  • Add the VMware Paravirtual Controller to the Virtual Machine. Do not change the disk controller assignment yet, only add the storage controller to the VM (screenshot 01).
  • Power-on the virtual machine.
  • Login with an account on the virtual machine (account must be able to obtain root access).
  • Start rebuilding the initial ramdisk image (screenshot 02):
    mkinitrd -f -v /boot/initramfs-$(uname -r).img $(uname -r)
  • Power-off the virtual machine.
  • Assign disks to the new storage controller and remove the old storage controller (screenshot 03).
  • Power-on the virtual machine.
  • Validate that everything is working and disks are mounted (screenshot 04).
  • Remove the virtual machine snapshot or backup after you are done.



At this point, I have swapped out three virtual machines from the LSI controller to the VMware Paravirtual SCSI Controller. The machines have been running now for about two weeks without any problems. So everything is compliant again ;).

If you encounter any problems or have any question about this subject please feel free to contact me on Twitter or the Reply option below.


Here are some interesting related articles that I used for creating this blog post:

Share this:

Deployment of VMware vCenter Server 6.7 Update 1

In this blog post, we are going to deploy VMware vCenter 6.7 Update 1 in my Lab environment. The deployment is fully covered with all the additional notes required to perform a successful installation, migration or upgrade. I also added some guidelines for designing your environment.

Now that vSphere 6.7 Update 1 is available since the announcement on VMworld 2018 US, it is a good time to start looking at vSphere 6.7 instead of vSphere 6.5.

Why should you look at vSphere 6.7 you might ask? vSphere 6.5 is still running like a charm! Yes you are correct but… there are a couple of items to consider:

If you are familiar with the VMware vCenter 6.5 graphical deployment it has been improved in VMware vCenter 6.7. In the past it was a web-based wizard, with 6.7 it is a binary executable. This means a lot faster and better-responding interface and it removes the browser dependency and browser plug-in on your workstation.


The checklist items can be verified, days or hours before the initial deployment. If you don’t have a plan before installing, migrating or upgrading things will turn out ugly…

With the checklist, you can determine if your environment is ready for vSphere 6.7 Update 1. It’s about checking and validation your current software and hardware and talking to your vendors about compatibility.

I have also added some design decision ideas. Because you can choose to install, upgrade or migrate without looking at your current architecture but maybe it is time to update your current architecture (design).

  • Make sure that all connected/used VMware products are compatible like (vRealize Automation / vRealize Orchestrator / vRealize Operations Manager / VMware Horizon and the list goes on). This can be verified on the VMware Product Interoperability Matrices page.
  • Make sure that all third-party products are compatible like (Backup & Replication software / Storage vendor software).
  • Determine the correct sizing for your environment. How many virtual machines and ESXi Hosts are going to be running underneath this vCenter Server. These figures determine your vCenter Server size.
  • An embedded Platform Services Controller (PSC) is the way to go in the future. An external Platform Services Controller will not be available anymore in the future.
  • Where is Windows? Please read this article from more than one year ago. Please do not deploy a vCenter Server on Windows. This is a thing of the past.
  • Verify the Hardware Requirements for the vCenter Appliance (depending on your chosen size and internal or external PSC).
  • Do you deploy against an ESXi Host or a vCenter Server?
  • Is your ESXi host hardware compatible with vSphere 6.7 Update 1?


The checklist is completed and you have determined that everything is working or is acceptable to continue. Let’s start with some basic stuff that is required:

  • Read the release notes (VMware vCenter Server 6.7 Update 1 Release Notes).
  • Download the latest release from the VMware website.
  • Create firewall rules for your new vCenter Server.
  • Create forward and reverse DNS records in your DNS Server.
  • Register your IP information in your IPAM system.
  • Save your passwords in your Password Management system (Appliance password / SSO password).
  • Have a workstation ready to perform the deployment with sufficient network access and administrative rights.


Let’s start the deployment of VMware vCenter 6.7 Update 1. I have chosen for a clean installation of VMware vCenter 6.7 Update 1. I have chosen for an embedded Platform Services Controller (PSC). Based on my total amount of virtual machines and ESXi Hosts I have selected a “Small” installation footprint.

The new deployment process for vCenter Server 6.7 Update 1 consists out of two stages, one is the deployment stage and one is the setup stage.

The first part is mainly responsible for delivering the full appliance with the operation system, network settings, and installation application bundles. The second part is configuring the applications that are running on the vCenter Server. A total installation takes about 45 minutes to complete.


  1. Mount the vCenter Server media (iso file).
  2. Navigate to the following path “X:\vcsa-ui-installer\win32\” (X stands for the CD-ROM drive label).
  3. Run the following application “installer.exe“.
  4. Follow the wizard, I have uploadedall screenshots for reference.

Stage 01 – Deployment

Here are the images of the first stage of the deployment of VMware vCenter 6.7 Update 1. I have no issues to report everything was working fine on the first try!

Stage 02 – Setup

Here are the images of the second stage of the deployment of VMware vCenter 6.7 Update 1. This part was also bug-free, so it was a good deployment.


After a successful deployment of the VCSA appliance, you need to configure at least some items to get vCenter Server production ready. The items listed below are a basic set of the most common items I see in the field:

  • Install the vCenter Server License.
  • Active Directory.
  • Assign rights & permissions.
  • Generate and installation of SSL Certificates.
  • Connect the required VMware products and third-party systems.
  • Create a datacenter object.
  • Create a cluster object.
  • Create a distributed switch.
  • Join ESXi Hosts to the newly created cluster.

Lab Environment

So how does my Lab environment look like?

  • VMware vCenter 6.5.0 Update 2. The target for the vCenter 6.7.0 Update 1 deployment.
  • VMware ESXi 6.5.0 Update 2 in the 24×7 environment. Known as the production cluster.
  • VMware ESXi 6.5.0 Update 2 in the Lab environment. Known as the lab cluster.

You might ask… why don’t you upgrade the current vCenter Server? Good question! The machine has been converted/upgraded multiple times. It started out in life as a VMware vCenter 5.5 machine, that was on the Windows Server 2012 platform. So it was a good moment to start clean after this many years.

Share this:

Upgrade to vRealize Orchestrator 7.5

How to upgrade vRealize Orchestrator 7.4 to vRealize Orchestrator 7.5 or to be more precise… migrate! Since the release of vRealize Orchestrator 7.5 a couple of weeks ago the update/upgrade option with the vamicli and appliance management interface is not available (to my surprise).

A quick introduction to how I got in that situation: In my Home Lab my main vRealize Orchestrator appliance is running version 7.4.0 and is responsible for some day-to-day Orchestration of my multiple environments.

Here is the official VMware statement about the upgrade option in vRealize Orchestrator 7.5. It can be found in the vRealize Orchestrator 7.5 release notes:

You can no longer upgrade to the Orchestrator appliance 7.5, you can only migrate to it.

vRealize Orchestrator 7.5 – Release notes

For the people that are is still running way older versions: Upgrading vRealize Orchestrator Appliance from version 5.5.x to 7.5 is not supported. You must upgrade your vRealize Orchestrator Appliance 5.5.x to 6.0.x first and then migrate to 7.5.


The following components were running in my environment and have been tested. Note: this part of my Lab environment is not running vRealize Automation. So I have not tested the migration with external vRO nodes in combination with vRealize Automation.

  • A single vCenter Server 6.5.0 Update 2 (with an embedded PSC)
  • A single vRealize Orchestrator 7.4.0 (external)


There are two options available. The first option is moving all data between the old and new vRealize Orchestrator. The second option is to migrate vRealize Orchestrator with the migration wizard. The second option is the one VMware recommends. The first option can be easier in some cases, some advantages are you retain your IP address, hostname and SSL certificates.

Both options are written down on this page.

vRO export data and redeploy

I have chosen for this scenario because this machine is only connected to a vCenter Server and can be reastablished very fast. Another reason is that the current vRO instance has been running since version vRO 7.0 and has been upgraded more than seven times in about 2.5 years. So a new clean install ain’t a bad thing!


  • Create a package in the vRealize Orchestrator Client with all your created workflows, actions and resource elements.
  • Save the package on a save place.
  • Remove the registration from vCenter Server (if they are connected). Workflows “Unregister a vCenter Server extension” & “Remove a vCenter Server instance“.
  • Poweroff the current vRO appliance.
  • Rename the appliance to %vm-name%.old (for example).
  • Deploy a new vRealize Orchestrator Appliance on the same IP address and FQDN.
  • Upgrade the virtual hardware.
  • Walkthrough the vRealize Orchestrator configuration wizard.
  • (Optional) install the SSL certificates.
  • Import the package.
  • Register with vCenter Server. Workflows “Add a vCenter Server instance” & “Register vCenter Orchestrator as a vCenter Server Extension“.


vRealize Orchestrator 7.5 - Old and New
The old and new vRealize Orchestrator appliances inside the vSphere Web Client.

vRO Migration

The migration path is performed in the following way (the official documentation is extensive, the link is listed below). The migration is a good option for an Orchestrator that is connected to a lot of extensibility and has a lot of plugins installed. The biggest issue for me was the new IP address, FQDN and new certificates required.

Note: Migrations with vRealize Orchestrator Clusters are not described here. There are a couple of small items you need to check in the migration manual.


  • Your source Orchestrator is running at least version 6.X.
  • Make sure no workflows are running.
  • Stop the Orchestrator services on the source Orchestrator.
  • Make sure SSH is enabled on both the source and destination Orchestrator.
  • Make sure no firewall is blocking traffic for the migration.
  • Create backups from the source and destination Orchestrator.


  • Register a new vRealize Orchestrator appliance in your IPAM solution.
  • Deploy a new vRealize Orchestrator next to the currently running.
  • Upgrade Virtual Machine Hardware
  • Power-on the vRealize Orchestrator appliance.
  • (Optional) Install new SSL Certificates.
  • Navigate with a browser to the Appliance Management interface (https://%FQDN%:5480).
  • Login with the root credentials.
  • Navigate to the migrate tab.
  • Insert the required information.
  • Start the validate and start the migration.

Source: Migrating vRealize Orchestrator – vRealize Orchestrator 7.5


The Migrate Tab in the Appliance Management Interface
The Migrate Tab in the Appliance Management Interface


Do not forget to configure and/or check the following items after a upgrade or migration:

  • Timezone configuration (Appliance Mangement Interface)
  • NTP server configuration (Appliance Management Interface)

Share this:

VMworld 2018 EU Announcements

This year VMworld 2018 Europe is held in Barcelona Spain from 5 November until 8 November. This has been the location for some time now. Only the date shifted a couple months backward compared to last year (September > November). This page will be updated multiple time in the coming days. I will add additional information and announcements multiple times a day.

I was able to attend myself in person thanks to my employer ITQ. This year we traveled with 33 ITQ colleagues to VMworld 2018 Europe. In total 12000 people attended VMworld 2018 Europe in Barcelona.

VMworld 2018 EU Logo

VMworld 2018 EU – Product Announcements

On VMworld 2018 in Europe, the following products and/or services were announced.

VMware Cloud Foundation (VCF) 3.5

In the opening keynote VMware Cloud Foundation (VCF) 3.5 has been released.

VMware Cloud Foundation is an automated release of the Software-Defined Datacenter with vSphere, NSX, and vSAN in the core. On top VMware is using there vRealize Suite for creating an entire Cloud Management Platform.

The VCF 3.5 release includes the following new features:

  • Clarity UI as for all VMware products.
  • More intuitive end-to-end UI driven workflows
  • Improved UI responsiveness and performance
  • Dashboard with widgets to more easily surface important SDDC information
  • Updated navigation and consistency between workflows
  • Extendable platform to ensure that we can incorporate new releases and functionality (ex. New workflow wizards, dashboard widgets)
  • Updated for vSphere 6.7 U1
  • NSX-T Workload domain
  • NFS Workload domain – use cases that requires NFS
  • Composable Infrastructure Support

Source: Announcing VMware Cloud Foundation 3.5

VMware Horizon 7 on VMC

Not my area of expertise (EUC) but VMware Horizon 7 will be available on VMware Cloud on AWS because of the overwhelming interest from the customers. The information can be found in the source below.

Source: What’s New with Horizon 7 on VMware Cloud on AWS

VMware Horizon 7.7

On the second day of VMworld 2018 in Europe, VMware Horizon 7.7 is announced. No information is yet available on the new features.

Source: What’s New with Horizon 7 on VMware Cloud on AWS

VMware App Volumes 2.15

On the second day of VMworld 2018 in Europe, VMware App Volumes 2.15 has been announced. No information is yet available on the new features.

Source: What’s New with Horizon 7 on VMware Cloud on AWS

vRealize Network Insight 4.0 (vRNI)

The new release of vRealize Network Insight (vRNI) is focussed on hybrid cloud visibility and analytics.

  • Support for application migration planning to public clouds
  • Support for Hybrid Cloud Troubleshooting
  • It provides NSX-T topology visualization and correlated troubleshooting for NSX-T events.
  • Users can also get P-P flow visibility through slow support in addition to Netflow and visibility into Cisco ACI underlay and Cisco ASA firewall

Source: vRealize Network Insight 4.0: Hybrid Cloud Visibility & Analytics

VMware vCloud Suite 2018 Platinum

On the first day, vCloud Suite 2018 Platinum was announced. It is a combination of vSphere Platinum and the vRealize Suite 2018 combined.

vSphere Platinum was announced in VMworld 2018 US and is available since last week. vSphere Platinum combines VMware vSphere with VMware AppDefense.

Source: Introducing VMware vCloud Suite 2018 Platinum!

VMware Cloud on AWS Region Expansion

On the first day of VMworld 2018 Europe, Pat Gelsinger announced a Region Expansion for VMware Cloud on AWS. There will be new regions in Europe and the US.

The new regions announced are the following:

  • Europe (Ireland)
  • US West (North California)
  • US East (Ohio)

Source: VMware Cloud on AWS expands across Europe and the U.S.

Amazon Relational Database Service (AWS RDS) 

In the VMworld 2018 Europe first day keynote, the following relational database services were announced:

  • PostgreSQL
  • MySQL
  • MariaDB
  • Microsoft SQL Server
  • Oracle

Source: Not yet available

VMware accuires Heptio

On the VMworld 2018 Europe keynote, Pat Gelsinger announced that VMware acquired Heptio as of today.

Heptio was founded by two creators of the Kubernetes project to support and advance the open Kubernetes ecosystem.


Source: Breaking: VMware Announces Intent to Acquire Heptio to Drive Kubernetes Across Multiple Clouds

VMware Cloud PKS

On the VMworld 2018 Europe keynote, Pat Gelsinger announced VMware Cloud PKS. To run Kubernetes on their VMware Cloud on AWS.

Source: Try VMware Cloud PKS (Formerly VKE) today and the first $150 is on us!

VMworld 2018 EU – Photo’s

I have created a gallery with some photo’s I have taken the last couple of days on VMworld 2018 Europe.

VMworld 2018 EU – Personal Highlights

To summarize my personal highlights that I encountered on VMworld 2018 Europe.

  • Meeting with the guys behind the VMware Validated Designs (VVD) and VMware Cloud Foundation (VCF) (Forbes Guthrie and Daniel Koeck).
  • The VMware Private Hands-On Lab (HOL) Tour provided for the VMware vExperts.
  • Connecting with the vCommunity.
  • Connecting with old colleagues and customers.
  • Connecting with the people that maintain and support the VMware Hands-On Labs (Cody de Arkland / Gavin Lees).
  • My colleague Ruurd Keizer that was called on stage to talk about his experience with PKS (Pivotal Container Service) and share that with other VMware Partners.

VMworld 2018 EU – Keynotes

Here are the recordings of the VMworld 2018 Europe Keynotes and live streams for the coming days:

VMworld 2018 EU – Blogs & Recordings

In this section, I have listed some interesting blogs and recordings from VMworld 2018 Europe. The people listed below are mostly people I work with or are vCommunity friends:

VMworld 2018 EU –  Session Recordings

Just like every year, William Lam from the website virtuallyGhetto creates a GIT repository with all the VMworld sessions. For each session, a recording and presentation is provided on his Git Repository. It will probably be a couple of days till weeks until all sessions become available:

Share this: