Home » vCenter

Tag: vCenter

vSphere 6.7 Convergence Tool: Failed to get vecs users and permissions

Last week I was converting a vSphere 6.7 Update 1 environment from external PSC to embedded PSC. After a couple of seconds running the conversion, it ended in an error message (Failed to get vecs users and permissions).

The customer was using the latest available vCenter 6.7 update 1 release available at this point vCenter Appliance 6.7 U1b (11727113). The environment consists of one Platform Services Controller (PSC) and one vCenter Server (VC) and a couple of VMware ESXi 6.7 Update 1 hosts.



Error Message

The error message in my PowerShell window displayed the following error message. Not really the best message (possible resolution is []) but it pointed me in the right direction.

### PowerShell output from vcsa-util.exe
2019-05-07 11:07:58,538 [loggable.py:102]: ================ [FAILED] Task: MonitorPSCDeployTask: Running MonitorPSCDeployTask execution failed at 11:07:58 ================
2019-05-07 11:07:58,553 [loggable.py:102]: Task 'MonitorPSCDeployTask: Running MonitorPSCDeployTask' execution failed because [ERROR: Converge Process Failed!], possible resolution is []
2019-05-07 11:07:58,553 [loggable.py:102]: ================================================================================
2019-05-07 11:07:58,631 [taskflow.py:943]: <MonitorPSCDeployTask - com.vmware.vcsa.installer.converge.monitor_psc_deploy(FAILED)> in <ConvergeTaskFlow - converge(FAILED)> status changed to: FAILED
2019-05-07 11:07:58,694 [taskflow.py:641]: Execution attempt 1 for Task <MonitorPSCDeployTask - com.vmware.vcsa.installer.converge.monitor_psc_deploy(FAILED)> FAILED with exception: ERROR: Converge Process Failed!
2019-05-07 11:07:58,694 [taskflow.py:672]: Finished executing <MonitorPSCDeployTask - com.vmware.vcsa.installer.converge.monitor_psc_deploy(FAILED)> and its status is FAILED
2019-05-07 11:07:58,694 [taskflow.py:675]: <ConvergeTaskFlow - converge(FAILED)> overall status is now FAILED

Inside the “converge_mgmt.log” logfile the following error was displayed see output below. The log file can be found on the following location on your local system: “C:\Users\User\AppData\Local\Temp\vcsaCliInstaller-2019-05-07-11-25-6pn5b67r\workflow_1557228307149\converge\converge_mgmt.log“. Keep in mind, the file path is dynamic and I was using Microsoft Windows.

2019-05-07T11:07:46.688Z ERROR converge Failed to get vecs users and permissions. Error: {
    "componentKey": null,
    "problemId": null,
    "detail": [
        {
            "id": "install.ciscommon.command.errinvoke",
            "localized": "An error occurred while invoking external command : 'Command: ['/usr/lib/vmware-vmafd/bin/vecs-cli', 'entry', 'getcert', '--store', 'APPLMGMT_PASSWORD', '--alias', 'location_password_default', '--output', '/root/velma/old_certs/APPLMGMT_PASSWORD.crt']\nStderr: Error: No certificates were found for entry [location_password_default] of type [Secret Key].\nvecs-cli failed. Error 87: Operation failed with error ERROR_INVALID_PARAMETER (87) \n'",
            "translatable": "An error occurred while invoking external command : '%(0)s'",
            "args": [
                "Command: ['/usr/lib/vmware-vmafd/bin/vecs-cli', 'entry', 'getcert', '--store', 'APPLMGMT_PASSWORD', '--alias', 'location_password_default', '--output', '/root/velma/old_certs/APPLMGMT_PASSWORD.crt']\nStderr: Error: No certificates were found for entry [location_password_default] of type [Secret Key].\nvecs-cli failed. Error 87: Operation failed with error ERROR_INVALID_PARAMETER (87) \n"
            ]
        }
    ],
    "resolution": null
}
2019-05-07T11:07:46.706Z INFO converge Cleanup successful with partial flag = True.


Solving the issue

After searching on Google on the string “ERROR converge Failed to get vecs users and permissions“. I got a hit on a VMware KB article. The VMware article can be found below and explained what was going wrong.

The solution is very simple… remove the vCenter Backup Schedule in the VAMI (VMware Appliance Management Interface):

Procedure:

  1. Log into the vCenter Server Appliance Management Interface (https://%vcenter-fqdn%:5480)
  2. Login with the root account.
  3. Navigate to the Backup view
  4. Next to Backup Schedule, click the Delete button to delete the current backup schedule
  5. Attempt the convergence process again!
  6. Once the convergence is complete, re-create the backup schedule. See Schedule a File-Based Backup for more information on creating a backup schedule.

Community Feedback

I got the following feedback on this article after publishing:

  • Update 08-04-2019: David Stamen reached out to me on Twitter with the response: This was fixed in #vSphere67U2.

Sources

The following websites were very usefull for troubleshooting this issue:

Share this:

vSAN: PBM error occurred during PreCloneCheckCallback

Lately, I encountered some issues related to VMware vSAN in my Lab environment. The error message that was popping up all the time was “PBM error occurred during PreCloneCheckCallback“.

So how did the problem occur? First, we start with some background information. My Lab environment is powered-on when needed and powered-off when not needed. This is, of course, a little bit different than a production 24×7 environment that you have in your datacenters worldwide.

The environment was booted successfully at first glance. We are talking about Domain Controllers, vCenter Server, VMware NSX-V, nested ESXi Hosts and vRealize Automation. When I started deploying virtual machines with a vRealize Automation (vRA) based on blueprints with vSphere Templates issues started to occur.

vRealize Automation was failing on the provisioning task and was cleaning up the deployment because of the failed state (default behavior). So it was time to dig into the underlying infrastructure.



Environment

When the issue occurred the following software versions were used in my lab environment:

  • VMware vCenter 6.5 Update 2B
  • VMware vRealize Automation 7.3.1
  • VMware ESXi 6.5 Update 2
  • VMware vSAN 6.6

Error message(s)

Here is all the information that can be found in various locations surrounding the issue.

Error message: Screenshots

Here are the screenshots, the first one is from VMware vCenter and the second one is from vRealize Automation. As you can see there is clearly a problem.

Error message: vRealize Automation

Here is the vRealize Automation log entry related to the VMware vSAN issue:

Error in Execute DynamicOps.Common.Client.HtmlResponseException: Service Unavailable (503)

Error message: vCenter Server

Here is the VMware vCenter log entry related to the VMware vSAN issue:

A general system error occurred - PBM error occurred during PreCloneCheckCallback (2118557)

Solution

The solution is quick but is more like a quick fix because it comes back every time I start up my lab environment.

Procedure:

  • Open a web browser.
  • Navigate to your vCenter Server URL (https://%vc%/vsphere-client).
  • Login with a user that has administrator credentials (administrator@vsphere.local).
  • Navigate to Hosts & Clusters > Select the vCenter Object.
  • Click on the Configure tab.
  • Click on the Storage Providers.
  • Click on the following two buttons:
    • Synchronizes all Storage Providers with the current state of the environment.
    • Rescan the storage provider for new storage systems and storage capabilities.
  • After pressing the buttons, you don’t see any tasks running on the vCenter Server (expected behavior). After 5 seconds everything should be working and provisioning should be possible.

Share this:

Deployment of VMware vCenter Server 6.7 Update 1

In this blog post, we are going to deploy VMware vCenter 6.7 Update 1 in my Lab environment. The deployment is fully covered with all the additional notes required to perform a successful installation, migration or upgrade. I also added some guidelines for designing your environment.

Now that vSphere 6.7 Update 1 is available since the announcement on VMworld 2018 US, it is a good time to start looking at vSphere 6.7 instead of vSphere 6.5.

Why should you look at vSphere 6.7 you might ask? vSphere 6.5 is still running like a charm! Yes you are correct but… there are a couple of items to consider:

If you are familiar with the VMware vCenter 6.5 graphical deployment it has been improved in VMware vCenter 6.7. In the past it was a web-based wizard, with 6.7 it is a binary executable. This means a lot faster and better-responding interface and it removes the browser dependency and browser plug-in on your workstation.



Checklist

The checklist items can be verified, days or hours before the initial deployment. If you don’t have a plan before installing, migrating or upgrading things will turn out ugly…

With the checklist, you can determine if your environment is ready for vSphere 6.7 Update 1. It’s about checking and validation your current software and hardware and talking to your vendors about compatibility.

I have also added some design decision ideas. Because you can choose to install, upgrade or migrate without looking at your current architecture but maybe it is time to update your current architecture (design).

  • Make sure that all connected/used VMware products are compatible like (vRealize Automation / vRealize Orchestrator / vRealize Operations Manager / VMware Horizon and the list goes on). This can be verified on the VMware Product Interoperability Matrices page.
  • Make sure that all third-party products are compatible like (Backup & Replication software / Storage vendor software).
  • Determine the correct sizing for your environment. How many virtual machines and ESXi Hosts are going to be running underneath this vCenter Server. These figures determine your vCenter Server size.
  • An embedded Platform Services Controller (PSC) is the way to go in the future. An external Platform Services Controller will not be available anymore in the future.
  • Where is Windows? Please read this article from more than one year ago. Please do not deploy a vCenter Server on Windows. This is a thing of the past.
  • Verify the Hardware Requirements for the vCenter Appliance (depending on your chosen size and internal or external PSC).
  • Do you deploy against an ESXi Host or a vCenter Server?
  • Is your ESXi host hardware compatible with vSphere 6.7 Update 1?

Preparation

The checklist is completed and you have determined that everything is working or is acceptable to continue. Let’s start with some basic stuff that is required:

  • Read the release notes (VMware vCenter Server 6.7 Update 1 Release Notes).
  • Download the latest release from the VMware website.
  • Create firewall rules for your new vCenter Server.
  • Create forward and reverse DNS records in your DNS Server.
  • Register your IP information in your IPAM system.
  • Save your passwords in your Password Management system (Appliance password / SSO password).
  • Have a workstation ready to perform the deployment with sufficient network access and administrative rights.

Deployment

Let’s start the deployment of VMware vCenter 6.7 Update 1. I have chosen for a clean installation of VMware vCenter 6.7 Update 1. I have chosen for an embedded Platform Services Controller (PSC). Based on my total amount of virtual machines and ESXi Hosts I have selected a “Small” installation footprint.

The new deployment process for vCenter Server 6.7 Update 1 consists out of two stages, one is the deployment stage and one is the setup stage.

The first part is mainly responsible for delivering the full appliance with the operation system, network settings, and installation application bundles. The second part is configuring the applications that are running on the vCenter Server. A total installation takes about 45 minutes to complete.

Procedure:

  1. Mount the vCenter Server media (iso file).
  2. Navigate to the following path “X:\vcsa-ui-installer\win32\” (X stands for the CD-ROM drive label).
  3. Run the following application “installer.exe“.
  4. Follow the wizard, I have uploadedall screenshots for reference.

Stage 01 – Deployment

Here are the images of the first stage of the deployment of VMware vCenter 6.7 Update 1. I have no issues to report everything was working fine on the first try!

Stage 02 – Setup

Here are the images of the second stage of the deployment of VMware vCenter 6.7 Update 1. This part was also bug-free, so it was a good deployment.



Configuration

After a successful deployment of the VCSA appliance, you need to configure at least some items to get vCenter Server production ready. The items listed below are a basic set of the most common items I see in the field:

  • Install the vCenter Server License.
  • Active Directory.
  • Assign rights & permissions.
  • Generate and installation of SSL Certificates.
  • Connect the required VMware products and third-party systems.
  • Create a datacenter object.
  • Create a cluster object.
  • Create a distributed switch.
  • Join ESXi Hosts to the newly created cluster.

Lab Environment

So how does my Lab environment look like?

  • VMware vCenter 6.5.0 Update 2. The target for the vCenter 6.7.0 Update 1 deployment.
  • VMware ESXi 6.5.0 Update 2 in the 24×7 environment. Known as the production cluster.
  • VMware ESXi 6.5.0 Update 2 in the Lab environment. Known as the lab cluster.

You might ask… why don’t you upgrade the current vCenter Server? Good question! The machine has been converted/upgraded multiple times. It started out in life as a VMware vCenter 5.5 machine, that was on the Windows Server 2012 platform. So it was a good moment to start clean after this many years.

Share this:

Cannot Remove Content Library in VCSA 6.5 Update 1

Share this:

Opening vSphere Web Client (Flash) on Windows Server 2016

Share this:

The VM Remote Console changed to VMware Workstation instead of VMRC

Lately, I discovered an annoying feature in combination with VMware vCenter and VMware Workstation. When installing VMware Workstation on your management computer it becomes the default Remote Console viewer. To be honest, I like the VMware Remote Console (VMRC) very much. The application has all the features and is quick and light. This compared to starting VMware Workstation to open a Remote Console.

What is VMware Remote Console: “The VMware Remote Console (VMRC) is a standalone console application for Windows. VMware Remote Console provides console access and client device connection to VMs on a remote host. You will need to download this installer before you can launch the external VMRC application directly from a VMware vSphere or vRealize Automation web client.”

In October 2017, I already fixed my problem on my management computer… but after a recent VMware Workstation update, it changed the Remote Console back to VMware Workstation. Currently, there is no option in the GUI to change the default Remote Console. Ok, but how do we get VMRC back?

When I was comparing the Windows Registry, I found out that the following registry keys were different between machines. To speed up to process I created some PowerShell one-liners to fix the problem.

### View settings in registry
Get-Item HKLM:\SOFTWARE\Classes\vmrc\DefaultIconGet-Item HKLM:\SOFTWARE\Classes\vmrc\shell\open\command

### Change settings to VMRC
Set-Item HKLM:\SOFTWARE\Classes\vmrc\DefaultIcon -Value '"C:\Program Files (x86)\VMware\VMware Remote Console\vmrc.exe",0'
Set-Item HKLM:\SOFTWARE\Classes\vmrc\shell\open\command -Value '"C:\Program Files (x86)\VMware\VMware Remote Console\vmrc.exe" "%1"'

When you change the registry keys, the settings are direct in effect. No Operating System reboot or browser restart is required. The change is instant. I hope the blog post helps some vSphere Administrators that also prefer VMRC above VMware Workstation for viewing Remote Consoles.

@VMware: I would like to have an option to control the behaviour without changing registry keys by hand… 🙂 Thanks!



Environment

The issues occurred with the following combination of software:

  • VMware vCenter Server 6.5 (Update 1e)
  • VMware VMRC (10.0.2-7096020)
  • VMware Workstation (12.5.9 build-7535481)
  • Management Workstation: Windows 10 X64

VMRC Screenshots

Some screenshots that display the changes when opening the Remote Console of a Virtual Machine in VMware vCenter.

  • Remote Console - VMware Workstation
  • Remote Console - VMRC
  • Remote Console - VMRC opening VM

Share this:

VMware vSphere 6.5 General Availability

vSphere 6.5 - Logo

VMware vSphere 6.5 is General Available (GA) so this means it is available for download on the VMware website!

For those who like to try out vSphere 6.5. The download mirrors are listed below.

It might be a good idea to wait a couple of months or even until vSphere 6.5 Update 1 is released to put into production.

For those who are interested. In vSphere 6.5 the following features are now available. Keep in mind there are a lot of more items new or changed in this release. I just listed the most interesting:

  • VMware vCenter Appliance Enhancements:
    • Migration Tool
    • Improved appliance management
    • Native high availability
    • Native backup and restore
    • New API Explorer
  • VMware vCenter Administrative Interfaces:
    • vSphere Web Client enhancements
    • vSphere Web Client integration with vSphere Update Manager
    • vSphere Web Client integration with Host Profiles
    • vSphere Web Client integration with Auto Deploy
    • vSphere Client is onboard (HTML5)
  • vSphere 6.5 – Availability Enhancements
    • Proactive HA
    • VMware vSphere High Availability Orchestrated Restart
    • vSphere HA Admission Control Improvements
    • vSphere HA Support for NVIDIA GRID vGPU Configured VMs
  • vSphere 6.5 – Fault Tolerance Enhancements
    • Improved integration with vSphere DRS
    • Reduced network latency
    • Support for multiple port groups
  • vSphere 6.5 – Storage Enhancements
    • Advanced Format Drives and 512e Mode
    • Automated UNMAP
    • LUN Scalability (maximum number of LUNs to 256 and paths to 1,024)
    • NFS 4.1 Support
    • Software iSCSI Static Routing Support
  • vSphere 6.5 – Network Enhancements
    • Dedicated Gateways for VMkernel Network Adapter
    • SR-IOV Provisioning
  • vSphere 6.5 – Security
    • Virtual Machine Encryption
    • Encrypted vMotion
    • Secure boot support
    • Enhanced logging

vSphere 6.5 Announcement

The official announcement can be found on the following page (Link: VMware Announces General Availability of vSphere 6.5). The complete overview can be found in the VMware White Paper – vSphere 6.5 – What’s New PDF file.

vSphere 6.5 Download

Here are the download locations on the VMware.com website.

Share this: