Lately, I encountered some issues related to VMware vSAN in my Lab environment. The error message that was popping up all the time was “PBM error occurred during PreCloneCheckCallback“.
So how did the problem occur? First, we start with some background information. My Lab environment is powered-on when needed and powered-off when not needed. This is, of course, a little bit different than a production 24×7 environment that you have in your datacenters worldwide.
The environment was booted successfully at first glance. We are talking about Domain Controllers, vCenter Server, VMware NSX-V, nested ESXi Hosts and vRealize Automation. When I started deploying virtual machines with a vRealize Automation (vRA) based on blueprints with vSphere Templates issues started to occur.
vRealize Automation was failing on the provisioning task and was cleaning up the deployment because of the failed state (default behavior). So it was time to dig into the underlying infrastructure.
When the issue occurred the following software versions were used in my lab environment:
VMware vCenter 6.5 Update 2B
VMware vRealize Automation 7.3.1
VMware ESXi 6.5 Update 2
VMware vSAN 6.6
Here is all the information that can be found in various locations surrounding the issue.
Error message: Screenshots
Here are the screenshots, the first one is from VMware vCenter and the second one is from vRealize Automation. As you can see there is clearly a problem.
Error message: vRealize Automation
Here is the vRealize Automation log entry related to the VMware vSAN issue:
Error in Execute DynamicOps.Common.Client.HtmlResponseException: Service Unavailable (503)
Error message: vCenter Server
Here is the VMware vCenter log entry related to the VMware vSAN issue:
A general system error occurred - PBM error occurred during PreCloneCheckCallback (2118557)
The solution is quick but is more like a quick fix because it comes back every time I start up my lab environment.
Open a web browser.
Navigate to your vCenter Server URL (https://%vc%/vsphere-client).
Login with a user that has administrator credentials (firstname.lastname@example.org).
Navigate to Hosts & Clusters > Select the vCenter Object.
Click on the Configure tab.
Click on the Storage Providers.
Click on the following two buttons:
Synchronizes all Storage Providers with the current state of the environment.
Rescan the storage provider for new storage systems and storage capabilities.
After pressing the buttons, you don’t see any tasks running on the vCenter Server (expected behavior). After 5 seconds everything should be working and provisioning should be possible.
This blog post is dedicated to the VMworld 2018 US announcements. In the post, you will find the articles, links and highlights. VMworld 2018 US is an event that is organized by VMware. The US version is a five-day event that is held in Las Vegas. It takes place from 26 August to 30 August 2018. This page will be updated multiple times to coming days to add additional information and the latest announcements.
Please reply underneath this blog post if you have some additional information. This can also be additional links or blogs posts.
VMworld 2018 US – Product Announcements
In this chapter are all the product announcements. I can tell you there are a lot of announcements made at VMworld 2018.
vRealize Automation (vRA) 7.5
One the first day of VMworld 2018 US vRealize Automation 7.5 was announced.
On the first day of VMworld 2018 US, a new vSphere edition was announced. The product is called vSphere Platinum and it has a tight integration with VMware AppDefense.
The key highlights are:
Benefits for vSphere Admins
Gain visibility into the intent of each virtual machine, and a detailed inventory of application assets and context.
Understand how applications behave and be alerted to potential issues and deviations.
Shrink the attack surface and reduce the risk of security compromise. Establish a simple and powerful way to collaborate with security, compliance and application teams.
Get better visibility and protection with a simple, light-weight and scalable security solution, with no agents to manage, and minimal overhead.
Use what you already own, understand, and run in your data center – vSphere – with its unique visibility, automation and isolation qualities.
Play a larger and critical role in the security of your entire IT environment – Be the Security Hero!
Benefits for Security Teams
Better visibility and situational awareness of application behaviours, and virtual machine purpose.
Faster detection, analysis, and time to response – quickly understand attacks and make fast decisions using application context and scope.
Enhance existing security tools and support compliance efforts through contextual visibility and alerts into application communications and deviations.
Lower false positives – integrated behavioural analytics and machine learning offer a more precise method to identify and respond to threats.
Big data correlation for better identification and context using cloud SaaS model.
Security as an agile business enabler – support DevOps environment through continuous learning and protection.
Easily Coordinate with vSphere Admins and Application teams for better security while respecting existing workflows & maintaining separation of duties.
VMware AppDefense – Protects the integrity of applications running on vSphere, using machine learning to monitor against threats and automate responses. AppDefense locks down the guest operating system for all applications, the VMware application stack and third-party applications. To accomplish this, AppDefense gathers inventory data on virtual machines and applications from vCenter Server, development tools, and automation frameworks and applies machine learning to discover the intended state and establish the known good behaviours for the application and machine. Any deviations from this state are detected and prevented, securing the integrity of the applications, infrastructure, and guest operating system. AppDefense provides detailed visibility for better change management and compliance reporting and also provides a rich set of automated or orchestrated incident response mechanisms to address attacks. Moreover, it leverages machine learning for a simple and automated way to conduct audits and reviews for applications.
FIPS 140-2 Validated VM Encryption, and cross-vCenter Encrypted vMotion – Secure against unauthorized data access both at rest and in motion, across the hybrid cloud. Secure Infrastructure
Secure Boot for ESXi – Allows only VMware and Partner signed code to run in your hypervisor. Secure Boot for Virtual Machines – Helps prevent images from being tampered with and prevents the loading of unauthorized components.
Support for TPM 2.0 for ESXi – Enables hypervisor integrity by validating the Secure Boot for ESXi process and enables remote host attestation.
Virtual TPM 2.0 – Provides the necessary support for guest operating system security features while retaining operational features such as vMotion and disaster recovery.
Support for Microsoft Virtualization Based Security – Supports Windows 10 and Windows 2016 security features, like Credential Guard, on vSphere.
Audit Quality Logging – Enables authorized administration and control by providing high fidelity visibility in vSphere operations.
Also, the VMware Validated Design (VVD) received some new features and changes to the documentation. Personally, the greatest value in this release is the Visio stencils that are available for everyone.
The key highlights are:
Official NSX-T Support
IT Automating IT Scenarios
Intelligent Operations Scenarios
Introduction to Security and Compliance
Certificate Replacement for 2-pod
Certificate Replacement for 1-pod
Architecture and Design of VMware PKS for Workload Domains
There were also a lot of announcements surrounding some new developments/projects.
Project Concord – Project Concord uses Byzantine fault-tolerant consensus protocols to deliver a functioning distributed trust system: one that is both “safe” and “alive.” Concord is a generic state machine replication library that can handle malicious (Byzantine) replicas.
Project Dimension – Project Dimension will extend VMware Cloud to deliver SDDC infrastructure and hardware as-a-service to on-premises locations.
Project Magna – Project Magna will make possible a self-driving data center based on machine learning.
RDS on VMware – VMware demonstrated how Amazon Web Service’s RDS service will run on VMware in a private data center, thus offering developers a familiar RDS Functionality available on VMware in a private data center or at the Edge.
Virtualization on 64-bit ARM for Edge – VMware demonstrated ESXi on 64-bit ARM running on a windmill farm at the Edge.
Just like every year, William Lam from the website virtuallyGhetto creates a GIT repository with all the VMworld sessions. For each session, a recording and presentation are provided. It will probably be a couple of days till weeks until all sessions become available.
This afternoon I took the VMware vSAN 2017 Specialist and I passed with a score of 456 points. The exam is about… guess what Virtual SAN (vSAN)! To be more precise the latest version of vSAN version 6.6. The exam is a mix of deployment & design questions. In my opinion it is a smaller version of the separated Deploy & Design exams as in the major exam tracks (DCV, NV, CMA, DTM). The questions were really good and realistic, some examples: like about space efficiency, cluster sizing and some customer use cases. The exam contains 60 questions and you have 105 minutes to complete the exam.
Over the years I have done a lot of vSAN deployments, starting in the vSAN 5.5 days. So personally it was not the most difficult exam to pass but a good one to add to my résumé.
The official exam description from VMware: “The VMware vSAN 2017 Specialist badge holder is a technical professional who understands the vSAN 6.6 architecture and its complete feature set, knows how to conduct a vSAN design and deployment exercise, can implement a live vSAN hyper-converged infrastructure environment based on certified hardware/software components and best practices, and can administer/operate a vSAN cluster properly.”
The Specialist Exams
At VMworld 2017 three new specialist exams were announced by VMware Education:
VMware vRealize Operations 2017 Specialist
VMware vSAN 2017 Specialist Exam
VMware Validated Designs for Software-Defined Data Center 2017
Currently the remaining two are on my “To Do” list because they all cover my area of expertise. The vSAN Specialist exam is the only one of the three that is done at a test center, the other two are online exams.
There is a lot of free vSAN 6.6 content available, especially the VMworld 2017 sessions are full of useful information. Take a look Duncan Epping his YouTube Channel (the URL listed below).
“Yesterday, with impeccable timing, VMware announced the general availability of vSphere 6.5. As always, we started work on the full integration of vSphere 6.5 since its beta, however, we now need adequate time to integrate the final VDDK code and then perform full regressive testing against the final vSphere 6.5 code to ensure the reliability of our advanced vSphere integrations. Therefore, full support for vSphere 6.5 in Veeam Availability Suite 9.5 will be delivered as a part of Veeam Availability Suite 9.5 Update 1. And while an exact support timeline will depend on the results of our testing, historically we deliver support for new vSphere releases approximately two months after the final code availability.” Source: https://www.veeam.com/blog/new-veeam-availability-suite-9-5-is-available-today.html
On 20 January, Veeam has finally released update 1 for the Backup & Replication software (download link below).
Dell EMC Data Domain OS 6.0 support, including synthetic full backup performances optimizations, backup retention and health check reliability improvements
HPE 3PAR 3.2.2 MU3 support, including multiple API interaction improvements for added reliability and performance
HPE StoreOnce 3.15.1 support, bringing Instant VM Recovery to Catalyst-based backup repositories
Veeam Agent for Linux 1.0 support
Veeam Agent for Microsoft Windows 2.0 Public Beta (build 188.8.131.524) support