Today I was checking my standalone vRealize Orchestator (vRO) appliance in my 24×7 Lab environment for scheduled task activities but something didn’t seem to be right. All scheduled tasks appeared to be running without problems, but there was no workflow output.
So what kind of schedule tasks is this Orchestrator running you might ask? This Orchestrator is responsible for maintaining my environment. Think about tasks like cleaning up snapshots, removing mounted CD-ROM media and adding new Virtual Machines to my backup schedule.
Let’s analyze the problem:
To check what was going-on I created a Workflow with a Scriptable Task that outputs text (the Workflow is displayed in the screenshot below). After realizing that vRO was not able to output any information to the screen it was time to start troubleshooting.
To start troubleshooting I connected with SSH to the vRealize Orchestrator appliance and started looking at the log files. Because vRO consists out of a lot of smaller sub systems it’s important to find the right log file.
First I ran a command to show all log files with the latest change date. It appeared that the “catalina.out” was very busy. Every time I ran the command the file had a new timestamp. ls -ltur /var/log/vmware/vco/app-server/
So lets look in the log file to figure out who is logging to this file: tail -f /var/log/vmware/vco/app-server/catalina.out
It appeared that Apache Lucene was generating a continuous stream of messages/errors. Caused by: java.lang.IllegalArgumentException: An SPI class of type org.apache.lucene.codecs.Codec with name 'Lucene6 0' does not exist. You need to add the corresponding JAR file supporting this SPI to your classpath. The current cl asspath supports the following names: [Lucene62]
... 103 more
19:30:49,588 WARN StubbornRetrier:38 - Retry 1/6 failed because of: Could not load codec 'Lucene60'. Did you forget to add lucene-backward-codecs.jar?
19:30:49,609 WARN StubbornRetrier:38 - Retry 5/6 failed because of: Could not load codec 'Lucene60'. Did you forget to add lucene-backward-codecs.jar?
19:30:49,611 WARN StubbornRetrier:38 - Retry 3/6 failed because of: Could not load codec 'Lucene60'. Did you forget to add lucene-backward-codecs.jar?
As I explained before vRO consists out of multiple smaller systems and Apache Lucene is one of them. So what is Apache Lucene exactly?
Apache LuceneTM is a high-performance, full-featured text search engine library written entirely in Java. It is a technology suitable for nearly any application that requires full-text search, especially cross-platform.
When searching on the log file error I directly found a VMware KB to resolve this issue.
Start a SSH session with the vRO Appliance.
Login with the root account.
Stop the vRO server service. service vco-server stop
Run the following command to remove the old lucene logs, this might be a single file or multiple files. rm -rf /var/log/vco/app-server/scripting.log_lucene*
Start the vRO server service. service vco-server start
vRealize Orchestrator 7.4 – Login
vRealize Orchestrator 7.4 – Applying Fix
To confirm the procedure/fix I ran the test workflow again. The screenshot is listed below and shows output to the screen. This is the expected behavior so we can conclude that the fix solved the problem.
Based on the information already available on the internet it seems that the upgrade from vRealize Orchestrator 7.3.1 to 7.4 is the root cause of the problem. The positive side of the problem is that the vRealize Orchestrator engine is just doing its work. In my lab environment I discovered the problem three weeks after the upgrade. To clarify: all my workflows that were running in these three weeks were not experiencing any problems.
Today I was dealing with an issue in vRealize Automation (vRA). The customer had deleted some virtual machines in the vCenter inventory that were managed by vRealize Automation (vRA). So this resulted in registered virtual machines in vRealize Automation that were reported as missing.
For this blog post I recreated the problem in my Lab environment to have the ability to share the screenshots.
Like explained in the beginning of this blog post. I recreated the problem in my home lab environment. To give you some background information about this environment. The Lab environment is running the following products and versions: – VMware vCenter 6.5 – VMware ESXi 6.5 – VMware vRealize Automation 7.3.1 – VMware vRealize Orchestrator 7.3.1 – VMware NSX-V 6.4.0
The clusters are split into a “Mgmt & Edge” Cluster and a “Payload” Cluster. The virtual machines that were managed by vRealize Automation were located on the “Payload” cluster. So to recreate the problem I removed the virtual machines in the “Payload” cluster.
Incase you are interested in more information about my Home Lab, you can go to the following page.
Download the latest version of the vRealize ClientCloud from the VMware website (the link can be found below).
When the download is done, unzip the zip file (VMware_vRealize_CloudClient-4.5.0-8227624.zip).
Navigate to the following directory (VMware_vRealize_CloudClient-4.5.0-8227624bin).
Start the CloudClient (Windows: cloudclient.bat / Linux: cloudclient.sh).
Log in to the vRA Appliance: Syntax: vra login userpass --user admin ID --tenant Tenant Name --server vRA FQDN --password Password
This afternoon I took the VMware vSAN 2017 Specialist and I passed with a score of 456 points. The exam is about… guess what Virtual SAN (vSAN)! To be more precise the latest version of vSAN version 6.6. The exam is a mix of deployment & design questions. In my opinion it is a smaller version of the separated Deploy & Design exams as in the major exam tracks (DCV, NV, CMA, DTM). The questions were really good and realistic, some examples: like about space efficiency, cluster sizing and some customer use cases. The exam contains 60 questions and you have 105 minutes to complete the exam.
Over the years I have done a lot of vSAN deployments, starting in the vSAN 5.5 days. So personally it was not the most difficult exam to pass but a good one to add to my résumé.
The official exam description from VMware: “The VMware vSAN 2017 Specialist badge holder is a technical professional who understands the vSAN 6.6 architecture and its complete feature set, knows how to conduct a vSAN design and deployment exercise, can implement a live vSAN hyper-converged infrastructure environment based on certified hardware/software components and best practices, and can administer/operate a vSAN cluster properly.”
The Specialist Exams
At VMworld 2017 three new specialist exams were announced by VMware Education:
VMware vRealize Operations 2017 Specialist
VMware vSAN 2017 Specialist Exam
VMware Validated Designs for Software-Defined Data Center 2017
Currently the remaining two are on my “To Do” list because they all cover my area of expertise. The vSAN Specialist exam is the only one of the three that is done at a test center, the other two are online exams.
There is a lot of free vSAN 6.6 content available, especially the VMworld 2017 sessions are full of useful information. Take a look Duncan Epping his YouTube Channel (the URL listed below).
On the 28 of February VMware PowerCLI 10.0.0 was released. The biggest change in this release is the multi-platform support which includes the support for Mac OS and Linux. Oh and they kind of increased the version number a bit… from 6.5.X to 10.0.0 ;).
So I decided to upgrade PowerCLI in my Lab environment. In my Lab environment, I have a Windows 10 virtual machine that runs as an Administrator Workstation. The upgrade was very smooth and it took about one minute but after the upgrade, no connections were possible with my vCenter server.
The following error was displayed in the PowerShell prompt, screenshots are displayed below: Error: Invalid server certificate. Use Set-PowerCLIConfiguration to set the value for the InvalidCertificateAction option to Prompt exception for this server.
It appears that they have changed the default PowerCLI behaviour regarding certificates.
Note:The solution I offer below is going ignore the SSL certificate checking, make sure this is something you want. A better way to validate this is to sign the certification of the vCenter Server with for example an Enterprise Certificate Authority. A write-up can be found here.
After the PowerCLI 10.0.0 upgrade on your system, you try to connect to your vCenter Server, but you receive the following error (image 01).
The error is pointing us to the following PowerCLI command-let “Get-PowerCLIConfiguration” (image 02). It appears that based on the output of the command the “InvalidCertificateAction” is configured as “Unset“.
Let’s look at the possible settings available for “InvalidCertificateAction” (image 03). In my case “Ignore” is the best option because I’m running a test environment.
Let’s change the setting to “Ignore” to disable the certificate check (Image 04).
Now it’s time to connect with your vCenter Server again (Image 05).
Everything should be working again without certificate warning messages.
PowerCLI 10.0.0 – 01 – Certificate Error
PowerCLI 10.0.0 – 02 – Show current SSL settings
PowerCLI 10.0.0 – 03 – List options
PowerCLI 10.0.0 – 04 – Configure settings for users
PowerCLI 10.0.0 – 05 – Connections are possible again
This week I attended the NLVMUG 2018 on the 20th of march. The NLVMUG is the largest VMUG (VMware User Group) in the world and is held in the Netherlands. More than a thousand people attend the NLVMUG. The keynote was given than by no other than Pat Gelsinger the Chief Executive Officer (CEO) of VMware. The afternoon keynote was given by Joshua McKenty the Vice President from Pivatal.
The NLVMUG 2018 is a wrap but a great success!! Thanks to the VMUG leaders: Joep Piscaer en Dennis Hoegen Dijkhof.
I’m very happy to announce that I passed the VCAP6-NV Deploy exam and unlocked the VCIX-NV accreditation!
About the VMware VCIX-NV:
The VCIX-NV exam consists of approximately 23 live lab activities and the passing score for this exam is 300 (scale is from 100 to 500). The total time for this exam is 210 minutes, but candidates who take the VCIX-NV Exam and have a home address in a country where English is not a primary language will have an additional 30 minutes added to the exam time.
For my study I used the following list of website’s, HOL Labs and Blogs. This helped me to pass the exam:
My best advice is: Build a Home Lab and deploy VMware NSX-V. After the deployment you start using all the features that NSX-V has to offer (yeah, I know that is a lot). Get familiar and deploy and design like you would in a production environment. This will help you to get the best understanding possible for the exam.
Last Friday VMware announced the vExperts for 2018. I’m pleased to announce that I have been rewarded with a vExpert 2018 award. This means I have been selected for a double in a row! I would like to congratulate everyone that has been awarded by VMware and VMware for supporting the community.
I’m also pleased to tell that my employer ITQhas fifteen vExperts for 2018. Congrats guys! For those interested ITQ is a VMware Premier Partner (the highest level in the VMware Partner Professional Services Program) and we are an independent VMware knowledge partner.
The VMware vExpert award?
The VMware vExpert program is VMware’s global evangelism and advocacy program. The program is designed to put VMware’s marketing resources towards your advocacy efforts. Promotion of your articles, exposure at our global events, co-op advertising, traffic analysis, and early access to beta programs and VMware’s roadmap.
Each year, we bring together in the vExpert Program the people who have made some of the most important contributions to the VMware community. These are the bloggers, book authors, VMUG leaders, speakers, tool builders, community leaders and general enthusiasts. They work as IT admins and architects for VMware customers, they act as trusted advisors and implementors for VMware partners or as independent consultants, and some work for VMware itself. All of them have the passion and enthusiasm for technology and applying technology to solve problems. They have contributed to the success of us all by sharing their knowledge and expertise over their days, nights, and weekends.
Today a blog about my Home Lab. At the end of 2017 it was time to replace the old Dell PowerEdge R710 servers with something new. Currently I was running two R710 servers for my lab environment.
These servers are ‘powered-on for a couple of hours a week to test new products and learn for certifications. My other environment described on this page is running 24/7 is providing a full set of infrastructure services.
Because of the price and I’m very familiar with the DL360 G8 it was a no brainer. Over the last couple of years all my virtualization projects were based for 75% on the DL360 Gen8… so we have a lot of history together ;).
Technical specifications – HP DL360e Gen8:
– Chassis: Small Form Factor (8-bays)
– CPU1: Intel® Xeon® Processor E5-2430 v2
– CPU2: Intel® Xeon® Processor E5-2430 v2
– Memory: 128 GB (8x 16GB DDR3 1600 mhz)
– Disks HDD: 2x Seagate Constellation SAS 1TB
– Disks SSD: 4x Samsung EVO 850 250 GB
– Storage controller: HP SmartArray P420 with 1 GB FBWC
– NIC: 4 port 1 Gbit
– Rack mounting kit, cable arm and security front bezel
The spinning drives provide “safe” storage, because they are configured as a mirrored volume. The SSD drives are configured as JBOD drives for performance without data protection (if I want protection I just create a virtual machine back-up to one of my storage arrays).
Lately, I discovered an annoying feature in combination with VMware vCenter and VMware Workstation. When installing VMware Workstation on your management computer it becomes the default Remote Console viewer. To be honest, I like the VMware Remote Console (VMRC) very much. The application has all the features and is quick and light. This compared to starting VMware Workstation to open a Remote Console.
What is VMware Remote Console: “The VMware Remote Console (VMRC) is a standalone console application for Windows. VMware Remote Console provides console access and client device connection to VMs on a remote host. You will need to download this installer before you can launch the external VMRC application directly from a VMware vSphere or vRealize Automation web client.”
In October 2017, I already fixed my problem on my management computer… but after a recent VMware Workstation update, it changed the Remote Console back to VMware Workstation. Currently, there is no option in the GUI to change the default Remote Console. Ok, but how do we get VMRC back?
When I was comparing the Windows Registry, I found out that the following registry keys were different between machines. To speed up to process I created some PowerShell one-liners to fix the problem.
When you change the registry keys, the settings are direct in effect. No Operating System reboot or browser restart is required. The change is instant. I hope the blog post helps some vSphere Administrators that also prefer VMRC above VMware Workstation for viewing Remote Consoles.
@VMware: I would like to have an option to control the behaviour without changing registry keys by hand… 🙂 Thanks!
The issues occurred with the following combination of software:
VMware vCenter Server 6.5 (Update 1e)
VMware VMRC (10.0.2-7096020)
VMware Workstation (12.5.9 build-7535481)
Management Workstation: Windows 10 X64
Some screenshots that display the changes when opening the Remote Console of a Virtual Machine in VMware vCenter.
Today I passed the VMware Certified Professional 7 Cloud Management and Automation Exam in short VCP7-CMA. The required score was 300 points and I scored 418 points! The exam is about automation of your cloud management platform with VMware Products: vRealize Automation (vRA), VMware vRealize Business (vRB) and vRealize Orchestrator (vRO) but there are also many questions about integrations with other VMware products like VMware NSX-V, VMware vCloud Air and of course VMware vSphere.
To be honest this was not my first attempt at the VCP7-CMA exam. In VMworld 2017 Barcelona, I already did the exam but I failed with just under 300 points, that was quite frustrating.
The VMware VCP7-CMA exam consists out of 85 multiple-choice questions and you need to score at least 300 points. The maximum score is 500 points and you have 110 minutes to complete the exam. In the VMware VCP7-CMA exam guide are all the exam subjects listed so make sure you understand all of them. The main focus is on the vRealize Automation product. Make sure you study the right version because the exam is based on vRealize Automation 7.X and at this time the newest version is vRealize Automation 7.3. A big surprise to me was the questions about vCloud Air… because at the time I took the exam vCloud Air was not a part anymore of VMware (it was sold of to OVH) and all the documentation related to vCloud Air was removed… mmm nice.
VCP7-CMA Study Tips:
For passing the exam, I have used the following websites: