Tag: VMware

VyOS Configuration Restore

1

In my home lab environment, I wanted to rebuild my VyOS virtual router/firewall. So I exported the configuration from the old appliance and I tried to perform a restore on the new virtual appliance. The question that arose was: how do you perform a VyOS configuration restore?

Somehow on the internet, I could not find any tutorial or manual that explained to me how this action could be performed. There are enough write-ups and articles surrounding the TFTP, FTP and SCP restore procedure but the VyOS appliance is empty… with a default configuration. I just wanted to restore the configuration without setting up all kinds of services and configuring by hand my interfaces on the VyOS appliance.

So it was time to examine the VyOS appliance and figure out what was going on under the covers.

Why do you need VyOS?

Before diving any further let‘s talk about VyOS! I use VyOS for my Lab environment because it is easily configurable and has an entire feature set of enterprise-grade network technology onboard by default. Like the routing protocols BGP and OSPF and high availability option VRRP.

So why do you need OSPF and BGP at home? I’m a VMware Consultant that is responsible for SDDC / SDN / NSX designs and implementations. I regularly need to perform tests in my Lab environment. VMware NSX likes to have a dynamic routing protocol to connect the virtual overlay network to the physical world. Both routing protocols can be used to perform this. An article about a detailed configuration can be found here at Jeffrey Kusters his blog (my ITQ colleague). I not going into further detail on VMware NSX, this blog post is focused on VyOS.

VyOS Virtual Hardware

My VyOS appliance is deployed on a VMware vSphere 6.5 infrastructure. I used the OVA file that is available on the VyOS website (vyos-1.1.8-amd64.ova). The virtual machine is called the “LAB-FW01” this hostname will appear in the video record. The YouTube video is listed below.

The Virtual Machine hardware is configured as default. I only assigned the virtual network cards to the right networks. An overview is listed here:

  • Public – Network adapter 1 – Connected to a WAN interface
  • Private – Network adapter 2 – Connected to a VLAN trunk

VyOS Configuration Restore

Now it is time for restoring the VyOS configuration file on a newly deployed VyOS appliance.

Preparation:

  1. Deploy a new VyOS appliance and make sure that the virtual networks are connected to the correct adapter.
  2. Verify and/or change the MAC addresses were needed. The MAC addresses should align between the configuration file and the new virtual appliance:
    • Option 01: Change the virtual network card MAC address to the ones that were used on the old appliance.
    • Option 02: Change the MAC addresses in the configuration file that is used for the restore. The MAC address should align with the newly deployed VyOS appliance.
  3. Create an ISO file with your latest configuration on it. I used the following opensource tooling as displayed below. Link to IsoCreator.
Creating an ISO file with VyOS Configuration
Creating an ISO file with VyOS Configuration

Procedure:

  1. Open your vSphere Infrastructure and navigate to the Virtual Machine. This would be in my case “LAB-FW01“.
  2. Assign the newly created ISO file to the Virtual Machine. You connect the ISO file to the CD-ROM drive.
  3. Power-on the Virtual Machine.
  4. Make sure you select in the GRUB bootloader the following mode to startup from: “VyOS 1.1.8 linux (USB console)”.
  5. When VyOS is booted, log in with the following default credentials:
    • Username: vyos
    • Password: vyos
    • You are now logged in into the Linux Shell.
  6. Now it’s time to mount the connected CD-ROM media:
    • sudo mount /dev/cdrom /mnt
  7. To make sure my configuration is available. I list the directory content with the following command:
    • ls -l /mnt
  8. Now it is time to copy my old configuration to the startup configuration location of VyOS. Use the following command to perform this action (keep in mind: My configuration is called “2018-06-05-vyos.config.boot”): 
    • cp /mnt/2018-06-05-vyos.config.boot /config/config.boot
  9. To verify the copy action, I run the following command to display my hostname that is  listed in the configuration file:
    • cat /config/config.boot | grep LAB-FW01
  10. Now it is time to reboot the VyOS appliance. At the next boot, the old configuration will be loaded and everything should be restored. The following commands are required for rebooting VyOS:
    • reboot
    • yes
  11. After the reboot is completed you should log in with your old credentials that belong to the restored configuration.
  12. To verify that the configuration is loaded correctly I run the following command to display all my interfaces and sub-interfaces:
    • show interfaces
  13. From this point, everything should be working.

Tips:

I have listed all the VyOS configuration locations that are important to this article. 

In case you messed up your VyOS configuration you can always restore the default out-of-the-box configuration with the procedure described above. You only need to change the copy action in step eight to the following: (cp /opt/vyatta/etc/config.boot.default /config/config.boot).

  • Startup configuration: /config/config.boot
  • Default configuration: /opt/vyatta/etc/config.boot.default

YouTube Recording:

Because there are a couple of steps involved I decided to record a video of me performing the procedure. Keep in mind: my VMware Remote Console is in dutch :).

Final word

About six months back I switched from pfSense to VyOS. The main reason was the BGP support and stability of the BGP routing process. I am happy I did. The VyOS appliance is just amazing and extremely reliable and robust.

If you are familiar with the Cisco CLI than you will be flying through the VyOS CLI in no-time.

vRealize Automation enabling the VMware Identity Manager GUI

4

In this blog post, I am going to show you how to enable the VMware Identity Manager GUI when it is located inside vRealize Automation. I am going to explain why you sometimes should and how the procedure is performed.

For some who don’t know, VMware vRealize Automation (vRA) is using under the covers VMware Identity Manager (vIDM). By default is the VMware Identity Manager GUI disabled after the vRealize Automation installation process. If you want to configure parts of vIDM you are configuring it through the vRealize Automation GUI.

Let’s get everybody on the same page: So you are talking about two GUIs the vRealize Automation GUI and the VMware Identity Manager GUI? Who is who, explain it to me!

vRealize Automation GUI - Displaying Active Directory Users
Here is the vRealize Automation (vRA) GUI – Displaying multiple Active Directory users

VMware Identity Manager GUI - Displaying Active Directory User
Here is the VMware Identity Manager (vIDM) GUI – Displaying an Active Directory user (by the way: the name Dick is a common name in The Netherlands)

Keep in mind: I’m not completely sure what will happen if you would configure items with the vIDM GUI. Because you are bypassing the default vRealize Automation GUI. I have done some tests and no problems were caused in my Lab environment but it might cause issues later on… So why do you enable it then? Because of the large amount of information provided throughout the vIDM GUI. The vRealize Automation GUI is only showing a small peace and not the big picture.

Product support: I have tested this procedure on vRealize Automation version 7.3 and vRealize Automation 7.3.1. I would expect it to work on newer and older vRealize Automation 7.X releases.

Enable the vIDM Interface

This procedure is for enabling the interface:

  • Step 01: Connect with an SSH session to the vRealize Automation Appliance. Use for example Putty.
  • Step 02: Login with the root credentials.
  • Step 03: Run the following command to start the VMware Identity Manager Interface: (vcac–vami horizon ui enable).
  • Step 04: Open a web browser and navigate to the following URL:(https://%FQDN%/SAAS/admin/).
VMware Identity Manager (vIDM) interface enable
VMware Identity Manager (vIDM) interface enable

Disable the vIDM Interface

This procedure is for disabling the interface:

  • Step 01: Connect with an SSH session to the vRealize Automation Appliance. Use for example Putty.
  • Step 02: Login with the root credentials.
  • Step 03: Run the following command to stop the VMware Identity Manager Interface: (vcac–vami horizon ui disable).
  • Step 04: When you navigate to the following URL no page should appear: (https://%FQDN%/SAAS/admin/).
VMware Identity Manager (vIDM) Interface/GUI disable
VMware Identity Manager (vIDM) interface disable

Status of the vIDM Interface

This procedure is for viewing the vIDM Interface status:

  • Step 01: Connect with an SSH session to the vRealize Automation Appliance. Use for example Putty.
  • Step 02: Login with the root credentials.
  • Step 03: Run the following command to view the current status of the VMware Identity Manager Interface: (vcac–vami horizon ui status).
  • Step 04: When you navigate to the following URL no page should appear: (https://%FQDN%/SAAS/admin/).
VMware Identity Manager (vIDM) interface status
VMware Identity Manager (vIDM) interface status

VMware Identity Manager URLs

The following URLs are available when the GUI is enabled (there are probably more URLs available than listed below):

  • Main page: https://%vRA-Appliance-FQDN%:8443
  • Tenant-specific page (Tenant vSphere.local): https://%vRA-Appliance-FQDN%/SAAS/t/vsphere.local
  • Tenant-specific page (Tenant Production): https://%vRA-Appliance-FQDN%/SAAS/t/production
  • Tenant-specific page (Tenant Development): https://%vRA-Appliance-FQDN%/SAAS/t/development

vRealize Automation Internal Proxy Explained

To explain what happens under the covers is the following: When you run the command to start or stop vIDM interface the reverse proxy configuration located in vRealize Automation Appliance (vRA) is changed. The command adds or removes some configuration files. Then the proxy daemon is reloaded to pick up the changes. At that point, the vIDM webpage becomes available or unavailable depending on the given command. Under the covers, vRealize Automation uses HAProxy for IT guys working in container or web hosting environments that might be a very familiar product. Here is a quote from the HAProxy website to explain what HAProxy is capable of:

HAProxy is a free, very fast and reliable solution offering high availability, load balancing, and proxying for TCP and HTTP-based applications. It is particularly suited for very high traffic websites and powers quite a number of the world’s most visited ones. Over the years it has become the de-facto standard opensource load balancer, is now shipped with most mainstream Linux distributions, and is often deployed by default in cloud platforms. Since it does not advertise itself, we only know it’s used when the admins report it.

https://www.haproxy.org/#desc

VMware Identity Manager GUI Advantages

The main advantages of enabling the GUI:

  • You can view the currently logged in users in vRealize Automation.
  • When logging into the default tenant in the vIDM GUI you can view the total amount of users and groups that are identified by vIDM (counters are from all tenants combined).
  • You can view the health status of VMware Identity Manager (vIDM).
  • You can view the user login history based on the last days and you have reporting functionality.

Final word

In this blog post, I explained how to enable and disable the VMware Identity Manager interface on a vRealize Automation Appliance. As noted before be careful! I personally only use this method for troubleshooting Identity Manager related problems with authentication and viewing user activity.

Content Update

The following items have been verified:

  • 2020-09-24: This is still working on vRealize Automation 7.6.

VMware vExpert NSX 2018

1

VMware vExpert NSX 2018 - Badge

I am happy to announce that as of today I entered the VMware vExpert NSX sub-program. Thanks, VMware for the award and congratulations to everyone else who also made it! Also an extraordinary achievement for our team at ITQ, we managed to get seven consultants in the vExpert NSX 2018 sub-program! This year VMware awarded 169 people worldwide for the vExpert NSX sub-program.

vExpert NSX:

  • Since when does the vExpert NSX sub-program exist?
    • The VMware vExpert NSX sub-program was introduced in the year 2016 and announcements are made every year.
  • So what is the difference between the vExpert NSX sub-program and vExpert program?
    • The program is built upon the vExpert program. To apply you must be a current vExpert as well as evangelizing VMware NSX.
  • What is VMware expecting of a vExpert NSX?
    • Each of these vExperts has demonstrated significant contributions to the community and a willingness to share their expertise with others. Contributing is not always blogging or Twitter as there are many public speakers, book authors, scriptwriters, VMUG leaders, VMTN community moderators and internal champions among this group.

Related links

vRealize Orchestrator Client and High-Resolution Screens

0

The vRealize Orchestrator Client is the required tool for vRealize Orchestrator (vRO) server. The vRealize Orchestrator client is used for designing, administering, setting permissions and running workflows. The vRealize Orchestrator client is Java 8 based and should be running locally on your Windows desktop or management server. 

The last couple of years a lot of new displays are released that are supporting resolutions above full high definition (HD) (1920 x 1080). As a result of these large displays with high resolutions (for example 3840 × 2160), the vRealize Orchestrator Client is not scaling correctly. The fonts are too small and some boxes are not readable anymore, this is mainly caused because of the scaling that is zoomed out like 300%. This is quite irritating when developing a workflow and you are just not sure if you typed “Input_Username”, “Input Username” or “Input__Username” for example. You are just guessing or copy and pasting between an application to view the text that is written.

So it was time to investigate the problem and look for a solution.

Option 01: Lowering your screen resolution

The easiest option is to lower your screen resolution to 1920 x 1080. This is not always the most prefered option but it is the fastest.

Option 02: Modifying the vRO Client File

This option modifies the file that is used to launch the vRealize Orchestrator Client.

Procedure:

1. Navigate to the vRealize Orchestrator webpage (https://%FQDN%:8281/vco/).
2. Click on the link “START THE ORCHESTRATOR CLIENT“.
3. Save the “client.jnlp” on your desktop or other location on your workstation.
4. Open the “client.jnlp” file with a code/text editor for example Notepad++.
5. Add the following line of code to the resources block at the end (see screenshot below).

<property name=“sun.java2d.dpiaware” value=“false”/>

6. Save the file and close the editor.
7. Now launch the “client.jnlp“. This will start-up the vRealize Orchestrator Client without the screen resolution problems.

Gallery

vRealize Orchestrator 7.4 – Start the Orchestrator Client

vRealize Orchestrator 7.4 – Client.jnlp

vRealize Orchestrator 7.4 – Without DPI aware setting

vRealize Orchestrator 7.4 – With DPI aware setting

Final Word

In this blog post, I described two methods to deal with large screen resolutions in combination with the vRealize Orchestrator Client. Personally, I prefer option two because it is more permanent and I don’t have to change my resolution which effects other applications. Keep in mind: I only tested this on Microsoft Windows and not on Apple OS X or Linux.

@VMware: It might be handy to add such information to the vRealize Orchestrator 7.4 – Using the VMware vRealize Orchestrator Client user manual.

Thanks for reading and if you have other solutions and/or tips? Please respond below :).

VMware VCAP6.5-DCV Design Exam Experience

0

VMware VCIX6.5-DCV Badge At VMware EMPOWER 2018 in Vienna, I passed the VMware Certified Advanced professional 6.5 Data center Virtualization Design exam or in sort VCAP6.5-DCV. This exam was already for a very long on my certification list, to be more precise for about four years. Because of the large number of new products VMware released the last couple of years, my priorities shifted many times because Data center Virtualization was not really too interesting to study anymore… (sorry Data center Virtualization for calling you not interesting). My experience with VMware vSphere started more than ten years ago and over the years I have been responsible for designing multiple infrastructures for customers.

The VMware way of describing the VMware Certified Advanced Professional 6.5 – Data Center Virtualization Design certification:

“The VCAP6.5-DCV Design certification validates advanced knowledge of simplifying data center operations through virtualization with vSphere 6.5 and its related components, and is able to recommend and design VMware solutions to meet specific goals and requirements. This industry-recognized certification improves your credibility among your peers and proves that you have a deep understanding of data center virtualization design principles and methodologies.”

 

So where is the exam about: The exam is about the VMware Design Methodology used for designing a VMware vSphere infrastructure the right way. To pass the exam you need to know everything about the following items and make sure you can apply them in real-life examples/use cases:
– AMPRS (Availability, Manageability, Performance, Recoverability and Security).
– RCAR (Requirement, Constraint, Assumption and Risk)
– Non-functional and functional requirements
– RPO (Recovery Point Objective)
– RTO (Recovery Time Objective)
– Conceptual, Logical and Physical Designs
– vSphere 6.5 features, what did the release of vSphere 6.5 provide to each feature and what are the requirements and constraints for implementation.
– vSAN 6.5 features, design and architecture
– Site Recovery Manager features, design and architecture
– vSphere Replication features, design and architecture

Preparation

For me, the preparation of the exam started with a couple of sessions about the vSphere Design Methodology. Jeffrey Kusters was so kind to host a couple of sessions in his spare time about the subject for all the ITQ colleges. Jeffrey Kusters is one of our VMware VCDXs that works at our firm and has a lot of years of experience designing vSphere Infrastructure as an IT architect.

Study Materials

I have used many different material/sources to prepare myself for the exam but watch out. There is so much information available that you can study until 2020 and then you are still not able to read all the information. Focus on the items listed above and you will have a good chance of passing. Important note, sometimes information on a blog or book is a couple of years old and you might expect it to be not relevant. This is not the case the VMware Design Mythology has not changed over the years.

Material list:
– Blog – Jeffrey Kusters – Breaking down the conceptual design, RCARs and AMPRS …. VCDX style
– Blog – Jeffrey Kusters – Passed VCAP6.5-DCV Design … Yes! Finally VCDX6
– Course – VMware – VMware vSphere: Design Workshop [V6.5]
– Documentation – VMware – vSphere 6.5 Availability Guide
– Documentation – VMware – vSphere 6.5 Installation Setup Guide
– Documentation – VMware – vSphere 6.5 Platform Services Controller Administration Guide
– Documentation – VMware – vSphere 6.5 Security Guide
– Documentation – VMware – vSphere 6.5 Upgrade Guide
– Ebook – VMware vSphere Design Second Edition by Scott Lowe, Forbes Guthrie and Kendrick Coleman
– Ebook – VMware vSphere 6.X Datacenter Design Cookbook SE by Hersey Cartwright
– Ebook – vSphere Design Pocketbook 2.0 Blog Edition
– Exam Guide – VMware – Link
– Video – Pluralsight – What’s New in vSphere 6.5
– Video – Youtube – vBrownbag – VCAP-DCV Design
– White Paper – VMware – Storage Protocol Comparison

Conclusion

I passed the exam on the first attempt with a good score. The questions are decent around 12 drag en drop and 48 multiple choices. The questions are decent and require you to have deep knowledge of the products. So basically you know the answer or you just don’t. For me passing the VCAP6.5-DCV Design made me a VCIX6.5-DCV, so I was extra pleased with the result.

Special thanks to Jeffrey Kusters! You have been an absolute help!

vRealize Suite Lifecycle Manager Internet Proxy URLs

1

Recently I was deploying the vRealize Suite Lifecycle Manager (vRSLCM) at a customer. One of their requirement was that the appliance only could reach the internet with their internet proxy configured. The deployment was based on the latest version of the vRSLCM appliance version 1.2. The main use case for deploying this appliance at the customer was to reduce the time spend on their Lifecycle Management proces. The vRLCM appliance is able to maintain vRealize Automation (vRA), vRealize Business (vRB), vRealize Log Insight (vRLI), and vRealize Operations Manager (vROPS).

So for the proxy configuration, we need to identify which proxy URLs are required. Luckily in the VMware Documentation, there is a list of URLs and ports which the vRealize Suite Lifecycle Manager uses. So I configured the URLs but it appeared to be not working. The vRSLCM was complaining about the URL (https://my.vmware.com). So after analyzing the proxy logging, I could conclude that the URL list was not complete. Below this section, I display the difference between the official statement and what was required to get it working.

Proxy URLs – Official List:

This is the official URL list from the vRealize Suite Lifecycle Manager (vRSLCM) v1.2 documentation:

PortsPort numberBase URLs
My VMware443https://apigw.vmware.com
Solutions Exchange443https://marketplace.vmware.com
Updates443https://vapp-updates.vmware.com
Compatibility443https://vapp-updates.vmware.com

Proxy URLs – Unofficial List:

This is the list that was required to get vRSLCM working through the proxy, keep in mind there are a lot of different functionalities in the vRSLCM appliance and I was not using the entire set of functionalities. There might be even more URLs required…

PortsPort numberBase URLs
My VMware443https://apigw.vmware.com
Solutions Exchange443https://marketplace.vmware.com
Updates443https://vapp-updates.vmware.com
Compatibility443https://simservice.vmware.com
My VMware443https://my.vmware.com
vConnect443https://vconnect.vmware.com
Download location443https://download2.vmware.com
Download location443https://download3.vmware.com
Icons for marketplace443https://marketplace-download.vmware.com

Proxy Configuration:

In the steps below I explain the proxy configuration in the vRealize Suite Lifecycle Manager web interface:

  1. Open the web browser.
  2. Navigate to the vRSLCM URL (https://%FQDN%).
  3. Log in with an administrator account (admin@localhost).
  4. Navigate to the following page (Settings > My VMware).
  5. Toggle the Configure Proxy to enabled.
  6. Enter the following proxy information:
    1. Proxy Server: %FQDN% or %IP%
    2. Proxy Port: %Port Number%
    3. Proxy Username: %Username%
    4. Proxy Password: %Password%
  7. Click on the Submit button.
  8. Verify if the My VMware communication is working and the appliance is able to download the media.

Note:

  • vRealize Suite Lifecycle Manager (vRSLCM) requires ICMP communication with the internet proxy. When configuring a proxy in vRSLCM version 1.3 the configuration validation fails without allowing ICMP.

Documentation:

Article Update:

Over time the article has been updated a couple of times:

  • 2018-07-16:
    • Added additional proxy URLs for the vRealize Suite Lifecycle Manager version 1.3.
    • Added link for the vRealize Suite Lifecycle Manager manual version 1.3.
    • Added additional note about ICMP.
  • 2021-03-05:
    • Fixed the proxy URL table layout.

Cannot Remove Content Library in VCSA 6.5 Update 1

0

Problem

At a customer I came across the following problem, the customer was not able to remove a Content Library from vCenter Server. They just created a Content Library and after that, they wanted to remove the item. When they tried to remove the content library it failed. We started troubleshooting the log files and tried to remove the Content Library in different ways with the vSphere Web Client, PowerShell and REST API but all ended with the same error. The error messages are listed below.

To add some more background information: the customer was running the environment with an external platform services controller and a vCenter Server (VCSA). The version that was being used was VCSA 6.5 Update 1e.

Content Library – Error messages

Cannot Remove Content Libary – 01

Cannot Remove Content Libary – 02

Cannot Remove Content Libary – 03

Solution

We ended up calling VMware Global Support Services (GSS) to resolve the issues. They were very helpful and fixed it within a couple of minutes. The knowledge base article listed below is only available for internal VMware personal.

The internal knowledge base article related to the issue:
– https://ikb.vmware.com/s/article/50121825 – Unable to delete the stale entry for the content library from the web client

Opening vSphere Web Client (Flash) on Windows Server 2016

1

Recently somebody asked me a question about VMware vCenter running on a Windows Server. The Windows Server was running VMware vCenter 6.5 and in case of a datacenter related problem, they wanted to get access to the vSphere Web Client (Flash) on the system locally.

It sounds easy right…? Just open the browser on the Windows Server and navigate to the vSphere Web Client page but that didn’t appear to be the case, because the system is missing the browser plugins required to open the vSphere Web Client.

So let’s dive into the problem.

Microsoft Browsers:
They are running Windows Server 2016 and you might expect it to have two browsers: Internet Explorer 11 and Microsoft Edge. That does not seem to be the case. Windows Server 2016 is only shipped with Internet Explorer 11. Why? Windows Server 2016 is marked as an LTSB (Long Time Service Branch) so this means no Microsoft Edge and it is also not available for manual installation.

Microsoft: “The Long-Term Servicing Branch (LTSB) versions of Windows, including Windows Server 2016, don’t include Microsoft Edge or many other Universal Windows Platform (UWP) apps. These apps and their services are frequently updated with new functionality, and can’t be supported on systems running the LTSB operating systems.”

Third-party browsers:
The company who was asking had a security policy that does not allow an installation of third-party browsers like Mozilla Firefox or Google Chrome. Alright, so this is not an option. Don’t have to look at that further.

Adobe Flash:
So let’s try Internet Explorer 11. It appears to be missing Adobe Flash and you can not download and install it from Adobe Website.

At this point, I was stuck and there did not seem to be a simple solution.

Screenshots

vSphere Web Client missing Adobe Flash

Adobe Flash not available for installation on Windows Server 2016

Solution

After searching for a solution for about an hour. I came across a Microsoft Blog article listed below. This article is talking about installing Adobe Flash on Windows Server 2016. It appears that all the software is already on the system but just needs to be installed.

Procedure:
– Step 01: Close all browsers
– Step 02: Start a PowerShell session with elevated rights.
– Step 03: Run the following command:
dism /online /add-package /packagepath:"C:\Windows\servicing\Packages\Adobe-Flash-For-Windows-Package~31bf3856ad364e35~amd64~~10.0.14393.0.mum"
– Step 04: Wait for the installation to complete.
– Step 05: Open a browser and navigate to the vSphere Web Client.
– Step 06: Everything should be working now.

Note: In the Microsoft Blog article they are talking about a reboot required in my case it was not required. Just a browser restart was enough.

Screenshots

Installing Adobe Flash on Windows Server 2016

vSphere Web Client with Adobe Flash

It sounded like an easy problem at first but it took some more time than I expected. The problem is solved with a simple one-liner and the customer is happy. I personally think that there might be other solution to the problem. If you know them please add a comment below.

Source(s):
– Blog.technet.microsoft.com – How to add the Adobe Flash Player / Shockwave Plugin for IE on Windows Server 2016
– Docs.microsoft.com – Microsoft Edge requirements and language support

No Workflow Output in vRealize Orchestrator (vRO) 7.4

2

Overview

Today I was checking my standalone vRealize Orchestator (vRO) appliance in my 24×7 Lab environment for scheduled task activities but something didn’t seem to be right. All scheduled tasks appeared to be running without problems, but there was no workflow output.

So what kind of schedule tasks is this Orchestrator running you might ask? This Orchestrator is responsible for maintaining my environment. Think about tasks like cleaning up snapshots, removing mounted CD-ROM media and adding new Virtual Machines to my backup schedule.

Let’s analyze the problem:

To check what was going-on I created a Workflow with a Scriptable Task that outputs text (the Workflow is displayed in the screenshot below). After realizing that vRO was not able to output any information to the screen it was time to start troubleshooting.

To start troubleshooting I connected with SSH to the vRealize Orchestrator appliance and started looking at the log files. Because vRO consists out of a lot of smaller sub systems it’s important to find the right log file.

  1. First I ran a command to show all log files with the latest change date. It appeared that the “catalina.out” was very busy. Every time I ran the command the file had a new timestamp.
    ls -ltur /var/log/vmware/vco/app-server/
  2. So lets look in the log file to figure out who is logging to this file:
    tail -f /var/log/vmware/vco/app-server/catalina.out
  3. It appeared that Apache Lucene was generating a continuous stream of messages/errors.
    Caused by: java.lang.IllegalArgumentException: An SPI class of type org.apache.lucene.codecs.Codec with name 'Lucene6 0' does not exist. You need to add the corresponding JAR file supporting this SPI to your classpath. The current cl asspath supports the following names: [Lucene62]
    at org.apache.lucene.util.NamedSPILoader.lookup(NamedSPILoader.java:116)
    at org.apache.lucene.codecs.Codec.forName(Codec.java:116)
    at org.apache.lucene.index.SegmentInfos.readCodec(SegmentInfos.java:424)
    ... 103 more
    19:30:49,588 WARN StubbornRetrier:38 - Retry 1/6 failed because of: Could not load codec 'Lucene60'. Did you forget to add lucene-backward-codecs.jar?
    19:30:49,609 WARN StubbornRetrier:38 - Retry 5/6 failed because of: Could not load codec 'Lucene60'. Did you forget to add lucene-backward-codecs.jar?
    19:30:49,611 WARN StubbornRetrier:38 - Retry 3/6 failed because of: Could not load codec 'Lucene60'. Did you forget to add lucene-backward-codecs.jar?

As I explained before vRO consists out of multiple smaller systems and Apache Lucene is one of them. So what is Apache Lucene exactly?

Apache LuceneTM is a high-performance, full-featured text search engine library written entirely in Java. It is a technology suitable for nearly any application that requires full-text search, especially cross-platform.

vRealize Orchestrator 7.4 – Version

vRealize Orchestrator 7.4 – Scriptable Task Output Logging

vRealize Orchestrator 7.4 – No log output

vRealize Orchestrator 7.4 – Catalina Log File

Procedure:

When searching on the log file error I directly found a VMware KB to resolve this issue.

  1. Start a SSH session with the vRO Appliance.
  2. Login with the root account.
  3. Stop the vRO server service.
    service vco-server stop
  4. Run the following command to remove the old lucene logs, this might be a single file or multiple files.
    rm -rf /var/log/vco/app-server/scripting.log_lucene*
  5. Start the vRO server service.
    service vco-server start

vRealize Orchestrator 7.4 – Login

vRealize Orchestrator 7.4 – Applying Fix

Test:

To confirm the procedure/fix I ran the test workflow again. The screenshot is listed below and shows output to the screen. This is the expected behavior so we can conclude that the fix solved the problem.

Based on the information already available on the internet it seems that the upgrade from vRealize Orchestrator 7.3.1 to 7.4 is the root cause of the problem. The positive side of the problem is that the vRealize Orchestrator engine is just doing its work. In my lab environment I discovered the problem three weeks after the upgrade. To clarify: all my workflows that were running in these three weeks were not experiencing any problems.

vRealize Orchestrator 7.4 – Output Working

Source:

Removing a Virtual Machine from vRealize Automation with the vRealize CloudClient

0

Overview

Today I was dealing with an issue in vRealize Automation (vRA). The customer had deleted some virtual machines in the vCenter inventory that were managed by vRealize Automation (vRA). So this resulted in registered virtual machines in vRealize Automation that were reported as missing.

For this blog post I recreated the problem in my Lab environment to have the ability to share the screenshots.

vRealize Automation Missing Virtual Machines

vRealize Automation 7.3 – Infrastructure – Managed Machines

Environment

Like explained in the beginning of this blog post. I recreated the problem in my home lab environment. To give you some background information about this environment. The Lab environment is running the following products and versions:
– VMware vCenter 6.5
– VMware ESXi 6.5
– VMware vRealize Automation 7.3.1
– VMware vRealize Orchestrator 7.3.1
– VMware NSX-V 6.4.0

The clusters are split into a “Mgmt & Edge” Cluster and a “Payload” Cluster. The virtual machines that were managed by vRealize Automation were located on the “Payload” cluster. So to recreate the problem I removed the virtual machines in the “Payload” cluster.

Incase you are interested in more information about my Home Lab, you can go to the following page.

Procedure:

  1. Download the latest version of the vRealize ClientCloud from the VMware website (the link can be found below).
  2. When the download is done, unzip the zip file (VMware_vRealize_CloudClient-4.5.0-8227624.zip).
  3. Navigate to the following directory (VMware_vRealize_CloudClient-4.5.0-8227624bin).
  4. Start the CloudClient (Windows: cloudclient.bat / Linux: cloudclient.sh).
  5. Log in to the vRA Appliance:
    Syntax:
    vra login userpass --user admin ID --tenant Tenant Name --server vRA FQDN --password Password

    Example:
    vra login userpass --user "M. Buijs" --password VMware1! --tenant Production --server https://cmp.lab.local

  6. Log in to the IaaS Machine:
    Syntax:
    vra login iaas --user service account --domain the domain --server web server --password Password

    Example:
    vra login iaas --user svc-vra-iaas --password VMware1! --domain lab.local --server iaasweb.lab.local

  7. Remove the virtual machine:
    Syntax:
    vra machines forceunregister --name %VMname%

    Example:
    vra machines forceunregister --name Prod-IT-00021

Keep in mind: It may take 5 to 10 minutes to remove the virtual machine from the interface.

vRealize CloudClient

vRealize CloudClient 4.5.0 – Started Application

vRealize CloudClient 4.5.0 – Authentication and removal

Links:

Here are some useful links related to the problem:
– VMware KB2144269 – Removing a virtual machine from vRealize Automation 7.x using CloudClient
– VMware vRealize CloudClient 4.5.0 – Download Page