Tag: VMware

NLVMUG 2022 Event

Today a blog about the NLVMUG 2022 event, which I attended on the fourteenth of June 2022. The event took place in the DeFabrique in Utrecht The Netherlands.

Lots of parking spaces and easy to reach as it is centrally located in the country. The location was the same location used for the NLVMUG in 2019 (the last physical VMUG before COVID).

For me personally, this was my first physical industry event since VMworld 2019 in Barcelona Spain. So it was great to be back at a physical conference and to actually meet up with a lot of familiar faces such as ITQ/VMware colleges and community members.

There was a great lineup and a lot of sessions you could choose from. Here is an overview of the available sessions of the NLVMUG 2022:

I personally attended a couple of sessions, as you can see in the photos below. Overall the sessions were good quality and with great content! Very nice to see that a lot of people love to share and spend time on prepare those sessions!

NLVMUG 2022 Photos:

I took some photos of the event to share on my blog to give people an idea of what a VMUG is and maybe inspire them to join the next NLVMUG in 2023 (if there isn’t a new … virus or something else).

So this wraps up my short blog post about the NLVMUG 2022. I would like to thank the NLVMUG team and my employer (ITQ) for making it possible for me to be there!

For me the next physical event will be VMworld … VMware Explore so maybe see you there? See you next time!

VMware Tanzu HAProxy Unattended Deployment

In this blog post, I am going to share my script to automate the HAProxy deployment for Tanzu with vSphere or in short TKGs. Because of my interest in the Tanzu product family, I ended up testing and redeploying parts of TKG many times. To reduce the change of mistakes and improve my speed I automated the HAProxy deployment part. To start with a special thanks to William Lam for this blog post which pointed me in the right direction for automating the HAProxy OVA file.

Goal

The reason behind the creation of the code was the following:

  • Automate as much as possible
  • Standardize deployment
  • Streamline the process
  • Improved the speed of the deployment process

Environment

My environment for performing this unattended deployment of HAProxy is listed below. All additional requirements can be found in the README.md file in the GIT Repository like DNS records etc.

  • Server:
    • VMware ESXi 7.0 Update 3
    • VMware vCenter 7.0 Update 3
  • Workstation:
    • OS: Windows 10
    • Components required: PowerShell and PowerCLI

Recording

Here is a recording of the HAProxy unattended deployment in my lab environment. I have changed the variables in the script to match my environment. You must change the variables in a way so that it matches your environment to perform a successful deployment.

Code

Here is an overview of the code and a link to the GIT repository. Keep in mind to always use the GIT repository version of the code because there could be new improvements.

HAProxy Tanzu Deployment:

<#
    Script: HAProxy Tanzu Deployment
    Author: M. Buijs
    Original concept developed by: William Lam - https://github.com/lamw/vmware-scripts/blob/master/powershell/deploy_3nic_haproxy.ps1
    version: 1.0 - 2021-12-17
    Execution: HAProxy_Deployment.ps1
#>

# Set variables

	# Script variables
	$global:script_name = "HAProxy_Tanzu_Deployment"
	$global:script_version = "v1.0"
	$global:debug = 0
    $global:temp_directory = "C:\Temp\"

    # vSphere
    $vCenter = "LAB-VC01.Lab.local"
    $ClusterName = "Lab"
    $DatastorePrefix = "iSCSI - Production - *" # datastore prefix
    $DiskProvisioning = "thin" # thin or thick
    $Hardware = "v14" # Virtual hardware

    # HAProxy General
    $HAProxyDisplayName = "LAB-HAProxy01"
    $HAProxyHostname = "lab-haproxy01.lab.local"
    $HAProxyDNS = "192.168.126.21, 192.168.126.22"
    $HAProxyPort = "5556" # 5556 default port

    # HAProxy Management
    $HAProxyManagementNetwork = "Management"
    $HAProxyManagementIPAddress = "192.168.151.40/24" # Format is IP Address/CIDR Prefix
    $HAProxyManagementGateway = "192.168.151.254"

    # HAProxy Frontend
    $HAProxyFrontendNetwork = "TKG - Frontend"
    $HAProxyFrontendIPAddress = "192.168.127.40/24" # Format is IP Address/CIDR Prefix
    $HAProxyFrontendGateway = "192.168.27.254"
    $HAProxyLoadBalanceIPRange = "192.168.127.128/26" # Format is Network CIDR Notation

    # HAProxy Workload
    $HAProxyWorkloadNetwork = "TKG - Workload"
    $HAProxyWorkloadIPAddress = "192.168.128.40/24" # Format is IP Address/CIDR Prefix
    $HAProxyWorkloadGateway = "192.168.128.254"

    # HAProxy Users
    $HAProxyUsername = "haproxy_api"

# Functions
function banner {
    # Clear
	Clear-Host

	# Clear errors
	$Error.clear()

    # Message
    Write-Host "`n---------------------------------------------------------" -foreground Red
    Write-Host "               $script_name - $script_version" -foreground Red
    Write-Host "---------------------------------------------------------" -foreground Red
}

function script_exit {
	Write-Host -Foreground Yellow ""
	Write-Host -Foreground Yellow "ERROR Message: $($Error[0].Exception.Message)"
	Write-Host -Foreground Yellow ""
	Write-Host -Foreground Cyan "Exiting PowerShell Script..."
	exit
}

function validate_media {
    ##### Message
    Write-Host "`nValidating media:"

        #### Locate temp directory
        If (-not (Test-Path "$($Temp_Directory)")) {
            Write-Host -ForegroundColor Red "- The temp directory is not created ($Temp_Directory)"
            script_exit
        }
        else {
            Write-Host -ForegroundColor Green "- Located the temp directory ($Temp_Directory)"
        }

        #### Locate OVA file
        Try {
            Write-Host -ForegroundColor Green  "- Searching for OVA file"
            $script:OVF_HAProxy = $(Get-ChildItem -Path "$Temp_Directory" -Include haproxy-v*.ova -File -Recurse -ErrorAction Stop | Sort-Object LastWriteTime | Select-Object -last 1)

            ### In case of no results
            if ([string]::IsNullOrEmpty($OVF_HAProxy.name)) {
                throw
            }
            #### Message
            Write-Host -ForegroundColor Green "- Located HAProxy OVA file ($($OVF_HAProxy.Name))"
        }
        Catch {
            Write-Host -ForegroundColor Red  "- Could not find HAProxy OVA file in location ($Temp_Directory)"
            script_exit
        }
}

function ask_passwords {
    # Banner
    Write-Host "`nPasswords:"

    # Ask passwords
    $script:HAProxyOSPassword = Read-Host -asSecureString "- Enter the HAProxy user password (root)"
    $script:HAProxyPassword = Read-Host -asSecureString "- Enter the HAProxy user password ($HAProxyUsername)"

    # Validation
    If ($HAProxyOSPassword.Length -eq 0) {
        Write-Host -ForegroundColor Red "- HAProxy root account password is empty"
        script_exit
    }
    # Validation
    If ($HAProxyPassword.Length -eq 0) {
        Write-Host -ForegroundColor Red "- HAProxy user account password is empty"
        script_exit
    }
}

function connect_vcenter {
    # Banner
    Write-Host "`nvCenter connection:"

        # Disable vCenter deprecation warnings
        Set-PowerCLIConfiguration -DisplayDeprecationWarnings $false -Confirm:$false | Out-Null

        # Disable vCenter certification errors
        Set-PowerCLIConfiguration -InvalidCertificateAction "ignore" -Confirm:$false | Out-Null

        # Determine script or user input
        if ($vCenter) {
            Write-Host -ForegroundColor Green "- Connecting with vCenter server ($vCenter)"
        }
        else {
            # Ask required vCenter information
            $script:vCenter = Read-Host "- Enter the vCenter IP address or hostname"
        }

        if ($global:DefaultVIServers.Count -gt 0) {
            Write-Host -ForegroundColor Green "- Session already established ($vCenter)"
        }
        else {
            # Check IP address for connectivity
            if (test-connection -computername $vCenter -count 1 -quiet -ErrorAction SilentlyContinue) {
                Write-Host -ForegroundColor Green "- Host is alive ($vCenter)"
            }
            else {
                Write-Host -ForegroundColor Red "- Host is not responding ($vCenter)"
                $vCenter = ""
                Break
            }

            # Connect with vCenter
            try {
                Write-host -ForegroundColor Green "- Connecting to vCenter, please wait..."

                # Connect to vCenter
                Connect-ViServer -server $vCenter -ErrorAction Stop | Out-Null
            }
            catch [Exception]{
                $status = 1
                $exception = $_.Exception
                Write-Host "- Could not connect to vCenter, exiting script" -foreground Yellow
                Write-Host ""
                Write-Host "Exit code: $status" -foreground Yellow
                Write-Host "Output: $exception" -foreground Yellow
                Break
            }
        }

        # Message
        Write-Host -ForegroundColor Green "- Connection successful"
}

function ovf_config {
    # Banner
    Write-Host "`nOVF Configuration:"

    # Start
    Write-Host -ForegroundColor Green "- Creating OVF Configuration"

    $script:ovfconfig = Get-OvfConfiguration $OVF_HAProxy

    # Three nic configuration
    $script:ovfconfig.DeploymentOption.value = "frontend"

    # General
    $script:ovfconfig.network.hostname.value = $HAProxyHostname
    $script:ovfconfig.network.nameservers.value = $HAProxyDNS
    $script:ovfconfig.loadbalance.dataplane_port.value = $HAProxyPort

    # Network port groups
    $script:ovfconfig.NetworkMapping.Management.value = $HAProxyManagementNetwork
    $script:ovfconfig.NetworkMapping.Frontend.value = $HAProxyFrontendNetwork
    $script:ovfconfig.NetworkMapping.Workload.value = $HAProxyWorkloadNetwork

    # Management
    $script:ovfconfig.network.management_ip.value = $HAProxyManagementIPAddress
    $script:ovfconfig.network.management_gateway.value = $HAProxyManagementGateway

    # Workload
    $script:ovfconfig.network.workload_ip.value = $HAProxyWorkloadIPAddress
    $script:ovfconfig.network.workload_gateway.value = $HAProxyWorkloadGateway
    $script:ovfconfig.loadbalance.service_ip_range.value = $HAProxyLoadBalanceIPRange

    # Accounts
    $script:ovfconfig.loadbalance.haproxy_user.value = $HAProxyUsername

    # Password root
    $BSTR1 = [System.Runtime.InteropServices.Marshal]::SecureStringToBSTR($HAProxyOSPassword)
    $HAProxyOSPassword = [System.Runtime.InteropServices.Marshal]::PtrToStringAuto($BSTR1)
    $script:ovfconfig.appliance.root_pwd.value = $HAProxyOSPassword

    # Password user
    $BSTR2 = [System.Runtime.InteropServices.Marshal]::SecureStringToBSTR($HAProxyPassword)
    $HAProxyPassword = [System.Runtime.InteropServices.Marshal]::PtrToStringAuto($BSTR2)
    $script:ovfconfig.loadbalance.haproxy_pwd.value = $HAProxyPassword

    # Finish
    Write-Host -ForegroundColor Green "- Completed OVF Configuration"
}

function pre_deployment {
    # Banner
    Write-Host "`nPre-deployment:"

    # Cluster
    $script:Cluster = Get-Cluster $ClusterName
    Write-Host -ForegroundColor Green "- Selected cluster ($Cluster)"

    # VMhost
    $script:VMHost = Get-VMHost | Where-Object { $_.ConnectionState -eq "Connected" } | Get-Random
    Write-Host -ForegroundColor Green "- Selected ESXi Host ($VMHost)"

    # Datastore
    $script:Datastore = Get-VMhost -Name $VMHost | Get-Datastore -Name $DatastorePrefix | Select-Object Name, FreeSpaceGB | Sort-Object FreeSpaceGB -Descending | Select-Object -first 1 | Select-Object Name -expandproperty name
    Write-Host -ForegroundColor Green "- Selected datatore ($Datastore)"

    # Check virtual machine name exists
    $VMname_check_query = Get-Cluster -Name $ClusterName | Get-VM -name $HAProxyDisplayName -ErrorAction SilentlyContinue

    if (! $VMname_check_query) {
        Write-Host -ForegroundColor Green "- Virtual machine name is not in use ($HAProxyDisplayName)"
    }
    else {
        Write-Host -ForegroundColor Red "- Virtual Machine with name ($HAProxyDisplayName) already exists. Exiting script cannot continue!"
        script_exit
    }

	#### Ask for conformation
	Write-Host "`nThis task is going to build the HAProxy virtual machine for TKGs."
	$confirmation = Read-Host "Are you sure you want to proceed? [y/n]"

	if ($confirmation -eq 'n') {
		Write-Host "Operation cancelled by user!" -Foreground Red
		base_exit
	}

	if (!$confirmation) {
		Write-Host -Foreground Red "No input detected!"
	    base_exit
	}
}

function deployment {
    # Banner
    Write-Host "`nDeployment:"

	# HAProxy deployment of OVF
	try {
		### Message
		Write-Host -ForegroundColor Green "- Starting HAProxy Deployment ($HAProxyHostname / $HAProxyManagementIPAddress)"

        $script:vm = Import-VApp -Source $OVF_HAProxy -OvfConfiguration $ovfconfig -Name $HAProxyDisplayName -Location $Cluster -VMHost $VMHost -Datastore $Datastore -DiskStorageFormat $DiskProvisioning

        ### Message
		Write-Host -ForegroundColor Green "- Finished HAProxy Deployment ($HAProxyHostname / $HAProxyManagementIPAddress)"
    }
	catch [Exception]{
		Write-Host -ForegroundColor Red "- HAProxy Deployment Failed ($HAProxyHostname / $HAProxyManagementIPAddress)"
		script_exit
	}
}

function post_deployment {
    # Banner
    Write-Host "`nPost-deployment:"

	# Configure OVF
	try {
		### Message
		Write-Host -ForegroundColor Green "- Starting HAProxy OVF Configuration ($HAProxyHostname / $HAProxyManagementIPAddress)"

        $vappProperties = $vm.ExtensionData.Config.VAppConfig.Property
        $spec = New-Object VMware.Vim.VirtualMachineConfigSpec
        $spec.vAppConfig = New-Object VMware.Vim.VmConfigSpec

        $ovfChanges = @{
            "frontend_ip"=$HAProxyFrontendIPAddress
            "frontend_gateway"=$HAProxyFrontendGateway
        }

        ### Message
		Write-Host -ForegroundColor Green "- Finished HAProxy OVF Configuration ($HAProxyHostname / $HAProxyManagementIPAddress)"
    }
	catch {
		Write-Host -ForegroundColor Red "- HAProxy OVF Configuration failed ($HAProxyHostname / $HAProxyManagementIPAddress)"
		script_exit
	}

    try {
        # Message
		Write-Host -ForegroundColor Green "- Starting HAProxy Update Specification ($HAProxyHostname / $HAProxyManagementIPAddress)"

        # Retrieve existing OVF properties from VM
        $vappProperties = $VM.ExtensionData.Config.VAppConfig.Property

        # Create a new Update spec based on the # of OVF properties to update
        $spec = New-Object VMware.Vim.VirtualMachineConfigSpec
        $spec.vAppConfig = New-Object VMware.Vim.VmConfigSpec
        $propertySpec = New-Object VMware.Vim.VAppPropertySpec[]($ovfChanges.count)

        # Find OVF property Id and update the Update Spec
        foreach ($vappProperty in $vappProperties) {
            if($ovfChanges.ContainsKey($vappProperty.Id)) {
                $tmp = New-Object VMware.Vim.VAppPropertySpec
                $tmp.Operation = "edit"
                $tmp.Info = New-Object VMware.Vim.VAppPropertyInfo
                $tmp.Info.Key = $vappProperty.Key
                $tmp.Info.value = $ovfChanges[$vappProperty.Id]
                $propertySpec+=($tmp)
            }
        }
        $spec.VAppConfig.Property = $propertySpec

        # Message
		Write-Host -ForegroundColor Green "- Finished HAProxy Update Specification ($HAProxyHostname / $HAProxyManagementIPAddress)"
    }

    catch {
        # Message
        Write-Host -ForegroundColor Red "- HAProxy Update Specification failed ($HAProxyHostname / $HAProxyManagementIPAddress)"
		script_exit
    }

    # HAProxy reconfigure task for virtual machine
    try {
        # Message
        Write-Host -ForegroundColor Green "- Start Reconfigure VM task ($HAProxyHostname / $HAProxyManagementIPAddress)"
        $task = $vm.ExtensionData.ReconfigVM_Task($spec)
        $task1 = Get-Task -Id ("Task-$($task.value)")
        $task1 | Wait-Task | Out-Null
    }
    catch {
        Write-Host -ForegroundColor Red "- Reconfigure VM task failed ($HAProxyHostname / $HAProxyManagementIPAddress)"
        script_exit
    }

    # Message
    Write-Host -ForegroundColor Green "- Completed the reconfigure VM task ($HAProxyHostname / $HAProxyManagementIPAddress)"
}

function boot {
    # Banner
    Write-Host "`nBoot:"

	# Upgrade Virtual Hardware
	Try {
		Write-Host -ForegroundColor Green "- Upgrade Virtual Hardware ($HAProxyHostname / $HAProxyManagementIPAddress)";
		Get-VM -Name $vm | Set-VM -Version $Hardware -Confirm:$false | Out-Null
	}
	Catch {
		Write-Host -ForegroundColor Red "- Upgrade Virtual Hardware failed ($HAProxyHostname / $HAProxyManagementIPAddress)";
		script_exit
	}

	# Power-On Virtual Machine
	Try {
		Write-Host -ForegroundColor Green "- Power-on HAProxy started ($HAProxyHostname / $HAProxyManagementIPAddress)"
		Get-VM $vm | Start-VM | Out-Null
	}
	Catch {
		Write-Host -ForegroundColor Red "- Starting HAProxy failed ($HAProxyHostname / $HAProxyManagementIPAddress)"
		script_exit
	}

    Write-Host -ForegroundColor Green "- Power-on HAProxy completed ($HAProxyHostname / $HAProxyManagementIPAddress)"
}

function check {
    # Banner
    Write-Host "`nCheck:"

    # Set total of retries
    $TOTAL = "10"

    # Host retry interval (seconds)
	$HOST_WAIT = "10";

    # Start loop
    For ($i=0; $i -le $TOTAL; $i++) {

        # Number conversion to 2 digit:
        $NUMBER = [INT]$i + 1
        $NUMBER = "{0:D2}" -f $NUMBER

        # Check Host
        $Host_check_query = Test-Connection -computername $HAProxyHostname -count 1 -quiet -ErrorAction SilentlyContinue

        # Validate, else retry after a wait
        if ($Host_check_query -eq $false) {
            Write-Host -Foregroundcolor green "- [$NUMBER/$TOTAL] Checking HAProxy availability ($HAProxyHostname)"
            Start-Sleep $HOST_WAIT
        }
        else {
            Write-Host -Foregroundcolor green "- [$NUMBER/$TOTAL] Checking HAProxy availability ($HAProxyHostname)"
            Write-Host -Foregroundcolor green "- [Ready] HAProxy is available ($HAProxyHostname)"
            break
        }
    }
}

function retrieve_certificate {
    # Banner
    Write-Host "`nRetrieve certificate:"

    # Build URL
    $script:url = "https://${HAProxyHostname}:${HAProxyPort}/v2/info"

    # Configure local system
    try {
        # Message
        Write-Host -ForegroundColor Green "- Disable certificate checking on local system"

        # Disable certificate check
        [System.Net.ServicePointManager]::ServerCertificateValidationCallback = {$true}
    }
	catch {
		Write-Host -ForegroundColor Red "- Could not disable certificate checking on local system"
		script_exit
	}

    # Download certificate
    try {
        # Message
        Write-Host -ForegroundColor Green "- Get HAProxy certificate ($url)"

        $req = [Net.HttpWebRequest]::Create($url)
        $req.ServicePoint | Out-Null

        # Authentication
        $req.Credentials = New-Object Net.NetworkCredential($HAProxyUsername, $HAProxyPassword);
    }
	catch {
		Write-Host -ForegroundColor Red "- Could not get HAProxy Certificate ($url)"
		script_exit
	}

    # Store error messages in variable to not crash a try and catch statement.
    $GetResponseResult = $req.GetResponse()

    # Store certificate as X.509 file
    try {
        # Message
        Write-Host -ForegroundColor Green "- Store HAProxy certificate as X.509 ($url)"

        $cert = $req.ServicePoint.Certificate
        $bytes = $cert.Export([Security.Cryptography.X509Certificates.X509ContentType]::Cert)
        set-content -value $bytes -encoding byte -path "$pwd\$HAProxyHostname.cer"
    }
    catch {
        Write-Host -ForegroundColor Red "- HAProxy X.509 certificate could not be saved ($url)"
        Write-Host -ForegroundColor Red "- Result from GetResponse: ($GetResponseResult)";
        script_exit
    }

    # Convert certificate to Base-64 file
    try {
        # Message
        Write-Host -ForegroundColor Green "- Store HAProxy certificate as Base-64 ($url)"

        $InsertLineBreaks=1
        $sMyCert="$pwd\$HAProxyHostname.cer"
        $oMyCert = New-Object System.Security.Cryptography.X509Certificates.X509Certificate2($sMyCert)
        $oPem = New-Object System.Text.StringBuilder
        $oPem.AppendLine("-----BEGIN CERTIFICATE-----") | Out-Null
        $oPem.AppendLine([System.Convert]::ToBase64String($oMyCert.RawData,$InsertLineBreaks)) | Out-Null
        $oPem.AppendLine("-----END CERTIFICATE-----") | Out-Null
        $oPem.ToString() | out-file "$pwd\$HAProxyHostname.pem"
    }
    catch {
        Write-Host -ForegroundColor Red "- HAProxy Base-64 certificate could not be saved ($url)"
        script_exit
    }
}

function complete_banner {
    # Message
    Write-Host -ForegroundColor Green "- HAProxy deployment completed successfully! ($HAProxyHostname / $HAProxyManagementIPAddress)"
}

##### Main
banner
validate_media
connect_vcenter
ask_passwords
ovf_config
pre_deployment
deployment
post_deployment
boot
check
retrieve_certificate
complete_banner

Wrap-up

I hope this blog about HAProxy unattended deployment was useful for some people or that the PowerShell code inspires people to deploy other types of OVA appliances.

If you got any improvements please commit them to the GIT repository and if you got any questions please respond below. Thank you for reading my blog post and see you next time!

VMware Tanzu HAProxy Troubleshooting

This blog post is dedicated to HAProxy Troubleshooting for vSphere with Tanzu or also known as TKGs. Based on your configuration and deployment and the various items you need to configure you can make mistakes or items are not correctly configured. In my case, there were multiple problems at different deployments with parameters and reachability related to the network. In the end, after all the hours of troubleshooting, I ended up with a list of commands that might help others out. So that is the topic of this blog post.

HAProxy Background

First an introduction about the product HAProxy. HAProxy is a load balancer that is used by vSphere with Tanzu. This is not mandatory but is a product to choose from. The main reason for HAProxy compared to the others is that it is completed free/open-source. The HAProxy OVA is packaged and delivered by VMware and can be found in the following repository. All commands below have been tested against the HAProxy v0.2.0 version (haproxy-v0.2.0.ova) that is at the moment of writing the most recent version available.

Appliance access (SSH)

After a successful deployment, you can access the HAProxy appliance with an SSH session. This session can be established with a tool like PuTTY. The user account that should be used in the root account.

Keep in mind: Do not change configuration unless you absolutely know what you are doing. Almost all the issues I ran into were related to entering incorrect information into the deployment wizard or firewall issues.

Troubleshooting Services

One of the first things to check at first is that all services are running on the HAProxy appliance. When services are not started this is mostly caused by an invalid/incomplete configuration that is filled by the deployment wizard of the OVA.

### Check failed services
systemctl list-units --state=failed

### Check primary services for HAProxy and Tanzu Integration
systemctl status anyip-routes.service
systemctl status haproxy.service

### Restart services
systemctl restart haproxy

Troubleshooting Configuration Files

There are multiple configuration files in use by HAProxy here are the most important ones. Also, keep in mind what I already said before… do not change anything unless…

### Anyip-routes configuration file
cat /etc/vmware/anyip-routes.cfg

### HAProxy configuration file
cat /etc/haproxy/haproxy.cfg

### HAProxy dataplane api configuration file
cat /etc/haproxy/dataplaneapi.cfg

### Validation of configuration file
haproxy -c -f /etc/haproxy/haproxy.cfg

Troubleshooting HAProxy process output

Sometimes it is good to check the latest messages generated by the HAProxy process. There will be information about the startup of the process and the pool members.

### Show logging
journalctl -u haproxy.service --since today --no-pager

Troubleshooting IP Settings

By entering wrong IP information in the deployment wizard the configuration files surrounding the IP address settings, gateway, etc can be configured incorrectly. What I noticed is there is not really a check inside the deployment that verifies if the address that is entered is valid in any sort of way.

### List IP Settings
ifconfig

### Config files (incase of three NIC configuration)
cat /etc/systemd/network/10-frontend.network
cat /etc/systemd/network/10-workload.network
cat /etc/systemd/network/10-management.network

### Routing check
route
ip route

Troubleshooting Certificates

Certificates files used by the HAProxy application are inside the HAProxy directory on the local system. The certificates are BASE-64 encoded!

### Certificate authority file:
cat /etc/haproxy/ca.crt

### Certificate server file:
cat /etc/haproxy/server.crt

### Certificate URL by default:
https://%HAProxy-Management-IP%:5556

Troubleshooting NTP

One of the all-time favorites that are notorious for disrupting IT systems is off course NTP. Here are some commands for troubleshooting on Photon OS.

### Check service status
systemctl status systemd-timesyncd

### Show NTP peers
ntpq -p

### Restart service
systemctl restart systemd-timesyncd

### Configuration file
cat /etc/systemd/timesyncd.conf

Troubleshooting the HAProxy API

The HAProxy API is used by Tanzu to configure HAProxy for the management and workload components. Authentication is set up when deploying the OVA and the credentials are entered in the wizard. With the second URL you can verify those credentials:

### Info page
https://%IP-address%:5556/v2/info

### Authentication should work with the HAProxy user account (specified in the deployment wizard)
https://%IP-address%:5556/v2/cluster

Wrapup

Thank you for reading this blog post about HAProxy troubleshooting for vSphere with Tanzu or in short TKGs. I hope it was useful to you! If you got something to add? Have additional tips or remarks please respond in the comment section below.

Have a nice day and see you next time.

Source

PowerCLI Update Fails on Certificate Issue

Today I was updating my PowerShell modules on my local system but I ran into an issue related to certificates. After some investigation, it appeared that the PowerCLI Update was failing. Every PowerCLI module that it tried to update returned an error and did not update to a newer version. In this blog post, I am showing you how to get rid of this error and you will be able to update PowerCLI again to the latest version.

Environment

Here is a quick summary of my environment where the issue occurred:

  • Current PowerCLI: 12.1.0.16997004
  • The system tried to update PowerCLI to: 12.4.0.18627054
  • Operating System: Windows 10 Pro (21H1)

Problem

Here is an overview of the PowerShell command I used to update my modules. Also, the error message is listed below in the code box. As you can see PowerShell is complaining about a DigiCert certificate and that it is not save to update because of the certificate change.

# Open a PowerShell command prompt with administrative permissions

# Enter the command to update the PowerShell modules
Update-Module

# The following error message appears
Install-Package: Authenticode issuer 'CN=DigiCert Trusted Root G4, OU=www.digicert.com, O=DigiCert INc, C=US' of the new module 
'VMware.VimAutomation.Sdk' with version '12.4.0.18627054' is not matching with the authenticode issuer 'CN=VeriSign Class 3 Public Primary Certification 
Authority - G5, OU="(c) 2006 VeriSign, Inc. - for authorized use only", OU=VeriSign Trust Network, O="VeriSign, Inc.", C=US' of the previously-installed
 module 'VMware.VimAutomation.Sdk' with version '12.1.0.16997004'.

Here is a screenshot of the error message, error message is listed in red text. It also shows the commands I used for the PowerCLI update on my local system.

Solution

The solution for fixing the issue is quite simple. Just reinstall the VMware PowerCLI modules on your local system. This needs to be done in a forced way but after that, you are done.

# Open a PowerShell command prompt with administrative permissions

# Enter the command to re-install the PowerShell modules
Install-Module VMware.PowerCLI -Force -SkipPublisherCheck

# Wait a couple of minutes and everything should be upgraded.
Get-Module -ListAvailable VMware.PowerCLI | Select-Object Name, Version

As you can see in the screenshots below the PowerCLI update is working and is returning no errors after the upgrade.

Wrap-up

This wraps up this small blog post about a PowerCLI update issue in PowerShell. Thank you for reading and I hope it helped you out. Please respond in the comment section below if you have any questions or remarks!

vRealize Orchestrator 8.X – Input Form Dropdown

Today a basic tutorial on vRealize Orchestrator 8.X drop-down boxes in a form. With a basic drop-down box, you can improve the user experience in selecting and requesting items from your cloud management portal (CMP). By using drop-down boxes you can leverage easy validation and responses based on other drop-down boxes in your form.

In this tutorial, we are going to create dropdown boxes that respond to each other based on the user’s selection. This can be handy for improving the user experience. Sometimes the list can become very big with numerous options. By sub-selecting a group and filtering to a smaller list of options the user can easier make his decision.

Keep in mind:

  • This tutorial is focused on vRealize Orchestrator 8.X but can still be leveraged in vRealize Orchestrator 7.X with some minor modifications.
  • This tutorial is also usable for vRealize Automation 8.X forms. This can be leveraged by the Service Broker component by importing vRealize Orchestrator workflows.

Use Case

To give you some background around the code and usability. Let’s assume we are developing a workflow for creating Virtual Machines in vSphere. Based on user input surrounding the Operating System information we can determine the type of virtual machine that will be created when the request is submitted. We can also limit the user to some standard options like only Windows 10 or Windows Server 2019.

Keep in mind: this blog post is only focused on the form part, not on the actual creation of the virtual machine in vSphere.

vRO Actions

The first action we are going to create is called “formVmOsFamily“. This will display three values in the form. Based on what you select here the second action will be triggered.

/*
Script name: formVmOsFamily

Inputs:
- None

Return Type:
- vRO 8.X: string:array

Description field:
Author: M. Buijs - ITQ
Developed by: M. Buijs - ITQ
Date: 2021-08-17
Version: 1.0.0

Description: This action returns the available Guest Family of the Operating Systems.
*/


// Operating System Family list
return [
    "Linux",
    "VMware",
    "Windows"
];

Here is the second action that is called “formVmOsGuest“. This will respond to the input provided by the operating system family in the interface.


/*
Script name: formVmOsGuest

Inputs:
- osFamily (string) = Operating System Family

Return Type:
- vRO 8.X: string:array

Description field:
Author: M. Buijs - ITQ
Developed by: M. Buijs - ITQ
Date: 2021-08-05
Version: 1.0.0

Description: This action returns the available Guest Operating Systems.
*/

// Input validation
if (osFamily == "" || osFamily == null) {
    return ["Please select the Operating System family first"];
}

// Linux
if (osFamily == "Linux")
return [
    "CentOS 6 (64-Bit)",
    "CentOS 7 (64-Bit)",
    "CentOS 8 (64-Bit)",
    "Debian 10 (64-Bit)"
];

// VMware
if (osFamily == "VMware")
return [
    "VMware ESXi 6.0",
    "VMware ESXi 6.5",
    "VMware ESXi 6.7",
    "VMware ESXi 7.0"
];

// Windows
if (osFamily == "Windows")
return [
    "Windows 10 (64-Bit)",
    "Windows Server 2016 (64-Bit)",
    "Windows Server 2019 (64-Bit)",
    "Windows Server 2022 (64-Bit)"
];

Here is an overview of screenshots of how it should look like when created the actions in vRealize Orchestrator:

vRO Workflow

Here is the vRealize Orchestrator workflow, I have created an empty workflow and only configured the input form dropdown part! This will help you to set up the workflow so that the actions will work in your environment. The important part is not to forget to configure the workflow inputs and listed below:

Inputs:

  • virtualMachineOsFamily (string)
  • virtualMachineOsGuest (string)

Recording

Here is a recorded video of the input form dropdown boxes in action. The video demonstrates the capability of dropdown boxes and what they can deliver for a customer. It also gives you an idea of what you will get after creating the workflow and actions.

Improvements/Guidance

If you are going to use this code in production… You might need to consider some important points:

  • Always create a list of supported Operating Systems that are used and allowed to be used in your Company. More options will not always simplify the deployment for an end-user.
  • You could store the values in a vRealize Orchestrator Configuration Element depending on how frequently this list is changed.

Summary

So this concludes my blog post about creating dropdown boxes in vRealize Orchestrator and reacting on the input. Hopefully, this was useful for somebody getting started with interfaces in vRealize Automation (vRA) or vRealize Orchestrator (vRO). Please respond in the comment section below if you have any questions or remarks!

vRealize Orchestrator Upgrade (8.X)

This blog post is about upgrading vRealize Orchestrator 8.X to a newer version. After a couple of vRealize Orchestrator Upgrades since the 8.0 release and getting stuck a couple of times I decided to do a simple write-up with some tips and tricks.

In my lab environment, I have got multiple orchestrators running embedded, standalone, and cluster. Most issues I encountered are related to the standalone version that is connected with the VMware vCenter Server.

vRO upgrade checks

Let’s start with some simple upgrade checks to make sure everything is working before the upgrade and to improve the chance of succeeding.

  • Make sure the root account is not expired on all nodes in the cluster.
  • Make sure you have the correct vCenter SSO password. Verify this by logging in with administrator@vsphere.local on the vCenter Server. The password is required for the standalone upgrade that is directly connected to the VMware vCenter Server.
  • Make sure the time sync is working on all the nodes in the cluster.

vRO upgrade

Let’s start with the vRealize Orchestrator Upgrade. Here is an overview of the procedure and the commands required to perform the upgrade.

Keep in mind: Step six is optional and is only required for the vRealize Orchestrator that is connected to the vCenter SSO. For the vRealize Automation connected upgrade, this step can be skipped.

Procedure:

  1. Create a virtual machine snapshot.
  2. Open an SSH session with the vRealize Orchestrator node.
  3. Login with the root account on the vRealize Orchestrator node.
  4. Mount the upgrade media to the virtual machine.
  5. Mount the media in the linux system (mount /dev/sr0 /mnt/cdrom).
  6. Enter the SSO password as a variable in the shell (export VRO_SSO_PASSWORD=your_sso_password).
  7. Start the upgrade (vracli upgrade exec -y –profile lcm –repo cdrom://).
  8. The upgrade will start. Depending on the size of the vRealize Orchestrator node it will take between 30 to 90 minutes.
  9. After the upgrade is completed restart the system (reboot).
  10. Verification:
    1. Check the virtual machine console for startup issues. Make sure the console is displaying a blue screen with information about the node.
    2. Check the virtual machine console for the version/build number on the blue screen that it is displaying.
    3. Check if the web interface is available and the interface is working.
    4. Login into the vRO interface and verify that authentication is working.
    5. Run a basic workflow.
  11. Remove the virtual machine snapshot.

Screenshot(s)

Here are a couple of screenshots of the upgrade process and the end result after a successful upgrade:

Summary

So that was my short blog post about the vRealize Orchestrator Upgrade experience so far for version 8.X. I hope it was useful. In most cases, there were problems with an expired account or an incorrect SSO password.

It would be nice if the upgrade process would validate the entered SSO password instead of hanging for hours in a crashed upgrade state without returning any error message to the console or shell session.

Thanks for reading and see you next time! Please respond in the comment section below if you got any remarks :).

Official documentation:

HPE ProLiant DL20 Gen9 SATADOM Installation

Today we are going to work on an HPE ProLiant DL20 Gen9 server. After the initial installation, I was using an SD card as boot media but I still had some Delock SATADOMs laying around from my older lab servers that were replaced. So it was time to improve the performance of the boot media in the servers. In this blog post, I am explaining in detail the SATADOM installation in an HPE ProLiant DL20 Gen9.

So what are the advantages compared to an SD card:

  • VMware ESXi boot time about 50% faster
  • VMware ESXi upgrade time about 70% faster
  • Inventory performance (very noticeable when clicking through the VMware vCenter or VMware ESXi web GUI)
  • The overall stability of the host, this because of the “high” failure rate of the SD card.

The summary of advantages is based on my own comparison between SD cards en SATADOMs in my ESXi Hosts in my Home Lab.

Delock SATADOM Specifications

Here are the specifications of the Delock Satadom devices I am using for both HPE ProLiant DL20 Gen9 servers. Here are some tips about what I have learned so far… I bought them in 2018 so they are not brand new anymore:

  • Buy them a little bit bigger because of the future proof > minimal 32GB I would suggest.
  • Verify before buying if you need the vertical or horizontal model (rack model server go for horizontal / tower model server no really important).

So here are the specifications from the Delock website:

ItemValue
VendorDelock
TypeSATA 6 Gb/s Flash Module 16 GB vertical
Part nr54655
Capacity16 GB
InterfaceSATA 6 Gb/s, SATA 3 Gb/s, SATA 1.5 Gb/s
Performance460 MB/s read – 160 MB/s write
Power usage1.0 W max. (5V x 200mA)

SATADOM Installation

So now it is time to install the device on the server. Of course, it is a little more complicated in a small half-size rack server. For example, there are no Molex power connections available by default. So in the end the cable kit is almost more expensive than the device itself. The preferred option should be to find an HPE cable kit, not sure which one you will need. So after some thinking and looking into the server I came up with the following solution to just plugin the SATADOM.

At first, I needed to find a SATA port on the motherboard. Both ports are available in my case but I used the one that is normally used for the DVD ROM drive number 14 (see the image from the HPE manual).

The storage device itself can be placed in the space of the storage controller battery pack. Both of my machines do not have the expensive storage controller option. Only the onboard default controller. So the space is completely empty and an easily accessible location for the SATADOM.

The power is the most difficult one. I ended up with converters to into the power connection from the storage backplane (keep in mind my server has no internal storage except the boot device (the SATADOM in this post…) If you have your storage filled with SSDs or HDDs you need to figure out a new solution where to get the power from. I have read something about a power kit for the DVD ROM for example. I have never seen it on a picture or in a server so I do not know which connectors are in that cable kit but it might be an option.

To make some more sense and pictures explain more than words… Here is a gallery with some pictures of the SATADOM installation:

DL20 Gen 9 BIOS Settings

After the physical installation, it was time to set up the BIOS. To be honest it was quite easy compared to the HPE Gen8 where I had a lot of problems because of the ports and bios settings.

Here are two screenshots. The first one is the activation of the internal storage controller. Note: make sure you power cycle the machine before the SATADOM is detected. After the power cycle, the VMware ESXi installer should detect the SATADOM when trying to install VMware ESXi.

After this point, the SATADOM installation is completed. Just continue your normal procedures and put your host into production when you are done.

Wrap-up

So that is it for today…! I hope it was useful for other people and interesting to read. Keep in mind this blog post was focused on the HPE ProLiant DL20 Gen9 but I think the procedure will be quite identical to other HPE Gen9 servers. The most difficult part will always be the cabling and after that, the BIOS settings to get the device detected correctly.

So far my hosts have been running for about 40+ days without any issues and are working perfectly fine. If you got additional questions or remarks please respond in the comment section below. Thanks for reading my blog post and see you next time.

HPE ProLiant DL20 Gen9 Home Lab

This blog post is about replacing my current 24×7 Lab with a new set of two HPE ProLiant DL20 Gen9 servers. In this blog post, I am going to tell you about the configuration of the machines and how they are running on VMware ESXi. Also, I am going to compare them to my other lab hardware and my past home lab equipment.

Hardware

So let’s kick off with the hardware! The HPE DL20 Gen 9 servers I bought were both new in the box from eBay and I changed the hardware components to my own liking.

A couple of interesting points I learned so far nearly all servers that you will find for sale are provided with an Intel Xeon E3-12XX v5 processor. One item you need to take into account: yes you can swap the CPU from a v5 to a v6 like I did but you need to replace the memory modules also! The memory modules are compatible with a v5 or v6 processor but not both ways. The Intel Xeon E3-12XX v5 CPUs are using 2133 MHz memory and the Intel Xeon E3-12XX v6 CPUs are using 2400 MHz memory. So keep that in mind when swapping the processor and/or buying memory.

In the end, after some swapping of components, I ended up with the following configuration. Both ProLiant servers have an equal configuration (like it should be in a vSphere cluster):

ComponentItem
Vendor:HPE
Model:DL20 Gen9
CPU:Intel® Xeon® Processor E3-1230 v6
Memory:64GB DDR4 ECC (4 x 16GB UDIMM @2400MHz)
Storage:32GB SD card on the motherboard
Storage controller:All disabled
Network card(s):HPE Ethernet 1Gb 2-port 332i Network Adapter
Expansion card(s):HPE 361T Dual-Port 2x Gigabit-LAN PCIe x4
Rackmount kit:HPE 1U Short Friction Rail Kit

Power usage

So far I have measured the power usage of the machines individually with the listed configuration in the hardware section. When measuring the power usage the machine was running VMware ESXi and on top of about seven virtual machines that were using about 30% of the total compacity. I was quite amazed by the low power consumption of 31.7 watts per host but I have to take into account that this is only the compute part! The hosts are not responsible for storage. Here is a photo of my power meter when performing the test:

Screenshot(s)

Here are some screenshot(s) of the servers running in my home lab environment and running some virtual machine workload:

  • Screenshot 01: Is displaying one of the hosts running VMware ESXi 6.7 (screenshot from HPE iLO).
  • Screenshot 02: Is displaying one of the hosts connected to VMware vCenter and running virtual machines.
  • Screenshot 03: Is displaying one of the hosts HPE iLO web page.

Positives & Negatives

To sum up, my experience I have created a list of positives and negatives to give you some insight into the HPE ProLiant DL20 Gen9 as a home lab server.

Positives:

  • A lot of CPU power compared to my previous ESXi hosts, link to the previous setup.
  • Rack-mounted servers (half-size deep with sliding rails).
  • Out of band management by default (HPE iLO).
  • Power usage is good for the amount of compute power delivered.
  • No additional drivers are required for VMware ESXi to run.
  • The HPE DL20 Gen9 has been on the VMware HCL, link.

Negatives:

  • Noisy compared to my previous setup (HPE ProLiant ML10 Gen8). For comparison, the HPE ProLiant DL360 Gen8 is in most cases “quiet” compared to the HPE ProLiant DL20 Gen9.
  • Would be nice if there was support for more memory because you can never have enough of that in a virtualization environment ;).

Photos

Here are some photos of the physical hardware and the internals, I did not take any pictures of the hardware when the components were all installed. I am sorry :(.

  • Screenshot 01 – Is displaying both machines running and installed in the 19″ server rack.
  • Screenshot 02 – Is displaying the internals of the DL20 Gen9. Keep in mind this one is empty. As you can see in that picture the chassis is just half-size!

Wrap-up

So that concludes my blog post. If you got additional questions or remarks please respond in the comment section below. Thanks for reading my blog post and see you next time.

VMware vCenter SNMP Configuration

VMware vSphere 6.7 Logo

In this last blog of the year, we are going to set up the SNMP agent on VMware vCenter Server. This blog will cover the vCenter SNMP configuration and I will show some debugging examples to verify the working of the SNMP Agent. In my case, I am using Zabbix Server as the monitoring program to verify the status of my VMware vCenter Server in my lab environments. This reduces the amount of manual troubleshooting and ensures that services are running as expected.

The reason why I did this write-up was because of the lack of documentation from the vendor’s website. As you can see in the source pages below there is a limited set of commands and nearly no examples. To set up my environment I needed some additional commands to get everything working correctly.

Environment

The environment where I configured the SNMP agent was on a VMware vCenter Server 6.7 update 3 (VCSA /appliance). I am monitoring the VMware vCenter Server with a Zabbix Server that is running on CentOS 8. I am currently using SNMP v2 in this example because it is used by most people.

Keep in mind: SNMP v1 and v3 are also supported by both products. My recommendation is to use SNMP v3 of course because of the security improvements like authentication & encryption :).

SNMP

A quick explanation about SNMP (thanks Wikipedia): Simple Network Management Protocol (SNMP) is an Internet Standard protocol for collecting and organizing information about managed devices on IP networks and for modifying that information to change device behavior. Devices that typically support SNMP include cable modems, routers, switches, servers, workstations, printers, and more.

SNMP is widely used in network management for network monitoring. SNMP exposes management data in the form of variables on the managed systems organized in a management information base (MIB) which describes the system status and configuration. These variables can then be remotely queried (and, in some circumstances, manipulated) by managing applications.

Commands

Here are the commands I have used for the vCenter SNMP configuration. Note: make sure you have access to the root account to perform the login.

# Step 1: Start an SSH connection with the vCenter Server (use Putty or something equivalent).

# Step 2: Login as the root user

# Step 3: After a successful login you should be in the appliance Shell.

# Step 4: View the current configuration for SNMP
snmp.get

# Step 5: Configure the SNMP Community (in this example I use MySnmpCommunity)
snmp.set --communities MySnmpCommunity

# Step 6: Allow a device to access the SNMP agent (192.168.10.10 = monitoring server)
snmp.set --targets 192.168.10.10@161/MySnmpCommunity,172.0.0.1@161/MySnmpCommunity,localhost@161/MySnmpCommunity

# Step 7: Enable the SNMP Agent
snmp.enable

# Step 7: Verify the SNMP Settings configured
snmp.get

# Step 8: Test the working (in my case it never works... not sure why? Has something to do with my access restrictions?)
snmp.test

# Step 9: Perform a test from the monitoring server (in my case a Linux machine with snmpwalk)
snmpwalk -v2c -c MySnmpCommunity %hostname-vcenter%

Screenshots

Here are some screenshots related to the SNMP configuration:

Wrap-up

So that is it! Hopefully, this blog post was useful and this wraps-up 2020. See you next year and if you have any comments please respond below.

Sources

Here are some sources I used when configuring SNMP on VMware vCenter Server:

vRealize Orchestrator Identifying Version Running

In this blog post, I am showing a simple vRealize Orchestrator action that receives information about vRealize Orchestrator nodes. This can also be used against remote nodes to compare orchestrator versions between different nodes. It displays the product version, product build, and API version.

So why do you want to verify that? Lately, a hot topic surrounding the vRealize Orchestrator software is migrations. This is because most customers are moving away from version 7 to version 8 (here you see vRO 8.X in action). So as a VMware consultant, you run into questions from customers about compatibility and integration use cases.

Below I will share the code and a video about using the action. You mean workflow right? No since vRO 8.0 you can run the action directly you do not need a workflow around it.

Code explained

Some explanation about the action called “troubleshootVroVersion“:

  • The action requires one input parameter that is called ‘fqdn’. Here you enter for example (vro.domain.local). This action detects which URL and port are required so it automatically supports the following scenarios:
    • This can be a standalone node, an embedded node (inside vRA), the central load balancer in front of the nodes.
    • There is support for the vRealize Orchestrator 7.X version and vRealize Orchestrator 8.X version.
  • No authentication is required because the leveraged API page is publically available without authentication.
  • The only port required between the Orchestrator that is executing the action and the remote Orchestrator is HTTPS TCP 443.

vRO Configuration

Here is an image of the configured vRO Action. You can see the input and return type configured. Also, you see the configured language that is used “JavaScript“.

Video

I have created a recording of a vRealize Orchestrator node running the action against itself. This can also be done against a remote vRealize Orchestrator node as explained before. This might also help somebody to create the action on his own orchestrator.

Code

Here is the code for the action and also the action configuration details for creating the action in vRealize Orchestrator:

// Input validation
if (!fqdn) {
    throw "The input variable 'fqdn' is null, this is not allowed!";
}

// Determine vRO Port
try {
    // Port 8181
    url = "https://" + fqdn + ":8181/vco/api/about";

    // Create URL object
    var urlObject = new URL(url);

    // Retrieve content
    var result = urlObject.getContent() ;

    // Message
    System.log ("Found a vRealize Orchestrator node on port 8181");
}
catch (error) {
    System.log ("No vRealize Orchestrator node found on port 8181 (" + error.message + ")");
}
try {
    // Port 443
    url = "https://" + fqdn + ":443/vco/api/about";

    // Create URL object
    var urlObject = new URL(url);

    // Retrieve content
    var result = urlObject.getContent() ;

    // Message
    System.log ("Found a vRealize Orchestrator node on port 443");
}
catch(error) {
    throw "Could not find any vRealize Orchestrator node on port 443 & 8181 (" + error.message + ")";
}

// JSON Parse
try {
    // Parse JSON data
    var jsonObject = JSON.parse(result);
}
catch (error) {
    throw "There is an issue with the JSON object (" + error.message + ")";
}

// Output data to screen
try {
    System.log("===== " + fqdn + " =====");
    System.log("Version: "+ jsonObject.version);
    System.log("Build number: "+ jsonObject["build-number"]);
    System.log("Build date: "+ jsonObject["build-date"]);
    System.log("API Version: "+ jsonObject["api-version"]);
}
catch (error) {
    throw "There is something wrong with the output, please verify the JSON input (" + error.message + ")";
}

GIT

Here is the Git Repository related to the code as shown above. The action used in the blog post is called “troubleshootVroVersion.js” inside the Git repository that is available on this URL.

Wrap Up

So that is it for today. In this blog post, I showed you an action to retrieve quickly some information about the Orchestrator version. As you can see in the code it is using a JSON object that is retrieved from a URL. This code is because that part easily usable for other items. So happy coding in vRO and see you next time!