Author: Mischa Buijs

HPE ProLiant Removing SD Card iLO Degraded

Recently I was removing an SD card from one of my lab servers but after removing it, the server kept complaining about it. The HPE ProLiant is equipped with HPE Integrated Lights-Out (iLO). This is an out-of-band management system to manage and configure the server. It also is responsible for monitoring the components inside the server.

This means it also monitors the health state of the SD card that is located on the motherboard slot. So when I removed the SD card it just kept checking the health of the component and causing health alerts.

In this blog post, I going to explain what I did to reset the HPE iLO to stop it from monitoring the SD card after permanent removal.

Environment

Here is a short list of information about the HPE ProLiant system that I used for this blog post:

  • Hardware: HPE Proliant DL360e Gen 8
  • HPE ilO version: 4
  • HPE SD card: HP 32GB SD card / Part nr: 700135-001
  • Firmware: HPE iLO version: 2.78
  • Software: VMware ESXi 7.0.3

Location – SD Card

To make the blog post complete I added the motherboard drawing from the HPE manual. The SD card slot is located on the HPE ProLiant DL360e Gen 8 motherboard and the slot is located at number 29 in the drawing below.

Problem – Removing SD card causes degraded state

The issue occurred when the SD card failed. After the SD card failure, I removed the SD card from the system and moved to an SSD-based boot media for VMware ESXi.

I performed some basic troubleshooting like removing the power from the server and restarting the HPE iLO but the health status was still degraded and it was still searching for the SD card.

Here are the error messages in the interface:

  • Error message on the login page: iLO Self-Test report a problem with: Embedded Flash/SD-CARD. View details on Diagnostics page.
  • Error message on diagnostics: Controller firmware revision 2.10.00 NAND read failure

Here are some screenshots related to the error messages:

Resolving – Resetting the SD card slot

Resolving the issue isn’t partially hard… if you know which buttons to push and in what order ;). Before starting, make sure the SD card is removed from the system and that the iLO has been rebooted.

To make sure that everything just works directly… open a clean browser and login into the iLO and directly follow the procedure described below.

Closing words

In the end, it cost me about three hours to get it fixed. The reason why I wanted it so badly fixed was that it kept triggering my monitoring system and that drove me crazy. This server in particular powers on and powers off regularly and during every power cycle, the health state resets and triggers monitoring alerts.

This wraps up the blog article hopefully it is useful for somebody, please respond below if you have any comments or additional information! See you next time! 🙂

Digitus Patch Cabinet Covering Fan Holes

For the last couple of months, I was busy making my new patch cabinets ready for use. One of the things I noticed with the Digitus patch cabinet is that there are openings on the top by default for the fan installation (fan holes/mounting holes). This is great for people that install the optional fan kit from Digitus but not everyone needs active fans and it is about 210/230 euros.

When you do not install the optional fan kit you will run the following problem… dust! The openings and holes on top of the cabinet allow dust to fall into the cabinet on top of your patch panels and switches. When cleaning the cabinet you need to be careful so you do not swipe the dust into large fan mounting holes and bolt hols.

So it was time to look for a solution without buying the Digitus expensive optional fan kit for basically no reason in my case… One of my requirements was no drilling or modifying of the cabinet because that would inevitably block me from installing the fan kit (in case I ever need them).

Part List

So let’s get started here is an overview of the part that I ordered online for one enclosure:

  • 2x Fan Filter (AABCOOLING Plastic Filter 120 Black OFG016 @ Amazon)
  • 8x Bolt (Bolt m5x12mm @ RVSpaleis)
  • 8x Nut (Nut with locking @ RVSpaleis)
  • 12x Plastic covers for 5 mm holes (Premium Afdekdopjes voor Boorgaten (specifications: 5 mm / RAL 9005 black) @ UwCover.nl)
  • Total costs: ~ 30 euro
  • Saving: ~ 220 – 30 = 190 euro
  • Percentage: ~ 86%

Keep in mind:

  • The items were ordered in The Netherlands (Yes, it is a country on planet earth).
  • The items were ordered in June 2022.
  • The prices might change over time.

Photos

In my home, I have two of the same Digitus Patch Cabinets mounted next to each other. That is why some of the pictures look double ;). No worries you do not have to go to the optometrist!

So here is a photo of the top side of the patch cabinet before the installation, as you can see large holes that attract a lot of dust. As you can see there are bolt holes and fan holes. Both of those we are going to cover up.

Here is a list of the parts that I ordered: Some basic product images to give you an idea of what you need. The links to the webshop are listed in the parts list.

Here are the photos when installing the parts for covering the fan holes in the cabinet. The last picture is the final result of both patch cabinets in a row.

So here is the result after installing all the components on both of the patch cabinets, the fan holes and bolt holes are covered and no dust gets in anymore:

Wrapup

In my opinion quite a nice retrofit for the Digitus patch cabinet and not too expensive. The nice thing is you can just remove the fan covering and everything is back to stock. This makes it possible to install the Digitus fan kit unit at a later stage without limiting yourself.

This wraps up the blog article hopefully it is useful for somebody, please respond below if you have any comments or additional information! See you next time! 🙂

NLVMUG 2022 Event

Today a blog about the NLVMUG 2022 event, which I attended on the fourteenth of June 2022. The event took place in the DeFabrique in Utrecht The Netherlands.

Lots of parking spaces and easy to reach as it is centrally located in the country. The location was the same location used for the NLVMUG in 2019 (the last physical VMUG before COVID).

For me personally, this was my first physical industry event since VMworld 2019 in Barcelona Spain. So it was great to be back at a physical conference and to actually meet up with a lot of familiar faces such as ITQ/VMware colleges and community members.

There was a great lineup and a lot of sessions you could choose from. Here is an overview of the available sessions of the NLVMUG 2022:

I personally attended a couple of sessions, as you can see in the photos below. Overall the sessions were good quality and with great content! Very nice to see that a lot of people love to share and spend time on prepare those sessions!

NLVMUG 2022 Photos:

I took some photos of the event to share on my blog to give people an idea of what a VMUG is and maybe inspire them to join the next NLVMUG in 2023 (if there isn’t a new … virus or something else).

So this wraps up my short blog post about the NLVMUG 2022. I would like to thank the NLVMUG team and my employer (ITQ) for making it possible for me to be there!

For me the next physical event will be VMworld … VMware Explore so maybe see you there? See you next time!

Domain Controller NTP Server Configuration

This blog post is about Microsoft Domain Controllers and NTP Server configuration for the FSMO PDC emulator role. In this blog, we are going to configure NTP and make sure it works correctly for all the clients.

Background

In this case, we were deploying an entire VMware environment at a customer for testing purposes including domain controllers. This was working fine until we wanted to configure the vCenter Server with domain authentication. It was not possible to login into VMware vCenter with domain accounts only with local accounts. After some looking and comparing we realized that the time synchronization in the environment was not working correctly.

I have been sitting on these simple commands for a long time without actually putting them on my blog but the customer that was suffering from this issue encouraged me to do this. So here we are ;).

Why an NTP Server is important

Short introduction on the Network Time Protocol or in short NTP. NTP is used for clock syncronization between IT systems. The NTP protocol uses port 123 and sends UDP packages. Now going back to Microsoft Domain Controllers ;).

One of the main things, when you are setting up a Domain Controller is that you need to make sure that time synchronization is working. This can be done with an NTP source or NTP GPS source. In most cases, an internet source is used, for example pool.ntp.org.

The domain controllers that are controlling the domain act as NTP sources for the domain members. This is the default setting by Microsoft. Keep in mind: that this can be changed with domain group policies.

So what is acceptable and when are things falling apart? By default a 300 seconds difference is allowed by Kerberos, this sounds a lot but when NTP is not working things quickly go beyond that limit. This 300 seconds interval means that you can be 300 seconds behind or 300 seconds in front of the domain controllers. When you are beyond this limit, Kerberos tickets are not working anymore and then there is a real problem: account login, log entry timestamps are not valid, permissions not working, etc.

Command support

I have tested these commands on the following Operating System versions over the years. It doesn’t matter if this is a server with a GUI or core. As long as the PowerShell terminal is accessible!

  • Windows Servers 2008
  • Windows Servers 2008 R2
  • Windows Servers 2012
  • Windows Servers 2012 R2
  • Windows Servers 2016
  • Windows Servers 2019
  • Windows Servers 2022

Commands

Here is an overview of the NTP commands for a domain controller. The most common way is to run this in a PowerShell terminal with administrative rights. I have listed the steps with a simple explanation:

# 1. Stop the NTP service
net stop w32time

# 2. Run the following command
w32tm /config /syncfromflags:manual /manualpeerlist:"0.nl.pool.ntp.org, 1.pool.ntp.org, 2.pool.ntp.org, 3.nl.pool.ntp.org"

# 3. Trust the time server
w32tm /config /reliable:yes

# 4. Start the NTP service
net start w32time

# 5. Check the configuration
w32tm /query /configuration

# 6. Force time resync
w32tm /resync

Troubleshooting

Here are some troubleshooting commands to validate the working and additional settings:

# Show timezone
w32tm /tz

# NTP Peers/Stratum
w32tm /query /peers

# Sync information
w32tm /query /peers /verbose 

# Monitor status
w32tm /monitor

# Read registry values
reg query HKLM\system\CurrentControlSet\Services\W32Time\TimeProviders\NtpServer

Wrap-up

So this is my blog post about Microsoft Domain Controllers in combination with an NTP server. I have explained the importance and the issues that can be caused by an invalid NTP Server configuration and provided the commands related to a good configuration.

Do you have additional tips or remarks please respond in the comment section below.

Have a nice day and see you next time!

VMware Tanzu HAProxy Unattended Deployment

In this blog post, I am going to share my script to automate the HAProxy deployment for Tanzu with vSphere or in short TKGs. Because of my interest in the Tanzu product family, I ended up testing and redeploying parts of TKG many times. To reduce the change of mistakes and improve my speed I automated the HAProxy deployment part. To start with a special thanks to William Lam for this blog post which pointed me in the right direction for automating the HAProxy OVA file.

Goal

The reason behind the creation of the code was the following:

  • Automate as much as possible
  • Standardize deployment
  • Streamline the process
  • Improved the speed of the deployment process

Environment

My environment for performing this unattended deployment of HAProxy is listed below. All additional requirements can be found in the README.md file in the GIT Repository like DNS records etc.

  • Server:
    • VMware ESXi 7.0 Update 3
    • VMware vCenter 7.0 Update 3
  • Workstation:
    • OS: Windows 10
    • Components required: PowerShell and PowerCLI

Recording

Here is a recording of the HAProxy unattended deployment in my lab environment. I have changed the variables in the script to match my environment. You must change the variables in a way so that it matches your environment to perform a successful deployment.

Code

Here is an overview of the code and a link to the GIT repository. Keep in mind to always use the GIT repository version of the code because there could be new improvements.

HAProxy Tanzu Deployment:

<#
    Script: HAProxy Tanzu Deployment
    Author: M. Buijs
    Original concept developed by: William Lam - https://github.com/lamw/vmware-scripts/blob/master/powershell/deploy_3nic_haproxy.ps1
    version: 1.0 - 2021-12-17
    Execution: HAProxy_Deployment.ps1
#>

# Set variables

	# Script variables
	$global:script_name = "HAProxy_Tanzu_Deployment"
	$global:script_version = "v1.0"
	$global:debug = 0
    $global:temp_directory = "C:\Temp\"

    # vSphere
    $vCenter = "LAB-VC01.Lab.local"
    $ClusterName = "Lab"
    $DatastorePrefix = "iSCSI - Production - *" # datastore prefix
    $DiskProvisioning = "thin" # thin or thick
    $Hardware = "v14" # Virtual hardware

    # HAProxy General
    $HAProxyDisplayName = "LAB-HAProxy01"
    $HAProxyHostname = "lab-haproxy01.lab.local"
    $HAProxyDNS = "192.168.126.21, 192.168.126.22"
    $HAProxyPort = "5556" # 5556 default port

    # HAProxy Management
    $HAProxyManagementNetwork = "Management"
    $HAProxyManagementIPAddress = "192.168.151.40/24" # Format is IP Address/CIDR Prefix
    $HAProxyManagementGateway = "192.168.151.254"

    # HAProxy Frontend
    $HAProxyFrontendNetwork = "TKG - Frontend"
    $HAProxyFrontendIPAddress = "192.168.127.40/24" # Format is IP Address/CIDR Prefix
    $HAProxyFrontendGateway = "192.168.27.254"
    $HAProxyLoadBalanceIPRange = "192.168.127.128/26" # Format is Network CIDR Notation

    # HAProxy Workload
    $HAProxyWorkloadNetwork = "TKG - Workload"
    $HAProxyWorkloadIPAddress = "192.168.128.40/24" # Format is IP Address/CIDR Prefix
    $HAProxyWorkloadGateway = "192.168.128.254"

    # HAProxy Users
    $HAProxyUsername = "haproxy_api"

# Functions
function banner {
    # Clear
	Clear-Host

	# Clear errors
	$Error.clear()

    # Message
    Write-Host "`n---------------------------------------------------------" -foreground Red
    Write-Host "               $script_name - $script_version" -foreground Red
    Write-Host "---------------------------------------------------------" -foreground Red
}

function script_exit {
	Write-Host -Foreground Yellow ""
	Write-Host -Foreground Yellow "ERROR Message: $($Error[0].Exception.Message)"
	Write-Host -Foreground Yellow ""
	Write-Host -Foreground Cyan "Exiting PowerShell Script..."
	exit
}

function validate_media {
    ##### Message
    Write-Host "`nValidating media:"

        #### Locate temp directory
        If (-not (Test-Path "$($Temp_Directory)")) {
            Write-Host -ForegroundColor Red "- The temp directory is not created ($Temp_Directory)"
            script_exit
        }
        else {
            Write-Host -ForegroundColor Green "- Located the temp directory ($Temp_Directory)"
        }

        #### Locate OVA file
        Try {
            Write-Host -ForegroundColor Green  "- Searching for OVA file"
            $script:OVF_HAProxy = $(Get-ChildItem -Path "$Temp_Directory" -Include haproxy-v*.ova -File -Recurse -ErrorAction Stop | Sort-Object LastWriteTime | Select-Object -last 1)

            ### In case of no results
            if ([string]::IsNullOrEmpty($OVF_HAProxy.name)) {
                throw
            }
            #### Message
            Write-Host -ForegroundColor Green "- Located HAProxy OVA file ($($OVF_HAProxy.Name))"
        }
        Catch {
            Write-Host -ForegroundColor Red  "- Could not find HAProxy OVA file in location ($Temp_Directory)"
            script_exit
        }
}

function ask_passwords {
    # Banner
    Write-Host "`nPasswords:"

    # Ask passwords
    $script:HAProxyOSPassword = Read-Host -asSecureString "- Enter the HAProxy user password (root)"
    $script:HAProxyPassword = Read-Host -asSecureString "- Enter the HAProxy user password ($HAProxyUsername)"

    # Validation
    If ($HAProxyOSPassword.Length -eq 0) {
        Write-Host -ForegroundColor Red "- HAProxy root account password is empty"
        script_exit
    }
    # Validation
    If ($HAProxyPassword.Length -eq 0) {
        Write-Host -ForegroundColor Red "- HAProxy user account password is empty"
        script_exit
    }
}

function connect_vcenter {
    # Banner
    Write-Host "`nvCenter connection:"

        # Disable vCenter deprecation warnings
        Set-PowerCLIConfiguration -DisplayDeprecationWarnings $false -Confirm:$false | Out-Null

        # Disable vCenter certification errors
        Set-PowerCLIConfiguration -InvalidCertificateAction "ignore" -Confirm:$false | Out-Null

        # Determine script or user input
        if ($vCenter) {
            Write-Host -ForegroundColor Green "- Connecting with vCenter server ($vCenter)"
        }
        else {
            # Ask required vCenter information
            $script:vCenter = Read-Host "- Enter the vCenter IP address or hostname"
        }

        if ($global:DefaultVIServers.Count -gt 0) {
            Write-Host -ForegroundColor Green "- Session already established ($vCenter)"
        }
        else {
            # Check IP address for connectivity
            if (test-connection -computername $vCenter -count 1 -quiet -ErrorAction SilentlyContinue) {
                Write-Host -ForegroundColor Green "- Host is alive ($vCenter)"
            }
            else {
                Write-Host -ForegroundColor Red "- Host is not responding ($vCenter)"
                $vCenter = ""
                Break
            }

            # Connect with vCenter
            try {
                Write-host -ForegroundColor Green "- Connecting to vCenter, please wait..."

                # Connect to vCenter
                Connect-ViServer -server $vCenter -ErrorAction Stop | Out-Null
            }
            catch [Exception]{
                $status = 1
                $exception = $_.Exception
                Write-Host "- Could not connect to vCenter, exiting script" -foreground Yellow
                Write-Host ""
                Write-Host "Exit code: $status" -foreground Yellow
                Write-Host "Output: $exception" -foreground Yellow
                Break
            }
        }

        # Message
        Write-Host -ForegroundColor Green "- Connection successful"
}

function ovf_config {
    # Banner
    Write-Host "`nOVF Configuration:"

    # Start
    Write-Host -ForegroundColor Green "- Creating OVF Configuration"

    $script:ovfconfig = Get-OvfConfiguration $OVF_HAProxy

    # Three nic configuration
    $script:ovfconfig.DeploymentOption.value = "frontend"

    # General
    $script:ovfconfig.network.hostname.value = $HAProxyHostname
    $script:ovfconfig.network.nameservers.value = $HAProxyDNS
    $script:ovfconfig.loadbalance.dataplane_port.value = $HAProxyPort

    # Network port groups
    $script:ovfconfig.NetworkMapping.Management.value = $HAProxyManagementNetwork
    $script:ovfconfig.NetworkMapping.Frontend.value = $HAProxyFrontendNetwork
    $script:ovfconfig.NetworkMapping.Workload.value = $HAProxyWorkloadNetwork

    # Management
    $script:ovfconfig.network.management_ip.value = $HAProxyManagementIPAddress
    $script:ovfconfig.network.management_gateway.value = $HAProxyManagementGateway

    # Workload
    $script:ovfconfig.network.workload_ip.value = $HAProxyWorkloadIPAddress
    $script:ovfconfig.network.workload_gateway.value = $HAProxyWorkloadGateway
    $script:ovfconfig.loadbalance.service_ip_range.value = $HAProxyLoadBalanceIPRange

    # Accounts
    $script:ovfconfig.loadbalance.haproxy_user.value = $HAProxyUsername

    # Password root
    $BSTR1 = [System.Runtime.InteropServices.Marshal]::SecureStringToBSTR($HAProxyOSPassword)
    $HAProxyOSPassword = [System.Runtime.InteropServices.Marshal]::PtrToStringAuto($BSTR1)
    $script:ovfconfig.appliance.root_pwd.value = $HAProxyOSPassword

    # Password user
    $BSTR2 = [System.Runtime.InteropServices.Marshal]::SecureStringToBSTR($HAProxyPassword)
    $HAProxyPassword = [System.Runtime.InteropServices.Marshal]::PtrToStringAuto($BSTR2)
    $script:ovfconfig.loadbalance.haproxy_pwd.value = $HAProxyPassword

    # Finish
    Write-Host -ForegroundColor Green "- Completed OVF Configuration"
}

function pre_deployment {
    # Banner
    Write-Host "`nPre-deployment:"

    # Cluster
    $script:Cluster = Get-Cluster $ClusterName
    Write-Host -ForegroundColor Green "- Selected cluster ($Cluster)"

    # VMhost
    $script:VMHost = Get-VMHost | Where-Object { $_.ConnectionState -eq "Connected" } | Get-Random
    Write-Host -ForegroundColor Green "- Selected ESXi Host ($VMHost)"

    # Datastore
    $script:Datastore = Get-VMhost -Name $VMHost | Get-Datastore -Name $DatastorePrefix | Select-Object Name, FreeSpaceGB | Sort-Object FreeSpaceGB -Descending | Select-Object -first 1 | Select-Object Name -expandproperty name
    Write-Host -ForegroundColor Green "- Selected datatore ($Datastore)"

    # Check virtual machine name exists
    $VMname_check_query = Get-Cluster -Name $ClusterName | Get-VM -name $HAProxyDisplayName -ErrorAction SilentlyContinue

    if (! $VMname_check_query) {
        Write-Host -ForegroundColor Green "- Virtual machine name is not in use ($HAProxyDisplayName)"
    }
    else {
        Write-Host -ForegroundColor Red "- Virtual Machine with name ($HAProxyDisplayName) already exists. Exiting script cannot continue!"
        script_exit
    }

	#### Ask for conformation
	Write-Host "`nThis task is going to build the HAProxy virtual machine for TKGs."
	$confirmation = Read-Host "Are you sure you want to proceed? [y/n]"

	if ($confirmation -eq 'n') {
		Write-Host "Operation cancelled by user!" -Foreground Red
		base_exit
	}

	if (!$confirmation) {
		Write-Host -Foreground Red "No input detected!"
	    base_exit
	}
}

function deployment {
    # Banner
    Write-Host "`nDeployment:"

	# HAProxy deployment of OVF
	try {
		### Message
		Write-Host -ForegroundColor Green "- Starting HAProxy Deployment ($HAProxyHostname / $HAProxyManagementIPAddress)"

        $script:vm = Import-VApp -Source $OVF_HAProxy -OvfConfiguration $ovfconfig -Name $HAProxyDisplayName -Location $Cluster -VMHost $VMHost -Datastore $Datastore -DiskStorageFormat $DiskProvisioning

        ### Message
		Write-Host -ForegroundColor Green "- Finished HAProxy Deployment ($HAProxyHostname / $HAProxyManagementIPAddress)"
    }
	catch [Exception]{
		Write-Host -ForegroundColor Red "- HAProxy Deployment Failed ($HAProxyHostname / $HAProxyManagementIPAddress)"
		script_exit
	}
}

function post_deployment {
    # Banner
    Write-Host "`nPost-deployment:"

	# Configure OVF
	try {
		### Message
		Write-Host -ForegroundColor Green "- Starting HAProxy OVF Configuration ($HAProxyHostname / $HAProxyManagementIPAddress)"

        $vappProperties = $vm.ExtensionData.Config.VAppConfig.Property
        $spec = New-Object VMware.Vim.VirtualMachineConfigSpec
        $spec.vAppConfig = New-Object VMware.Vim.VmConfigSpec

        $ovfChanges = @{
            "frontend_ip"=$HAProxyFrontendIPAddress
            "frontend_gateway"=$HAProxyFrontendGateway
        }

        ### Message
		Write-Host -ForegroundColor Green "- Finished HAProxy OVF Configuration ($HAProxyHostname / $HAProxyManagementIPAddress)"
    }
	catch {
		Write-Host -ForegroundColor Red "- HAProxy OVF Configuration failed ($HAProxyHostname / $HAProxyManagementIPAddress)"
		script_exit
	}

    try {
        # Message
		Write-Host -ForegroundColor Green "- Starting HAProxy Update Specification ($HAProxyHostname / $HAProxyManagementIPAddress)"

        # Retrieve existing OVF properties from VM
        $vappProperties = $VM.ExtensionData.Config.VAppConfig.Property

        # Create a new Update spec based on the # of OVF properties to update
        $spec = New-Object VMware.Vim.VirtualMachineConfigSpec
        $spec.vAppConfig = New-Object VMware.Vim.VmConfigSpec
        $propertySpec = New-Object VMware.Vim.VAppPropertySpec[]($ovfChanges.count)

        # Find OVF property Id and update the Update Spec
        foreach ($vappProperty in $vappProperties) {
            if($ovfChanges.ContainsKey($vappProperty.Id)) {
                $tmp = New-Object VMware.Vim.VAppPropertySpec
                $tmp.Operation = "edit"
                $tmp.Info = New-Object VMware.Vim.VAppPropertyInfo
                $tmp.Info.Key = $vappProperty.Key
                $tmp.Info.value = $ovfChanges[$vappProperty.Id]
                $propertySpec+=($tmp)
            }
        }
        $spec.VAppConfig.Property = $propertySpec

        # Message
		Write-Host -ForegroundColor Green "- Finished HAProxy Update Specification ($HAProxyHostname / $HAProxyManagementIPAddress)"
    }

    catch {
        # Message
        Write-Host -ForegroundColor Red "- HAProxy Update Specification failed ($HAProxyHostname / $HAProxyManagementIPAddress)"
		script_exit
    }

    # HAProxy reconfigure task for virtual machine
    try {
        # Message
        Write-Host -ForegroundColor Green "- Start Reconfigure VM task ($HAProxyHostname / $HAProxyManagementIPAddress)"
        $task = $vm.ExtensionData.ReconfigVM_Task($spec)
        $task1 = Get-Task -Id ("Task-$($task.value)")
        $task1 | Wait-Task | Out-Null
    }
    catch {
        Write-Host -ForegroundColor Red "- Reconfigure VM task failed ($HAProxyHostname / $HAProxyManagementIPAddress)"
        script_exit
    }

    # Message
    Write-Host -ForegroundColor Green "- Completed the reconfigure VM task ($HAProxyHostname / $HAProxyManagementIPAddress)"
}

function boot {
    # Banner
    Write-Host "`nBoot:"

	# Upgrade Virtual Hardware
	Try {
		Write-Host -ForegroundColor Green "- Upgrade Virtual Hardware ($HAProxyHostname / $HAProxyManagementIPAddress)";
		Get-VM -Name $vm | Set-VM -Version $Hardware -Confirm:$false | Out-Null
	}
	Catch {
		Write-Host -ForegroundColor Red "- Upgrade Virtual Hardware failed ($HAProxyHostname / $HAProxyManagementIPAddress)";
		script_exit
	}

	# Power-On Virtual Machine
	Try {
		Write-Host -ForegroundColor Green "- Power-on HAProxy started ($HAProxyHostname / $HAProxyManagementIPAddress)"
		Get-VM $vm | Start-VM | Out-Null
	}
	Catch {
		Write-Host -ForegroundColor Red "- Starting HAProxy failed ($HAProxyHostname / $HAProxyManagementIPAddress)"
		script_exit
	}

    Write-Host -ForegroundColor Green "- Power-on HAProxy completed ($HAProxyHostname / $HAProxyManagementIPAddress)"
}

function check {
    # Banner
    Write-Host "`nCheck:"

    # Set total of retries
    $TOTAL = "10"

    # Host retry interval (seconds)
	$HOST_WAIT = "10";

    # Start loop
    For ($i=0; $i -le $TOTAL; $i++) {

        # Number conversion to 2 digit:
        $NUMBER = [INT]$i + 1
        $NUMBER = "{0:D2}" -f $NUMBER

        # Check Host
        $Host_check_query = Test-Connection -computername $HAProxyHostname -count 1 -quiet -ErrorAction SilentlyContinue

        # Validate, else retry after a wait
        if ($Host_check_query -eq $false) {
            Write-Host -Foregroundcolor green "- [$NUMBER/$TOTAL] Checking HAProxy availability ($HAProxyHostname)"
            Start-Sleep $HOST_WAIT
        }
        else {
            Write-Host -Foregroundcolor green "- [$NUMBER/$TOTAL] Checking HAProxy availability ($HAProxyHostname)"
            Write-Host -Foregroundcolor green "- [Ready] HAProxy is available ($HAProxyHostname)"
            break
        }
    }
}

function retrieve_certificate {
    # Banner
    Write-Host "`nRetrieve certificate:"

    # Build URL
    $script:url = "https://${HAProxyHostname}:${HAProxyPort}/v2/info"

    # Configure local system
    try {
        # Message
        Write-Host -ForegroundColor Green "- Disable certificate checking on local system"

        # Disable certificate check
        [System.Net.ServicePointManager]::ServerCertificateValidationCallback = {$true}
    }
	catch {
		Write-Host -ForegroundColor Red "- Could not disable certificate checking on local system"
		script_exit
	}

    # Download certificate
    try {
        # Message
        Write-Host -ForegroundColor Green "- Get HAProxy certificate ($url)"

        $req = [Net.HttpWebRequest]::Create($url)
        $req.ServicePoint | Out-Null

        # Authentication
        $req.Credentials = New-Object Net.NetworkCredential($HAProxyUsername, $HAProxyPassword);
    }
	catch {
		Write-Host -ForegroundColor Red "- Could not get HAProxy Certificate ($url)"
		script_exit
	}

    # Store error messages in variable to not crash a try and catch statement.
    $GetResponseResult = $req.GetResponse()

    # Store certificate as X.509 file
    try {
        # Message
        Write-Host -ForegroundColor Green "- Store HAProxy certificate as X.509 ($url)"

        $cert = $req.ServicePoint.Certificate
        $bytes = $cert.Export([Security.Cryptography.X509Certificates.X509ContentType]::Cert)
        set-content -value $bytes -encoding byte -path "$pwd\$HAProxyHostname.cer"
    }
    catch {
        Write-Host -ForegroundColor Red "- HAProxy X.509 certificate could not be saved ($url)"
        Write-Host -ForegroundColor Red "- Result from GetResponse: ($GetResponseResult)";
        script_exit
    }

    # Convert certificate to Base-64 file
    try {
        # Message
        Write-Host -ForegroundColor Green "- Store HAProxy certificate as Base-64 ($url)"

        $InsertLineBreaks=1
        $sMyCert="$pwd\$HAProxyHostname.cer"
        $oMyCert = New-Object System.Security.Cryptography.X509Certificates.X509Certificate2($sMyCert)
        $oPem = New-Object System.Text.StringBuilder
        $oPem.AppendLine("-----BEGIN CERTIFICATE-----") | Out-Null
        $oPem.AppendLine([System.Convert]::ToBase64String($oMyCert.RawData,$InsertLineBreaks)) | Out-Null
        $oPem.AppendLine("-----END CERTIFICATE-----") | Out-Null
        $oPem.ToString() | out-file "$pwd\$HAProxyHostname.pem"
    }
    catch {
        Write-Host -ForegroundColor Red "- HAProxy Base-64 certificate could not be saved ($url)"
        script_exit
    }
}

function complete_banner {
    # Message
    Write-Host -ForegroundColor Green "- HAProxy deployment completed successfully! ($HAProxyHostname / $HAProxyManagementIPAddress)"
}

##### Main
banner
validate_media
connect_vcenter
ask_passwords
ovf_config
pre_deployment
deployment
post_deployment
boot
check
retrieve_certificate
complete_banner

Wrap-up

I hope this blog about HAProxy unattended deployment was useful for some people or that the PowerShell code inspires people to deploy other types of OVA appliances.

If you got any improvements please commit them to the GIT repository and if you got any questions please respond below. Thank you for reading my blog post and see you next time!

Windows Server 2022 Core DC Boot Loop

On 11 January 2022 Microsoft released an update that puts some Microsoft Domain Controllers in a boot loop. This is related to Windows Server 2016, Windows Server 2019, and Windows Server 2022. When my lab environment was running into issues I noticed that I could not find the updates listed online. After some searching, I realized that Windows Server 2022 Core is different and has different updates installed.

In my environment, I am running two Windows Server 2022 Core edition domain controllers. On 19 January both domain controllers came into a boot loop at night after installing the updates automatically with my automated patching tools.

Server with Desktop Experience

As listed on the websites the following updates are the ones you need to delete from the system:

# Remove the updates
wusa /uninstall /kb:5009595 /quiet 
wusa /uninstall /kb:5009624 /quiet

When running the command the Windows Server 2022 domain controller told me that the updates were not installed on this system… So that was quite weird… after running the following command and checking the KB article description on the Microsoft website I found the correct one (the articles I am referring to are listed below).

# List installed updates on the machine
wmic qfe list

Server Core

I have created a simple procedure and added some screenshots. Here is the procedure that I performed on each domain controller in my environment:

  1. Disconnect the virtual network card.
  2. Connect with the virtual machine console through the hypervisor.
  3. Log in with the administrator account.
  4. Open the PowerShell prompt.
  5. Remove the update, see the command below.
  6. Reboot the system.
  7. Connect the virtual network card.
  8. Log in with the administrator account.
  9. Install update KB5010796.
  10. Reboot the system.
  11. Check the available updates to make sure it is installed.
  12. Everything should be working fine again.
# Remove the update (the quiet option was not working and the removal required mouse interaction to work)
wusa /uninstall /kb:5009555

Screenshot(s)

I kind of forgot to screenshot everything because of the time between the boot loops but the end result is captured on the screenshot(s). Also added the screenshots in VMware vCenter for disabling and enabling the network card connectivity.

Wrap-up

A couple of things I have learned so far, Windows Updates KB numbers are not identical between core and desktop experience (full) versions. It was strange that the domain controllers still listed the faulty update as the first update to install and would go into the same state back…

This wraps up the blog article hopefully it is useful for somebody, please respond below if you have any comments or additional information!

Sources

VMware Tanzu HAProxy Troubleshooting

This blog post is dedicated to HAProxy Troubleshooting for vSphere with Tanzu or also known as TKGs. Based on your configuration and deployment and the various items you need to configure you can make mistakes or items are not correctly configured. In my case, there were multiple problems at different deployments with parameters and reachability related to the network. In the end, after all the hours of troubleshooting, I ended up with a list of commands that might help others out. So that is the topic of this blog post.

HAProxy Background

First an introduction about the product HAProxy. HAProxy is a load balancer that is used by vSphere with Tanzu. This is not mandatory but is a product to choose from. The main reason for HAProxy compared to the others is that it is completed free/open-source. The HAProxy OVA is packaged and delivered by VMware and can be found in the following repository. All commands below have been tested against the HAProxy v0.2.0 version (haproxy-v0.2.0.ova) that is at the moment of writing the most recent version available.

Appliance access (SSH)

After a successful deployment, you can access the HAProxy appliance with an SSH session. This session can be established with a tool like PuTTY. The user account that should be used in the root account.

Keep in mind: Do not change configuration unless you absolutely know what you are doing. Almost all the issues I ran into were related to entering incorrect information into the deployment wizard or firewall issues.

Troubleshooting Services

One of the first things to check at first is that all services are running on the HAProxy appliance. When services are not started this is mostly caused by an invalid/incomplete configuration that is filled by the deployment wizard of the OVA.

### Check failed services
systemctl list-units --state=failed

### Check primary services for HAProxy and Tanzu Integration
systemctl status anyip-routes.service
systemctl status haproxy.service

### Restart services
systemctl restart haproxy

Troubleshooting Configuration Files

There are multiple configuration files in use by HAProxy here are the most important ones. Also, keep in mind what I already said before… do not change anything unless…

### Anyip-routes configuration file
cat /etc/vmware/anyip-routes.cfg

### HAProxy configuration file
cat /etc/haproxy/haproxy.cfg

### HAProxy dataplane api configuration file
cat /etc/haproxy/dataplaneapi.cfg

### Validation of configuration file
haproxy -c -f /etc/haproxy/haproxy.cfg

Troubleshooting HAProxy process output

Sometimes it is good to check the latest messages generated by the HAProxy process. There will be information about the startup of the process and the pool members.

### Show logging
journalctl -u haproxy.service --since today --no-pager

Troubleshooting IP Settings

By entering wrong IP information in the deployment wizard the configuration files surrounding the IP address settings, gateway, etc can be configured incorrectly. What I noticed is there is not really a check inside the deployment that verifies if the address that is entered is valid in any sort of way.

### List IP Settings
ifconfig

### Config files (incase of three NIC configuration)
cat /etc/systemd/network/10-frontend.network
cat /etc/systemd/network/10-workload.network
cat /etc/systemd/network/10-management.network

### Routing check
route
ip route

Troubleshooting Certificates

Certificates files used by the HAProxy application are inside the HAProxy directory on the local system. The certificates are BASE-64 encoded!

### Certificate authority file:
cat /etc/haproxy/ca.crt

### Certificate server file:
cat /etc/haproxy/server.crt

### Certificate URL by default:
https://%HAProxy-Management-IP%:5556

Troubleshooting NTP

One of the all-time favorites that are notorious for disrupting IT systems is off course NTP. Here are some commands for troubleshooting on Photon OS.

### Check service status
systemctl status systemd-timesyncd

### Show NTP peers
ntpq -p

### Restart service
systemctl restart systemd-timesyncd

### Configuration file
cat /etc/systemd/timesyncd.conf

Troubleshooting the HAProxy API

The HAProxy API is used by Tanzu to configure HAProxy for the management and workload components. Authentication is set up when deploying the OVA and the credentials are entered in the wizard. With the second URL you can verify those credentials:

### Info page
https://%IP-address%:5556/v2/info

### Authentication should work with the HAProxy user account (specified in the deployment wizard)
https://%IP-address%:5556/v2/cluster

Wrapup

Thank you for reading this blog post about HAProxy troubleshooting for vSphere with Tanzu or in short TKGs. I hope it was useful to you! If you got something to add? Have additional tips or remarks please respond in the comment section below.

Have a nice day and see you next time.

Source

PowerCLI Update Fails on Certificate Issue

Today I was updating my PowerShell modules on my local system but I ran into an issue related to certificates. After some investigation, it appeared that the PowerCLI Update was failing. Every PowerCLI module that it tried to update returned an error and did not update to a newer version. In this blog post, I am showing you how to get rid of this error and you will be able to update PowerCLI again to the latest version.

Environment

Here is a quick summary of my environment where the issue occurred:

  • Current PowerCLI: 12.1.0.16997004
  • The system tried to update PowerCLI to: 12.4.0.18627054
  • Operating System: Windows 10 Pro (21H1)

Problem

Here is an overview of the PowerShell command I used to update my modules. Also, the error message is listed below in the code box. As you can see PowerShell is complaining about a DigiCert certificate and that it is not save to update because of the certificate change.

# Open a PowerShell command prompt with administrative permissions

# Enter the command to update the PowerShell modules
Update-Module

# The following error message appears
Install-Package: Authenticode issuer 'CN=DigiCert Trusted Root G4, OU=www.digicert.com, O=DigiCert INc, C=US' of the new module 
'VMware.VimAutomation.Sdk' with version '12.4.0.18627054' is not matching with the authenticode issuer 'CN=VeriSign Class 3 Public Primary Certification 
Authority - G5, OU="(c) 2006 VeriSign, Inc. - for authorized use only", OU=VeriSign Trust Network, O="VeriSign, Inc.", C=US' of the previously-installed
 module 'VMware.VimAutomation.Sdk' with version '12.1.0.16997004'.

Here is a screenshot of the error message, error message is listed in red text. It also shows the commands I used for the PowerCLI update on my local system.

Solution

The solution for fixing the issue is quite simple. Just reinstall the VMware PowerCLI modules on your local system. This needs to be done in a forced way but after that, you are done.

# Open a PowerShell command prompt with administrative permissions

# Enter the command to re-install the PowerShell modules
Install-Module VMware.PowerCLI -Force -SkipPublisherCheck

# Wait a couple of minutes and everything should be upgraded.
Get-Module -ListAvailable VMware.PowerCLI | Select-Object Name, Version

As you can see in the screenshots below the PowerCLI update is working and is returning no errors after the upgrade.

Wrap-up

This wraps up this small blog post about a PowerCLI update issue in PowerShell. Thank you for reading and I hope it helped you out. Please respond in the comment section below if you have any questions or remarks!

vRealize Orchestrator 8.X – Input Form Dropdown

Today a basic tutorial on vRealize Orchestrator 8.X drop-down boxes in a form. With a basic drop-down box, you can improve the user experience in selecting and requesting items from your cloud management portal (CMP). By using drop-down boxes you can leverage easy validation and responses based on other drop-down boxes in your form.

In this tutorial, we are going to create dropdown boxes that respond to each other based on the user’s selection. This can be handy for improving the user experience. Sometimes the list can become very big with numerous options. By sub-selecting a group and filtering to a smaller list of options the user can easier make his decision.

Keep in mind:

  • This tutorial is focused on vRealize Orchestrator 8.X but can still be leveraged in vRealize Orchestrator 7.X with some minor modifications.
  • This tutorial is also usable for vRealize Automation 8.X forms. This can be leveraged by the Service Broker component by importing vRealize Orchestrator workflows.

Use Case

To give you some background around the code and usability. Let’s assume we are developing a workflow for creating Virtual Machines in vSphere. Based on user input surrounding the Operating System information we can determine the type of virtual machine that will be created when the request is submitted. We can also limit the user to some standard options like only Windows 10 or Windows Server 2019.

Keep in mind: this blog post is only focused on the form part, not on the actual creation of the virtual machine in vSphere.

vRO Actions

The first action we are going to create is called “formVmOsFamily“. This will display three values in the form. Based on what you select here the second action will be triggered.

/*
Script name: formVmOsFamily

Inputs:
- None

Return Type:
- vRO 8.X: string:array

Description field:
Author: M. Buijs - ITQ
Developed by: M. Buijs - ITQ
Date: 2021-08-17
Version: 1.0.0

Description: This action returns the available Guest Family of the Operating Systems.
*/


// Operating System Family list
return [
    "Linux",
    "VMware",
    "Windows"
];

Here is the second action that is called “formVmOsGuest“. This will respond to the input provided by the operating system family in the interface.


/*
Script name: formVmOsGuest

Inputs:
- osFamily (string) = Operating System Family

Return Type:
- vRO 8.X: string:array

Description field:
Author: M. Buijs - ITQ
Developed by: M. Buijs - ITQ
Date: 2021-08-05
Version: 1.0.0

Description: This action returns the available Guest Operating Systems.
*/

// Input validation
if (osFamily == "" || osFamily == null) {
    return ["Please select the Operating System family first"];
}

// Linux
if (osFamily == "Linux")
return [
    "CentOS 6 (64-Bit)",
    "CentOS 7 (64-Bit)",
    "CentOS 8 (64-Bit)",
    "Debian 10 (64-Bit)"
];

// VMware
if (osFamily == "VMware")
return [
    "VMware ESXi 6.0",
    "VMware ESXi 6.5",
    "VMware ESXi 6.7",
    "VMware ESXi 7.0"
];

// Windows
if (osFamily == "Windows")
return [
    "Windows 10 (64-Bit)",
    "Windows Server 2016 (64-Bit)",
    "Windows Server 2019 (64-Bit)",
    "Windows Server 2022 (64-Bit)"
];

Here is an overview of screenshots of how it should look like when created the actions in vRealize Orchestrator:

vRO Workflow

Here is the vRealize Orchestrator workflow, I have created an empty workflow and only configured the input form dropdown part! This will help you to set up the workflow so that the actions will work in your environment. The important part is not to forget to configure the workflow inputs and listed below:

Inputs:

  • virtualMachineOsFamily (string)
  • virtualMachineOsGuest (string)

Recording

Here is a recorded video of the input form dropdown boxes in action. The video demonstrates the capability of dropdown boxes and what they can deliver for a customer. It also gives you an idea of what you will get after creating the workflow and actions.

Improvements/Guidance

If you are going to use this code in production… You might need to consider some important points:

  • Always create a list of supported Operating Systems that are used and allowed to be used in your Company. More options will not always simplify the deployment for an end-user.
  • You could store the values in a vRealize Orchestrator Configuration Element depending on how frequently this list is changed.

Summary

So this concludes my blog post about creating dropdown boxes in vRealize Orchestrator and reacting on the input. Hopefully, this was useful for somebody getting started with interfaces in vRealize Automation (vRA) or vRealize Orchestrator (vRO). Please respond in the comment section below if you have any questions or remarks!

vRealize Orchestrator Upgrade (8.X)

This blog post is about upgrading vRealize Orchestrator 8.X to a newer version. After a couple of vRealize Orchestrator Upgrades since the 8.0 release and getting stuck a couple of times I decided to do a simple write-up with some tips and tricks.

In my lab environment, I have got multiple orchestrators running embedded, standalone, and cluster. Most issues I encountered are related to the standalone version that is connected with the VMware vCenter Server.

vRO upgrade checks

Let’s start with some simple upgrade checks to make sure everything is working before the upgrade and to improve the chance of succeeding.

  • Make sure the root account is not expired on all nodes in the cluster.
  • Make sure you have the correct vCenter SSO password. Verify this by logging in with administrator@vsphere.local on the vCenter Server. The password is required for the standalone upgrade that is directly connected to the VMware vCenter Server.
  • Make sure the time sync is working on all the nodes in the cluster.

vRO upgrade

Let’s start with the vRealize Orchestrator Upgrade. Here is an overview of the procedure and the commands required to perform the upgrade.

Keep in mind: Step six is optional and is only required for the vRealize Orchestrator that is connected to the vCenter SSO. For the vRealize Automation connected upgrade, this step can be skipped.

Procedure:

  1. Create a virtual machine snapshot.
  2. Open an SSH session with the vRealize Orchestrator node.
  3. Login with the root account on the vRealize Orchestrator node.
  4. Mount the upgrade media to the virtual machine.
  5. Mount the media in the linux system (mount /dev/sr0 /mnt/cdrom).
  6. Enter the SSO password as a variable in the shell (export VRO_SSO_PASSWORD=your_sso_password).
  7. Start the upgrade (vracli upgrade exec -y –profile lcm –repo cdrom://).
  8. The upgrade will start. Depending on the size of the vRealize Orchestrator node it will take between 30 to 90 minutes.
  9. After the upgrade is completed restart the system (reboot).
  10. Verification:
    1. Check the virtual machine console for startup issues. Make sure the console is displaying a blue screen with information about the node.
    2. Check the virtual machine console for the version/build number on the blue screen that it is displaying.
    3. Check if the web interface is available and the interface is working.
    4. Login into the vRO interface and verify that authentication is working.
    5. Run a basic workflow.
  11. Remove the virtual machine snapshot.

Screenshot(s)

Here are a couple of screenshots of the upgrade process and the end result after a successful upgrade:

Summary

So that was my short blog post about the vRealize Orchestrator Upgrade experience so far for version 8.X. I hope it was useful. In most cases, there were problems with an expired account or an incorrect SSO password.

It would be nice if the upgrade process would validate the entered SSO password instead of hanging for hours in a crashed upgrade state without returning any error message to the console or shell session.

Thanks for reading and see you next time! Please respond in the comment section below if you got any remarks :).

Official documentation: