VMware – esxtop – troubleshooting VM CPU performance

To display VM only press V (VM only view):

then to expand specific VM press e (expand) then enter the GID number:

As this VM has 3 CPU I can see CPU usage on all of them, RDY, USED and CSTP times.

%RDY is a Key Performance Indicator!  This one defines how much time your virtual machine wanted to execute CPU cycles but could not get access to the physical CPU. It tells you how much time did you spend in a “queue”. I normally expect this value to be better than 5%. In this case we can see that this VM is struggling a bit or it is very busy.

%USED tells you how much time did the virtual machine spend executing CPU cycles on the physical CPU.

%CSTP tells you how much time a virtual machine is waiting for a virtual machine with multiple vCPU to catch up. If this number is higher than 3% you should consider lowering the amount of vCPU in your virtual machine.

VMware – capturing network packets

There are two utilities available on ESXi host out of the box that enable us to capture network traffic:

  • tcpdump (limited only to capturing traffic from vmkernel adapters)
  • pktcap
tcpdump

To list vmkernel adapters use esxcfg-vmknic -l command

tcpdump-uw -i vmk0 -s 0 -nn -e
notes:
-s 0 – indicates that we capture the entire packet (as opposed truncated packets)
-nn – indicates that we want to use numbers instead of names for the IP addresses, and for the portnumber a number instead of the service name
-e – will list ethernet headers in addition to all other information

To filter the traffic I can list i.e. port number:
tcpdump-uw -i vmk0 -s 0 -nn -e port 80

To generate traffic on that port I could use for example
nc -z host.IP.address 80
from another host/system

To save the output to a file use -w switch
tcpdump-uw -i vmk0 -s 0 -nn -e port 80 -w /vmfs/volumes/share/capture.pcap

Then to analize it copy it to system with i.e. Wireshar and open it from within.

 

pktcap

It is used to monitor traffic that flows through physical network adapters, VMkernel adapters, and virtual machines adapters, and analyze packet information by using the graphical user interface of network analysis tools such as Wireshark.

Example:
To capture packets on a switch port
First get the switch port from esxtop (press n to get the networking view) and look at the PORT-ID column
pktcap-uw --switchport 33554433
to save the output to a file use the -o switch followed by file location/name.pcap

WMware – netcat – testing connectivity

To check if a port is open on VM

nc -z 192.168.1.2 80
Connection to 192.168.1.72 80 port [tcp/http] succeeded! 

check if UDP port is open:

 nc -z -u 192.168.1.51 53
Connection to 192.168.1.51 53 port [udp/domain] succeeded! 

Send data between hosts:
Host 1 (192.168.1.222) set to listen mode:

nc -lp 1234

Host 2 send data to host 1

nc 192.168.1.222 1234
test 123

If the port (1234) is open the text (test 123) will be displayed in the console. This trick can be used a simple chat

The same can be used to send for example a log or a text file using this syntax:

nc 192.168.1.222 1234 < /var/log/vmkernel.log

this way content of that file will be displayed on Host 1 (if conditions specified above are met).

ESXi – How to list OS discrepancies

As VMs get upgraded to newer OSes the VM Guest OS setup on the ESXi hosts migth no longer be the same as the OS version the VM is running (i.e. VM got upgraded from Win 7 to 10 but ESXi host still thinks it is running 7).

Here is a one liner that shows what is set and what is actually there:
Get-View -ViewType "VirtualMachine" -Property @("Name", "Config.GuestFullName", "Guest.GuestFullName") | Where-Object {($_.Config.GuestFullName -ne $_.Guest.GuestFullName) -and ($_.Guest.GuestFullName -ne $null)} | Select-Object -Property Name, @{N="Configured OS";E={$_.Config.GuestFullName}}, @{N="Running OS";E={$_.Guest.GuestFullName}} | Format-Table -AutoSize

The output look something like that:

Name                                  Configured OS                       Running OS
----                                  -------------                       ----------
win2K                                 Microsoft Windows 2000 Server       Microsoft Windows 2000 Professional
RH3                                   Red Hat Enterprise Linux 3 (32-bit) Red Hat Enterprise Linux 3 (64-bit)
suse51                                SUSE Linux Enterprise 11 (64-bit)   SUSE Linux Enterprise 12 (64-bit)