VMware – esxtop – troubleshooting VM CPU performance

To display VM only press V (VM only view):

then to expand specific VM press e (expand) then enter the GID number:

As this VM has 3 CPU I can see CPU usage on all of them, RDY, USED and CSTP times.

%RDY is a Key Performance Indicator!  This one defines how much time your virtual machine wanted to execute CPU cycles but could not get access to the physical CPU. It tells you how much time did you spend in a “queue”. I normally expect this value to be better than 5%. In this case we can see that this VM is struggling a bit or it is very busy.

%USED tells you how much time did the virtual machine spend executing CPU cycles on the physical CPU.

%CSTP tells you how much time a virtual machine is waiting for a virtual machine with multiple vCPU to catch up. If this number is higher than 3% you should consider lowering the amount of vCPU in your virtual machine.

VMware – capturing network packets

There are two utilities available on ESXi host out of the box that enable us to capture network traffic:

  • tcpdump (limited only to capturing traffic from vmkernel adapters)
  • pktcap
tcpdump

To list vmkernel adapters use esxcfg-vmknic -l command

tcpdump-uw -i vmk0 -s 0 -nn -e
notes:
-s 0 – indicates that we capture the entire packet (as opposed truncated packets)
-nn – indicates that we want to use numbers instead of names for the IP addresses, and for the portnumber a number instead of the service name
-e – will list ethernet headers in addition to all other information

To filter the traffic I can list i.e. port number:
tcpdump-uw -i vmk0 -s 0 -nn -e port 80

To generate traffic on that port I could use for example
nc -z host.IP.address 80
from another host/system

To save the output to a file use -w switch
tcpdump-uw -i vmk0 -s 0 -nn -e port 80 -w /vmfs/volumes/share/capture.pcap

Then to analize it copy it to system with i.e. Wireshar and open it from within.

 

pktcap

It is used to monitor traffic that flows through physical network adapters, VMkernel adapters, and virtual machines adapters, and analyze packet information by using the graphical user interface of network analysis tools such as Wireshark.

Example:
To capture packets on a switch port
First get the switch port from esxtop (press n to get the networking view) and look at the PORT-ID column
pktcap-uw --switchport 33554433
to save the output to a file use the -o switch followed by file location/name.pcap

WMware – netcat – testing connectivity

To check if a port is open on VM

nc -z 192.168.1.2 80
Connection to 192.168.1.72 80 port [tcp/http] succeeded! 

check if UDP port is open:

 nc -z -u 192.168.1.51 53
Connection to 192.168.1.51 53 port [udp/domain] succeeded! 

Send data between hosts:
Host 1 (192.168.1.222) set to listen mode:

nc -lp 1234

Host 2 send data to host 1

nc 192.168.1.222 1234
test 123

If the port (1234) is open the text (test 123) will be displayed in the console. This trick can be used a simple chat

The same can be used to send for example a log or a text file using this syntax:

nc 192.168.1.222 1234 < /var/log/vmkernel.log

this way content of that file will be displayed on Host 1 (if conditions specified above are met).

PowerShell – Set VM permissions

Here is my PS way of granting user access to a VM in vSphere

$vmname = read-host "Enter Virtual Machine name"
$username = read-host "Enter User name (i.e. DOMAIN\username)"
$roles = Get-VIRole | select name | % {$counter = -1} {$counter++; $_ | Add-Member -Name Role_ID -Value $counter -MemberType NoteProperty -PassThru}
$roles | ft -auto
$myRole = read-host "select Role ID"
$role_selection = $roles[$myRole]
get-vm $vmname | New-VIPermission -Role (Get-VIRole -Name $role_selection.name) -Principal $username