ESXI 6.0 monitoring SSD drive TBW

“Terabytes Written” is the total amount of data that can be written into an SSD before it is likely to fail.

There is a nicely written article on this blog explaining how to get the TBW for a SSD drive in ESXI host – open here.

However since I am a lazy kind of person I like to get stuff scripted and emailed to me.

In essence here is what I have done:

Install smartctl

  1. Download smartctl-6.6-4321.x86_64.vib
  2. Copy the VIB to the /tmp/ directory of an ESXi host
  3. SSH to the ESXi host
  4. Set the VIB acceptance level to CommunitySupported
    # esxcli software acceptance set --level=CommunitySupported
  5. Install the package (Maintenance Mode or Reboot is not required)
    #esxcli software vib install -v /tmp/smartctl-6.6-4321.x86_64.vib

The tool is located at /opt/smartmontools/smartctl and works just like the Linux version.
Locate physical disks with ls -l /dev/disks/

Create the script and automate
Save the below lines in a script somwehere:

#!/bin/sh
 var=`/opt/smartmontools/smartctl -d sat --all /dev/disks/t10.ATA_____Samsung_SSD_850_EVO_M.2_250GB___________S24BNXAH119741P_____ | grep Total_LBAs | cut -d"-" -f2`
 TBW=`awk "BEGIN {print $var*512/1099511627776}"`
 echo "Total TB Writen so far: " $TBW

var2=`/opt/smartmontools/smartctl -d sat --all /dev/disks/t10.ATA_____Samsung_SSD_850_EVO_M.2_250GB___________S24BNXAH119741P_____ | grep Power_On | cut -d"-" -f2`
 POWERON=`awk "BEGIN {print $var2/24}"`
 echo "Device in use since [days]: " $POWERON

echo "Device has 5 Years Limited Warranty or 75TBW Limited "
 REMAIN=`awk "BEGIN {print (75*$POWERON)/$TBW}"`
 echo "Device has approximately: " $REMAIN "days left"
 REMAINY=`awk "BEGIN {print $REMAIN/360}"`
 echo "that is approx. $REMAINY years left"

 

To see how much life my SSD drive has got left I added that script to /etc/profile.local file so whenever I login I get something like that:

Samsung SSD 850 Evo m.2 250GB Remaining Disk LifeTime
Tue May 24 16:30:12 UTC 2016

Total TB Written so far: 1.26013
Device in use since [days]: 97.9167
Device has 5 Years Limited Warranty or 75TBW Limited
Device has approximately: 5827.77 days left
that is approx. 16.1883 years left

As I don’t often login to my esxi host via ssh I created a cron job – /var/spool/cron/crontabs/root :

0    0   1,15 *  *    source /opt/smartmontools/report.sh > /opt/smartmontools/disklifetime.txt

and specified it to run every two weeks and save the results to a file. That file is then picked up (via scp) by my linux server and emailed to me.

Job done.

SSH and Secure Access with RSA certificate

Prerequisites:
1. Two linux systems
2. Someone who is fed up of constantly entering ssh username and password

There comes a time when you had enough of constantly entering your username and password:

user@server1:~/.ssh$ ssh user@172.31.27.99
user@172.31.27.99's password:
Linux 3.2.0-4-amd64 #1 SMP Debian 3.2.78-1 x86_64
Last login: Sat Apr 23 09:52:10 2016 from 172.31.100.158
user@server2:~$

Luckily there is another way using RSA certs. Here is a quick way of setting it all up:
1. On you normal/daily workstation generate pair of certificates:

user@server1:~/.ssh$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/user/.ssh/id_rsa): [ENTER]
Enter passphrase (empty for no passphrase): yourpass_phrase [ENTER]
Enter same passphrase again: yourpass_phrase [ENTER]
Your identification has been saved in /home/user/.ssh/id_rsa.
Your public key has been saved in /home/user/.ssh/id_rsa.pub.
The key fingerprint is:
da:fe:08:91:5f:63:89:8f:27:74:59:c1:19:d6:9f:2c user@server1
The key's randomart image is:
+--[ RSA 2048]----+
|           .++   |
|           .o..  |
|            . ...|
|       . . + E o.|
|      o S B   .  |
|       * * .     |
|      o = o      |
|       o +       |
|        o..      |
+-----------------+
user@server1:~/.ssh$

This will generate two files id_rsa and id_rsa.pub –> those are your private and public keys.

2. Copy your public key to the destination server

user@server1:~/.ssh$ ssh-copy-id user@172.31.27.99
user@172.31.27.99's password: 
Now try logging into the machine, with "ssh 'user@172.31.27.99'", and check in:

  ~/.ssh/authorized_keys

to make sure we haven't added extra keys that you weren't expecting.

if you go to your destination server and check the ~/.ssh/authorized_keys you will find that it has exactly the same content as your id_rsa.pub key:

root@server2:/home/user/.ssh# ls -al
total 12
drwx------ 2 user user 4096 Apr 23 10:03 .
drwxr-xr-x 3 user user 4096 Apr 23 09:13 ..
-rw------- 1 user user  394 Apr 23 10:03 authorized_keys
root@server2:/home/user/.ssh# cat authorized_keys 
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyN1oh1L9dVBOGgb5QVSoJ4Cls/l+uCSjwUeH7Jr2NYyTz/0VeLQSDmvAOlyhy/S26KY8wT41z9coT+O8TDWo4F+Wvz1M27fYvscaAQO3cY5iIIEHTV0BpORDHTKvHd/YnP0CVitE65sbTssUGApG9iHyE/yTDpl+g7xe/9NwSxjPYSn2ZGxcG0vWkIUPLFProDK5VPSYo4FI27s5F+uqsWK60Ey+SuotPp6BDIKqe6jnNWjmxYbPnVWyU4Qb0DiQNWX1HmmaxehknnJM7NZWIIOzY8kSsTC8hdxcZu1IGHO6N9IDn+bQUUz7OSzfzPwDvadchScD3vzUuRdGq10d1 user@server1
root@server2:/home/user/.ssh#
user@server1:~/.ssh$ cat id_rsa.pub 
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyN1oh1L9dVBOGgb5QVSoJ4Cls/l+uCSjwUeH7Jr2NYyTz/0VeLQSDmvAOlyhy/S26KY8wT41z9coT+O8TDWo4F+Wvz1M27fYvscaAQO3cY5iIIEHTV0BpORDHTKvHd/YnP0CVitE65sbTssUGApG9iHyE/yTDpl+g7xe/9NwSxjPYSn2ZGxcG0vWkIUPLFProDK5VPSYo4FI27s5F+uqsWK60Ey+SuotPp6BDIKqe6jnNWjmxYbPnVWyU4Qb0DiQNWX1HmmaxehknnJM7NZWIIOzY8kSsTC8hdxcZu1IGHO6N9IDn+bQUUz7OSzfzPwDvadchScD3vzUuRdGq10d1 user@server1
user@server1:~/.ssh$

No to test that this bit is working fine:

user@server1:~/.ssh$ ssh user@172.31.27.99
Enter passphrase for key '/home/user/.ssh/id_rsa': yourpass_phrase [ENTER]
Linux 3.2.0-4-amd64 #1 SMP Debian 3.2.78-1 x86_64
Last login: Sat Apr 23 09:52:56 2016 from 172.31.100.158
user@server2:~$

You can of course leave the passphrase empty and on this stage you are all done. However if you have set the passphrase do not despair as there is a way of telling your machine to remember it for you.

3. Using ssh-agent to remember the passphrase
DESCRIPTION
ssh-agent is a program to hold private keys used for public key authentication
(RSA, DSA, ECDSA). The idea is that ssh-agent is started in the beginning of
an X-session or a login session, and all other windows or programs are started
as clients to the ssh-agent program. Through use of environment variables the
agent can be located and automatically used for authentication when logging in
to other machines using ssh(1).

I tend to add this line to .bashrc file under my user profile:
eval `ssh-agent -s`

then check that it is running:

user@server1:~$ ps aux | grep ssh-agent
user      6088  0.0  0.0  12480   332 ?        Ss   10:22   0:00 ssh-agent -s
user      6090  0.0  0.0   7812   608 pts/0    S+   10:22   0:00 grep ssh-agent

Now that the authentication agent is running last remaining thing to do is to add the private key identities to the agent:

user@server1:~$ ssh-add
Enter passphrase for /home/user/.ssh/id_rsa: yourpass_phrase [ENTER]
Identity added: /home/user/.ssh/id_rsa (/home/user/.ssh/id_rsa)
user@server1:~$ 

Now for the rest of the time that we remain log in to our normal/daily workstation passphrases that we might have setup on hundreds of servers will be forwarded so we no longer need to type them in. To verify/list the added fingerprints of all identities currently represented by the agent just run:

user@server1:~$ ssh-add -l
2048 da:fe:08:91:5f:63:89:8f:27:74:59:c1:19:d6:9f:2c /home/user/.ssh/id_rsa (RSA)
user@server1:~$ ssh user@172.31.27.99
Linux 3.2.0-4-amd64 #1 SMP Debian 3.2.78-1 x86_64
Last login: Sat Apr 23 10:07:14 2016 from 172.31.100.158
user@server2:~$

if you are using the same username on both ends, you can skip the user name:

user@server1:~$ ssh 172.31.27.99
Linux 3.2.0-4-amd64 #1 SMP Debian 3.2.78-1 x86_64
Last login: Sat Apr 23 10:07:14 2016 from 172.31.100.158
user@server2:~$

4.Troubleshooting

If you get stuck and something isn’t working the way it should be connect using the verbose switch -v (or if you want to go nuts go extra verbose -vvv):

user@server1:~$ ssh 172.31.27.99 -v
OpenSSH_6.0p1 Debian-4+deb7u4, OpenSSL 1.0.1e 11 Feb 2013
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug1: Connecting to 172.31.27.99 [172.31.27.99] port 22.
debug1: Connection established.
debug1: identity file /home/user/.ssh/id_rsa type 1
debug1: Checking blacklist file /usr/share/ssh/blacklist.RSA-2048
debug1: Checking blacklist file /etc/ssh/blacklist.RSA-2048
debug1: identity file /home/user/.ssh/id_rsa-cert type -1
debug1: identity file /home/user/.ssh/id_dsa type -1
debug1: identity file /home/user/.ssh/id_dsa-cert type -1
debug1: identity file /home/user/.ssh/id_ecdsa type -1
debug1: identity file /home/user/.ssh/id_ecdsa-cert type -1
debug1: Remote protocol version 2.0, remote software version OpenSSH_6.0p1 Debian-4+deb7u4
debug1: match: OpenSSH_6.0p1 Debian-4+deb7u4 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.0p1 Debian-4+deb7u4
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-ctr hmac-md5 none
debug1: kex: client->server aes128-ctr hmac-md5 none
debug1: sending SSH2_MSG_KEX_ECDH_INIT
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ECDSA f9:c4:2a:ee:20:5e:66:c2:fc:76:12:63:53:13:9e:dc
debug1: Host '172.31.27.99' is known and matches the ECDSA host key.
debug1: Found key in /home/user/.ssh/known_hosts:1
debug1: ssh_ecdsa_verify: signature correct
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,password
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /home/user/.ssh/id_rsa
debug1: Server accepts key: pkalg ssh-rsa blen 279
debug1: Authentication succeeded (publickey).
Authenticated to 172.31.27.99 ([172.31.27.99]:22).
debug1: channel 0: new [client-session]
debug1: Requesting no-more-sessions@openssh.com
debug1: Entering interactive session.
debug1: Sending environment.
debug1: Sending env LANG = en_GB.UTF-8
Linux 3.2.0-4-amd64 #1 SMP Debian 3.2.78-1 x86_64
Last login: Sat Apr 23 10:34:20 2016 from 172.31.100.158
user@server2:~$ 

How to Tunnel Remote Desktop Through SSH

Recently I wanted to set up a remote desktop connection when I was away from home to my home server.  While VPN is great I found that SSH tunnels are even better. I use my good old Raspberry Pi as the “gateway”. So here goes…

My Raspberry PI SSH address:     raspberry.address.com:12345

Server I want to RDP to:        10.10.10.10:3389

Open Putty and first enter the IP address of the remote SSH host (in my case RPi), name the connection and click Save:

ssh_tunnel_1

Then in the Category menu scroll down to Connection/SSH/Tunnels and enter the Source port (you will be connecting to that port locally) then the destination IP address and port, click Add button, go back to Session and Save it all.

ssh_tunnel_2

And that is nearly it. Click Open to establish the SSH tunnel, once connected open Remote Desktop Client and in the Computer field type in: localhost:9090

SSH_tunnel_3

And that is it. As simple as that.

When working on a linux workstation the same thing can be acomplished with this command:

ssh -f -L 9090:10.10.10.10:3389 raspberry.address.com:12345 -N

and then user RDP client to connect to localhost port 9090.

 

Side note: the same tunnel can be used to connect to local proxy, mysql server and anything else that is running on the other side of the SSH tunnel.

ESXI – How to find out VMs IP address from SSH

Sometimes I need to get the IP address of a running VM from ESXi SSH. It is pretty straightforward and easy thing to do:
1. Connect to your host via SSH
2. Establish your VM ID

[root@esxi:~] vim-cmd vmsvc/getallvms

3. Now to find out the VMs IP address run:

[root@esxi:~] vim-cmd vmsvc/get.guest 13 | grep ipAddress

OUTPUT:
   ipAddress = "10.10.1.109", 
         ipAddress = (string) [
            ipAddress = (vim.net.IpConfigInfo.IpAddress) [
                  ipAddress = "10.10.1.109", 
                  ipAddress = "fe80::20c:29ff:fec8:db22", 
            ipAddress = (string) [
                     ipAddress = "10.10.1.254", 
                     ipAddress = , 
                     ipAddress = , 
                     ipAddress = "fe80::221:55ff:fefb:da4", 
                     ipAddress = ,

Job done.

ESXI – how to power on/off a VM from SSH

Working from linux I miss the luxury of having vSphere client on hand to power on/off VMs as I need them. There is only one solution – do that via SSH.

Of course for this to work SSH needs to be enabled on the ESXi host – see here.

First list all VMs:

[root@esxi:~] vim-cmd vmsvc/getallvms
Vmid      Name            File                  Guest OS       Version                  Annotation                   
13   CentOS2     [SSD] CentOS2/CentOS.vmx      centos64Guest      vmx-11 Linked-Clone

then verify that the VM is powered down:

[root@esxi:~] vim-cmd vmsvc/power.getstate 13
Retrieved runtime info
Powered off

now to power it on and check the status run:

[root@esxi:~] vim-cmd vmsvc/power.on 13
Powering on VM:
[root@esxi:~] vim-cmd vmsvc/power.getstate 13
Retrieved runtime info
Powered on

Next to find out the IP address of the VM that we have just powered on run:

[root@esxi:~] vim-cmd vmsvc/get.guest 13 | grep ipAddress
   ipAddress = "10.10.1.109", 
         ipAddress = (string) [
            ipAddress = (vim.net.IpConfigInfo.IpAddress) [
                  ipAddress = "10.10.1.109", 
                  ipAddress = "fe80::20c:29ff:fec8:db22", 
            ipAddress = (string) [
                     ipAddress = "10.10.1.254", 
                     ipAddress = , 
                     ipAddress = , 
                     ipAddress = "fe80::221:55ff:fefb:da4", 
                     ipAddress = , 

Now we can RDP or SSH into that box or power it off by running:

[root@esxi:~] vim-cmd vmsvc/power.off 13
Powering off VM: