cli remote management of ESXi virtual machines over ssh

One of the ways to manage ESXi virtual machines is through vim-cmd commands. You can look at a quick tutorial of the command here.

Let's say there are a multitude of ESXi servers in your environment and you'd like to manage some virtual machines through cli.
One way to do this is to run vim-cmd commands over ssh. For example, I have 2 ESXis named in the example esxi1 and esxi2. On both of them I have one VM:

~$ ssh root@esxi1.localdomain "vim-cmd vmsvc/getallvms"
Vmid             Name                                         File                                  Guest OS       Version             Annotation
7      Linux                       [datastore1] Linux/Linux.vmx                                 debian6_64Guest    vmx-08

~$ ssh root@esxi2.localdomain "vim-cmd vmsvc/getallvms"
Password: 
Vmid           Name                                      File                                  Guest OS        Version             Annotation
13     VSRX                   [datastore1] VSRX/VSRX.vmx                                   otherGuest          vmx-09    VSRX OVF Template

So, this works fine and nice and actually there's nothing special about it. It's just running remote commands over ssh.

If you'd like to make things more easy, you can use ssh public key authentication for the remote esxi hosts (no need to type in the password every time you want to run a command).
I wrote some bash functions to make it even easier (to remember) and shorter to type.
This is what I have among other functions and things in my .bashrc file:

function start_vm () { ssh root@${1}.localdomain "vim-cmd vmsvc/power.on" "$2";}
function stop_vm () { ssh root@${1}.localdomain "vim-cmd vmsvc/power.off" "$2";}
function reboot_vm () { ssh root@${1}.localdomain "vim-cmd vmsvc/power.reboot" "$2";}
function getallvm () { ssh root@${1}.localdomain "vim-cmd vmsvc/getallvms" ;}
function powerstate_vm () { ssh root@${1}.localdomain "vim-cmd vmsvc/power.getstate" "$2";}
function getnetwrorks () { ssh root@${1}.localdomain "vim-cmd vmsvc/get.networks" "$2";}

ssh tunnels set up and port forwarding

SSH tunnels allow you to forward a local TCP port to a remote machine and vice versa. The tunnel option is available in many ssh clients. I will give the example here on how to create SSH tunnels with putty and the openssh-client.
I will consider in the below that we want to access the SERVER on port 80 (http server).

Scenario 1. SSH tunnel setup with local port forwarding.

The SSH tunnel is shown with the red arrow. In order to access the SERVER through the ssh tunnel the connection will have to be made on the CLIENT's local forwarded port (2000 in the example). The traffic between the CLIENT and SSH-HELPER is encrypted by ssh, the traffic between the SSH-HELPER and the SERVER is not encrypted.

SSH tunnel  is set up on the CLIENT:

openssh-client:

ssh -L *:2000:server:80 ssh-helper

The '*' before the local port to be forwarded 2000 denotes that the port 2000 should be listening on all available interfaces on the client. This goes according to the openssh-client configuration.

putty:

The tunnel configuration is done under Connection - SSH - Tunnels. Source port is the local port, destination is where the connection will be forwarded after exiting the SSH tunnel.
After you specify source port and destination, you need to click "Add" for the configuration to take effect.
If you want to access the remote server from other hosts, make sure you check the box  "Local ports accept connections from other hosts", otherwise the port 2000 will be opened only for the loopback address (127.0.0.1)

ssh keepalives and tcp keepalives in openssh

The SSH connection can be kept alive either with SSH keepalive packets (encrypted) or with TCP keepalive packets. This allows also to detect hanging sessions and disconnect the hanging client/server when a connection has become inactive.

On a open SSH server, to control the SSH keepalive packets the parameters are:
ClientAliveCountMax 3 (default)
ClientAliveInterval 0 (default) - means the SSH keepalive packets will not be sent by the server