ssh keepalives and tcp keepalives in openssh

The SSH connection can be kept alive either with SSH keepalive packets (encrypted) or with TCP keepalive packets. This allows also to detect hanging sessions and disconnect the hanging client/server when a connection has become inactive.

On a open SSH server, to control the SSH keepalive packets the parameters are:
ClientAliveCountMax 3 (default)
ClientAliveInterval 0 (default) - means the SSH keepalive packets will not be sent by the server

Replaying packets with tcpreplay

Tcpreplay is a suite of tools that allows editing and replaying previously captured traffic in libpcap format. This can come handy in many situations, one common use is traffic pattern based behavior re-creation in a lab environment.
Tcpreplay suite comes with the following tools:

• tcpprep - multi-pass pcap file pre-processor which determines packets as client or server and creates cache files used by tcpreplay and tcprewrite
• tcprewrite - pcap file editor which rewrites TCP/IP and Layer 2 packet headers
• tcpreplay - replays pcap files at arbitrary speeds onto the network
• tcpliveplay - Replays network traffic stored in a pcap file on live networks using new TCP connections
• tcpreplay-edit - replays; edits pcap files at arbitrary speeds onto the network
• tcpbridge - bridge two network segments with the power of tcprewrite
• tcpcapinfo - raw pcap file decoder and debugger

To exemplify the use of tcpreplay, let's say we have the following setup:

Now in this setup we're interested in how our DUT device (Device Under Test) is reacting given a specific traffic pattern that is let's say very specific to this environment. I will assume the DUT is a Layer 3 device. 

How to delete files from CSS11501

The scope of this post is to show how to delete archived files on a Cisco CSS11501. In order to delete other files (core dumps, logs, etc) the procedure is similar, you just need to know what you want to delete. In order to manipulate the files we need to go to debug mode:

CSS11501# llama

Then the ap_file command provides the "File commands using Complete pathnames". To delete the archived_rc_file from the Archive directory, issue:

CSS11501(debug)# ap_file delete c:/Archive/archived_rc_file

To see the files which are present in the Archive directory you can use the Tab after you type ap_file delete c:/Archive/ I am using the sg0820001 WebNS, and the dir - to list the contents of the Archive directory - command did not yeld any results when ran.

CSS11501(debug)# dir c:/Archive/
CSS11501(debug)#

mdadm tips on Linux software RAID

mdadm is a tool for managing, creating and reporting on Linux software RAID arrays.

I will describe some tips which I found useful at the moment.

Improve RAID1 re-sync time with write-intent bitmap
The RAID driver writes out periodically bitmap information recording which areas of the RAID component have been modified since the RAID array was last in sync.

If, for example one of two members of a RAID1 array fails and is removed from the array, md (the multiple disk software RAID drive) will record bits to the bitmap relating to the changes the active member is undertaking since the two members were last in sync. If the same failed/removed drive is re-added to the RAID1 array, md will notice and will recover only the portions indicated by the bitmap. In this way a lengthy re-sync is avoided (a full re-sync is normally needed if the drives are not in sync when the array starts up).

Cisco CSS11501 source groups and ACL

Few days ago I had to reconfigure a running CSS 11501 loadbalancer from an existing configuration in which traffic initiated from some services running in one VLAN towards any destination was source NAT-ed to a selective NAT which was based on the destination IP address.
What I wanted to do is almost exactly to what is described in this official Cisco document with the difference that I had more IP destination addresses for my ACL based source NAT, I had only 2 VLANs and the IP addresses were different than in the example given.
Below is an excerpt similar to my original configuration:

service SERV11
ip address 192.168.0.3
protocol tcp
keepalive type tcp
redundant-index 111
keepalive port 11501
active

service SERV12
ip address 192.168.0.4
protocol tcp
keepalive type tcp
redundant-index 112
keepalive port 11501
active

....

owner OWNER1

content SERV_BAL
vip address 10.0.0.1
add service SERV11
add service SERV12
redundant-index 11
balance leastconn
flow-reset-reject
flow-timeout-multiplier 20
active

....

group GROUP1
add service SERV11
add service SERV12
vip address 10.0.0.1
redundant-index 21
active

A subset of the IP addresses which I wanted to bypass the source NAT when the connection was initiated by the configured services (SERV11 and SERV12) were 10.0.0.11 and 10.0.0.12 So I wrote an ACL as documentation recommends, and applied it to the circuit VLAN of the configured services (VLAN 1), while on the other VLAN there was an ACL which allowed all traffic.

acl enable

....

acl 1
clause 5 bypass any 192.168.0.3 255.255.255.255 destination 10.0.0.11 255.255.255.255 
clause 10 bypass any 192.168.0.3 255.255.255.255 destination 10.0.0.12 255.255.255.255
clause 15 bypass any 192.168.0.4 255.255.255.255 destination 10.0.0.11 255.255.255.255 
clause 20 bypass any 192.168.0.4 255.255.255.255 destination 10.0.0.12 255.255.255.255
clause 101 permit any 192.168.0.3 255.255.255.255 destination any sourcegroup GROUP1
clause 102 permit any 192.168.0.4 255.255.255.255 destination any sourcegroup GROUP1
clause 254 permit any any destination any
apply circuit-(VLAN1)

However, checking on 10.0.0.11 and 10.0.0.12 for incoming sessions from SERV11, SERV12 I could see that the source IP address was still NAT-ed (packets were arriving with 10.0.0.1 as the source IP address). There's a catch which for me was not obvious from the documentation (probably my non-native English has contributed to that :) ). In order for the packets to arrive on 10.0.0.11, 10.0.0.12 with their real IP address and for the rest of the destinations to be NAT-ed the services had to be removed from the source group. In the end desired configuration looked like below:

acl enable

....

service SERV11
ip address 192.168.0.3
protocol tcp
keepalive type tcp
redundant-index 111
keepalive port 11501
active

service SERV12
ip address 192.168.0.4
protocol tcp
keepalive type tcp
redundant-index 112
keepalive port 11501
active

....

owner OWNER1

content SERV_BAL
vip address 10.0.0.1
add service SERV11
add service SERV12
redundant-index 11
balance leastconn
flow-reset-reject
flow-timeout-multiplier 20
active

....

group GROUP1
vip address 10.0.0.1
redundant-index 21
active
....

acl 1
clause 5 bypass any 192.168.0.3 255.255.255.255 destination 10.0.0.11 255.255.255.255 
clause 10 bypass any 192.168.0.3 255.255.255.255 destination 10.0.0.12 255.255.255.255
clause 15 bypass any 192.168.0.4 255.255.255.255 destination 10.0.0.11 255.255.255.255 
clause 20 bypass any 192.168.0.4 255.255.255.255 destination 10.0.0.12 255.255.255.255
clause 101 permit any 192.168.0.3 255.255.255.255 destination any sourcegroup GROUP1
clause 102 permit any 192.168.0.4 255.255.255.255 destination any sourcegroup GROUP1
clause 254 permit any any destination any
apply circuit-(VLAN1)

acl 2
clause 254 permit any any destination any
apply circuit-(VLAN2)

Where VLAN2 is the network towards 10.0.0 and the rest of the clients.

Reference: CSS Content Load-Balancing Configuration Guide (Software Version 8.10)