m.(A.TT)/er
Well Defined and Parallized Environment Control
Friday 07/29/11 at 09:46:35 PM

The latest Massh RPM is : massh-2.0-58.noarch.rpm

The latest Massh OSX Package is : massh-2.0-58.zip

The latest Massh Tar/Gzipped Source is : massh-2.0-57.tgz

Massh makes it possible to perform the following on hundreds, or even thousands of hosts in a parallelized fashion:

Massh uses Ambit to enumerate a list of targets for which it will execute one of the aforementioned operations. Hosts can be logically grouped and dynamically managed allowing for consistent operational changes and overall management of identically purposed hosts.

Massh can display the full output resulting from its various operations or it can be set to display an operation's success or failure for individual target hosts. The full ouput produced by Massh is completely organized and labeled by host, regardless of what order output is returned from target hosts.

Starting with version 2.x Massh combines flexible option control management and a simplified command + subcommand sytax. This reduces the complexity of typical Massh commands and makes it much easier to learn how to perform specific operations with specific options with Massh.

Contents

Simplified CLI Syntax

Massh 2.x introduces a new simplified CLI sytax that is best broken down in the following way:

COMMAND  =>  OPTION    +   SUBCOMMAND  =>  SUBOPTION
=====================================================================
massh        allhosts      verbose         gpasswd -a mm wheel
massh        db[1..30]     push            my.cnf
massh        hosts.txt     worked          sudo service restart httpd

COMMAND

The Massh executable itself.

OPTION

Currently the values that are valid for OPTION are those that can be understood, expanded and enumerated by Ambit or some other hostlist generator:

Command ARG

A command line argument in the form of an Ambit expandable string.

$ massh [2,3].a.tt verbose whoami
3.a.tt : mm
2.a.tt : mm

$ massh [1..3].a.tt verbose sudo whereami
2.a.tt : Fremont, CA, US
1.a.tt : Atlanta, GA, US
3.a.tt : Dallas, TX, US

Hostfile in $PWD

If a file in the user's current working directory ($PWD) contains hostnames (one per line) or Ambit expandable strings it can be passed to OPTION by its name (full path not needed).

1 $ cat somehosts.txt 
1.a.tt
2.a.tt
3.a.tt

$ massh somehosts.txt verbose sudo whereami
1.a.tt : Atlanta, GA, US
3.a.tt : Dallas, TX, US
2.a.tt : Fremont, CA, US

Path to Hostfile

If a file located somewhere on the current host contains hostnames (one per line) or Ambit expandable strings it can be passed to OPTION by listing the files full path.

1 $ cat /usr/tmp/otherhosts.txt 
db[1..3].a.tt

1 $ ambit /usr/tmp/otherhosts.txt 
db1.a.tt
db2.a.tt
db3.a.tt

$ massh /usr/tmp/otherhosts.txt verbose 'sudo monit summary | grep bind'
db1.a.tt : Process 'bind'                      running
db3.a.tt : Process 'bind'                      running
db2.a.tt : Process 'bind'                      running

User HostGroup

Users can create personal or User HostGroups (via Ambit) that can be passed to OPTION by name.

1 $ ambit create hostgroup
HostGroup Name : myradservers
HostGroup Summary : My Rad Servers 
Enter Hostnames Below [Ctrl-d to End]: 
db[1..3].a.tt  

1 $ ambit list hostgroups

User HostGroups
----------------
myradservers:   My Rad Servers

1 $ massh myradservers push /etc/hosts
db1.a.tt : Push Succeeded
db3.a.tt : Push Succeeded
db2.a.tt : Push Succeeded

System HostGroup

Users with root access can create System HostGroups (via Ambit) that all users on the system can pass the OPTION by name.

Network HostGroup

Network accessible HostGroups can be established by creating a DNS TXT record (alas they have a use ;^) containing an Ambit expandable string. Ambit will check for such records if the 'Domain' Ambit option is set. The shortname for the aforementioned DNS TXT records can be passed to OPTION. Example: The domain a.tt has the following TXT record to establish a Network HostGroup for the Yum package repositories:

$ host -t TXT yum
yum.a.tt descriptive text "[1..3].a.tt"

This mean the HostGroup's name (yum) can be passed to OPTION.

SUBCOMMAND and SUBOPTION

The SUBCOMMAND is a one word descriptive indicator that best describes the overall operation that Massh will be performing. Unfortunately the initial goal of establishing SUBCOMMAND's that were gramatically consistent (i.e. all verbs or even adverbs) was sacrificed in in favor of memorable SUBCOMMAND names.

A SUBOPTION will either be a command (to be run on target hosts) or a filename (i.e. file to be pulled, pushed or containing a script to be executed). The following are possible SUBCOMMANDS:

terse

SSH to all targets enumerated from OPTION, execute the command stipulated by SUBOPTION and display only the success or failure of the executed command.

$ massh east10 terse uptime 
easthost5 : Command Succeeded
easthost7 : Command Succeeded
easthost1 : Command Succeeded
easthost3 : Command Succeeded
easthost8 : Command Succeeded
easthost2 : Command Succeeded
easthost9 : Command Succeeded
easthost6 : Command Succeeded
easthost10 : Command Succeeded
easthost4 : Command Succeeded

verbose

SSH to all targets enumerated from OPTION, execute the command stipulated by SUBOPTION and display the full output resulting from the command.

$ massh east10 verbose uname -vr
easthost10 : 2.6.9-34.ELsmp #1 SMP Fri Feb 24 16:54:53 EST 2006
easthost3 : 2.6.9-34.ELsmp #1 SMP Fri Feb 24 16:54:53 EST 2006
easthost5 : 2.6.9-34.ELsmp #1 SMP Fri Feb 24 16:54:53 EST 2006
easthost6 : 2.6.9-34.ELsmp #1 SMP Fri Feb 24 16:54:53 EST 2006
easthost8 : 2.6.9-34.ELsmp #1 SMP Fri Feb 24 16:54:53 EST 2006
easthost1 : 2.6.9-67.ELsmp #1 SMP Wed Nov 7 13:58:04 EST 2007
easthost9 : 2.6.9-34.ELsmp #1 SMP Fri Feb 24 16:54:53 EST 2006
easthost2 : 2.6.9-34.ELsmp #1 SMP Fri Feb 24 16:54:53 EST 2006
easthost4 : 2.6.9-34.ELsmp #1 SMP Fri Feb 24 16:54:53 EST 2006
easthost7 : 2.6.9-34.ELsmp #1 SMP Fri Feb 24 16:54:53 EST 2006

worked

SSH to all targets enumerated from OPTION, execute the command stipulated by SUBOPTION and display only the hostnames where the command executed successfully.

$ ambit west10 | wc -l
10

$ massh west10 worked uname -vr
westhost1
westhost2
westhost4
westhost3
westhost7
westhost10
westhost6
westhost9
westhost8

$ massh west10 worked uname -vr | wc -l 
9

The example above shows that the HostGroup 'west10' is made up of ten hosts. However the Massh run only returned 9 hostnames.

bombed

SSH to all targets enumerated from OPTION, execute the command stipulated by SUBOPTION and display only the hostnames where the command failed.

$ massh west10 bombed uname -vr
westhost5

This example demonstrates how the option 'bombed' only returns hostnames that are not able to fulfill the Massh run. The host 'westhost5' was the only host that is not listed in the results from the previous example. These options are very useful for large scale operations where success is expected across 100% of hosts (production code releases and production package updates come to mine). Capturing all hosts that fail a code release provides the immediate opportunity to fix the issue or the immediate opportunity to make sure the failed hosts are effectively taken out of service.

push

SCP the file stipulated by SUBOPTION, to all targets enumerated from OPTION.

$ massh east10 push /etc/passwd 
easthost7 : Push Succeeded
easthost2 : Push Succeeded
easthost6 : Push Succeeded
easthost5 : Push Succeeded
easthost10 : Push Succeeded
easthost3 : Push Succeeded
easthost4 : Push Succeeded
easthost8 : Push Succeeded
easthost1 : Push Succeeded
easthost9 : Push Succeeded

pull

SCP the file stipulated by SUBOPTION, from all targets enumerated from OPTION.

$ massh east10 pull /etc/resolv.conf
easthost9 : Pull Succeeded
easthost2 : Pull Succeeded
easthost1 : Pull Succeeded
easthost7 : Pull Succeeded
easthost4 : Pull Succeeded
easthost8 : Pull Succeeded
easthost10 : Pull Succeeded
easthost3 : Pull Succeeded
easthost5 : Pull Succeeded
easthost6 : Pull Succeeded

The pull subcommand saves each file separately in directories named after the hosts:path they were pulled from:

$ find .massh/pull/ -name resolv.conf
.massh/pull/easthost10/etc/resolv.conf
.massh/pull/easthost1/etc/resolv.conf
.massh/pull/easthost7/etc/resolv.conf
.massh/pull/easthost2/etc/resolv.conf
.massh/pull/easthost8/etc/resolv.conf
.massh/pull/easthost9/etc/resolv.conf
.massh/pull/easthost6/etc/resolv.conf
.massh/pull/easthost5/etc/resolv.conf
.massh/pull/easthost3/etc/resolv.conf
.massh/pull/easthost4/etc/resolv.conf

execute

SSH to all targets enumerated from OPTION, piping the script stipulated by SUBOPTION to all targets enumerated from OPTION, execute the script and display the full output of the executed command.

$ cat dateup.sh 
#!/bin/bash -e

MyDate=$(date '+%a %B %d')
MyTime=$(sed 's/^.*up\ \(.*days\).*$/\1/' <(uptime))

echo "The Current Date is ${MyDate}"
echo "The System Has Been Up For ${MyTime}"

$ massh east10 execute dateup.sh 
easthost1 : The Current Date is Sun July 24
easthost1 : The System Has Been Up For 829 days
easthost2 : The Current Date is Sun July 24
easthost2 : The System Has Been Up For 1752 days
easthost4 : The Current Date is Sun July 24
easthost4 : The System Has Been Up For 544 days
easthost7 : The Current Date is Sun July 24
easthost7 : The System Has Been Up For 1752 days
easthost8 : The Current Date is Sun July 24
easthost8 : The System Has Been Up For 1752 days
easthost5 : The Current Date is Sun July 24
easthost5 : The System Has Been Up For 1752 days
easthost6 : The Current Date is Sun July 24
easthost6 : The System Has Been Up For 1752 days
easthost9 : The Current Date is Sun July 24
easthost9 : The System Has Been Up For 1752 days
easthost3 : The Current Date is Sun July 24
easthost3 : The System Has Been Up For 544 days
easthost10 : The Current Date is Sun July 24
easthost10 : The System Has Been Up For 1752 days

The example above demonstrates that things work quite well - with one slight annoyance. Massh is excellent at ensuring that multi-line output is always returned and displayed just like it would be displayed directly on each host. It also ensures that multi-line output is accurately separated by host regardless of the order that output is returned Massh. These capabilities alone separate Massh from most other parallelized shells, but they don't equate to perfect or even optimal output. The addition of the option 'Separator' goes a long way towards achieving well formatted, multi-line parallel output:

$ massh ed opt Separator
Updating Option : Separator
Current Value   : no
Enter New Value : yes


 $ massh east10 verbose df -h
easthost2 : Filesystem            Size  Used Avail Use% Mounted on
easthost2 : /dev/sda3              72G  9.8G   58G  15% /
easthost2 : /dev/sda1              99M   17M   78M  18% /boot
easthost2 : none                  2.0G     0  2.0G   0% /dev/shm
-
easthost6 : Filesystem            Size  Used Avail Use% Mounted on
easthost6 : /dev/sda3              72G   13G   56G  19% /
easthost6 : /dev/sda1              99M   17M   78M  18% /boot
easthost6 : none                  2.0G     0  2.0G   0% /dev/shm
-
easthost4 : Filesystem            Size  Used Avail Use% Mounted on
easthost4 : /dev/sda3              72G   13G   55G  19% /
easthost4 : /dev/sda1              99M   17M   78M  18% /boot
easthost4 : none                  2.0G     0  2.0G   0% /dev/shm
-
easthost7 : Filesystem            Size  Used Avail Use% Mounted on
easthost7 : /dev/sda3              72G   13G   56G  19% /
easthost7 : /dev/sda1              99M   17M   78M  18% /boot
easthost7 : none                  2.0G     0  2.0G   0% /dev/shm
-
easthost3 : Filesystem            Size  Used Avail Use% Mounted on
easthost3 : /dev/sda3              72G   13G   55G  19% /
easthost3 : /dev/sda1              99M   17M   78M  18% /boot
easthost3 : none                  2.0G     0  2.0G   0% /dev/shm
-
easthost5 : Filesystem            Size  Used Avail Use% Mounted on
easthost5 : /dev/sda3              72G   13G   56G  19% /
easthost5 : /dev/sda1              99M   17M   78M  18% /boot
easthost5 : none                  2.0G     0  2.0G   0% /dev/shm
-
easthost1 : Filesystem            Size  Used Avail Use% Mounted on
easthost1 : /dev/sda1             3.7G  1.4G  2.2G  38% /
easthost1 : /dev/sda3              63G  2.6G   57G   5% /home
easthost1 : none                  2.0G     0  2.0G   0% /dev/shm
-
easthost8 : Filesystem            Size  Used Avail Use% Mounted on
easthost8 : /dev/sda3              72G   13G   55G  19% /
easthost8 : /dev/sda1              99M   17M   78M  18% /boot
easthost8 : none                  2.0G     0  2.0G   0% /dev/shm
-
easthost9 : Filesystem            Size  Used Avail Use% Mounted on
easthost9 : /dev/sda3              72G  6.0G   62G   9% /
easthost9 : /dev/sda1              99M   17M   78M  18% /boot
easthost9 : none                 1014M     0 1014M   0% /dev/shm
-
easthost10 : Filesystem            Size  Used Avail Use% Mounted on
easthost10 : /dev/sda3              72G  6.0G   62G   9% /
easthost10 : /dev/sda1              99M   17M   78M  18% /boot
easthost10 : none                 1014M     0 1014M   0% /dev/shm
-

CONFIGURABLE OPTIONS

Starting with version 2.x Massh establishes a system wide option baseline that can only be changed with root privileges, but can be overridden by users on a per user basis. Massh provides the ability to view and edit options via simple interactive dialogues.

Option Controls

list options

List both System and User Tunable Options.

$ massh list options 

System Default Options
----------------------
Debug="no"
ForcePseudoTTY="no"
SyslogMassh="yes"
SSHLogLevel="Error"
TimeOut="30"
Parallel="30"
Random="$RANDOM"
ControlPath="$HOME/.ssh/masshter-%r@%h:%p"
ControlMaster="auto"
ControlPersist="300"
HostsFrom="ambit"
Separator="no"
SubCommand=""

User Specified Options
----------------------
SubCommand=""
ControlMaster="no"
SSHLogLevel="error"
TimeOut="5"
Separator="yes"

edit option

Edit or Set a User Option. User options override System Options.

Let's say I want to set the subcommand to always show verbose output (making for an even shorter command):

$ massh edit option SubCommand
Updating Option : SubCommand
Current Value   : 
Enter New Value : verbose
SubCommand="verbose"

$ massh list options 

System Default Options
----------------------
Debug="no"
ForcePseudoTTY="no"
SyslogMassh="yes"
SSHLogLevel="Error"
TimeOut="30"
Parallel="30"
Random="$RANDOM"
ControlPath="$HOME/.ssh/masshter-%r@%h:%p"
ControlMaster="auto"
ControlPersist="300"
HostsFrom="ambit"
Separator="no"
SubCommand=""

User Specified Options
----------------------
ControlMaster="no"
SSHLogLevel="error"
TimeOut="5"
Separator="yes"
SubCommand="verbose"

$ massh east10 uname -vr
easthost1 : 2.6.9-67.ELsmp #1 SMP Wed Nov 7 13:58:04 EST 2007
easthost5 : 2.6.9-34.ELsmp #1 SMP Fri Feb 24 16:54:53 EST 2006
easthost10 : 2.6.9-34.ELsmp #1 SMP Fri Feb 24 16:54:53 EST 2006
easthost2 : 2.6.9-34.ELsmp #1 SMP Fri Feb 24 16:54:53 EST 2006
easthost9 : 2.6.9-34.ELsmp #1 SMP Fri Feb 24 16:54:53 EST 2006
easthost3 : 2.6.9-34.ELsmp #1 SMP Fri Feb 24 16:54:53 EST 2006
easthost6 : 2.6.9-34.ELsmp #1 SMP Fri Feb 24 16:54:53 EST 2006
easthost8 : 2.6.9-34.ELsmp #1 SMP Fri Feb 24 16:54:53 EST 2006
easthost4 : 2.6.9-34.ELsmp #1 SMP Fri Feb 24 16:54:53 EST 2006
easthost7 : 2.6.9-34.ELsmp #1 SMP Fri Feb 24 16:54:53 EST 2006

SHORT COMMANDS

Massh supports abbreviate command sytax for its Option Controls:

SPECIAL HOSTGROUP

There's a special User HostGroup called 'results' ($HOME/.massh/hosts/results) that will contain the results of the SUBCOMMAND's 'worked' and 'bombed', along with a time stamp of the time the command was run and the command itself. This is mostly to address anything that 'bombed' during Massh runs where 100% success was either expected or required. Ambit will not show this group by default when running: ambit list hostgroups. Nor will Ambit know it exists since it is not in any of Ambit's data paths. If you would like this group to be listed simply hard link it to ($HOME/.ambit/hosts/results):

$ ln $HOME/.massh/hosts/results $HOME/.ambit/hosts/results

$ ambit list hostgroups | grep results
results:   2011-07-11 09:26:10 grep massh grep Marschall /etc/passwd

EXAMPLES

This site will always use valid XHTML and valid CSS.

© AfterTheTweet.com 2014