Automate password less SSH between thousands of linux servers

Last Updated on December 16, 2019 by Paresh Gupta

There is no shortage of information on web on setting up password less ssh. But I had to automate the process to be executed between 1000s of linux servers, to be completed within seconds or at most minutes. The high level steps are:

1. Create private and public keys
2. Copy public keys to .ssh/authorized_keys file to the remote machine

For two servers, above steps are required twice, once on each server. For 10 servers, above steps are required 10 x 9 = 90 times. Repeating the same set of commands 90 times is just so un-cool. As a solution, here is a script I wrote, which:

1. Expects a list of servers. Just separate the server names or IP addresses by space. See the code for details.
2. Expects you to enter the password to login to the remote server. This is required to copy the public keys to the remote server. This script assumes same password on all servers and hence, just asks once and re-uses.
3. Goes in loop over the list of servers and:
   1. Uses ssh-keygen -t rsa to generate public and private keys
   2. Goes in loop over all other servers in the list and
      1. Creates .ssh directory on remote servers
      2. Copies the public keys to the remote server using ssh-copy-id -i.

Overall, for 10 servers, the script opens a ssh session to all of them (one at time), create ssh keys and copies the public keys to rest of the 9 servers.

It has been coded in Expect and hence, you need expect on your machine. For CentOS 7 (that is what I was using):

yum -y install expect

Here is the script.

#!/usr/bin/expect

set prompt {[#>$] }
set user $env(USER)
set this_host $env(HOSTNAME)
set server_list {server1 server2 server3 server4 server5 server6 server7 server8 server9 server10}

# Ask user for password, in case password-less SSH not enabled
stty -echo
send "Password:"
expect {
    -re "(.*)\n" {
        send "\n"
        set passwd $expect_out(1,string)
        send "Confirm:"
        expect {
            -re "(.*)\n" {
                send "\n"
                if ![string match $passwd $expect_out(1,string)] {
                    send_user "Password mismatch\n"
                    stty echo
                    exit
                }
            }
            timeout {
                send_user "\n---Timeout---\n"
                exit
            }
        }
    }
    timeout {
        send_user "\n---Timeout---\n"
        exit
    }
}
stty echo

foreach server_host $server_list {
    spawn ssh $user@$server_host
    expect {
        "*\\(yes/no\\)? " {
            send "yes\n"
            exp_continue
        }
        "*password:" {
            send "$passwd\n"
            exp_continue
        }
        timeout {
            send_user "SSH timed out from $this_host to $server_host"
            continue
        }
        -re $prompt {
            # Generate RSA Key
            send "ssh-keygen -t rsa\n"
            expect -re "id_rsa\\): "
            send "\n"
            expect {
               "Overwrite (y/n)" {
                    send "n\n"
                }
                expect -re "passphrase\\): " {
                    send "\n"
                    exp_continue
                }
                expect -re "again: " {
                    send "\n"
                }
            }
            expect -re $prompt

            # Create .ssh directory. Ideally, this directory should already exist
            foreach server $server_list {
                if [string match $server_host $server] {
                    # DO not create directory on local machine
                    continue
                }
                send "ssh $user@$server mkdir -p .ssh\n"
                expect {
                    "*\\(yes/no\\)? " {
                        send "yes\n"
                        exp_continue
                    }
                    "*password:" {
                        send "$passwd\n"
                    }
                    timeout {
                        send_user "SSH timed out from $server_host to $server"
                        continue
                    }
                    -re $prompt {}
                }
            }

            # Copy public key to authorized_keys on remote server
            foreach server $server_list {
                if [string match $server_host $server] {
                    # Do not copy keys to local machine
                    continue
                }
                send "ssh-copy-id -i .ssh/id_rsa.pub $user@$server\n"
                expect {
                    "*\\(yes/no\\)? " {
                        send "yes\n"
                        exp_continue
                    }
                    "*password: " {
                        send "$passwd\n"
                    }
                    timeout {
                        send_user "SSH timed out from $server_host to $server"
                        continue
                    }
                    -re $prompt {}
                }
            }
        }
    }
    send "\n"
    send "exit\n"
    expect -re $prompt
}

Don Libes’s book Exploring Expect helped me to write this script. Respect! (http://shop.oreilly.com/product/9781565920903.do)