Automating Scripts Via SSH

Have you ever logged on to a Linux machine, only to be greeted by a prompt requesting you make some choice? It’s probably a script being called from .bash_profile or .profile, and that script doesn’t let you exit until a ‘proper’ choice is registered, probably something chosen from a menu.

 

 

It’s a bit annoying when it happens. When running automated scripts via ssh, the result can be more than annoying, as it may prevent the scripts from working.

At Pythian, we have a framework of scripts that we run on RAC systems for health and performance assessments. Collecting data from the database is fairly straightforward for most SQL scripts, as the GV$ views show data for all nodes, so a login to the local database instance can provide all the needed data in that case.

When collecting data from the OS, or Oracle instance only data (X$ ‘tables’ for instance), it’s necessary to run a script on all the nodes in the cluster.

Typically, a single node is chosen to stage the scripts, and all data is collected in one place.

Piping a script to SSH

It’s not necessary to copy the scripts to the remote node for either For both shell orand SQL scripts: the scripts can be run via ssh, like this:

scriptOutput=$(cat some-script.sh | ssh node01)

This is a somewhat simplified example, but this will work from the command line as seen later in this article.

Recently however, a problem was encountered.

The scripts that relied on SSH had no results when run on a client’s 19c RAC, which is running on an Oracle Linux 7 server.

That seemed rather odd, so I reran them locally on a 19c RAC–everything worked as expected.

As it was only the scripts that relied on SSH that were the problem, I suspected there was a script in .bash_profile or .bashrc that was asking for user input.

Prompts in login files

As it turned out, that was the case, as there was a script in .profile that required user input.

In this case the account was configured to use .profile rather than .bash_profile. Bash will read .profile only if .bash_profile is not available.

Here is a test with both .profile, .bash_profile and .bashrc all available:

[oracle@ora192rac02 tmp]$ strace -o bash-03.strc bash --login
[oracle@ora192rac02 tmp]$ grep -E '(stat|open).*(\.bash_profile|\.bashrc|\.profile)' bash-03.strc
open("/home/oracle/.bash_profile", O_RDONLY) = 3
stat("/home/oracle/.bashrc", {st_mode=S_IFREG|0644, st_size=444, ...}) = 0
open("/home/oracle/.bashrc", O_RDONLY) = 3

The .profile is ignored.

When .bash_profile is removed however, .profile is read:

[oracle@ora192rac02 tmp]$ strace -o bash-04.strc bash --login
[oracle@ora192rac02 tmp]$ grep -E '(stat|open).*(\.bash_profile|\.bashrc|\.profile)' bash-04.strc
open("/home/oracle/.bash_profile", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/home/oracle/.profile", O_RDONLY) = 3

So, whether .bash_profile or .profile is used, the login process looks the same.

Circumventing the login files

As I dug into this issue, it seem that the RAC scripts should have worked fine, as they were non-interactive SSH sessions, and .bash_profile (or .profile) should not have run. SSH documentation states that non-interactive sessions should not run the login files, such as .bashrc and .bash_profile. As .bashrc is called from .bash_profile, it would be expected that if .bash_profile is not executed, then .bashrc should also not be executed.

Let’s find out if that is true.

Testing will be done with the oracle account on two different servers:

• ora10gR2 – Oracle Linux 5.5
• ora12102b – Oracle Linux 6.10

The .bashrc and .bash_profile files each have an echo statement

• echo “this is .bashrc”
• echo “this is .bash_profile”

echo this is .bashrc
if [ -f /etc/bashrc ]; then
        . /etc/bashrc
fi
alias l='ls -la'
set -o vi

 

echo this is .bash_profile
if [ -f ~/.bashrc ]; then
        . ~/.bashrc
fi
PATH=$PATH:$HOME/bin
export PATH

.bashrc warning

Please note that any echo statements, and/or interactive commands in .bashrc will probably prevent scp from working.
There will be no error message, it just won’t work.

The exit code will indicate that some issue occurred:

$ scp test.txt oracle@ora192rac02:~/
This is .bashrc
(oci) jkstill@benson ~ $
$ echo $?
1

When the “echo This is .bashrc” line is removed from .bashrc, it works as expected:

$ scp test.txt oracle@ora192rac02:~/
test.txt 100% 0 0.0KB/s 00:00
(oci) jkstill@benson ~ $
$ echo $?
0

Under what circumstances are the login files executed?

The reason for using these two older Linux versions is that that login behavior changed with the Oracle Linux 6 release.

Here are the SSH command options used in several following examples:

• -T do not allocate a terminal
• -tt force allocation of a pseudo terminal

Any session that does not login is non-interactive.   Here is an SSH connection as oracle to a server running Oracle Linux 5.

The first is a standard interactive SSH session:

$ ssh oracle@ora10gr2
Last login: Fri Mar 25 10:18:25 2022 from benson
this is .bash_profile
this is .bashrc
[oracle@ora10gR2 ~]$

Notice the output from each of the echo statements.

Now run a non-interactive session to the same server ( it is non-interactive as a command is being sent, “id”), and you can see that .bashrc is executed when there are no SSH options included on the command line.

When -tt is used, the “this is .bashrc” output is absent, and so we know that .bashrc was not executed:

$ ssh oracle@ora10gr2 id
this is .bashrc
uid=300(oracle) gid=301(oinstall) groups=300(dba),301(oinstall)

$  ssh -tt oracle@ora10gr2 id
uid=300(oracle) gid=301(oinstall) groups=300(dba),301(oinstall)
Connection to ora10gr2 closed.

Neither .bash_profile nor .bashrc were processed when there was no login and the use of a pseudo terminal was forced.

Here is the same login process on Linux 6, first with a standard login:

$  ssh oracle@ora12102b
Last login: Fri Mar 25 10:15:37 2022 benson
this is .bash_profile
This is .bashrc
[oracle@ora12102b ~]$

… and now just running the id command as before on Linux 5:

$ ssh oracle@ora12102b id
This is .bashrc
uid=54321(oracle) gid=54324(asmdba) groups=54324(asmdba),492(vboxsf),54321(oinstall),54322(dba),54325(sysasm)

$  ssh -tt oracle@ora12102b id
This is .bashrc
uid=54321(oracle) gid=54324(asmdba) groups=54324(asmdba),492(vboxsf),54321(oinstall),54322(dba),54325(sysasm)
Connection to ora12102b closed.

Note that even when no login was performed, the .bashrc file was still processed.

The following table summarizes what we have seen so far:

Server	Linux Version	Interactive Login	Force Psuedo Terminal	.bashrc executed	.bash_profile executed
ora10Gr2	5.5	Y	N	Y	Y
ora10Gr2	5.5	N	N	Y	N
ora10Gr2	5.5	N	Y	N	N
ora12102b	6.10	Y	N	Y	Y
ora12102b	6.10	N	N	Y	N
ora12102b	6.10	N	Y	Y	N

And now here is something very interesting.  The .bash_profile file has been removed for the oracle account on each of these servers. Remember that .bashrc is called from .bash_profile.

What do you think will happen when running a non-interactive session?

First on the Linux 5 server, again without and with -T (no terminal) :

$ ssh oracle@ora10gr2 id
this is .bashrc
uid=300(oracle) gid=301(oinstall) groups=300(dba),301(oinstall)

$ ssh -T oracle@ora10gr2 id
uid=300(oracle) gid=301(oinstall) groups=300(dba),301(oinstall)
$

When logging onto Linux 5, and using the  -T option, the .bashrc was not executed.

And now on the Linux 6 server:

$  ssh oracle@ora12102b  id
This is .bashrc
uid=54321(oracle) gid=54324(asmdba) groups=54324(asmdba),492(vboxsf),54321(oinstall),54322(dba),54325(sysasm)

$  ssh -T oracle@ora12102b  id
This is .bashrc
uid=54321(oracle) gid=54324(asmdba) groups=54324(asmdba),492(vboxsf),54321(oinstall),54322(dba),54325(sysasm)
$

For non-interactive logins, even when the .bash_profile file is missing, the .bashrc login gets executed by default on both versions of Linux. On versions of Linux older than 6, that behavior can be changed by including the -T flag.

This behavior changes for interactive logins.  If .bash_profile is missing, .bashrc does not get executed:

[oracle@ora12102b ~]$ rm .bash_profile
[oracle@ora12102b ~]$ logout
Connection to ora12102b closed.

$  ssh oracle@ora12102b
Last login: Fri Mar 25 13:13:28 2022 from benson
-bash-4.1$

The behavior for interactive vs non-interactive logins is clearly different as regards .bashrc.  If the file exists, and it is a non-interactive login, .bashrc will always get executed on Linux 6+.

It seems that on Linux 6 and later, there is no way to circumvent the execution of the .bashrc script for non-interactive logins.

Dealing with .bashrc

Normally one would not expect to find any menu prompts or other interactive code in .bashrc.  Usually that is done in .bash_profile, if at all.

But it does happen, and I suspect that is the case with the client where our scripts didn’t work properly.

As implemented in Red Hat Linux and derivatives such as Oracle Linux, .bashrc cannot be circumvented with RH Linux 6+.

And that is a problem when working with automated scripts.

To simulate the situation encountered, a script menu-loop.sh has been placed in .bash_profile in the ora12102b:~oracle/.bash_profile.  This is the Oracle Linux 6 server:

#!/usr/bin/env bash

caughtSig () {
   echo
   echo "Trapped signal!"
   echo
}

trap "caughtSig" INT TERM HUP

declare answer=''

while [ -z "$answer" ]
do
   echo your choice:
   read choice
   case $choice in
      1) answer=1; break;;
      2) answer=2; break;;
      3) answer=3; break;;
      *) echo incorrect!;echo "You Entered "'$choice'"; echo;;
   esac
done

Menu-loop.sh expects a response of 1, 2 or 3.

Signals are trapped, so the only way to exit is to answer the prompt correctly:

$  ssh oracle@ora12102b
Last login: Fri Mar 25 12:02:16 2022 from benson
This is .bashrc
your choice:
4
incorrect!
You Entered '4'

your choice:
^C
Trapped signal!

2
[oracle@ora12102b ~]$

By default, neither /bin/bash nor /bin/sh will process .bash_profile for a non-interactive shell.

When accepting input from STDIN such as via a pipe, the session is non-interactive.  This is an important distinction, as Bash operates differently when non-interactive, as it will not process the .bash_profile file as discussed earlier.

We have also learned that on Oracle Linux versions 6 and later, .bashrc is always processed.  So there is nothing we can do to stop the menu from running.

Here this is tested by piping the contents of a one-line shell script to ssh.

The file t.sh contains 1 line: ‘id’

Watch what  happens when we try to run the contents of t.sh via SSH without forcing the use of a pseudo tty:

$ cat t.sh | ssh oracle@ora192rac02
This is .bashrc
your choice:
incorrect!

your choice:
incorrect!

your choice:
incorrect!
...

That output continues until pressing CTL-C.

As this was by default a non-interactive login, there was not any way to accept user input, and the menu-loop.sh went into an uncontrolled loop.

Now let’s run it with the -tt flag, forcing the use of a terminal:

$  cat t.sh | ssh -tt  oracle@ora12102b
id
Last login: Fri Mar 25 12:08:05 2022 from benson
This is .bashrc
your choice:
incorrect!
You Entered 'id'

your choice:

^C
$

The menu-loop.sh script consumed the ‘id’ line from t.sh as a response, flagged it as incorrect and is awaiting another response.

However, the script has hung, as even though we have forced the use of a pseudo tty, this is a non-interactive session.

The only way out is CTL-C.

Bash options

There some two flags for bash that looked promising for dealing with this issue.

• --noprofile – do not run .bash_profile
• --norc – do not run .bashrc

Initially it seemed these could be used to circumvent .bashrc.  While --noprofile does prevent .bash_profile from being executed, again, we are powerless to stop .bashrc from being run.  The .bash_profile is never really a problem for this use case, as it does not get run, but as you have already seen, Linux 6+ always executes .bashrc.

Here’s an example of using bash to try and circumvent .bashrc:

$ ssh -tt oracle@ora12102b 'bash --norc '
This is .bashrc
your choice:
1
bash-4.1$

.bashrc is always executed.

Using a subshell

There is hope, even though there still is the problem of that script in .bashrc. As it is not my machine (at least in the client case), I cannot alter the .bashrc file.

Valid input can be passed to the script  by including an echo command along with the contents of the script t.sh when logging in via ssh.  This does require that you know the appropriate valid response to the prompt.  In this case, that value will be ‘2’.

The echo and cat commands must be in a subshell for this to work.  An exit command must also be included, or the shell will hang waiting for input.

As there is really no need for user input, and therefore, there’s no need for a terminal, the -T argument will be used, which tells ssh to not allocate a pseudo tty:

$  (echo 2; cat t.sh;echo exit) | ssh -T  oracle@ora12102b
This is .bashrc
your choice:
uid=54321(oracle) gid=54324(asmdba) groups=54324(asmdba),492(vboxsf),54321(oinstall),54322(dba),54325(sysasm)
$

Now the prompt in .bashrc has been correctly answered, the expected output (from the ‘id’ command) has appeared, and the local shell prompt has reappeared.

Additional commands can be entered into the shell script ‘t.sh’, such as ‘hostname’ and ‘who’:

$  (echo 2; cat t.sh;echo exit) | ssh -T oracle@ora12102b
This is .bashrc
your choice:
this is .bash_profile
uid=54321(oracle) gid=54324(asmdba) groups=54324(asmdba),492(vboxsf),54321(oinstall),54322(dba),54325(sysasm)
ora12102b.jks.com
root     tty1         2022-01-24 13:38 (:0)
root     pts/0        2022-01-24 13:38 (:0)
root     pts/1        2022-01-24 13:38 (:0)
root     pts/2        2022-01-24 13:38 (:0)
root     pts/3        2022-01-24 13:38 (:0)
root     pts/4        2022-01-24 13:38 (:0)
root     pts/5        2022-03-25 08:41 (benson)

Conclusion

There is not any method I can find on RedHat/Oracle Linux 6+ to NOT process .bashrc when connecting via ssh.

If it is not possible to alter the .bashrc file, then the workaround is shown in the previous example, and summarized here.

• In a subshell:
  • echo the expected response
  • cat the file containing commands to run on the remote server
  • echo ‘exit’ to exit the spawned process
• pipe it to ‘ssh -T username@hostname’
  • -T eliminates allocating a pseudo tty
  • this also eliminates doing a full login.
  • (echo 2; cat commands.sh;echo exit) | ssh -T username@server

I have learned some things in the process of writing this blog.  While some similar techniques were used in the Pythian scripts, I believe the ssh commands can now be simplified somewhat, making the code easier to maintain.

One final note:  If you find any of your Linux servers are performing user interaction in .bashrc, it would be wise to move that to .bash_profile.  It would be even better however to have no interaction in Linux login files.

 

I hope this post useful. Let me know if you have any questions in the comments, and don’t forget to sign up for the next post.