How to replace string with shell metacharacters in bash

Its very tricky to replace a string containing shell metacharacters. Following is how I did it. This is not the complete list but just what was required for me

::::::::::::::
encode.sed
::::::::::::::s/\[/START_SQUARE/g
s/\]/END_SQUARE/g
s/\}/END_CURLY/g
s/{/START_CURLY/g
s/[$]/DOLLAR_SIGN/g
s/["]/DOUBLE_QUOTES/g
s/[|]/PIPE_SIGN/g
s/[/]/FRONT_SLASH/g
::::::::::::::
decode.sed
::::::::::::::s/START_SQUARE/[/g
s/END_SQUARE/]/g
s/END_CURLY/}/g
s/START_CURLY/{/g
s/DOLLAR_SIGN/$/g
s/DOUBLE_QUOTES/"/g
s/PIPE_SIGN/|/g
s/FRONT_SLASH/\//g
::::::::::::::
rep.sh
::::::::::::::old='[ -n ${MY_HOME:=""} ] || source /home/${USER}/app/myvariables.sh'
new='[ -n ${MY_APPS:=""} ] || source ${MY_HOME}/myvariables.sh'
oldx=`echo "${old}" | sed -f encode.sed`
newx=`echo "${new}" | sed -f encode.sed`
cat myshell.sh | sed -f encode.sed | sed 's/'"${oldx}"'/'"${newx}"'/g' | sed -f decode.sed

How to see the memory stats of a box running linux

Following file contains all the information  about the memory of the machine running linux
       cat /proc/meminfo
The output of this command is something like

MemTotal:     74178200 kB
MemFree:      14204988 kB
Buffers:       7106472 kB
Cached:       43864060 kB
SwapCached:          0 kB
Active:       20877324 kB
Inactive:     36139244 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     74178200 kB
LowFree:      14204988 kB
SwapTotal:     2096472 kB
SwapFree:      2096196 kB
Dirty:          278512 kB
Writeback:           0 kB
AnonPages:     6045916 kB
Mapped:          71672 kB
Slab:          2836748 kB
PageTables:      38336 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  39185572 kB
Committed_AS: 33337792 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    264000 kB
VmallocChunk: 34359473963 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

How to see calendar for any month or year in Linux Bash command line

Following command returns calendar for the current month with current day highlighted in black background [tikku@tata ~]$ cal

   November 2011
Su Mo Tu We Th Fr Sa
       1  2  3  4  5
 6  7  8  9 10 11 12
13 14 15 16 17 18 19
20 21 22 23 24 25 26
27 28 29 30

Following command returns calendar for Jan 2012.
[tikku@tata ~]$ cal 1 2012

    January 2012
Su Mo Tu We Th Fr Sa
 1  2  3  4  5  6  7
 8  9 10 11 12 13 14
15 16 17 18 19 20 21
22 23 24 25 26 27 28
29 30 31

Following command returns calendar for year 2012
[tikku@tata ~]$ cal 2012

                               2012

       January               February                 March
Su Mo Tu We Th Fr Sa   Su Mo Tu We Th Fr Sa   Su Mo Tu We Th Fr Sa
 1  2  3  4  5  6  7             1  2  3  4                1  2  3
 8  9 10 11 12 13 14    5  6  7  8  9 10 11    4  5  6  7  8  9 10
15 16 17 18 19 20 21   12 13 14 15 16 17 18   11 12 13 14 15 16 17
22 23 24 25 26 27 28   19 20 21 22 23 24 25   18 19 20 21 22 23 24
29 30 31               26 27 28 29            25 26 27 28 29 30 31

        April                   May                   June
Su Mo Tu We Th Fr Sa   Su Mo Tu We Th Fr Sa   Su Mo Tu We Th Fr Sa
 1  2  3  4  5  6  7          1  2  3  4  5                   1  2
 8  9 10 11 12 13 14    6  7  8  9 10 11 12    3  4  5  6  7  8  9
15 16 17 18 19 20 21   13 14 15 16 17 18 19   10 11 12 13 14 15 16
22 23 24 25 26 27 28   20 21 22 23 24 25 26   17 18 19 20 21 22 23
29 30                  27 28 29 30 31         24 25 26 27 28 29 30

        July                  August                September
Su Mo Tu We Th Fr Sa   Su Mo Tu We Th Fr Sa   Su Mo Tu We Th Fr Sa
 1  2  3  4  5  6  7             1  2  3  4                      1
 8  9 10 11 12 13 14    5  6  7  8  9 10 11    2  3  4  5  6  7  8
15 16 17 18 19 20 21   12 13 14 15 16 17 18    9 10 11 12 13 14 15
22 23 24 25 26 27 28   19 20 21 22 23 24 25   16 17 18 19 20 21 22
29 30 31               26 27 28 29 30 31      23 24 25 26 27 28 29
                                              30
       October               November               December
Su Mo Tu We Th Fr Sa   Su Mo Tu We Th Fr Sa   Su Mo Tu We Th Fr Sa
    1  2  3  4  5  6                1  2  3                      1
 7  8  9 10 11 12 13    4  5  6  7  8  9 10    2  3  4  5  6  7  8
14 15 16 17 18 19 20   11 12 13 14 15 16 17    9 10 11 12 13 14 15
21 22 23 24 25 26 27   18 19 20 21 22 23 24   16 17 18 19 20 21 22
28 29 30 31            25 26 27 28 29 30      23 24 25 26 27 28 29
                                              30 31

How to get detailed information about the machine's CPU in Linux

You can run the following command to see all information about the processors of the machine
    less /proc/cpuinfo

Note: This file shows logical cpu's seen by the OS on the machine, i.e. the CPU shown can be
- physical CPU
- hyperthreading 
- shared (e.g. dual or quad) core


Following command will return number of physical CPU's
    cat /proc/cpuinfo | egrep '(physical id).*:'  | sort -u | wc -l

Following command will return number of shared cores
    cat /proc/cpuinfo | egrep '(cpu cores).*:' | awk -F ":" '{print $2}'   | sort | uniq

If following command returns 1 then hyper-threading is disabled on then machine, if  >1 then hyper threading is enabled on the machine
    echo $(( `cat /proc/cpuinfo | egrep '(siblings).*:' | awk -F ":" '{print $2}'   | sort | uniq` \
           / `cat /proc/cpuinfo | egrep '(cpu cores).*:' | awk -F ":" '{print $2}'   | sort | uniq` ))

To get total virtual processors available to the OS, multiply above 3 values.

How to perform an FTP transaction using Bash scripting in Linux

If you want to automate an FTP operation then you can easily use Bash scripting for that.


For example, following script gets file file.txt from ftp server


ftp -ivn ftp.internet.com<<FTP_COMMANDS
        user <username> <password>
        type binary
        cd /
        lcd /home/<username>
        get file.txt
        quit
FTP_COMMANDS

Redirections and file descriptors in bash

Following are some redirection options in bash that are not widely used but are very useful

- To redirect both stdout and stdin to a file in bash
     echo tan &> filename 
 OR
     echo tan > filename 2>&1

- To redirect stdout of a command to stderr
       echo tan 2>&1

- Following command will open a file for reading and writing and assign the interger file description N to it. If the file does not exist then it will be created
    9<>filename
Following is an example of how you can use this
     exec 9<>temp     # open file temp and assign file descriptor 9 to it
     echo aman >&9  # Write aman to the file
     exec 9>&-         # Close the file descriptor Now read the file and it will contain string aman
     cat temp

Before closing the file descriptor, you can see it opened

    cd /proc/$$/fd 



Closing File Descriptors

n<&-

Close input file descriptor n.

0<&-, <&-

Close stdin.

n>&-

Close output file descriptor n.

1>&-, >&-

Close stdout

How to convert a binary number to decimal in Bash

Following is how you can convert a binary number to decimal in Linux Bash
    echo $(( 2#101 ))
This  command will return
    5

 

How to create a daemon in Bash in Linux

Following is how you can write a daemon in Bash in Linux

- Create script, say runner.sh
- Make it executable
  chmod 500 runner.sh
- Copy to directory /etc/init.d/
- Create link file to the script to have this file up and running
     ln -s /etc/init.d/runner.sh.sh /etc/rc.d/rc3.d/S43runner.sh
     ln -s /etc/init.d/runner.sh.sh /etc/rc.d/rc3.d/K43runner.sh

How to see which version of Linux I am running

Run the following command to see all information about which version of LInux you are running
uname -a && cat /etc/*release

Calculate IP's between specified range

If you want to calculate all IP's between the specified range then following perl script can be used.

#!/usr/bin/perl
$numArgs = $#ARGV + 1;
if ($numArgs ne "2"){
        print "USAGE: perl $0 <start ip address> <end ip address>\n";
        exit;
}

$sip=$ARGV[0];
$eip=$ARGV[1];

($sip1,$sip2,$sip3,$sip4)=split(/[.]/,$sip);
($eip1,$eip2,$eip3,$eip4)=split(/[.]/,$eip);

$next_ip=$sip;
$end_ip=$eip;
while ($next_ip ne $end_ip){

        $next_ip="$sip1.$sip2.$sip3.$sip4";
        print $next_ip."\n";

        $sip4++;
        if ( $sip4 > 255){
                $sip4=0;
                $sip3 =$sip3 + 1;
        }

        if ( $sip3 > 255){
                $sip3=0;
                $sip2 =$sip2 + 1;
        }

       if ( $sip2 > 255){
                $sip2=0;
                $sip1 =$sip1 + 1;
        }

}

Usage

perl create_ip_ranges.pl 192.168.0.0  192.168.0.2

will return

192.168.0.0
192.168.0.1
192.168.0.2

How to run mysql from Linux command line

Following code connects to MySql server and show all databases in mysql with column delimited by tab

mysql -hHOST -uUSER -pPASSWORD -e "show databases" -N -s

-N means do *not* display the column names

-s option means display raw data with no formatting

To run multiple commands, delimit them by semicolon.

To read command from a file run

mysql -hHOST -uUSER -pPASSWORD < batch-fileor

mysql -hHOST -uUSER -pPASSWORD-e "source batch-file"

How to encrypt and decrypt a file in Linux

You can use command gpg to encrypt and decrypt a file in Linux

Following is an example of how you can encrypt a file using gpg
     gpg -c test.txt
This command will ask for a passwphrase and on successful execution will create file test.txt.gpg

Following is an example of how you can decrypt a file using gpg
   gpg -d test.txt.gpg > test.txt.tmp
NOTE: Do *not* decrypt file to test.txt because if you accidently provided wrong passphrase , decryption will fail and file test.txt will be empty and if you forgot the passphrase, there is no way to receover test.txt file

How to use getopts with single character and multicharacter switches

#!/bin/bash

# A small example program for using the new getopt(1) program.
# This program will only work with bash(1)
# An similar program using the tcsh(1) script language can be found
# as parse.tcsh

# Example input and output (from the bash prompt):
# ./parse.bash -a par1 'another arg' --c-long 'wow!*\?' -cmore -b " very long "
# Option a
# Option c, no argument
# Option c, argument `more'
# Option b, argument ` very long '
# Remaining arguments:
# --> `par1'
# --> `another arg'
# --> `wow!*\?'

# Note that we use `"$@"' to let each command-line parameter expand to a 
# separate word. The quotes around `$@' are essential!
# We need TEMP as the `eval set --' would nuke the return value of getopt.
TEMP=`getopt -o ab:c:: --long a-long,b-long:,c-long:: \
     -n 'example.bash' -- "$@"`

if [ $? != 0 ] ; then echo "Terminating..." >&2 ; exit 1 ; fi

# Note the quotes around `$TEMP': they are essential!
eval set -- "$TEMP"

while true ; do
 case "$1" in
  -a|--a-long) echo "Option a" ; shift ;;
  -b|--b-long) echo "Option b, argument \`$2'" ; shift 2 ;;
  -c|--c-long) 
   # c has an optional argument. As we are in quoted mode,
   # an empty parameter will be generated if its optional
   # argument is not found.
   case "$2" in
    "") echo "Option c, no argument"; shift 2 ;;
    *)  echo "Option c, argument \`$2'" ; shift 2 ;;
   esac ;;
  --) shift ; break ;;
  *) echo "Internal error!" ; exit 1 ;;
 esac
done
echo "Remaining arguments:"
for arg do echo '--> '"\`$arg'" ; done

How to replace a string in a string variable in bash without using sed, awk or perl

Bash provides a way to replace a string
${<STRING>//<PATTERN TO FIND IN STRING>/<STRING TO REPLACE>}

How to specify control characters in sed

Sometimes you might have to process data, using sed, that has control characters like ^A or ^B. Following is how you do that

The following command echo's a string with ^B and will replace ^B by "|"
echo -e "Tiger^BLepo" | sed 's/\x02/|/g'

How to find length of a string in Linux

Suppose you have following variables in your shell
str="foobar"
and you want to output the length of this string (length of this string is 6) then run the following command
echo ${#str}

Calculate IP's between specified range

If you want to calculate all IP's between the specified range then following perl script can be used.

--------------------------------------------------------------------------

#!/usr/bin/perl
$numArgs = $#ARGV + 1;
if ($numArgs ne "2"){
        print "USAGE: perl $0 <start ip address> <end ip address>\n";
        exit;
}

$sip=$ARGV[0];
$eip=$ARGV[1];

($sip1,$sip2,$sip3,$sip4)=split(/[.]/,$sip);
($eip1,$eip2,$eip3,$eip4)=split(/[.]/,$eip);

$next_ip=$sip;
$end_ip=$eip;
while ($next_ip ne $end_ip){

        $next_ip="$sip1.$sip2.$sip3.$sip4";
        print $next_ip."\n";

        $sip4++;
        if ( $sip4 > 255){
                $sip4=0;
                $sip3 =$sip3 + 1;
        }

        if ( $sip3 > 255){
                $sip3=0;
                $sip2 =$sip2 + 1;
        }

       if ( $sip2 > 255){
                $sip2=0;
                $sip1 =$sip1 + 1;
        }

}
-------------------------------------------------------------------------------

Usage

perl create_ip_ranges.pl 192.168.0.0  192.168.0.2

will return

192.168.0.0
192.168.0.1
192.168.0.2

Following site can also be used. 

To find all IP's within a range, use http://www.ipaddresslocation.org/ip-address-ranges.php

How to see non printable special and control characters in Unix

Sometimes while troubleshooting data issues I want to see non printable characters on the stdin.
Following is how to do that

1. Create a file with non printable characters
     echo -e "Quick brown fox\tjumped over lazy\0002dog"  > test

2. View the file
     cat test
You will see
Quick brown fox jumped over lazydog

3. Now to see non printable speical characters use command od as

      od -c test

Output will be
0000000   Q   u   i   c   k       b   r   o   w   n       f   o   x  \t
0000020   j   u   m   p   e   d       o   v   e   r       l   a   z   y
0000040 002   d   o   g  \n
0000045