Tovili TWS log addition info

Display additional information for Tivloi logs -

• Job Stream Job listing
• Successor 
• Predecessor

Simple Linux Socket

My first venture to socket.

• https://idea.popcount.org/2014-04-03-bind-before-connect/

• cat /proc/sys/net/ipv4/ip_local_port_range
• http://users.pja.edu.pl/~jms/qnx/help/tcpip_4.25_en/prog_guide/sock_advanced_tut.html 
• http://www.yolinux.com/TUTORIALS/Sockets.html
• http://www.softlab.ntua.gr/facilities/documentation/unix/unix-socket-faq/unix-socket-faq.html#toc2

Access Env of anothe process

• Read env of another process: http://unix.stackexchange.com/questions/29128/how-to-read-environment-variables-of-a-process

1. xargs --null --max-args=1 echo < /proc/<pid>/environ
2.  ps wwwe <pid> | tr  ' ' '\n'

• Orphan vs Zombie vs Daemon processes: http://gmarik.info/blog/2012/08/15/orphan-vs-zombie-vs-daemon-processes

•  How To Kill Defunct Or Zombie Process: source: http://www.linuxnix.com/how-to-kill-defunct-or-zombie-process/

 

Process killed due to OOM

Original article: 
http://unix.stackexchange.com/questions/128642/debug-out-of-memory-with-var-log-messages

" The kernel will have logged a bunch of stuff before this happened, but most of it will probably not be in /var/log/messages, depending on how your (r)syslogd is configured. Try:

grep oom /var/log/*
grep total_vm /var/log/*

The former should show up a bunch of times and the latter in only one or two places. That is the file you want to look at.
Find the original "Out of memory" line in one of the files that also contains total_vm. Thirty second to a minute (could be more, could be less) before that line you'll find something like:

kernel: foobar invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0

You should also find a table somewhere between that line and the "Out of memory" line with headers like this:

[ pid ]   uid  tgid total_vm      rss nr_ptes swapents oom_score_adj name

This may not tell you much more than you already know, but the fields are:

• pid The process ID.
• uid User ID.
• tgid Thread group ID.
• total_vm Virtual memory use (in 4 kB pages)
• rss Resident memory use (in 4 kB pages)
• nr_ptes Page table entries
• swapents Swap entries
• oom_score_adj Usually 0; a lower number indicates the process will be less likely to die when the OOM killer is invoked.

You can mostly ignore nr_ptes and swapents although I believe these are factors in determining who gets killed. This is not necessarily the process using the most memory, but it very likely is. For more about the selection process, see here. Basically, the process that ends up with the highest oom score is killed -- that's the "score" reported on the "Out of memory" line; unfortunately the other scores aren't reported but that table provides some clues in terms of factors.
Again, this probably won't do much more than illuminate the obvious: the system ran out of memory and mysqld was choosen to die because killing it would release the most resources. This does not necessary mean mysqld is doing anything wrong. You can look at the table to see if anything else went way out of line at the time, but there may not be any clear culprit: the system can run out of memory simply because you misjudged or misconfigured the running processes.

"

> free -m

sed

good info: https://www.grymoire.com/Unix/Sed.html

• change multiple values at column position for multiple records

sed -e 's/./X/10' -e '/^000/s/./X/18' -e '/^001/s/./X/819' input

• grep uppercase words preceded by '-' & followed by ','

grep -oP "\-\w[A-Z]+\w," xxx

* print lines between patterns(inclusive) & stop
sed -n '/^pattern1/,/^pattern2/p;/^pattern2/q'

* print lines between patterns(exclusive)
sed -n '1,/^pattern1/d;/^pattern2/,$d;p'

* count # of lines between patterns (EMPLID)
awk '/EMPLID/ {if (count) print count; print; count=0; next} {count++} END {print count}' file

* replace char if pattern found
 sed '/^Box 14/ s/#/^/g' file

*insert line after pattern
sed '/pattern/a insert_text_line' file

*insert line before pattern
(https://fabianlee.org/2018/10/28/linux-using-sed-to-insert-lines-before-or-after-a-match/)

sed '/^anothervalue=.*/i before=me' test.txt

insert multiple lines

sed '/^anothervalue=.*/i before=me\nbefore2=me2' test.txt

* add text to pattern:
echo '123 pattern 456' | sed 's/\<pattern\>/xxx&xxx/' 
echo '123 pattern 456' | sed 's/\(pattern\)/xxx\1xxx/' 
echo '123 pattern 456' | sed 's/\(pattern\) 4\(.*\)6/xxx\1xxx \2/' 

* replace extended ascii with spaces:

SET KK_CLOSE_FLAG =Â 'N' | LANG=C sed 's/[\x80-\xFF]//g' 

* print from match to end of file:
> sed -ne '/pattern/,$ p'
  # alternatively, if you don't want to print the match:

> sed -e '1,/pattern/ d'

* Branching to Label:
> echo -e "a\nb\nc\nd\ne\nf\ng\nh" | 

sed '3b action; /e/{p; b action;}; b end; : action; a \
lalala
: end'

https://unix.stackexchange.com/questions/117511/is-it-possible-to-match-multiple-specific-line-numbers-not-range-with-sed

* print lines between two patterns, inclusive or exclusive (in sed, AWK or Perl)

https://stackoverflow.com/questions/38972736/how-to-print-lines-between-two-patterns-inclusive-or-exclusive-in-sed-awk-or

Display Tivoli Job Name next to logs

This ksh function lt (list tivoli) display additional Job Stream+ Job names next to log file list so we can see which job created the logs. It based on "ls -ltr" with Job names displayed next to log file name.

Tivoli logs are stored in ~maestroX/stdlist/YYYY.MM.DD. They keeps the job execution details but is hard to tell which job it is based on.  This function makes it easy to investigate the log files.

In a stdlist/YYYY.MM.DD folder and run lt, Output:

-rw-r--r-- 1 user1 uGroup 1543 Jun 24 00:05 O33964244.0004 (JOBSTREAM1.JOB1:0/1)
-rw-r--r-- 1 user1 uGroup 1060 Jun 24 00:05 O33964246.0005 (JOBSTREAM1.JOB2:1/0)
-rw-r--r-- 1 user1 uGroup 1578 Jun 24 00:10 O31059994.0010 (JOBSTREAM2.JOB1:0/4)
-rw-r--r-- 1 user1 uGroup 1543 Jun 24 00:10 O34291814.0009 (JOBSTREAM3.JOB1:0/1)
-rw-r--r-- 1 user1 uGroup 1060 Jun 24 00:15 O35487936.0015 (JOBSTREAM3.JOB2:1/0)
-rw-r--r-- 1 user1 uGroup 1060 Jun 24 00:15 O35389680.0015 (JOBSTREAM2.JOB2:1/10)
-rw-r--r-- 1 user1 uGroup 1058 Jun 24 00:15 O34934840.0015 (JOBSTREAM4.JOB1:1/12)
-rw-r--r-- 1 user1 uGroup 1059 Jun 24 00:15 O34775842.0015 (JOBSTREAM4.JOB1:0/36)    
-rw-r--r-- 1 user1 uGroup 1059 Jun 24 00:15 O34775841.0015 (JOBSTREAM5.JOB1:-/-)

• First # after Job Name is the Exit Status. '-' if still running
• 2nd # after Job Name is  Elapsed Time.  '-' if still running
• grep on a Job Stream to see all Jobs within and processing time
• grep on a Job across dates to see overall performance

linux/ksh:

lt(){ for x in `ls -tr $* | grep  -E "O[0-9]*.[0-9]{4}$"`; do; echo -e `ls -l $x` "\c" ; grep ^"= JOB" $x | sed "s/[:#\\[,.]/ /g" | awk '{printf "(%s.%s", $4,$8}'; echo `grep '^= Exit Status' $x`|sed 's/ : /:/g'| awk -F: '{printf ": %s/", (length($0) == 0)?"-":$2}'; echo `grep 'Elapsed' $x`|sed 's/ : /:/g'| awk -F: '{printf "%s)\n", (length($0) == 0)?"-":$3}';done; }


linux/bash:
lt(){ for x in `ls -tr $* | grep  -E "O[0-9]*.[0-9]{4}$"`; do
echo -e `ls -l $x` "\c" ;grep ^"= JOB" $x | sed "s/[:#\\[,.]/ /g" | awk '{printf "(%s.%s", $4,$8}';
echo `grep '^= Exit Status' $x`|sed 's/ : /:/g'| awk -F: '{printf ": %s/", (length($0) == 0)?"-":$2}'; echo `grep 'Elapsed' $x`|sed 's/ : /:/g'| awk -F: '{printf "%s)\n", (length($0) == 0)?"-":$3}';done;}


aix/ksh:
lt{for x in `ls -tr $* | grep  -E "O[0-9]*.[0-9]{4}$"`; do
   echo  `ls -l $x` "\c" ;
   grep ^"= JOB" $x | sed "s/[:#\\[,.]/ /g" | awk '{printf "(%s.%s", $4,$8}'; echo `grep '^= Exit Status' $x`|sed 's/ : /:/g'| awk -F: '{printf ": %s/", (length($0) == 0)?"-":$2}'; echo `grep 'Elapsed' $x`|sed 's/ : /:/g'| awk -F: '{printf "%s)\n", (length($0) == 0)?"-":$3}';
done; }

Misc Linux/Shell

Get rid of ^M

• cat xxx | tr -d "\r"
• :%s/^M//g (in vi, ^M = Ctrl-V + Ctrl-M)
• sed -e 's/^M//g' 
• tr -d $'\r' < xx > yy

 Process ID

http://unix.stackexchange.com/questions/32576/how-to-get-subshells-pid-in-korn-shell-equivalent-of-bashpid

• sub-shell id: 

1. readlink /proc/self
2. p=$(exec sh -c 'echo $PPID')

• current shell id:

1. $$
2. read -r xxx _ </proc/self/stat (xxx=1st argv, _=rest of line)

• parent shell id:

1. $PPID  
2. ps -p 12345678 -o ppid=

How to tell Korn shell version

https://community.oracle.com/thread/2251850

1. [ "${ERRNO}" ] && echo ksh88 || echo ksh99

How to tell if login or interactive shell

• [[ $- == *i* ]] && echo 'Interactive' || echo 'Not interactive'
• case $- in *i*) echo "This shell is interactive";; *) echo "This is a script";; esac
• http://unix.stackexchange.com/questions/38175/difference-between-login-shell-and-non-login-shell: interactive+login,  non-interactive+login(rare),  non-interactive+non-login,  interactive+non-login,  
• to see all options: echo $- (http://www.tutorialspoint.com/unix_commands/ksh.htm)

Ksh startup

• /etc/profile
•  $HOME/.profile
•  ksh's startup file(any file pointed to by ENV variable whether or not in login shell)

Print lines between patterns

Source: www.shellhacks.com/en/Using-SED-and-AWK-to-Print-Lines-Between-Two-Patterns+&cd=5&hl=en&ct=clnk&gl=us

File:

I Love Linux 
***** BEGIN ***** 
BASH is awesome 
BASH is awesome 
***** END ***** 
I Love Linux

• sed 

Syntax :

sed -n '/StartPattern/,/EndPattern/p' FileName  <<<---inclusive both

Option	Description
-n, --quiet, --silent	Suppress automatic printing of pattern space
p	Print the current pattern space

Example :

sed -n '/BEGIN/,/END/p' info.txt 

***** BEGIN *****
BASH is awesome
BASH is awesome
***** END *****

•  awk

Syntax :

awk '/StartPattern/,/EndPattern/' FileName

Example :

awk '/BEGIN/,/END/' info.txt

***** BEGIN *****
BASH is awesome
BASH is awesome
***** END ***** 

Another VERY Cool snipet:

source: 
http://stackoverflow.com/questions/9476018/split-text-file-into-parts-based-on-a-pattern-taken-from-the-text-file 

BEGIN { fn=0 }
NR==1 { next }
NR==2 { delim=$1 }
$1 == delim {
    f=sprintf("test%02d.txt",fn++);
    print "Creating " f
}

{ print $0 > f }

1. initialize output file number
2. ignore the first line
3. extract the delimiter from the second line
4. for every input line whose first token matches the delimiter, set up the output file name
5. for all lines, write to the current output file

---------------------------------------- another ex:

* file: TEST_TAF PREF: RAC1 RAC2 RAC3 ...... AVAIL: RAC4  (PREF, AVAIL)

   >  sed 's/.*PREF//;s/AVAIL.*//' yourfile

   >  sed 's/.*PREF: //;s/ AVAIL.*//;s/  */,/g' yourfile

* https://nixtip.wordpress.com/2010/10/12/print-lines-between-two-patterns-the-awk-way/

file: 

test -3

test -2

test -1

OUTPUT

top 2

bottom 1

left 0

right 0

page 66

END

test 1

test 2

test 3

> awk '/OUTPUT/ {flag=1;next} /END/{flag=0} flag {print}'

-------------------------- Another Ex: starting with multiple patterns

sed -n '/\(12:05:43.376\|Begin FS_BP.Init.*Init.OnExecute\)/,/12:05:43.605/p'  AE_FS_BP_2740745.trc 

More examples ------ https://stackoverflow.com/questions/18185771/extract-nth-line-after-matching-pattern

To extract the Nth line after a matching pattern you want:

awk 'c&&!--c;/pattern/{c=N}' file

e.g.

awk 'c&&!--c;/Revision:/{c=5}' file

would print the 5th line after the text "Revision:"/.

FYI the following idioms describe how to select a range of records given a specific pattern to match:

a) Print all records from some pattern:

awk '/pattern/{f=1}f' file

b) Print all records after some pattern:

awk 'f;/pattern/{f=1}' file

c) Print the Nth record after some pattern:

awk 'c&&!--c;/pattern/{c=N}' file

d) Print every record except the Nth record after some pattern:

awk 'c&&!--c{next}/pattern/{c=N}1' file

e) Print the N records after some pattern:

awk 'c&&c--;/pattern/{c=N}' file

f) Print every record except the N records after some pattern:

awk 'c&&c--{next}/pattern/{c=N}1' file

g) Print the N records from some pattern:

awk '/pattern/{c=N}c&&c--' file

I changed the variable name from "f" for "found" to "c" for "count" where appropriate as that's more expressive of what the variable actually IS.

Source:  http://hacktux.com/bash/bashrc/bash_profile

An Explanation of .bashrc and .bash_profile

---

Both the ~/.bashrc and ~/.bash_profile are scripts that might be executed when bash is invoked. The ~/.bashrc file gets executed when you run bash using an interactive shell that is not a login shell. The ~/.bash_profile only gets executed during a login shell. What does this all mean? The paragraphs below explains interactive shells, login shells, .bashrc, .bash_profile and other bash scripts that are executed during login.

Login Shells (.bash_profile)

A login shell is a bash shell that is started with - or --login. The following are examples that will invoke a login shell.

sudo su -
bash --login
ssh user@host

When BASH is invoked as a login shell, the following files are executed in the displayed order.

1. /etc/profile
2. ~/.bash_profile
3. ~/.bash_login
4. ~/.profile

Although ~/.bashrc is not listed here, most default ~/.bash_profile scripts run ~/.bashrc.

Purely Interactive Shells (.bashrc)

Interactive shells are those not invoked with -c and whose standard input and output are connected to a terminal. Interactive shells do not need to be login shells. Here are some examples that will evoke an interactive shell that is not a login shell.

sudo su
bash
ssh user@host /path/to/command

In this case of an interactive but non-login shell, only ~/.bashrc is executed. In most cases, the default ~/.bashrc script executes the system's /etc/bashrc.

Be warned that you should never echo output to the screen in a ~/.bashrc file. Otherwise, commands like 'ssh user@host /path/to/command' will echo output unrelated to the command called.

Non-interactive shells

Non-interactive shells do not automatically execute any scripts like ~/.bashrc or ~/.bash_profile. Here are some examples of non-interactive shells.

su user -c /path/to/command
bash -c /path/to/command