| Current File : //proc/thread-self/root/usr/share/doc/bpfcc-tools/examples/doc/exitsnoop_example.txt |
Demonstrations of exitsnoop.
This Linux tool traces all process terminations and reason, it
- is implemented using BPF, which requires CAP_SYS_ADMIN and
should therefore be invoked with sudo
- traces sched_process_exit tracepoint in kernel/exit.c
- includes processes by root and all users
- includes processes in containers
- includes processes that become zombie
The following example shows the termination of the 'sleep' and 'bash' commands
when run in a loop that is interrupted with Ctrl-C from the terminal:
# ./exitsnoop.py > exitlog &
[1] 18997
# for((i=65;i<100;i+=5)); do bash -c "sleep 1.$i;exit $i"; done
^C
# fg
./exitsnoop.py > exitlog
^C
# cat exitlog
PCOMM PID PPID TID AGE(s) EXIT_CODE
sleep 19004 19003 19004 1.65 0
bash 19003 17656 19003 1.65 code 65
sleep 19007 19006 19007 1.70 0
bash 19006 17656 19006 1.70 code 70
sleep 19010 19009 19010 1.75 0
bash 19009 17656 19009 1.75 code 75
sleep 19014 19013 19014 0.23 signal 2 (INT)
bash 19013 17656 19013 0.23 signal 2 (INT)
#
The output shows the process/command name (PCOMM), the PID,
the process that will be notified (PPID), the thread (TID), the AGE
of the process with hundredth of a second resolution, and the reason for
the process exit (EXIT_CODE).
A -t option can be used to include a timestamp column, it shows local time
by default. The --utc option shows the time in UTC. The --label
option adds a column indicating the tool that generated the output,
'exit' by default. If other tools follow this format their outputs
can be merged into a single trace with a simple lexical sort
increasing in time order with each line labeled to indicate the event,
e.g. 'exec', 'open', 'exit', etc. Time is displayed with millisecond
resolution. The -x option will show only non-zero exits and fatal
signals, which excludes processes that exit with 0 code:
# ./exitsnoop.py -t --utc -x --label= > exitlog &
[1] 18289
# for((i=65;i<100;i+=5)); do bash -c "sleep 1.$i;exit $i"; done
^C
# fg
./exitsnoop.py -t --utc -x --label= > exitlog
^C
# cat exitlog
TIME-UTC LABEL PCOMM PID PPID TID AGE(s) EXIT_CODE
13:20:22.997 exit bash 18300 17656 18300 1.65 code 65
13:20:24.701 exit bash 18303 17656 18303 1.70 code 70
13:20:26.456 exit bash 18306 17656 18306 1.75 code 75
13:20:28.260 exit bash 18310 17656 18310 1.80 code 80
13:20:30.113 exit bash 18313 17656 18313 1.85 code 85
13:20:31.495 exit sleep 18318 18317 18318 1.38 signal 2 (INT)
13:20:31.495 exit bash 18317 17656 18317 1.38 signal 2 (INT)
#
USAGE message:
# ./exitsnoop.py -h
usage: exitsnoop.py [-h] [-t] [--utc] [-p PID] [--label LABEL] [-x] [--per-thread]
Trace all process termination (exit, fatal signal)
optional arguments:
-h, --help show this help message and exit
-t, --timestamp include timestamp (local time default)
--utc include timestamp in UTC (-t implied)
-p PID, --pid PID trace this PID only
--label LABEL label each line
-x, --failed trace only fails, exclude exit(0)
--per-thread trace per thread termination
examples:
exitsnoop # trace all process termination
exitsnoop -x # trace only fails, exclude exit(0)
exitsnoop -t # include timestamps (local time)
exitsnoop --utc # include timestamps (UTC)
exitsnoop -p 181 # only trace PID 181
exitsnoop --label=exit # label each output line with 'exit'
exitsnoop --per-thread # trace per thread termination
Exit status:
0 EX_OK Success
2 argparse error
70 EX_SOFTWARE syntax error detected by compiler, or
verifier error from kernel
77 EX_NOPERM Need sudo (CAP_SYS_ADMIN) for BPF() system call
About process termination in Linux
----------------------------------
A program/process on Linux terminates normally
- by explicitly invoking the exit( int ) system call
- in C/C++ by returning an int from main(),
...which is then used as the value for exit()
- by reaching the end of main() without a return
...which is equivalent to return 0 (C99 and C++)
Notes:
- Linux keeps only the least significant eight bits of the exit value
- an exit value of 0 means success
- an exit value of 1-255 means an error
A process terminates abnormally if it
- receives a signal which is not ignored or blocked and has no handler
... the default action is to terminate with optional core dump
- is selected by the kernel's "Out of Memory Killer",
equivalent to being sent SIGKILL (9), which cannot be ignored or blocked
Notes:
- any signal can be sent asynchronously via the kill() system call
- synchronous signals are the result of the CPU detecting
a fault or trap during execution of the program, a kernel handler
is dispatched which determines the cause and the corresponding
signal, examples are
- attempting to fetch data or instructions at invalid or
privileged addresses,
- attempting to divide by zero, unmasked floating point exceptions
- hitting a breakpoint
Linux keeps process termination information in 'exit_code', an int
within struct 'task_struct' defined in <linux/sched.c>
- if the process terminated normally:
- the exit value is in bits 15:8
- the least significant 8 bits of exit_code are zero (bits 7:0)
- if the process terminates abnormally:
- the signal number (>= 1) is in bits 6:0
- bit 7 indicates a 'core dump' action, whether a core dump was
actually done depends on ulimit.
Success is indicated with an exit value of zero.
The meaning of a non zero exit value depends on the program.
Some programs document their exit values and their meaning.
This script uses exit values as defined in <include/sysexits.h>
References:
https://github.com/torvalds/linux/blob/master/kernel/exit.c
https://github.com/torvalds/linux/blob/master/arch/x86/include/uapi/asm/signal.h
https://code.woboq.org/userspace/glibc/misc/sysexits.h.html