| Current File : //proc/thread-self/root/usr/share/doc/bpfcc-tools/examples/doc/syscount_example.txt |
Demonstrations of syscount, the Linux/eBPF version.
syscount summarizes syscall counts across the system or a specific process,
with optional latency information. It is very useful for general workload
characterization, for example:
# syscount
Tracing syscalls, printing top 10... Ctrl+C to quit.
[09:39:04]
SYSCALL COUNT
write 10739
read 10584
wait4 1460
nanosleep 1457
select 795
rt_sigprocmask 689
clock_gettime 653
rt_sigaction 128
futex 86
ioctl 83
^C
These are the top 10 entries; you can get more by using the -T switch. Here,
the output indicates that the write and read syscalls were very common, followed
immediately by wait4, nanosleep, and so on. By default, syscount counts across
the entire system, but we can point it to a specific process of interest:
# syscount -p $(pidof dd)
Tracing syscalls, printing top 10... Ctrl+C to quit.
[09:40:21]
SYSCALL COUNT
read 7878397
write 7878397
^C
Indeed, dd's workload is a bit easier to characterize. Occasionally, the count
of syscalls is not enough, and you'd also want an aggregate latency:
# syscount -L
Tracing syscalls, printing top 10... Ctrl+C to quit.
[09:41:32]
SYSCALL COUNT TIME (us)
select 16 3415860.022
nanosleep 291 12038.707
ftruncate 1 122.939
write 4 63.389
stat 1 23.431
fstat 1 5.088
[unknown: 321] 32 4.965
timerfd_settime 1 4.830
ioctl 3 4.802
kill 1 4.342
^C
The select and nanosleep calls are responsible for a lot of time, but remember
these are blocking calls. This output was taken from a mostly idle system. Note
the "unknown" entry -- syscall 321 is the bpf() syscall, which is not in the
table used by this tool (borrowed from strace sources).
Another direction would be to understand which processes are making a lot of
syscalls, thus responsible for a lot of activity. This is what the -P switch
does:
# syscount -P
Tracing syscalls, printing top 10... Ctrl+C to quit.
[09:58:13]
PID COMM COUNT
13820 vim 548
30216 sshd 149
29633 bash 72
25188 screen 70
25776 mysqld 30
31285 python 10
529 systemd-udevd 9
1 systemd 8
494 systemd-journal 5
^C
This is again from a mostly idle system over an interval of a few seconds.
Sometimes, you'd only care about failed syscalls -- these are the ones that
might be worth investigating with follow-up tools like opensnoop, execsnoop,
or trace. Use the -x switch for this; the following example also demonstrates
the -i switch, for printing at predefined intervals:
# syscount -x -i 5
Tracing failed syscalls, printing top 10... Ctrl+C to quit.
[09:44:16]
SYSCALL COUNT
futex 13
getxattr 10
stat 8
open 6
wait4 3
access 2
[unknown: 321] 1
[09:44:21]
SYSCALL COUNT
futex 12
getxattr 10
[unknown: 321] 2
wait4 1
access 1
pause 1
^C
Similar to -x/--failures, sometimes you only care about certain syscall
errors like EPERM or ENONET -- these are the ones that might be worth
investigating with follow-up tools like opensnoop, execsnoop, or
trace. Use the -e/--errno switch for this; the following example also
demonstrates the -e switch, for printing ENOENT failures at predefined intervals:
# syscount -e ENOENT -i 5
Tracing syscalls, printing top 10... Ctrl+C to quit.
[13:15:57]
SYSCALL COUNT
stat 4669
open 1951
access 561
lstat 62
openat 42
readlink 8
execve 4
newfstatat 1
[13:16:02]
SYSCALL COUNT
lstat 18506
stat 13087
open 2907
access 412
openat 19
readlink 12
execve 7
connect 6
unlink 1
rmdir 1
^C
Sometimes, you'd only care about a single syscall rather than all syscalls.
Use the --syscall option for this; the following example also demonstrates
the --syscall option, for printing at predefined intervals:
# syscount --syscall stat -i 1
Tracing syscall 'stat'... Ctrl+C to quit.
[12:51:06]
SYSCALL COUNT
stat 310
[12:51:07]
SYSCALL COUNT
stat 316
^C
USAGE:
# syscount -h
usage: syscount.py [-h] [-p PID] [-t TID] [-i INTERVAL] [-d DURATION] [-T TOP]
[-x] [-e ERRNO] [-L] [-m] [-P] [-l] [--syscall SYSCALL]
Summarize syscall counts and latencies.
optional arguments:
-h, --help show this help message and exit
-p PID, --pid PID trace only this pid
-t TID, --tid TID trace only this tid
-c PPID, --ppid PPID trace only child of this pid
-i INTERVAL, --interval INTERVAL
print summary at this interval (seconds)
-d DURATION, --duration DURATION
total duration of trace, in seconds
-T TOP, --top TOP print only the top syscalls by count or latency
-x, --failures trace only failed syscalls (return < 0)
-e ERRNO, --errno ERRNO
trace only syscalls that return this error (numeric or
EPERM, etc.)
-L, --latency collect syscall latency
-m, --milliseconds display latency in milliseconds (default:
microseconds)
-P, --process count by process and not by syscall
-l, --list print list of recognized syscalls and exit
--syscall SYSCALL trace this syscall only (use option -l to get all
recognized syscalls)