Changes

Remove namespace code from libbpf to fix USDT

This removes the namespace code from libbpf, as they are no longer necessary
since #2324 was merged, and have caused regressions on older Kernels that do
not use the new API for creating probes. This also deletes the dead code for
namespace handling in libbpf, as this was the last use of it.

This also introduces regression tests to ensure that processes in containers
can be USDT probed, by adding tests that unshare the mount and process
namespaces.
 (commit: c6a3f02)

fix a bug in networking/simulation.py

test test_brb.py failed on fc31 with the following error messages:
  Traceback (most recent call last):
    File "./test_brb.py", line 162, in test_brb
      disable_ipv6=True)
    File "/home/yhs/work/bcc/tests/python/simulation.py", line 94, in _create_ns
      disable_ipv6)
    File "/home/yhs/work/bcc/tests/python/simulation.py", line 68, in _ns_add_ifc
      ns_ipdb.interfaces.lo.up().commit()
    File "/usr/local/lib/python3.7/site-packages/pyroute2/ipdb/interfaces.py", line 1078, in commit
      raise error
    File "/usr/local/lib/python3.7/site-packages/pyroute2/ipdb/interfaces.py", line 859, in commit
    transaction.wait_all_targets()
    File "/usr/local/lib/python3.7/site-packages/pyroute2/ipdb/transactional.py", line 507, in wait_all_targets
      raise CommitException('target %s is not set' % key)
  pyroute2.ipdb.exceptions.CommitException: target state is not set

The reason is in networking/simulation.py, if the interface 'lo'
inside the namespace is already up and it is tried to commit
to 'up' state again, the pyroute2 library will cause an exception.

The fix is to avoid to 'up' interface 'lo' again if the interface
is already in 'up' state.

Signed-off-by: Yonghong Song <yhs@fb.com>
 (commit: dbfb188)

fix line TypeError

./profile.py -adf -p `pgrep -n main` 5
Traceback (most recent call last):
  File "./profile.py", line 342, in <module>
    print("%s %d" % (b";".join(line).decode('utf-8', 'replace'), v.value))
TypeError: sequence item 5: expected a bytes-like object, str found
 (commit: 0cafe55)

sync to latest libbpf repo

Sync to latest libbpf repo. New helper bpf_jiffies64
and new program type BPF_PROG_TYPE_EXT is added.

Signed-off-by: Yonghong Song <yhs@fb.com>
 (commit: 83b67c2)

Run BCC test suite with github actions

With this commit, pushes to branches can trigger tests directly on
Github Actions.

Here it will be able to test against kernel 4.14 and 5.0.0, which correspond
to the latest Ubuntu LTS kernel releases.

Tests are run for both Debug and Release modes.

Tests run inside docker, in an ubuntu 19.04 container base.

For the github workflow:

- The test_libbcc suite is run first, to potentially fail fast on these faster
unit tests.
- The Python test suite is run

Some of these tests are allowed to fail, but the failure is still reported:

- In catch2 tests, using the [!mayfail] tag, where this will be displayed in
the test summary
- In Python unittests, using the `@mayFail("Reason")` decorator, which is
introduce for consistency with catch2. These will report the reason for the
failure, and log this to an artifact of the test.
 (commit: a47c44f)

Make inode-only matching default, reverse USDT kludge
 (commit: b090f5f)

remove dead link

solves #2722
 (commit: 215b724)

tools/klockstat.py: Do not display symbol twice for stack

Currently we display the caller symbol in stack,
which ends up with output below when we enable stack:

                                  Caller   Avg Hold  Count   Max hold Total hold
                  b'flush_to_ldisc+0x22'      56112      2     103914     112225
                  b'flush_to_ldisc+0x22'
               b'process_one_work+0x1b0'
                   b'worker_thread+0x50'
                         b'kthread+0xfb'

Skipping one more symbol in stack to fix that:

                                  Caller   Avg Hold  Count   Max hold Total hold
                  b'flush_to_ldisc+0x22'       1893      2       2765       3787
               b'process_one_work+0x1b0'
                   b'worker_thread+0x50'
                         b'kthread+0xfb'

Signed-off-by: Jiri Olsa <jolsa@kernel.org>
 (commit: 90b2382)

fix compilation error due to latest llvm change

llvm 11.0 required explicit conversion from StringRef to std::string.
The patch is
  https://github.com/llvm/llvm-project/commit/777180a32b61070a10dd330b4f038bf24e916af1

This patch made a compatible change which works for old llvm
as well.

Signed-off-by: Yonghong Song <yhs@fb.com>
 (commit: 0e63c5c)

Add XSKMAP support (#2730)

* add XSKMAP support
* enable lookup for XSKMAP available from kernel 5.3
* add section for XSKMAP in reference guide
 (commit: 46965c6)

bcc.BPF.cleanup(): Ensure self.funcs items get deleted during cleanup

Since commit 115b959d86
("Fix a file descriptor leak when module is deleted (#2530)"), we
observe the following exceptions during python exit:

  Error in atexit._run_exitfuncs:
  Traceback (most recent call last):
    File "/usr/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
      func(*targs, **kargs)
    File "/usr/lib/python2.7/dist-packages/bcc/__init__.py", line 1366, in cleanup
      os.close(fn.fd)
  OSError: [Errno 9] Bad file descriptor

which occurs for python programs issuing a call to 'cleanup()', or using
the 'with bcc.BPF(...)' code pattern.

BPF's 'cleanup' is registered to be invoked atexit. Alas, commit
115b959d86 introduced an 'os.close(fn.fd)' call for each func loaded (to
prevent accidental FD leakage).

Problem is, that the 'self.funcs' dict entries are NOT deleted, making
subsequent calls to 'cleanup' to attempt closing the same 'fn.fd' again
and again.

It is expected from 'cleanup' to operate correctly when called
repeatedly; Therefore, it should "nullify" references to closed
resources.

Fix, by deleting the reference to each unloaded function from the
'self.func' dictionary.

Fixes: 115b959d86 ("Fix a file descriptor leak when module is deleted (#2530)")
Reported-by: Dana Rubin <drubin@metanetworks.com>
Signed-off-by: Shmulik Ladkani <shmulik.ladkani@gmail.com>
 (commit: f727a00)

define CC_USING_FENTRY if CONFIG_FUNCTION_TRACER is defined

Fix issue #2734.

In 4.18 and later, when CONFIG_FUNCTION_TRACER is defined, kernel Makefile adds
-DCC_USING_FENTRY. Let do the same for bpf programs.

Signed-off-by: Yonghong Song <yhs@fb.com>
 (commit: 6878949)

snap: update and cleanup snapcraft.yaml

A couple of issues fixed:

- put snapcraft.yaml in the snap directory so it is
  picked up by snapcraft when run from the project root.
- use snap/local for local snapcraft assets.
- setup.py.in should not add root for sdist targets
- cleanup snap/README.md.
- re-arrange parts in snapcraft.yaml and move to python3.

Signed-off-by: Sergio Schvezov <sergio.schvezov@canonical.com>
 (commit: 685ec23)

prevent rewriting for array element type

Fix issue #2352.
Corresponding llvm bug: https://bugs.llvm.org/show_bug.cgi?id=41918.

If the element type is array type, the rewriter may generate code like
  addr = ({ typeof(__u8 [16]) _val; __builtin_memset(&_val, 0, sizeof(_val));
            bpf_probe_read(&_val, sizeof(_val), (u64)&daddr->s6_addr); _val; })
for something like
  addr = daddr->s6_addr;
where s6_addr is an array.

The above code has an issue then as addr is pointing to some data
which is out of scope, which meaning compiler is free to use the space.

Let us disable such transformation until we find a good solution.

Signed-off-by: Yonghong Song <yhs@fb.com>
 (commit: a4834a6)

Do not rewrite array subscripts if invalid sourceloc range

Fix #2739.

The issue exposed an issue that to rewrite
  CODE1: __u8 byte = daddr->s6_addr[4];
will segfault and to rewrite
  CODE2: __u8 byte = (daddr->s6_addr)[4];
will be okay.

For CODE1, the clang did not give enough information to find the text
which contains the left bracket "[", given base "daddr->s6_addr"
and subscript "4". For CODE2, the clang is able to get the information
successfuly.

I think if we really go inside the base "daddr->s6_addr" and gets to
its member field "s6_addr", we can find the needed information
for the text range containing "[". Let us fix the segfault first
and if really desirable, we can try to enhance later for CODE1 patterns.

Signed-off-by: Yonghong Song <yhs@fb.com>
 (commit: f149ca5)

Fix offwaketime regression introduced by #ac00ac5
 (commit: 0a7da74)

support sockmap/sockhash/cgroup_local_storage maps

This patch supports sockmap, sockhash and cgroup_local_storage
maps. Two C++ APIs, attach_func and detach_func are also
added to allow to attach program to cgroups. So this makes
using C++ APIs for cgroup based networking applications
easier to integrate with bpf programs.

The unit testing is rough as it needs some work to set up
cgroups and establish TCP connections to really test the
result of map operations. But still all supported map
operations in kernel and in C++ APIs are tested syntacically.

Signed-off-by: Yonghong Song <yhs@fb.com>
 (commit: 6cacc41)

libbcc debian package is architecture-dependent
 (commit: 397ff37)

correctly describe installation situation for Ubuntu
 (commit: 0f92c84)

snapcraft: add missing libfl-dev dependency

Also need libfl-dev to build on other non-x86 targets

Signed-off-by: Colin Ian King <colin.king@canonical.com>
 (commit: c82aa51)

libbpf tools: check in bpftool for use by libbpf-based tool build

Check in bpftool binary to be used for BPF skeleton generation for
libbpf-based tools.

Signed-off-by: Andrii Nakryiko <andriin@fb.com>
 (commit: 4281c62)

tools: add vmlinux.h header with all kernel types

Check in vmlinux.h generated from Linux 5.5 version with default config. This
is used from libbpf-based tools.

Signed-off-by: Andrii Nakryiko <andriin@fb.com>
 (commit: 7fcb335)

Allow specifying clang base directory
 (commit: 2d099cd)

tools: add libbpf-based tools to BCC w/ runqslower as first converted tool

Add runqslower as a first tool converted from BCC to libbpf, utilizing BPF
CO-RE (Compile Once - Run Everywhere) approach and BPF skeleton for
interfacing with BPF programs. Current set up is Makefile based and is set up
in such a way as to enable easy addition for more tools, based on a simple and
convenient naming pattern. General build infrastructure takes case of BPF
skeleton generation, tracking dependencies, clean up, etc.

Signed-off-by: Andrii Nakryiko <andriin@fb.com>
 (commit: e1496e1)

fix pid filter bug

1. Wirte a simple go, just print pid and get some website.
func main() {
	 pid := os.Getpid()
         fmt.Println(pid)
         response, err := http.Get("http://www.baidu.com")
        ....
}

./main
3581

2. But when run `tcpconnlat`, we just go tid 3585, (not pid 3581)
#./tcpconnlat
PID    COMM         IP SADDR            DADDR            DPORT LAT(ms)
3585   main         4  10.0.2.15        180.101.49.11    80    60.68

3. So run `./tcpconnlat -p 3581` not work
I have tested this situation under kernel 3.10 and 5.0.9.
 (commit: a28ad05)

tools/stackcount: Fix address resolution for user-space stack

This patch fixes issue #2748. The bug was that address-to-symbol resolution
didn't work for user-space stacks without the `-P` (per-pid) flag when tracing
single, isolated processes. The current documentation of the `-P` option
indicates that it's used to "display stacks separately for each process",
and this doesn't match with the intended usage.

This patch has two parts:

 - Fix `tools/stackcount.py` to explicitly set perpid to True if `-p <pid>`
   is used
 - Remove the `-P` option from the example of tracing single, isolated process
   in `tools/stackcount_example.txt`, since the usage of the option can
   be confusing (and unnecessary after the current change)
 (commit: 9b6a0e3)

Update dddos.py

Fix a few more typos
 (commit: 1332e68)

bindsnoop BCC tool (#2749)

bindsnoop BCC utility

bindsnoop tool traces the kernel function performing socket binding and
print socket options set before the system call invocation that might
impact bind behavior and bound interface
 (commit: 8dd4b5a)

sync to latest libbpf

sync to latest libbpf (libbpf release v0.0.7) to
prepare bcc release.

Signed-off-by: Yonghong Song <yhs@fb.com>
 (commit: 51be481)

tools/profile: fix kernel delimiter when folding (#2758)

retain delimiter with folding enabled
 (commit: 29aa619)

tools: fix confusion between execsnoop and opensnoop

The execsnoop documentation mispelled 'execsnoop' as 'opensnoop'.

Bug introduced by https://github.com/iovisor/bcc/pull/2651
 (commit: b26b429)

debian changelog for v0.13.0 tag

  * Support for kernel up to 5.5
  * bindsnoop tool to track tcp/udp bind information
  * added compile-once run-everywhere based libbpf-tools, currently
    only runqslower is implemented.
  * new map support: sockhash, sockmap, sk_storage, cgroup_storage
  * enable to run github actions on the diff
  * cgroupmap based cgroup filtering for opensnoop, execsnoop and bindsnoop.
  * lots of bug fixes.

Signed-off-by: Yonghong Song <yhs@fb.com>
 (commit: 9422274)

tools: add option --cgroupmap to tcp tools

List of tcp tools updated: tcpaccept, tcpconnect, tcptracer
 (commit: 1ce868f)

man pages: add documentation about --cgroupmap
 (commit: 5e3f9e4)

c++ api: add wrapper for raw tracepoints
 (commit: 5681ea9)

cpp examples: use a raw tracepoint for RandomRead
 (commit: 2a2e729)

libbpf-tools: add links to BPF CO-RE posts

Add links to BPF CO-RE blog posts, explainint what it is, how to use it, and
giving practical recommendations on how to convert BCC BPF program to
libbpf-based one.

Signed-off-by: Andrii Nakryiko <andriin@fb.com>
 (commit: 7ad1763)

tools: execsnoop add -U and -u flags

Add flags to display UID and filter by UID
 (commit: c949f61)

tools/oomkill: output PID instead of thread ID (user's view)
 (commit: 9edbae3)

Fix the python part of the http-parse-complete example

The example wasn't working on my machine, just like in issue #2169
 (commit: 6f3c33b)

Fix a table of contents of maps

Signed-off-by: Masanori Misono <m.misono760@gmail.com>
 (commit: 08de253)

Fix bpf linking if specified via LIBBPF_LIBRARIES variable

We need to use LIBBPF_LIBRARIES variable instead of just using bpf
for libbpf linking, so it user specified library gets linked in.

Signed-off-by: Jiri Olsa <jolsa@kernel.org>
 (commit: 18da40c)

Unite libbpf includes

Currently we include headers from local libbpf subpackage,
which does not work if user specify LIBBPF_INCLUDE_DIR.

Adding HAVE_EXTERNAL_LIBBPF macro, that gets defined when
user specifies libbpf header path. This macro is checked
in bcc_libbpf_inc.h and proper files are included.

Using bcc_libbpf_inc.h in place of libbpf includes.

Signed-off-by: Jiri Olsa <jolsa@kernel.org>
 (commit: 1e7862f)

Do not initialize kern_version for TRACING/EXT programs

The TRACING/EXT programs use attach_btf_id and attach_prog_fd
fields from struct bpf_load_program_attr.

The attach_prog_fd field shares space with kern_version,
so by setting kern_version unconditionally we also set
attach_prog_fd to bogus value and kernel fails the load
because it tries to look it up.

Setting kern_version only for programs other than TRACING/EXT
type.

Signed-off-by: Jiri Olsa <jolsa@kernel.org>
 (commit: 550706a)

Introduce {attach|detach}_kfunc API

Kernel added new probe called trampoline allowing to probe
almost any kernel function when BTF info is available in
the system.

Adding the interface to define trampoline function for given
kernel function via BPF_PROG macro, like:

  KFUNC_PROBE(do_sys_open, int dfd, const char *filename, int flags, int mode)
  {
    ...
  }

which defines trampoline function with the 'kfunc__do_sys_open'
name, that instruments do_sys_open kernel function before the
function is executed.

or:

  KRETFUNC_PROBE(do_sys_open, int dfd, const char *filename, int flags, int mode, int ret)
  {
    ...
  }

which defines trampoline function with the 'kfunc__do_sys_open'
name, that instruments do_sys_open kernel function after the
function is executed.

The main benefit is really lower overhead for trampolines (please
see following commit for klockstat.py with perf comparison).

Another benefit is the ability of kretfunc probe to access
function arguments, so some tools might need only one program
instead of entry/exit ones (please see following commit for
opensnoop.py changes).

Currently the interface does not allow to define function
of different name than:
  kfunc__<function_name> or kretfunc__<function_name>

which is sufficient by now, and can be easily changed in future
if needed.

Signed-off-by: Jiri Olsa <jolsa@kernel.org>
 (commit: 572478b)

Add support_kfunc function to BPF object

Adding support_kfunc function to BPF object to return
state of kfunc trampolines support.

Signed-off-by: Jiri Olsa <jolsa@kernel.org>
 (commit: 1ad2656)

Support kfunc in opensnoop.py

Adding kfunc return trampoline probe if available instead of
kprobe/kretprobe probes.

The return trampoline has access to function entry arguments,
so we are good with just single eBPF program.

The kfunc trampolines are also faster - less intrusive.

Below are stats for compiling linux kernel while running
opensnoop.py on the background for kprobes and kfuncs.

Without opensnoop.py:

 Performance counter stats for 'system wide':

   849,741,782,765      cycles:k

     162.235646336 seconds time elapsed

With opensnoop.py using kprobes:

 Performance counter stats for 'system wide':

   941,615,199,769      cycles:k

     164.355032879 seconds time elapsed

With opensnoop.py using trampolines:

 Performance counter stats for 'system wide':

   913,437,005,488      cycles:k

     163.742914795 seconds time elapsed

Signed-off-by: Jiri Olsa <jolsa@kernel.org>
 (commit: c347fe6)

Support kfunc in klockstat.py

Adding kfunc return trampoline probe if available instead of
kprobe/kretprobe probes.

The kfunc trampolines are faster - less intrusive. The benchmark
without klockstat.py script on background:

	$ perf bench sched messaging -l 50000
	# Running 'sched/messaging' benchmark:
	# 20 sender and receiver processes per group
	# 10 groups == 400 processes run

	     Total time: 18.571 [sec]

With kprobe tracing:
	$ perf bench sched messaging -l 50000
	# Running 'sched/messaging' benchmark:
	# 20 sender and receiver processes per group
	# 10 groups == 400 processes run

	     Total time: 183.395 [sec]

With kfunc tracing:
	$ perf bench sched messaging -l 50000
	# Running 'sched/messaging' benchmark:
	# 20 sender and receiver processes per group
	# 10 groups == 400 processes run

	     Total time: 39.773 [sec]

Signed-off-by: Jiri Olsa <jolsa@kernel.org>
 (commit: da7cac7)

Support kfunc in vfsstat.py

Adding kfunc return trampoline probe if available instead of
kprobe/kretprobe probes.

The kfunc trampolines are faster - less intrusive.

Below are stats for running perf bench sched pipe benchamark while
running vfsstat.py on the background for kprobes and kfuncs.

With kprobes:
         Performance counter stats for './perf bench sched pipe -l 5000000' (3 runs):

           112,520,853,574      cycles:k

                    48.674 +- 0.672 seconds time elapsed  ( +-  1.38% )

With kfuncs:
         Performance counter stats for './perf bench sched pipe -l 5000000' (3 runs):

           106,304,165,424      cycles:k

                    46.820 +- 0.197 seconds time elapsed  ( +-  0.42% )

Signed-off-by: Jiri Olsa <jolsa@kernel.org>
 (commit: 2fa54c0)

Update StatusTuple to use enum Code
 (commit: 6de96c7)

doc: add python3 doc to "Ubuntu - Source" section in INSTALL.md
 (commit: 440016f)

usdt.py: improve error messags for enable_probe()
 (commit: c46c7b3)

libbpf-tools: update bpftool

Newer bpftool generates vmlinux.h with header guard `#define`, which plays
nicely with bpf_tracing.h header and PT_REGS_PARM macros.

Signed-off-by: Andrii Nakryiko <andriin@fb.com>
 (commit: 17ff392)

libbpf-tools: covert BCC drsnoop to BPF CO-RE version

Signed-off-by: Wenbo Zhang <ethercflow@gmail.com>
 (commit: c421970)

usdt.py: print errors to stderr in enable_probe_or_bail()
 (commit: aeea0e9)

usdt: fix parsing sp register in arguments on AArch64

One of the USDT probes for OpenJDK on AArch64 has an argument as an
offset from the stack pointer register like "8@[sp, 112]". This causes
the argument parser to fail:

  Parse error:
      8@x22 8@x20 8@x23 8@x0 8@x26 8@x27 8@[sp, 112] 8@[sp, 120]
  ------------------------------------------^

The error message then repeats forever.

Changed ArgumentParser_aarch64::parse_register so it accepts either
"xNN" or "sp" and outputs the register name rather than the register
number. The stack pointer is in a separate field `sp` in `struct
pt_regs` rather than in the `regs[]` array [1].

Note that the parser currently accepts "x31" and converts that into a
reference to `regs[31]' but that array only has 31 elements. Made x31 an
alias for `sp` to avoid undefined behaviour from reading past the end of
the array.

[1]: https://elixir.bootlin.com/linux/latest/source/arch/arm64/include/asm/ptrace.h#L160

Change-Id: I88b6ff741914b5d06ad5798a55bd21ea03f69825
Signed-off-by: Nick Gasson <nick.gasson@arm.com>
 (commit: 5011f99)

Replace StatusTuple(0) with StatusTuple::OK()

With the following shell command:
$ find src -type f -regex '.*\.\(h\|cc\)' -exec sed -i 's/StatusTuple(0)/StatusTuple::OK()/' {} \;
 (commit: e508059)

print error messages to stderr and exit(1) instead of exit(0) for fatal errors
 (commit: 98c18b6)

GitHub Actions: optionally publish container image

This is optional: this GitHub Action will only attempt to publish the
container image if the GitHub repository has been configured with GitHub
secrets (e.g. https://github.com/iovisor/bcc/settings/secrets).

The GitHub secrets can be configured as follows:
- DOCKER_PUBLISH = 1
- DOCKER_IMAGE = docker.io/myorg/bcc
- DOCKER_USERNAME = username
- DOCKER_PASSWORD = password

This is intended to make it easy for anyone to fork iovisor/bcc on
GitHub and publish custom container images with bcc.
 (commit: cf20a49)

libbpf-tools: optimize ksyms cache

Re-implement internals of ksyms cache to minimize memory overhead and
allocations. Addr is also changed to be unsigned long.

fscanf() can be further optimized into manual parsing, but it's low enough
overhead right now that I felt like readibility is more important.

Benchmarking ksyms loading/parsing parts:

BEFORE:
$ /usr/bin/time ./test
0.03user 0.04system 0:00.08elapsed 98%CPU (0avgtext+0avgdata 6968maxresident)k
0inputs+0outputs (0major+1512minor)pagefaults 0swaps

AFTER:
$ /usr/bin/time ./test
0.02user 0.03system 0:00.06elapsed 100%CPU (0avgtext+0avgdata 9508maxresident)k
  0inputs+0outputs (0major+2106minor)pagefaults 0swaps

RSS goes down from 9.5MB to <7MB, while CPU time went up about 20ms.

Signed-off-by: Andrii Nakryiko <andriin@fb.com>
 (commit: b7541d0)

python binding: allow fully-specified probe names for enable_probe()
 (commit: 37cd183)

trace.py: support an optional provider name for the probe spec
 (commit: 2162516)

softirqs: Combined CPU as part of the key is necessary to avoid amiss value.

In the environment of massive software interrupts.
<idle>-0     [003] ..s1   106.421020: softirq_entry: vec=6 [action=TASKLET]
<idle>-0     [000] ..s1   106.421063: softirq_entry: vec=3 [action=NET_RX]
<idle>-0     [003] ..s1   106.421083: softirq_exit: vec=6 [action=TASKLET]

Follow the above ftrace logs, we know the correct vec-6 start timestamp is replaced with incorrect vec-3.
Because PID is idle-0. It will produce the wrong result after calculating delta.
 (commit: 9465f8c)

tools: add option --cgroupmap to capable.py

Documentation (man page and example text) updated.
 (commit: 15e998d)

tools: add option --unique to capable.py

Some processes can do a lot of security capability checks, generating a
lot of ouput. In this case, the --unique option is useful to only print
once the same set of capability, pid (or cgroup if --cgroupmap is used)
and kernel/user stacks (if -K or -U are used).

  # ./capable.py -K -U --unique

Documentation (man page and example text) updated.
 (commit: 9d7feee)

Reduce container image size with multi-stage builds

Before this patch, Dockerfile.ubuntu was generating an oversized image.

                       Uncompressed size      Compressed size
Before this patch:     1.3GB                  390MB
After this patch:      250MB                  110MB
 (commit: c9cfb52)

libbpf-tools: add CO-RE opensnoop (#2778)

* libbpf-tools: add CO-RE opensnoop

* libbpf-tools/opensnoop: feedback
 (commit: 5e123df)

tools: add option --cgroupmap to tcptop
 (commit: 7d62656)

libbpf-tools: update vmlinux.h with BPF helper flags generated from UAPI

Update vmlinux.h to a version generated from same v5.5 tag and default config
with cherry-picked 1aae4bdd7879 ("bpf: Switch BPF UAPI #define constants used
from BPF program side to enums") on top of it. This adds lots of BPF helper
flags often useful from BPF program side.

Signed-off-by: Andrii Nakryiko <andriin@fb.com>
 (commit: 8b6a7db)

libbpf: sync to latest version

Sync libbpf to latest revision. It brings latest BPF headers, among other
things.

Signed-off-by: Andrii Nakryiko <andriin@fb.com>
 (commit: 77d60b1)

libbpf-tools: small clean ups across all tools

Remove BPF_F_CURRENT_CPU definitions, which are now provided by vmlinux.h
after 1aae4bdd7879 ("bpf: Switch BPF UAPI #define constants used from BPF
program side to enums") commit in kernel.

Fix potential uninitialized read warning in opensnoop.

Also add opensnoop to .gitignore.

Signed-off-by: Andrii Nakryiko <andriin@fb.com>
 (commit: 454b138)

libbpf-tools: determine target host architecture

BPF code is compiled with -target bpf, but for PT_REGS_PARM macro (and by
induction for BPF_KPROBE/BPF_KRETPROBE macros as well), it's important to know
what's the target host original architecture was, to use correct definition of
struct pt_regs. Determine that based on output of `uname -m` (taking into
account that both x86_64 and x86 are defined as x86 internally for kernel).

Signed-off-by: Andrii Nakryiko <andriin@fb.com>
 (commit: 2b5fcc6)

Backport tcpstates to kernels < 4.15

The tracepoint inet_sock_set_state only exists in kernels 4.15.
Backported the bpf tracepoint to use kprobes on older kernels.
 (commit: d0ec8a2)

libbpf-tools: adjust Kconfig and re-build vmlinux.h

Default v5.5 kernel config doesn't have most of BPF-related functionality
enabled, which leads to vmlinux.h not containing a lot of useful constants.
This patch contains re-generated vmlinux.h from kernel built with default
config plus minimal changes to enable most (all?) BPF-relevant parts of
kernel. Here's a list of added options:

CONFIG_BPF_EVENTS=y
CONFIG_BPFILTER_UMH=m
CONFIG_BPFILTER=y
CONFIG_BPF_JIT=y
CONFIG_BPF_KPROBE_OVERRIDE=y
CONFIG_BPF_STREAM_PARSER=y
CONFIG_BPF_SYSCALL=y
CONFIG_CC_HAS_ASM_INLINE=y
CONFIG_CC_HAS_KASAN_GENERIC=y
CONFIG_CC_HAS_SANCOV_TRACE_PC=y
CONFIG_CGROUP_BPF=y
CONFIG_GCC_VERSION=70300
CONFIG_IPV6_MULTIPLE_TABLES=y
CONFIG_IPV6_SEG6_BPF=y
CONFIG_IPV6_SEG6_LWTUNNEL=y
CONFIG_LIBCRC32C=y
CONFIG_LWTUNNEL_BPF=y
CONFIG_LWTUNNEL=y
CONFIG_NET_ACT_BPF=y
CONFIG_NET_CLS_BPF=y
CONFIG_NETFILTER_ADVANCED=y
CONFIG_NETFILTER_XT_MATCH_BPF=y
CONFIG_NET_SOCK_MSG=y
CONFIG_NF_CT_PROTO_DCCP=y
CONFIG_NF_CT_PROTO_SCTP=y
CONFIG_NF_CT_PROTO_UDPLITE=y
CONFIG_SOCK_CGROUP_DATA=y
CONFIG_STREAM_PARSER=y
CONFIG_XDP_SOCKETS_DIAG=y
CONFIG_XDP_SOCKETS=y

To make this vmlinux.h generation process easier for future adjustments (e.g.,
if some of the tools would need types that default config compiles out), check
in used Kconfig along the vmlinux.h itself.

Signed-off-by: Andrii Nakryiko <andriin@fb.com>
 (commit: cd43be4)

Factor out ebpf::parse_tracepoint function

And moving it to common.cc in order to be able to make
automated tests for it. Following patches are adding
automated test for this function and it seems too much
to link in all the clang/llvm stuff to the test binary
just for single function test.

Adding ebpf::parse_tracepoint that takes istream of the
tracepoint format data and returns tracepoint struct
as std::string.

No functional change is intended or expected.

Signed-off-by: Jiri Olsa <jolsa@kernel.org>
 (commit: 7fd2fa6)

Add test for ebpf::parse_tracepoint function

The test prepares tracepoint format file and run
ebpf::parse_tracepoint on it. Then it compares
expected struct output with actual function result.

Signed-off-by: Jiri Olsa <jolsa@kernel.org>
 (commit: b9099c5)

Strengthen tracepoint format parsing

There's issue in current RHEL real time kernel with tracepoint format,
which makes bcc-tools to return wrong data.

Two new 'common_' fields were added and it causes 2 issues for tracepoint
format parser.

First issue
  - is the gap between common fields and other fields, which is not
    picked up by the parser, so the resulted struct is not aligned with
    the data.

Second issue
  - is the fact that current parser covers common fields with:
      u64 __do_not_use__
    so the new common fields are not accounted for.

    This issue is solved in the following patch. I kept both
    issues and fixes separated to make the change readable.

There's a 'not described gap' in the sched_wakeup's format file and
probably in other formats as well:

Having:
  # cat /sys/kernel/debug/tracing/events/sched/sched_wakeup/format
  name: sched_wakeup
  ID: 310
  format:
          field:unsigned short common_type;       offset:0;       size:2; signed:0;
          field:unsigned char common_flags;       offset:2;       size:1; signed:0;
          field:unsigned char common_preempt_count;       offset:3;       size:1; signed:0;
          field:int common_pid;   offset:4;       size:4; signed:1;
          field:unsigned char common_migrate_disable;     offset:8;       size:1; signed:0;
          field:unsigned char common_preempt_lazy_count;  offset:9;       size:1; signed:0;

          field:char comm[16];    offset:12;      size:16;        signed:1;
          field:pid_t pid;        offset:28;      size:4; signed:1;
          field:int prio; offset:32;      size:4; signed:1;
          field:int success;      offset:36;      size:4; signed:1;
          field:int target_cpu;   offset:40;      size:4; signed:1;

There's "common_preempt_lazy_count" field on offset 9 with size 1:
        common_preempt_lazy_count;  offset:9;       size:1;

and it's followed by "comm" field on offset 12:
        field:char comm[16];    offset:12;      size:16;        signed:1;

which makes 2 bytes gap in between, that might confuse some applications
like bpftrace or bcc-tools library.

The tracepoint parser makes struct out of the field descriptions,
but does not account for such gaps.

I posted patch to fix this [1] in RT kernel, but that might take a while,
and we could easily fix our tracepoint parser to workaround this issue.

Adding code to detect this gaps and add 1 byte __pad_X fields, where X is
the offset number.

[1] https://lore.kernel.org/linux-rt-users/20200221153541.681468-1-jolsa@kernel.org/
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
 (commit: b8423e6)

Add support for new common fields

Current parser covers common fields with:
  u64 __do_not_use__
so the new common fields are not accounted for.

Keeping the 'u64 __do_not_use__' field for backward compatibility
(who knows who's actualy using it) and adding new fields, like:
  char __do_not_use__X

for each byte of extra common fields, where X is the offset of the
field.

With this fix the bcc-tools correctly parses tracepoints on RT kernel
and it's usable again.

Signed-off-by: Jiri Olsa <jolsa@kernel.org>
 (commit: 0a9d6db)

libbpf: update to latest libbpf version

Pull in latest libbpf changes. In particular, containing BPF_KRETPROBE fix.

Signed-off-by: Andrii Nakryiko <andriin@fb.com>
 (commit: 5302614)

adding example/networking/net_monitor.py

Aggregates incoming network traffic
outputs source ip, destination ip, the number of their network traffic, and current time

Co-authored-by: gist-banana <gist.banana@gist.ac.kr>
 (commit: ca1b0af)

Make -DCMAKE_INSTALL_PREFIX=/usr as the default (#2800)

* make -DCMAKE_INSTALL_PREFIX=/usr as the default
* remove -DCMAKE_INSTALL_PEREFIX=/usr from INSTALL.md  as it gets the default
 (commit: 4c1136e)

tools: add option --cgroupmap to profile.py
 (commit: f82ea45)

Do not prepend /proc multiple times in bcc_resolve_symname
 (commit: e1d53a3)

Add regression test for bcc resolve symbol names
 (commit: 1cd3952)

libbpf-tools: add CO-RE xfsslower

Signed-off-by: Wenbo Zhang <ethercflow@gmail.com>
 (commit: c2772a3)

opensnoop: fix --cgroupmap with kfunc

Commit c347fe6c9f75 ("Support kfunc in opensnoop.py") introduces an
alternative probe on do_sys_open() with kfuncs instead of kprobes. This
new implementation is used if the kernel supports it. But it removed the
--cgroupmap filter added in commit b2aa29fa3269 ("tools: cgroup
filtering in execsnoop/opensnoop").

This patch adds the --cgroupmap filter in the kfunc implementation.
 (commit: 510fc74)

chore: include Amazon Linux 2 binary option in INSTALL
 (commit: 35c9940)

tools: fix alignment of ipv6_key_t in tcptop

Fixes the following error on aarch64:

bpf: Failed to load program: Permission denied
; struct sock *sk = ctx->regs[0]; int copied = ctx->regs[1];
0: (79) r8 = *(u64 *)(r1 +8)
...
; struct ipv6_key_t ipv6_key = {.pid = pid};
79: (63) *(u32 *)(r10 -48) = r7
; struct ipv6_key_t ipv6_key = {.pid = pid};
80: (7b) *(u64 *)(r10 +8) = r9
invalid stack off=8 size=8
processed 96 insns (limit 1000000) max_states_per_insn 0 total_states 7 peak_states 7 mark_read 4
 (commit: 6b8a896)

Change the default sort order to Descending
currently with ascending sort the useful information
is commonly beyond the bottom of the terminal window
and it is necessary to reverse the sort manually every execution.

Signed-off-by: Mark Kogan <mkogan@redhat.com>
 (commit: 1b0fe40)

tools/capable: fix compilation error

9d7feeed87c5 ("tools: add option --unique to capable.py") introduced a
compilation error when the --unique flag is passed.

Signed-off-by: Mauricio Vásquez <mauricio@kinvolk.io>
 (commit: 9adda70)

libbpf-tools: add CO-RE filelife

Signed-off-by: Wenbo Zhang <ethercflow@gmail.com>
 (commit: 85f5e7d)

make -lrt linking optional

On Android there is no standalone rt library and relevant symbols are provided
by libc (bionic).
 (commit: e323254)

correcting input to hex using ord funtion
 (commit: 915d6d7)

mallocstacks: validating before stacktrace walk
 (commit: 472f9b3)

http-parse-simple.py: fix wrong start position

Signed-off-by: Jean-Philippe Menil <jpmenil@gmail.com>
 (commit: 4f6ecea)

libbpf-tools: CO-RE opensnoop minor improvements

 * allow passing 0 as valid uid
 * fix type for targ_uid
 * improve uid validation (uid >= 4294967295 now invalid)
 (commit: ab54de2)

[C++] add BPF::enable_usdt_all() and disble_usdt_all() to handle all the USDT at once (#2854)

* add BPF::enable_usdt_all() to enable all the USDT
* add detach_usdt_all() to detach all the USDTs
 (commit: 689f900)

examples/tracing: add nflatency - netfilter hook metrics

This adds a kprobe / kretprobe for chasing down slow netfilter hooks.
Since, often, the slowest hooks are on connection initialization and
teardown, it is bucketed by TCP flags. (rather than dive deep in to
conntrack)

Signed-off-by: Casey Callendrello <cdc@redhat.com>
 (commit: 495a4a3)

add test_lpm_trie.py test

test_lpm_trie.py has been in the repo for quite some time,
but is not included in the unit test.

The issue https://github.com/iovisor/bcc/issues/2860
exposed an issue involved in using together with BTF,
which requires the key type to be a structure.

Let add it as a unit test so we can be sure
lpm_trie map is supported properly by bcc.

Signed-off-by: Yonghong Song <yhs@fb.com>
 (commit: 533391e)

adjust layout string in JIT with llvm11 128bit spec support

To fix the issue (https://github.com/iovisor/bcc/issues/2827)
which exposed a problem with aarch64 frontend and bpf backend
regarding __int128 type, the following llvm patch
   https://reviews.llvm.org/D76587
landed to explicitly support i128 type in bpf layout spec.

Adjust the layout string in bpf_module JIT compilation
accordingly.

Signed-off-by: Yonghong Song <yhs@fb.com>
 (commit: f35dae0)

Translate virtual addresses to binary addresses for shared libs as well.

Quoting https://refspecs.linuxbase.org/elf/elf.pdf:
```
Symbol table entries for different object file types have slightly different
interpretations for the st_value member.
- In relocatable files, st_value holds alignment constraints for a symbol
whose section index is SHN_COMMON.
- In relocatable files, st_value holds a section offset for a defined symbol.
That is, st_value is an offset from the beginning of the section that st_shndx
identifies.
- In executable and shared object files, st_value holds a virtual address. To
make these files' symbols more useful for the dynamic linker, the section offset
(file interpretation) gives way to a virtual address (memory interpretation) for
which the section number is irrelevant.
```

This is a problem in practice as well. I run into this while tracing shared
libraries on Android with bpftrace. Some of them have text sections mmapped
at different offset than file offset which results in probes being placed
at wrong offsets.
 (commit: 007ee49)

examples/tracing: add io_latencies - IO latency distribution monitor

This adds flexible low overhead latency percentile monitoring for block
devices which can be useful in understanding and debugging IO related
issues.

Signed-off-by: Tejun Heo <tj@kernel.org>
 (commit: 5057bbe)

Update pid filter

Update pid filter;  u32 pid = bpf_get_current_pid_tgid() >> 32;
 (commit: 15d7955)

doc: fix wrong references to get_syscall_name
 (commit: 7a38543)

tools: Move io_iolatencies.py example to tools/biolatpcts.py
 (commit: 1515f28)

tools/biolatpcts: Use #!/usr/bin/python instead of python3

Nothing in the script requires python3.
 (commit: 66ab787)

tools/biolatpcts: --pcts handling improvements

* Multiple arguments parsing conflicts with positional argument and can become
  confusing. Use a single comma separated list instead.

* Instead of converting to floats while parsing arguments, keep the strings
  verbatim and use them when outputting results. This allows matching the same
  target percentile string in the output and is helpful when it's consumed by
  other programs.
 (commit: 7214903)

tools/biolatpcts: Accept device name/node for target device selection

MAJ:MIN is cumbersome. Update it to accept DEVNAME and /dev/DEVNAME too.
 (commit: fbf0392)

tools/biolatpcts: Add the example file and man page
 (commit: 1e9c56f)

examples: Add biolatpcts.py example

This is simplified version of tools/biolatpcts.py and demonstrates how to
obtain and calculate latency percentiles.
 (commit: 10519c7)

add attribute readers to class USDT (C++ API)
 (commit: f8f230c)

build: fix clang 10 build

some distros already packaging clang 10 (checked fedora and arch) no longer
ship all the individual libclang*.so component libraries.
Instead, clang from 9.0 onwards provides a new lib, libclang-cpp.so, which
includes everything we need.

Tell cmake to use it if the individual libraries are no longer found.
(Build-wise, if both are present it is more efficient to use the individual
components so keep these first)
 (commit: 1599c2e)

Added kernel recompile guidance for  libbpf CO-RE (#2876)

Added kernel recompile guidance for  libbpf CO-RE
 (commit: c1d3a65)

Add `bcc_version.h` to access LIBBCC_VERSION from C/C++ (#2856)

 add bcc/bcc_version.h to install
 (commit: 6d61622)

Improve ubuntu installation description
 (commit: d3359b2)

fix llvm 11 compilation issues

The llvm CreateCall used in bcc is deprecated in llvm 11:
  https://reviews.llvm.org/D76269
The llvm CreateMemCpy is changed in llvm 11 as well:
  https://reviews.llvm.org/D71473

This caused bcc compilation error.

  /home/yhs/work/bcc/src/cc/frontends/b/codegen_llvm.cc: In member function
     ‘ebpf::StatusTuple ebpf::cc::CodegenLLVM::emit_log(ebpf::cc::Method CallExprNode*)’:
  /home/yhs/work/bcc/src/cc/frontends/b/codegen_llvm.cc:691:39: error: no matching function for call to
     ‘llvm::IRBuilder<>::CreateCall(llvm::Value*&, std::vector<llvm::Value*, std::allocator<llvm::Value*> >&)’
     expr_ = B.CreateCall(printk_fn, args);
                                       ^
  ...

  /home/yhs/work/bcc/src/cc/frontends/b/codegen_llvm.cc: In member function
     ‘virtual ebpf::StatusTuple ebpf::cc::CodegenLLVM::visit_string_exp_node(ebpf::cc::StringExprNode*)’:
  /home/yhs/work/bcc/src/cc/frontends/b/codegen_llvm.cc:440:55: error: no matching function for call to
     ‘llvm::IRBuilder<>::CreateMemCpy(llvm:Value*&, int, llvm::Value*&, int, std::__cxx11::basic_string<char>::size_type)’
   B.CreateMemCpy(ptr, 1, global, 1, n->val_.size() + 1);
                                                       ^
  ...

This patch fixed the compilation issue.
 (commit: 45e63f2)

sync to latest libbpf

sync to libbpf v0.0.8. Add newer helpers to
helpers.h, libbpf.c error reporting and docs.
 (commit: df3e7d6)

prepare for release v0.14.0

added changelog for release v0.14.0
 (commit: ceb458d)