Changes

Adjust the order of linux kernel header include paths

The current order of include paths could cause the following error:

In file included from /virtual/main.c:3:
In file included from include/linux/sched.h:13:
In file included from include/linux/pid.h:4:
In file included from include/linux/rculist.h:10:
In file included from include/linux/rcupdate.h:38:
In file included from include/linux/spinlock.h:50:
In file included from include/linux/preempt.h:80:
In file included from /lib/modules/4.12.14-197.45-default/build/arch/s390/include/generated/asm/preempt.h:1:
include/asm-generic/preempt.h:10:42: error: no member named 'preempt_count' in 'struct thread_info'
        return READ_ONCE(current_thread_info()->preempt_count);
               ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~
include/linux/compiler.h:349:34: note: expanded from macro 'READ_ONCE'
                     ~~~~~~~~~~~~^~~~~
include/linux/compiler.h:342:17: note: expanded from macro '__READ_ONCE'
        union { typeof(x) __val; char __c[1]; } __u;
\
...
7 errors generated.

It's supposed to load "asm/preempt.h" from "source/"(*) but accidentally
loaded the one from "build/", so the error showed.
(x86_64 didn't suffer this error because it didn't install the extra
 "asm/preempt.h" to "build/".)

For the distros, e.g. SUSE/openSUSE and debian,  with separate "build/"
and "source/", all those "generated/" paths only exist in "build/". To
avoid the potential compilation issue, this commit adjusts the include
order for those with split headers to align the include order from
kernel top Makefile.

(*) /lib/modules/4.12.14-197.45-default/source/arch/s390/include/

Signed-off-by: Gary Lin <glin@suse.com>
 (commit: c31e9d6)

libbpf-tools: update bpftool and fix .rodata hack

Update bpftool to the latest version, handing const volatile arrays properly
when generating BPF skeletons. Fix tcpconnect tool hack to work around that
issue.

Signed-off-by: Andrii Nakryiko <andriin@fb.com>
 (commit: a02663b)

Add return value check and handling for xattr loading

libbpf_find_vmlinux_btf_id may return error code for different reasons. We
should check and report the error we have just got, rather than let it slip
down the path and give an ambiguous error later.

Signed-off-by: He Zhe <zhe.he@windriver.com>
 (commit: ba0bacf)

tools/opensnoop: Snoop all open related syscall stubs

kernel v5.6 introduces fddb5d430ad9 ("open: introduce openat2(2) syscall").
Even though do_sys_open still exists, it might be optimized off final binary
depending on compilers. So we can't catch do_sys_open in some cases.

This patch uses ksymname to try to get entries of open, openat and openat2,
and changes the definitions of the trace functions to snoop them all. This
works for both kprobe and kfunc.

Credit to Yonghong Song for better code organization.

Signed-off-by: He Zhe <zhe.he@windriver.com>
 (commit: 6889afe)

man: fix tcpconnect man page

Add -u and -U options to the synopsis.
Move their description frome the example to the description section.
Add a description to --mntnsmap option.
 (commit: 7ce6f64)

Add example for new helper to documentation.
 (commit: 7928ea0)

libbpf-tools: add CO-RE biopattern

Signed-off-by: Wenbo Zhang <ethercflow@gmail.com>
 (commit: ca5bcd2)

libbpf-tools: add CO-RE tcpconnlat

Signed-off-by: Wenbo Zhang <ethercflow@gmail.com>
 (commit: b6ffa24)

sync with latest libbpf repo

sync with latest libbpf repo

Signed-off-by: Yonghong Song <yhs@fb.com>
 (commit: 8a164f9)

libbpf-tools: add CO-RE biosnoop

Signed-off-by: Wenbo Zhang <ethercflow@gmail.com>
 (commit: e0e2215)

 libbpf-tools: fix tcpconnect compile errors

Signed-off-by: Wenbo Zhang <ethercflow@gmail.com>
 (commit: f8ac3c6)

Offset support for uprobe and kprobe
 (commit: d7b427e)

Update the documentation of the k/uprobe offset support
 (commit: 8105317)

typos
 (commit: 6432324)

sync with latest libbpf repo

Signed-off-by: Yonghong Song <yhs@fb.com>
 (commit: 9ef20e7)

Fixes a problem when LLVM_DEFINITIONS contains a definition with an `=` in it.
add_definitions doesn't seem to parse this type of argument correctly.
 (commit: 3fbb664)

bcc/tools/fileslower: Attach to vfs_read after failing to attach __vfs_read

__vfs_read has been removed since kernel v5.8-rc5
775802c0571f ("fs: remove __vfs_read"). Then try vfs_read instead.

Signed-off-by: He Zhe <zhe.he@windriver.com>
 (commit: b61e65f)

libbpibbpf-tools: fix tcpconnect types usage

The __int128 is not defined for 32-bit platforms, see [1], so replace it by
a portable array of __u8.

The usage of this type came from the original tcpconnect.py, and __int128 is
still used in most (if not all) original BCC tools to store IPv6 addresses.

[1] https://github.com/iovisor/bcc/issues/3044

Signed-off-by: Anton Protopopov <a.s.protopopov@gmail.com>
 (commit: 5c27222)

libbpf-tools: add CO-RE readahead

Signed-off-by: Wenbo Zhang <ethercflow@gmail.com>
 (commit: fb54a82)

libbpf-tools: fix tcpconnlat types usage

Signed-off-by: Wenbo Zhang <ethercflow@gmail.com>
 (commit: 7d081eb)

docs: fix broken links
 (commit: 639e83f)

Forbid trampolines for archs other than x86_64

The trampoline support check in bcc does not work properly,
so the feature is detected even on architectures that do not
support it - all archs other than x86_64.

We are checking for bpf_trampoline_link_prog to exist in
kernel, which works fine on x86_64 to check if the feature
is supported, but it's global function, so it exists also
in other archs even when the feature is not supported
so it returns True also on other archs.

Adding explicit x86_64 check to support_kfunc function.

Signed-off-by: Jiri Olsa <jolsa@kernel.org>
 (commit: 49beed8)

kernel-versions.md: document map UAPI features (#3058)

Document kernel versions for userspace API features regarding map manipulation.
 (commit: e73e1f6)

tools/funccount: support funccount on specified CPU (#3059)

A typical case of this feature: count timer setting on a x86 server
for each CPU:
 for i in `seq 0 39`;
   do ./funccount.py -i 1 lapic_next_deadline -d 5 -c $i;
 done

Then we can know the timer setting is balanced of not and do some
futher work.

Signed-off-by: zhenwei pi <pizhenwei@bytedance.com>
 (commit: f4e65ac)

Extend PerfType members

You can use

    b.attach_perf_event(ev_type=PerfType.RAW, ...)

to attach perf raw event for profiling.

Signed-off-by: Edward Wu <edwardwu@realtek.com>
 (commit: 785924b)

libbpf-tools: add CO-RE biostacks

Signed-off-by: Wenbo Zhang <ethercflow@gmail.com>
 (commit: 71c22fc)

sync libbpf repo release 0.1.0

Signed-off-by: Yonghong Song <yhs@fb.com>
 (commit: a79ec23)

prepare for release v0.16.0

add changelog for release v0.16.0

Signed-off-by: Yonghong Song <yhs@fb.com>
 (commit: fecd934)

sync with libbpf repo

sync with latest libbpf repo.

Strut definition for btf_ext_header is defined in libbpf/src/btf.h
previously and used by bcc. Now, the struct is moved to
libbpf/src/libbpf_internal.h and not available to bcc.
We do not want to include libbpf/src/libbpf_internal.h as
it is really libbpf internal. Let us define bcc version
of btf_ext_header with struct name bcc_btf_ext_header.
The new name is to avoid conflict when compiling with
old libbpf package.

Signed-off-by: Yonghong Song <yhs@fb.com>
 (commit: 43910c4)

fix the exits abnormal due to trace buffer

Some data may be left in the trace buffer, and the format cannot be
parsed by trace_fields(), which whill cause an abnormality in the
system.

Re-running the program cannot solved, only restart kernel.

Call trace:
      File "/usr/lib/python2.7/site-packages/kwaibcc/__init__.py", line 1273, in trace_fields
        pid, cpu, flags, ts = line[:ts_end].split()
    ValueError: need more than 2 values to unpack

Prevent problems with exception handling.

Signed-off-by: Chen Guanqiao <chen.chenchacha@foxmail.com>
 (commit: 60de171)

doc: fix mismatched link

Signed-off-by: zhuangqh <zhuangqhc@gmail.com>
 (commit: fc92caa)

tools/capable: support new capabilities

This commit adds the support of below capabilities.
* CAP_PERFMON
* CAP_BPF
* CAP_CHECKPOINT_RESTORE

Signed-off-by: Kenta Tada <Kenta.Tada@sony.com>
 (commit: 0e7304e)

fix retruning typo
 (commit: 892475f)

tests: only run arg parser test on supported arches

Fixes #3077

Signed-off-by: Rafael Fonseca <r4f4rfs@gmail.com>
 (commit: 4344034)

Add instructions on how to install amazon linux 2 from source
 (commit: 7ac370b)

memleak: fix false positive on failed allocations

memleak tool reports a failed allocation as a leak,
because the NULL address is never freed.

Signed-off-by: Amir Goldstein <amir73il@gmail.com>
 (commit: b8e661f)

tools/tcprtt: add tcprtt to trace the RTT of TCP

This program traces TCP RTT(round-trip time) to analyze the quality of
network, then help us to distinguish the network latency trouble is
from user process or physical network.

Currently, support source address/port and destination address/port as
tcp filter.

Suggested-by: Edward Wu <edwardwu@realtek.com>
Suggested-by: Martin KaFai Lau <kafai@fb.com>
Signed-off-by: zhenwei pi <pizhenwei@bytedance.com>
 (commit: e42ac41)

Netqtop 3037 (#3048)

The tool netqtop uses tracepoints NET_DEV_START_XMIT and NETIF_RECEIVE_SKB to intercept every transmitted and received packet, as a result, considerable performance descent is expected.
Details for some performance evaluation can be found at
   https://github.com/iovisor/bcc/pull/3048
 (commit: ba73657)

add threadsnoop
 (commit: 3f95637)

add tcpsynbl
 (commit: 12c234f)

add swapin
 (commit: da8c6a9)

libbpf-tools: add hardirqs

Signed-off-by: Wenbo Zhang <ethercflow@gmail.com>
 (commit: 3953c60)

libbpf-tools: add softirqs

Signed-off-by: Wenbo Zhang <ethercflow@gmail.com>
 (commit: bf4668d)

libbpf-tools: add runqlat

Signed-off-by: Wenbo Zhang <ethercflow@gmail.com>
 (commit: 48b7d61)

libbpf-tools: add runqlen

Signed-off-by: Wenbo Zhang <ethercflow@gmail.com>
 (commit: 911ce07)

libbpf-tools: add numamove

Signed-off-by: Wenbo Zhang <ethercflow@gmail.com>
 (commit: caaeb0a)

Add uretprobe example for indirect function

Already there is strlen_hist.py as a nice example of uretprobe. But it can
not work correctly if strlen is indirect function(IFUNC). So
strlen_hist_ifunc.py is introduced to show how to use uprobe/uretprobe for
indirect functions.
 (commit: 39a2a8f)

tools/tcpstates: remove unnecessary checking of TCP
 (commit: e548dd5)

tool: port readahead tool to bcc
 (commit: 561a71e)

tcprtt: add --byladdr/--byraddr (#3106)

On a server side, a server listens a local port and accepts lots of
connections. To seperate histogram into views by each remote address,
so add --byraddr to support this.

Suggested-by: Brendan Gregg <brendan.d.gregg@gmail.com>
Signed-off-by: zhenwei pi <pizhenwei@bytedance.com>
 (commit: 4cbcd9a)

add average function latency (#3061)

* add average function latency
* update example.txt
 (commit: af90bd4)

slabratetop: Add memcg_cache_params struct def

struct memcg_cache_params moved from include/linux/slab.h to
mm/slab.h in kernel v5.4, causing a compiler error when including
slub_def.h or slab_def.h in slabratetop's bpf program.

It has been removed completely from kernel version 5.9.

Add an empty memcg_cache_params struct in slabratetop's bpf
program so it will compile with kernel versions 5.4 to 5.8.
 (commit: 788303e)

Fix v6 source (remote) and dest (local) address

For v6, tcpdrop.py would report the source and destination addresses incorrectly - tcp_drop() covers the input path, where the source of the received packet is the daddr stored in the socket, while the destination of the received packet is the local address.   This commit swaps them to be correct and leaves a comment since it is not obvious.
 (commit: 7e3f0c0)

tools: tcptracer: fix alignement in tcp_ipv6_event_t

On IPv6, tcptracer ports always appears as zeros:

Tracing TCP established connections. Ctrl-C to end.
T  PID    COMM             IP SADDR            DADDR            SPORT  DPORT
X  7055   nc               4  127.0.0.1        127.0.0.1        49476  9999
C  7074   nc               4  127.0.0.1        127.0.0.1        49478  9999
X  7074   nc               4  127.0.0.1        127.0.0.1        49478  9999
C  7085   nc               6  [::]             [0:0:0:1::]      0      0
X  7085   nc               6  [::]             [0:0:0:1::]      0      0
C  7086   nc               6  [::]             [0:0:0:1::]      0      0

This seems related to alignment issue wrt to the __int128 type in
tcp_ipv6_event_t structure. Moving the u8 field ip to the end of the
structure fixes the issue.

Fixes #2781
 (commit: 21b810a)

tools/biolatency: Handle signals from user
 (commit: 87792ce)

libbpf-tools: make some minor refactoring

Signed-off-by: Wenbo Zhang <ethercflow@gmail.com>
 (commit: 73cf23b)

Revert "tools/biolatency: Handle signals from user"

This reverts commit 87792ce6b783941ea1d8bcd29812a55520930c95.

Let us have a consensus about how bcc tools should interact with
external signals etc. before applying such changes to this tool
and other tools.
 (commit: 17e14ef)

sync with latest libbpf repo

11 new helpers are added so far in 5.10, added helper function prototypes
and related documentation, etc.

Signed-off-by: Yonghong Song <yhs@fb.com>
 (commit: d0e1c93)

add C++ bpf_iter support

bpf iterator is introduced in linux kernel 5.8.
  https://lore.kernel.org/bpf/20200509175859.2474608-1-yhs@fb.com/
In 5.8, iterator support for task, task_file, bpf_map, netlink_sock
and ipv6_route. In 5.9, tcp, udp and hash/array/sk_local_storage
map iterators are implemented.

This patch added necessary interface to support bpf_iter in bcc.
A few C++ APIs are added to bcc. Two bpf_iter examples,
task and sk_local_storage_map, are added to illustrate how
bpf iterator can be implemented. Python interface can be added
later if there is a need.

Signed-off-by: Yonghong Song <yhs@fb.com>
 (commit: 5fe3576)

docs: Add missing code block closure

Signed-off-by: Mauricio Vásquez <mauricio@kinvolk.io>
 (commit: 5e55c22)

tools: Use correct key key when deleting item from map

The correct key to use is "tig" instead of "pid".

Signed-off-by: Mauricio Vásquez <mauricio@kinvolk.io>
 (commit: 884799f)

sslsniff: fix comparing bytes to string

Signed-off-by: keyolk <chanhun.jeong@navercorp.com>
 (commit: 49fdec6)

Update installation instructions for Arch Linux

The bcc packages were recently moved from the AUR to the standard Arch
repos, so the installation instructions for Arch should be updated to
reflect that.

Additionally, the Arch "linux-lts" package was upgraded to 4.4.7 in
April 2016, so it is safe to assume that any Arch installations that are
still in use have been upgraded by this point. Therefore, the note about
upgrading to kernel 4.3.1 is removed.
 (commit: 6ce68b9)

set larger range to get probes evnet for test_usdt2.py

I test test_usdt2.py on aarch64 with kernel-5.9-rc5,
gcc-9.2.1 and glibc2.32.
with the current range of 3, always get failure:
======================================================================
FAIL: test_attach1 (__main__.TestUDST)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_usdt2.py", line 170, in test_attach1
    self.assertTrue(self.evt_st_6 == 1)
AssertionError: False is not true

----------------------------------------------------------------------
Ran 1 test in 1.068s

FAILED (failures=1)

Signed-off-by: Chunmei Xu <xuchunmei@linux.alibaba.com>
 (commit: 4354c9c)

Add install instructions for Debian
 (commit: 5f53dbe)

libbpf-tools: initialize global structs in runqlen and softirqs

Signed-off-by: Juraj Vijtiuk <juraj.vijtiuk@sartura.hr>
 (commit: 8c2d67e)

libbpf-tools: add llcstat

Signed-off-by: Wenbo Zhang <ethercflow@gmail.com>
 (commit: 6929626)

Fix python2.7 support for dirtop tool

* Catch TypeError raised by Python3 glob and fall back to Python2 interface
 (commit: 1efba05)

fix python3 compatible issue for netqtop and tcprtt tools

Otherwise, we have:
  ...
    File "bcc/tools/netqtop.py", line 54
      print(hd.center(COL_WIDTH)),
                                 ^
  TabError: inconsistent use of tabs and spaces in indentation
  ...

    File "bcc/tools/tcprtt.py", line 117, in <module>
    bpf_text = bpf_text.replace(b'LPORTFILTER', b'')
  TypeError: replace() argument 1 must be str, not bytes
  ...

Signed-off-by: Yonghong Song <yhs@fb.com>
 (commit: 6ba4dc1)

fix netqtop python3 compatible (#3140)

* fix netqtop python3 compatible
* delete import types
 (commit: ec3747e)

uprobe: Add ref_cnt_offset arg to bpf_attach_uprobe

This argument allows callers to tell the kernel to manage USDT semaphore
counts. It's better than fiddling with the counts in userspace b/c if
the userspace program crashes then the semaphore count does not get
decremented.

It's also better b/c the kernel can activate the semaphore for all
processes that have the target mapped into memory. Currently with bcc,
you have to provide the pid of each process you want to activate a
semaphore for and then bcc will go poke /proc/pid/mem at the right
offset.
 (commit: 50f2009)

usdt: Calculate and suppply semaphore_offset values in APIs

`semaphore_offset` is the offset from the start of the file the
usdt semaphore can be found. This is different than the already exiting
`semaphore` value which is the virtual address the usdt semaphore can be
found at (after load).

The reason we need to calculate and export this `semaphore_offset` value
is b/c the kernel's uprobe refcount API requires the offset and not the
vaddr.
 (commit: 3f0c5f0)

usdt: Use uprobe refcnt if available in usdt C++ API
 (commit: 28a5cef)

folly: Update tracing headers

Sync to upstream folly.

Also turn all users of the headers to be C++ programs instead of C b/c
folly now #includes <cstddef>.
 (commit: 8287f17)

usdt: Add test for uprobe refcnt support
 (commit: 50ad6fa)

Refactor docker image publishing

This adds support to push docker images to quay.io, like other projects in
the iovisor org.

It separates docker image builds into a separate github workflow, and
refactors the package building process slightly, to be generic, in order to
create builds for both ubuntu 16.04 and ubuntu 18.04.

This provides a means to distribute intermediate apt packages between releases,
and also enables uploading these as CI artifacts.

As recent releases have not annotated their tags, it drops the requirement for
tags to be annotated in selecting the version to use.
 (commit: d4aecc6)

Add ubuntu 20.04 build support
 (commit: 20b098a)

Guard artifact upload with environment variable
 (commit: d35a2dc)

Prefer newer llvm/clang versions in debian packaging
 (commit: 9269e48)

Do not permit errors in docker build
 (commit: 4967a61)

python: import ABCs from collections.abc

Importing Abstract Base Classes (ABCs) from the collections module
is deprecated since Python 3.3, it emits a warning since Python 3.8,
and it will stop working in Python 3.10.

Try importing MutableMapping from collections.abc (the preferred way since
Python 3.3) and, in case of error (Python < 3.3) fall back to importing it
from collections.
 (commit: 48946d2)

add DNS correlation to connect tracking
 (commit: 33817e6)

Update INSTALL.md

For ubuntu focal fossa only instructions given for Eoan worked for me. If we don't add Focal fossa to list people will go to "For other versions" section. which refers to libllvm3.7 which could not be installed on ubuntu focal. Hence this proposed change
 (commit: 1591469)

docs(INSTALL): add libfl-dev to ubuntu dependencies

To compile the software on Ubuntu 20.04 I had trouble compiling the source because my OS is missing the `FlexLexer.h` library. This package is supplied by `libfl-dev`.
 (commit: 5ab0183)

sync with latest libbpf repo

sync with libbpf v0.2.0

Signed-off-by: Yonghong Song <yhs@fb.com>
 (commit: 15efd46)

update debian changelog for release v0.17.0

 * Support for kernel up to 5.9
 * usdt: add uprobe refcnt support
 * use newer llvm/clang versions in debian packaging if possible
 * add bpf iterator C++ support
 * new bcc tools: tcprtt, netqtop, swapin, tcpsynbl, threadsnoop
 * tcpconnect: add DNS correlation to connect tracking
 * new libbpf-tools: llcstat, numamove, runqlen, runqlat, softirgs, hardirqs
 * doc update, bug fixes and some additional arguments for tools

Signed-off-by: Yonghong Song <yhs@fb.com>
 (commit: ad5b82a)

use correct arch register for the 4th param of x86_64 syscalls

Fix #3150

On x86_64, for syscalls, the calling convension is to use
r10 instead of rcx for the 4th parameter. I have verified this
with disassembling vmlinux codes.
  https://www.systutorials.com/x86-64-calling-convention-by-gcc/

bcc previously used rcx for the 4th parameter for all cases.
This patch fixed the issue by using r10 for syscalls.
A macro PT_REGS_PARM4_SYSCALL() is also introduced in helpers.h
to access the 4th parameter for r10.

Signed-off-by: Yonghong Song <yhs@fb.com>
 (commit: 12107c6)

fix compilation issues with latest llvm12 trunk

With latest llvm12 trunk, we got two compilation bugs.

Bug #1:
  /home/yhs/work/bcc/src/cc/frontends/clang/b_frontend_action.cc:
     In member function ‘void ebpf::BFrontendAction::DoMiscWorkAround()’:
  /home/yhs/work/bcc/src/cc/frontends/clang/b_frontend_action.cc:1706:31:
     error: ‘class clang::SourceManage’ has no member named ‘getBuffer’; did you mean ‘getBufferData’?
     rewriter_->getSourceMgr().getBuffer(rewriter_->getSourceMgr().getMainFileID())->getBufferSize(),
                               ^~~~~~~~~
                               getBufferData

  This is due to upstream change https://reviews.llvm.org/D89394.
  To fix, follow upstream examples in https://reviews.llvm.org/D89394.

Bug #2:
  /home/yhs/work/bcc/src/cc/bpf_module.cc: In member function ‘int ebpf::BPFModule::finalize()’:
  /home/yhs/work/bcc/src/cc/bpf_module.cc:470:11:
    error: ‘class llvm::EngineBuilder’ has no member named ‘setUseOrcMCJITReplacement’
   builder.setUseOrcMCJITReplacement(false);
           ^~~~~~~~~~~~~~~~~~~~~~~~~

  This is due to upstream
    https://github.com/llvm/llvm-project/commit/6154c4115cd4b78d0171892aac21e340e72e32bd

  It seems builder.setUseOrcMCJITReplacement() is not needed any more. So just remove it
  from bcc.

Signed-off-by: Yonghong Song <yhs@fb.com>
 (commit: 11c93f6)