Demonstrations of slabratetop, the Linux eBPF/bcc version. slabratetop shows the rate of allocations and total bytes from the kernel memory allocation caches (SLAB or SLUB), in a top-like display that refreshes. For example: # ./slabratetop 07:01:35 loadavg: 0.38 0.21 0.12 1/342 13297 CACHE ALLOCS BYTES kmalloc-4096 3554 14557184 kmalloc-256 2382 609536 cred_jar 2568 493056 anon_vma_chain 2007 128448 anon_vma 972 77760 sighand_cache 24 50688 mm_struct 49 50176 RAW 52 49920 proc_inode_cache 59 38232 signal_cache 24 26112 dentry 135 25920 sock_inode_cache 29 18560 files_cache 24 16896 inode_cache 13 7696 TCP 2 3840 pid 24 3072 sigqueue 17 2720 ext4_inode_cache 2 2160 buffer_head 16 1664 xfs_trans 5 1160 By default the screen refreshes every one second, and only the top 20 caches are shown. These can be tuned with options: see USAGE (-h). The output above showed that the kmalloc-4096 cache allocated the most, about 14 Mbytes during this interval. This is a generic cache; other caches have more meaningful names ("dentry", "TCP", "pid", etc). slabtop(1) is a similar tool that shows the current static volume and usage of the caches. slabratetop shows the active call rates and total size of the allocations. Since "kmalloc-4096" isn't very descriptive, I'm interested in seeing the kernel stacks that led to this allocation. In the future (maybe by now) the bcc trace tool could do this. As I'm writing this, it can't, so I'll use my older ftrace-based kprobe tool as a workarond. This is from my perf-tools collection: https://github.com/brendangregg/perf-tools. # ./perf-tools/bin/kprobe -s 'p:kmem_cache_alloc name=+0(+96(%di)):string' 'name == "kmalloc-4096' | head -100 Tracing kprobe kmem_cache_alloc. Ctrl-C to end. kprobe-3892 [002] d... 7888274.478331: kmem_cache_alloc: (kmem_cache_alloc+0x0/0x1b0) name="kmalloc-4096" kprobe-3892 [002] d... 7888274.478333: => kmem_cache_alloc => user_path_at_empty => vfs_fstatat => SYSC_newstat => SyS_newstat => entry_SYSCALL_64_fastpath kprobe-3892 [002] d... 7888274.478340: kmem_cache_alloc: (kmem_cache_alloc+0x0/0x1b0) name="kmalloc-4096" kprobe-3892 [002] d... 7888274.478341: => kmem_cache_alloc => user_path_at_empty => vfs_fstatat => SYSC_newstat => SyS_newstat => entry_SYSCALL_64_fastpath kprobe-3892 [002] d... 7888274.478345: kmem_cache_alloc: (kmem_cache_alloc+0x0/0x1b0) name="kmalloc-4096" kprobe-3892 [002] d... 7888274.478346: => kmem_cache_alloc => user_path_at_empty => vfs_fstatat => SYSC_newstat => SyS_newstat => entry_SYSCALL_64_fastpath kprobe-3892 [002] d... 7888274.478350: kmem_cache_alloc: (kmem_cache_alloc+0x0/0x1b0) name="kmalloc-4096" kprobe-3892 [002] d... 7888274.478351: => kmem_cache_alloc => user_path_at_empty => vfs_fstatat => SYSC_newstat => SyS_newstat => entry_SYSCALL_64_fastpath kprobe-3892 [002] d... 7888274.478355: kmem_cache_alloc: (kmem_cache_alloc+0x0/0x1b0) name="kmalloc-4096" kprobe-3892 [002] d... 7888274.478355: => kmem_cache_alloc => user_path_at_empty => vfs_fstatat => SYSC_newstat => SyS_newstat => entry_SYSCALL_64_fastpath kprobe-3892 [002] d... 7888274.478359: kmem_cache_alloc: (kmem_cache_alloc+0x0/0x1b0) name="kmalloc-4096" kprobe-3892 [002] d... 7888274.478359: => kmem_cache_alloc => user_path_at_empty => vfs_fstatat => SYSC_newstat => SyS_newstat => entry_SYSCALL_64_fastpath [...] This is just an example so that you can see it's possible to dig further. Please don't copy-n-paste that kprobe command, as it's unlikely to work (the "+0(+96(%di))" text is specific to a kernel version and architecture). So these allocations are coming from user_path_at_empty(), which calls other functions (not seen in the stack: I suspect it's a tail-call compiler optimization). USAGE: # ./slabratetop -h usage: slabratetop [-h] [-C] [-r MAXROWS] [interval] [count] Kernel SLAB/SLUB memory cache allocation rate top positional arguments: interval output interval, in seconds count number of outputs optional arguments: -h, --help show this help message and exit -C, --noclear don't clear the screen -r MAXROWS, --maxrows MAXROWS maximum rows to print, default 20 examples: ./slabratetop # kmem_cache_alloc() top, 1 second refresh ./slabratetop -C # don't clear the screen ./slabratetop 5 # 5 second summaries ./slabratetop 5 10 # 5 second summaries, 10 times only