GitHub pull request #2579 of commit 0ada26541b72b973ae253fe4590b095d779a8c24, no merge conflicts.sha1origin/pr/2579/mergeghprbActualCommit0ada26541b72b973ae253fe4590b095d779a8c24ghprbActualCommitAuthorJiri OlsaghprbActualCommitAuthorEmailjolsa@kernel.orgghprbAuthorRepoGitUrlhttps://github.com/olsajiri/bcc.gitghprbTriggerAuthorghprbTriggerAuthorEmailghprbTriggerAuthorLoginyonghong-songghprbTriggerAuthorLoginMention@yonghong-songghprbPullId2579ghprbTargetBranchmasterghprbSourceBranchlockstatGIT_BRANCHlockstatghprbPullAuthorEmailghprbPullAuthorLoginolsajirighprbPullAuthorLoginMention@olsajirighprbPullDescriptionGitHub pull request #2579 of commit 0ada26541b72b973ae253fe4590b095d779a8c24, no merge conflicts.ghprbPullTitleAdd lockstat toolghprbPullLinkhttps://github.com/iovisor/bcc/pull/2579ghprbPullLongDescriptionAdding lockstat tool to trace kernel mutex lock events and\r\ndisplay locks statistics and displays following data:\r\n\r\n```\r\n Caller Average Spin Count Max spin Total spin\r\n pipe_wait+0xa9 708 13062320 86318 9254917935\r\n pipe_write+0x35 465 13062622 86105 6081839337\r\n pipe_read+0x3f 408 13062615 83390 5335451713\r\n unix_dgram_recvmsg+0xd0 2613 52 20699 135900\r\n rtnetlink_rcv_msg+0x264 2083 68 18544 141688\r\n psi_avgs_work+0x2e 2536 437 16887 1108366\r\n\r\n Caller Average Hold Count Max hold Total hold\r\n _vm_unmap_aliases+0xdb 98228 2 106752 196457\r\n _vm_unmap_aliases+0xdb 96857 3 102253 290573\r\n pipe_write+0x35 654 13062622 96473 8554734697\r\n __fdget_pos+0x42 17499 31 89222 542480\r\n rtnetlink_rcv_msg+0x264 13614 68 85603 925767\r\n pipe_wait+0xa9 788 13062320 81304 10305658481\r\n\r\n```\r\n\r\nEvery caller to using kernel's mutex is displayed on every line.\r\n\r\nFirst portion of lines show the lock acquiring data, showing the amount\r\nof time it took to acquired given lock.\r\n\r\n 'Caller' - symbol acquiring the mutex\r\n 'Average Spin' - average time to acquire the mutex\r\n 'Count' - number of times mutex was acquired\r\n 'Max spin' - maximum time to acquire the mutex\r\n 'Total spin' - total time spent in acquiring the mutex\r\n\r\nSecond portion of lines show the lock holding data, showing the amount\r\nof time it took to hold given lock.\r\n\r\n 'Caller' - symbol holding the mutex\r\n 'Average Hold' - average time mutex was held\r\n 'Count' - number of times mutex was held\r\n 'Max hold' - maximum time mutex was held\r\n 'Total hold' - total time spent in holding the mutex\r\n\r\nThis works by tracing mutex_lock/unlock kprobes, udating the lock stats\r\nin maps and processing them in the python part.\r\n\r\nExamples:\r\n```\r\n lockstats # trace system wide\r\n lockstats -d 5 # trace for 5 seconds only\r\n lockstats -i 5 # display stats every 5 seconds\r\n lockstats -p 123 # trace locks for PID 123\r\n lockstats -t 321 # trace locks for PID 321\r\n lockstats -c pipe_ # display stats only for lock callers with 'pipe_' substring\r\n lockstats -S acq_count # sort lock acquired results on acquired count\r\n lockstats -S hld_total # sort lock held results on total held time\r\n lockstats -S acq_count,hld_total # combination of above\r\n lockstats -n 3 # display 3 locks\r\n lockstats -s 3 # display 3 levels of stack\r\n```ghprbCommentBodynullghprbGhRepositoryiovisor/bccghprbCredentialsId6d3daf13-69b8-48b1-9c8f-ec5353264113false<a title="Add lockstat tool" href="https://github.com/iovisor/bcc/pull/2579">PR #2579</a>: Add lockstat tool#13912437167460664bcc-pr #139139false1391140SUCCESS1574066935110https://buildbot.iovisor.org/jenkins/user/dalehamel/my-views/view/all/job/bcc-pr/139/git139https://buildbot.iovisor.org/jenkins/user/dalehamel/my-views/view/all/job/bcc-pr/label=fc25/139/139https://buildbot.iovisor.org/jenkins/user/dalehamel/my-views/view/all/job/bcc-pr/label=fc26/139/139https://buildbot.iovisor.org/jenkins/user/dalehamel/my-views/view/all/job/bcc-pr/label=fc27/139/139https://buildbot.iovisor.org/jenkins/user/dalehamel/my-views/view/all/job/bcc-pr/label=fc28/139/139https://buildbot.iovisor.org/jenkins/user/dalehamel/my-views/view/all/job/bcc-pr/label=ubuntu1604/139/139https://buildbot.iovisor.org/jenkins/user/dalehamel/my-views/view/all/job/bcc-pr/label=ubuntu1710/139/139https://buildbot.iovisor.org/jenkins/user/dalehamel/my-views/view/all/job/bcc-pr/label=ubuntu1804/139/