{ "_class" : "hudson.matrix.MatrixBuild", "actions" : [ { "_class" : "hudson.model.CauseAction", "causes" : [ { "_class" : "org.jenkinsci.plugins.ghprb.GhprbCause", "shortDescription" : "GitHub pull request #3576 of commit ea67a86314cd2d097f33a1b298666aa2073ba60a, no merge conflicts." } ] }, { "_class" : "org.jenkinsci.plugins.ghprb.GhprbParametersAction", "parameters" : [ { "_class" : "hudson.model.StringParameterValue", "name" : "sha1", "value" : "origin/pr/3576/merge" }, { "_class" : "hudson.model.StringParameterValue", "name" : "ghprbActualCommit", "value" : "ea67a86314cd2d097f33a1b298666aa2073ba60a" }, { "_class" : "hudson.model.StringParameterValue", "name" : "ghprbActualCommitAuthor", "value" : "" }, { "_class" : "hudson.model.StringParameterValue", "name" : "ghprbActualCommitAuthorEmail", "value" : "" }, { "_class" : "hudson.model.StringParameterValue", "name" : "ghprbAuthorRepoGitUrl", "value" : "https://github.com/mstange/bcc.git" }, { "_class" : "hudson.model.StringParameterValue", "name" : "ghprbTriggerAuthor", "value" : "" }, { "_class" : "hudson.model.StringParameterValue", "name" : "ghprbTriggerAuthorEmail", "value" : "" }, { "_class" : "hudson.model.StringParameterValue", "name" : "ghprbTriggerAuthorLogin", "value" : "yonghong-song" }, { "_class" : "hudson.model.StringParameterValue", "name" : "ghprbTriggerAuthorLoginMention", "value" : "@yonghong-song" }, { "_class" : "hudson.model.StringParameterValue", "name" : "ghprbPullId", "value" : "3576" }, { "_class" : "hudson.model.StringParameterValue", "name" : "ghprbTargetBranch", "value" : "master" }, { "_class" : "hudson.model.StringParameterValue", "name" : "ghprbSourceBranch", "value" : "offcputime-sampling" }, { "_class" : "hudson.model.StringParameterValue", "name" : "GIT_BRANCH", "value" : "offcputime-sampling" }, { "_class" : "hudson.model.StringParameterValue", "name" : "ghprbPullAuthorEmail", "value" : "mstange.moz@gmail.com" }, { "_class" : "hudson.model.StringParameterValue", "name" : "ghprbPullAuthorLogin", "value" : "mstange" }, { "_class" : "hudson.model.StringParameterValue", "name" : "ghprbPullAuthorLoginMention", "value" : "@mstange" }, { "_class" : "hudson.model.StringParameterValue", "name" : "ghprbPullDescription", "value" : "GitHub pull request #3576 of commit ea67a86314cd2d097f33a1b298666aa2073ba60a, no merge conflicts." }, { "_class" : "hudson.model.StringParameterValue", "name" : "ghprbPullTitle", "value" : "Proof of concept: Add offcpu-sampling capability to offcputime.py " }, { "_class" : "hudson.model.StringParameterValue", "name" : "ghprbPullLink", "value" : "https://github.com/iovisor/bcc/pull/3576" }, { "_class" : "hudson.model.StringParameterValue", "name" : "ghprbPullLongDescription", "value" : "This adds a -F flag to offcputime.py.\\r\\n\\r\\nExample usage:\\r\\n\\r\\n```\\r\\n$ sudo ./tools/offcputime.py -F 1000 2 # Sample at 1000 Hz for 2 seconds\\r\\n```\\r\\n\\r\\nThis achieves something like offcpu sampling without actually using\\r\\na wall-clock time based timer interrupt:\\r\\nFirst, we calculate a sample period from the supplied frequency, in nanoseconds.\\r\\nThen, whenever a thread wakes up, we accumulate its sleep time into a per-thread field.\\r\\nWhenever this accumulated sleep time counter overflows a multiple of the sample\\r\\nperiod, we sample.\\r\\nThis is an alternative to using --min-block-time: It achieves low overhead\\r\\nwithout missing short sleeps.\\r\\n\\r\\n## Backstory\\r\\n\\r\\nThis proof of concept was born out of confusion after reading https://www.brendangregg.com/offcpuanalysis.html\\r\\nThe page has this warning:\\r\\n\\r\\n> WARNING: With off-CPU tracing, be aware that scheduler events can be very frequent – in extreme cases, millions of events per second – and although tracers may only add a tiny amount of overhead to each event, due to the event rate that overhead can add up and become significant. Off-CPU sampling has overhead issues too, as systems may have tens of thousands of threads which must be constantly sampled, orders of magnitude higher overhead than CPU sampling across the CPU count only.\\r\\n\\r\\nThere is also a section about \\\"Off-CPU Sampling\\\", which says:\\r\\n\\r\\n> An off-CPU sampler must work in a different way: either setting up timers in each application thread to wake them up and capture stacks, or having the kernel walk all threads at an interval and capture their stacks.\\r\\n\\r\\nThe approach in this PR doesn't have any of the mentioned drawbacks:\\r\\n 1. It is tracing-based, it only does work when the thread starts running. There are no extra wakeups.\\r\\n 2. It has low overhead, because it only samples at the specified rate.\\r\\n\\r\\n## Comparison\\r\\n\\r\\nI ran the following three commands, one after the other, with the same workload running in the background. Specifically, I had Firefox Nightly running which was displaying an animation and also running its own built-in profiler which calls `sleep` in ~0.7ms increments in each process's SamplerThread.\\r\\n\\r\\n```\\r\\nsudo ./tools/offcputime.py -uUf 2 | ../FlameGraph/flamegraph.pl --color=io --title=\\\"Off-CPU Time Flame Graph\\\" --countname=us > out-traced.svg\\r\\nsudo ./tools/offcputime.py -F 1000 -uUf 2 | ../FlameGraph/flamegraph.pl --color=io --title=\\\"Off-CPU Time Flame Graph\\\" --countname=us > out-sampled.svg\\r\\nsudo ./tools/offcputime.py -m 1000 -uUf 2 | ../FlameGraph/flamegraph.pl --color=io --title=\\\"Off-CPU Time Flame Graph\\\" --countname=us > out-with-minimum.svg\\r\\n```\\r\\n\\r\\nResults:\\r\\n\\r\\n - [out-traced.svg](http://tests.themasta.com/tmp/out-traced.svg)\\r\\n - [out-sampled.svg](http://tests.themasta.com/tmp/out-sampled.svg)\\r\\n - [out-with-minimum.svg](http://tests.themasta.com/tmp/out-with-minimum.svg)\\r\\n\\r\\nYou can see that `out-traced.svg` and `out-sampled.svg` look very similar. Importantly, they both show off-cpu time in Firefox's `SamplerThread`. In contrast, `out-with-minimum.svg` does not show the off-cpu time in the `SamplerThread` because it filtered out those short sleep times." }, { "_class" : "hudson.model.StringParameterValue", "name" : "ghprbCommentBody", "value" : "[buildbot, test this please]" }, { "_class" : "hudson.model.StringParameterValue", "name" : "ghprbGhRepository", "value" : "iovisor/bcc" }, { "_class" : "hudson.model.StringParameterValue", "name" : "ghprbCredentialsId", "value" : "6d3daf13-69b8-48b1-9c8f-ec5353264113" } ] }, { }, { }, { "_class" : "org.jenkinsci.plugins.displayurlapi.actions.RunDisplayAction" } ], "artifacts" : [ ], "building" : False, "description" : "PR #3576: Proof of concept: Add offcp...", "displayName" : "#1111", "duration" : 1206068, "estimatedDuration" : 7460664, "executor" : None, "fullDisplayName" : "bcc-pr #1111", "id" : "1111", "keepLog" : False, "number" : 1111, "queueId" : 8890, "result" : "FAILURE", "timestamp" : 1628837709142, "url" : "https://buildbot.iovisor.org/jenkins/job/bcc-pr/1111/", "builtOn" : "", "changeSet" : { "_class" : "hudson.plugins.git.GitChangeSetList", "items" : [ ], "kind" : "git" }, "culprits" : [ ], "runs" : [ { "number" : 1111, "url" : "https://buildbot.iovisor.org/jenkins/job/bcc-pr/label=fc25/1111/" }, { "number" : 1111, "url" : "https://buildbot.iovisor.org/jenkins/job/bcc-pr/label=fc26/1111/" }, { "number" : 1111, "url" : "https://buildbot.iovisor.org/jenkins/job/bcc-pr/label=fc27/1111/" }, { "number" : 1111, "url" : "https://buildbot.iovisor.org/jenkins/job/bcc-pr/label=fc28/1111/" }, { "number" : 1111, "url" : "https://buildbot.iovisor.org/jenkins/job/bcc-pr/label=ubuntu1604/1111/" }, { "number" : 1111, "url" : "https://buildbot.iovisor.org/jenkins/job/bcc-pr/label=ubuntu1710/1111/" }, { "number" : 1111, "url" : "https://buildbot.iovisor.org/jenkins/job/bcc-pr/label=ubuntu1804/1111/" } ] }