{ "_class" : "hudson.matrix.MatrixBuild", "actions" : [ { "_class" : "hudson.model.CauseAction", "causes" : [ { "_class" : "org.jenkinsci.plugins.ghprb.GhprbCause", "shortDescription" : "GitHub pull request #2143 of commit 3d629761021825f0ecb9919e435a25765af8eb6e, no merge conflicts." } ] }, { "_class" : "org.jenkinsci.plugins.ghprb.GhprbParametersAction", "parameters" : [ { "_class" : "hudson.model.StringParameterValue", "name" : "sha1", "value" : "origin/pr/2143/merge" }, { "_class" : "hudson.model.StringParameterValue", "name" : "ghprbActualCommit", "value" : "3d629761021825f0ecb9919e435a25765af8eb6e" }, { "_class" : "hudson.model.StringParameterValue", "name" : "ghprbActualCommitAuthor", "value" : "" }, { "_class" : "hudson.model.StringParameterValue", "name" : "ghprbActualCommitAuthorEmail", "value" : "" }, { "_class" : "hudson.model.StringParameterValue", "name" : "ghprbAuthorRepoGitUrl", "value" : "https://github.com/th0m/bcc.git" }, { "_class" : "hudson.model.StringParameterValue", "name" : "ghprbTriggerAuthor", "value" : "Danny Kulchinsky" }, { "_class" : "hudson.model.StringParameterValue", "name" : "ghprbTriggerAuthorEmail", "value" : "dannyk@tuenti.com" }, { "_class" : "hudson.model.StringParameterValue", "name" : "ghprbTriggerAuthorLogin", "value" : "dannyk81" }, { "_class" : "hudson.model.StringParameterValue", "name" : "ghprbTriggerAuthorLoginMention", "value" : "@dannyk81" }, { "_class" : "hudson.model.StringParameterValue", "name" : "ghprbPullId", "value" : "2143" }, { "_class" : "hudson.model.StringParameterValue", "name" : "ghprbTargetBranch", "value" : "master" }, { "_class" : "hudson.model.StringParameterValue", "name" : "ghprbSourceBranch", "value" : "master" }, { "_class" : "hudson.model.StringParameterValue", "name" : "GIT_BRANCH", "value" : "master" }, { "_class" : "hudson.model.StringParameterValue", "name" : "ghprbPullAuthorEmail", "value" : "" }, { "_class" : "hudson.model.StringParameterValue", "name" : "ghprbPullAuthorLogin", "value" : "th0m" }, { "_class" : "hudson.model.StringParameterValue", "name" : "ghprbPullAuthorLoginMention", "value" : "@th0m" }, { "_class" : "hudson.model.StringParameterValue", "name" : "ghprbPullDescription", "value" : "GitHub pull request #2143 of commit 3d629761021825f0ecb9919e435a25765af8eb6e, no merge conflicts." }, { "_class" : "hudson.model.StringParameterValue", "name" : "ghprbPullTitle", "value" : "Add udp_rcvbuf_errors.py to tools to help debug UDP RcvbufErrors packet drops" }, { "_class" : "hudson.model.StringParameterValue", "name" : "ghprbPullLink", "value" : "https://github.com/iovisor/bcc/pull/2143" }, { "_class" : "hudson.model.StringParameterValue", "name" : "ghprbPullLongDescription", "value" : "`udp_rcvbuf_errors` prints information about UDP packets that were dropped by the\\r\\nkernel due to the receive buffer being full.\\r\\nThis is useful for debugging high rates of UDP drops due to RcvbufErrors and finding which socket is responsible for them in real time." }, { "_class" : "hudson.model.StringParameterValue", "name" : "ghprbCommentBody", "value" : "Apologies in advance for piggy-backing on this PR :pray: but I came across it while investigating an illusive issue with UDP, our problem though is on the transmit path for which I'm not able to find any relevant tracepoints to try and narrow it down (so I'm trying to play with the code in this PR to make it work on the send path - not much luck so far...).\\r\\n\\r\\nWe have a Java application that transmits relatively small (100-150 bytes) datagrams in a containerized deployment (Kubernetes), we are testing the same application on a newer kernel (moved from 4.14.96 -> 4.19.28) and hitting send failures for which we can't seem to find any explanation.\\r\\n\\r\\nThe application uses `java.nio.channels.DatagramChannel` to transmit the datagrams (using a non-blocking channel and 4096 bytes `SO_SNDBUF`), we also limit the `ByteBuffer` to 1500 so that we never try to send something that might exceed the send buffer size.\\r\\n\\r\\nWhen the load on the system increases (around 2~3k sends per second) the `send` method seem to be failing to send some percentage of the datagrams and returns 0 which (based on the docs) suggests: \\r\\n>if this channel is non-blocking, may be zero if there was insufficient room for the datagram in the underlying output buffer\\r\\n\\r\\nhttps://docs.oracle.com/javase/7/docs/api/java/nio/channels/DatagramChannel.html#send(java.nio.ByteBuffer,%20java.net.SocketAddress)\\r\\n\\r\\nfor example: \\r\\n```Failed to send stat java.nio.HeapByteBuffer[pos=0 lim=1500 cap=1500] to host . Only sent 0 bytes out of 107 bytes```\\r\\n\\r\\nthe buffer to be sent has a capacity of 1500 bytes, and current position in the buffer is 0 as expected since we send each buffer independently.\\r\\n\\r\\nin this case, it tried to transmit 107 bytes and `send` returned 0, this datagram doesn't exceed the `ByteBuffer` limit, nor does it exceed the SO_SNDBUF limit, so the only thing I could think of is that it failed to transmit this datagram somewhere down the line.\\r\\n\\r\\nBased on everything I understand on how this works, the only reason that a 0 would be returned here is in case the various underlying buffers in the Kernel/NIC are saturated.\\r\\n\\r\\nWe do see increased counter in the container for `UdpSndbufErrors` stat which seem to roughly correlate with the number of application exceptions we see, but I can't seem to find a clear way to trace this more accurately.\\r\\n\\r\\nWe have compared all the possible tuning points between the two systems running the different kernel versions, and as far as we can tell - they are identical, specifically:\\r\\n```\\r\\n$ sysctl net.ipv4.udp_mem\\r\\nnet.ipv4.udp_mem = 768285\u00091024383\u00091536570\\r\\n```\\r\\nwhich I think the one that plays the biggest role here.\\r\\n\\r\\nKernel 4.19 introduced 2 additional namespaced tunables:\\r\\n```\\r\\nudp_rmem_min\\r\\nudp_rmem_min\\r\\n```\\r\\nhowever, these have the same value on both deployments (4096).\\r\\n\\r\\n@th0m maybe a pointer on how we could use similar approach for the send path?\\r\\n@brendangregg any ideas or suggestions would be greatly appreciated\\r\\n\\r\\nI again apologize for jumping on here with our issue." }, { "_class" : "hudson.model.StringParameterValue", "name" : "ghprbGhRepository", "value" : "iovisor/bcc" }, { "_class" : "hudson.model.StringParameterValue", "name" : "ghprbCredentialsId", "value" : "6d3daf13-69b8-48b1-9c8f-ec5353264113" } ] }, { "_class" : "hudson.plugins.git.GitTagAction" }, { }, { }, { "_class" : "org.jenkinsci.plugins.displayurlapi.actions.RunDisplayAction" } ], "artifacts" : [ ], "building" : False, "description" : "PR #2143: Add udp_rcvbuf_errors.py to...", "displayName" : "#143", "duration" : 1217583, "estimatedDuration" : 7460664, "executor" : None, "fullDisplayName" : "bcc-pr #143", "id" : "143", "keepLog" : False, "number" : 143, "queueId" : 1172, "result" : "SUCCESS", "timestamp" : 1574290815927, "url" : "https://buildbot.iovisor.org/jenkins/job/bcc-pr/143/", "builtOn" : "", "changeSet" : { "_class" : "hudson.plugins.git.GitChangeSetList", "items" : [ ], "kind" : "git" }, "culprits" : [ ], "runs" : [ { "number" : 143, "url" : "https://buildbot.iovisor.org/jenkins/job/bcc-pr/label=fc25/143/" }, { "number" : 143, "url" : "https://buildbot.iovisor.org/jenkins/job/bcc-pr/label=fc26/143/" }, { "number" : 143, "url" : "https://buildbot.iovisor.org/jenkins/job/bcc-pr/label=fc27/143/" }, { "number" : 143, "url" : "https://buildbot.iovisor.org/jenkins/job/bcc-pr/label=fc28/143/" }, { "number" : 143, "url" : "https://buildbot.iovisor.org/jenkins/job/bcc-pr/label=ubuntu1604/143/" }, { "number" : 143, "url" : "https://buildbot.iovisor.org/jenkins/job/bcc-pr/label=ubuntu1710/143/" }, { "number" : 143, "url" : "https://buildbot.iovisor.org/jenkins/job/bcc-pr/label=ubuntu1804/143/" } ] }