Uploaded image for project: 'SonarQube'
  1. SonarQube
  2. SONAR-7311

Under high CE load, SQ leaks open files to CE logs files

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 5.4
    • Component/s: None
    • Labels:
      None

      Description

      Deployment of SQ 5.4-RC1 on Nemo lead to a drastic improvement in the speed the CE processes developers refresh.
      This also lead to the following error which caused the unexpected stop of SQ on Nemo:

      2016.02.09 17:49:53 ERROR web[o.a.t.u.n.NioEndpoint] Socket accept failed
      java.io.IOException: Too many open files
              at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) ~[na:1.8.0_71]
              at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422) ~[na:1.8.0_71]
              at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250) ~[na:1.8.0_71]
              at org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:682) ~[tomcat-embed-core-8.0.30.jar:8.0.30]
              at java.lang.Thread.run(Thread.java:745) [na:1.8.0_71]
      

      Investigation by the ops team identified a huge number of open files on CE logs files (see INFRA-1375).

      sudo lsof -p 23446 | wc -l
      4427
      

      About all of them are related to CE

      java    23446 root 4078w   REG             202,81     1340 2492780 /vol/sonar/sonarqube-5.4-RC1/data/ce/logs/DEV_REFRESH/AVIeo3bvKL44VIxfFrJt/AVLG1QLe6jaGX2q3DtIW.log
      java    23446 root 4079w   REG             202,81     1357 2492782 /vol/sonar/sonarqube-5.4-RC1/data/ce/logs/DEV_REFRESH/AVIckUGhKL44VIxfFc4X/AVLG1QLe6jaGX2q3DtIX.log
      java    23446 root 4080w   REG             202,81     1369 2492784 /vol/sonar/sonarqube-5.4-RC1/data/ce/logs/DEV_REFRESH/AVIgoQEHKL44VIxfF5en/AVLG1QLe6jaGX2q3DtIY.log
      java    23446 root 4081w   REG             202,81     1392 2492786 /vol/sonar/sonarqube-5.4-RC1/data/ce/logs/DEV_REFRESH/0e512cad-6423-4a9c-9966-5090527f43db/AVLG1QLe6jaGX2q3DtIZ.log
      java    23446 root 4082w   REG             202,81     1382 2492788 /vol/sonar/sonarqube-5.4-RC1/data/ce/logs/DEV_REFRESH/AVIdwxxRKL44VIxfFmgO/AVLG1QLe6jaGX2q3DtIa.log
      java    23446 root 4083w   REG             202,81     1361 2492790 /vol/sonar/sonarqube-5.4-RC1/data/ce/logs/DEV_REFRESH/AVIfKkDMKL44VIxfFro_/AVLG1QLe6jaGX2q3DtIb.log
      java    23446 root 4084w   REG             202,81     1371 2492794 /vol/sonar/sonarqube-5.4-RC1/data/ce/logs/DEV_REFRESH/febc29b4-a07e-4f36-8f39-070031540183/AVLG1QLe6jaGX2q3DtId.log
      java    23446 root 4085w   REG             202,81     1363 2492796 /vol/sonar/sonarqube-5.4-RC1/data/ce/logs/DEV_REFRESH/AVIdYPfpKL44VIxfFjcR/AVLG1QLe6jaGX2q3DtIe.log
      java    23446 root 4086w   REG             202,81     1354 2492798 /vol/sonar/sonarqube-5.4-RC1/data/ce/logs/DEV_REFRESH/f5fdac3c-b9b1-4772-b07e-c7bbb976ae3d/AVLG1QLe6jaGX2q3DtIf.log
      java    23446 root 4087w   REG             202,81     1354 2492800 /vol/sonar/sonarqube-5.4-RC1/data/ce/logs/DEV_REFRESH/AVIdpc4xKL44VIxfFl-M/AVLG1QLe6jaGX2q3DtIg.log
      java    23446 root 4088w   REG             202,81     1433 2492802 /vol/sonar/sonarqube-5.4-RC1/data/ce/logs/DEV_REFRESH/4a2dc044-f52f-4a95-83cd-c8ed55162b4c/AVLG1QLe6jaGX2q3DtIh.log
      java    23446 root 4089w   REG             202,81     1365 2492804 /vol/sonar/sonarqube-5.4-RC1/data/ce/logs/DEV_REFRESH/AVIjntOQgO41Dm3GpYYo/AVLG1QLe6jaGX2q3DtIi.log
      java    23446 root 4090w   REG             202,81     1358 2492806 /vol/sonar/sonarqube-5.4-RC1/data/ce/logs/DEV_REFRESH/AVIjntPLgO41Dm3GpYYq/AVLG1QLe6jaGX2q3DtIj.log
      java    23446 root 4091w   REG             202,81     1445 2492808 /vol/sonar/sonarqube-5.4-RC1/data/ce/logs/DEV_REFRESH/36980b12-3a97-430d-bd9f-531a68645308/AVLG1QLe6jaGX2q3DtIk.log
      java    23446 root 4092w   REG             202,81     1365 2492810 /vol/sonar/sonarqube-5.4-RC1/data/ce/logs/DEV_REFRESH/AVJcmKLwK5v7qBQevbcT/AVLG1QLe6jaGX2q3DtIl.log
      java    23446 root 4093w   REG             202,81     1355 2492812 /vol/sonar/sonarqube-5.4-RC1/data/ce/logs/DEV_REFRESH/AVIdYMamKL44VIxfFjWQ/AVLG1QLe6jaGX2q3DtIm.log
      java    23446 root 4094w   REG             202,81     1346 2492814 /vol/sonar/sonarqube-5.4-RC1/data/ce/logs/DEV_REFRESH/AVIgLXh1KL44VIxfF1pG/AVLG1QLe6jaGX2q3DtIn.log
      java    23446 root 4095w   REG             202,81     1460 2492816 /vol/sonar/sonarqube-5.4-RC1/data/ce/logs/DEV_REFRESH/AVIdXSKXKL44VIxfFid7/AVLG1QLe6jaGX2q3DtIo.log
      

      This indicates a leak of opened files related to the logback appender used to log CE's tasks logs into specific files.

        Attachments

          Activity

            People

            • Assignee:
              thomas.verin Thomas VĂ©rin
              Reporter:
              sebastien.lesaint Sebastien Lesaint
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Due:
                Created:
                Updated:
                Resolved: