Rebooting node at end of Jenkins pipeline



  • I have a pipeline that runs on a dedicated bare metal node for automated performance benchmarking. Using bare metal is necessary due to the nature of the project.

    Near the end of the pipeline, the results are sent to users (simply seeing a pass or fail status is not sufficient).

    To provide each instance of the pipeline a clean slate, the node is rebooted at the end of the pipeline. This makes the pipeline appear stuck in Jenkins. Several minutes after Jenkins is able to reconnect to the node, Jenkins realizes the pipeline that was running ended abruptly, and marks it failed:

    + sudo reboot
    Cannot contact baremetal: hudson.remoting.ChannelClosedException: Channel "hudson.remoting.Channel@35ca6b21: baremetal": Remote call on baremetal failed. The channel is closing down or has closed down
    wrapper script does not seem to be touching the log file in /home/jenkins/workspace/bare-metal-benchmarks@tmp/durable-e96e9611
    (JENKINS-48300: if on an extremely laggy filesystem, consider -Dorg.jenkinsci.plugins.durabletask.BourneShellScript.HEARTBEAT_CHECK_INTERVAL=86400)
    [Pipeline] }
    [Pipeline] // node
    [Pipeline] End of Pipeline
    ERROR: script returned exit code -1
    Setting status of *** to FAILURE with url *** and message: ' '
    Using context: Bare Metal Benchmarks
    Finished: FAILURE
    

    The next instance of this pipeline can then start running on the rebooted bare metal node.

    Is there a way to safely reboot a node at the end of a pipeline so Jenkins does not mark the pipeline status as failed?

    The basic structure of the scripted pipeline is:

    node('baremetal') {
        checkout scm
    
    try { 
        stage('Benchmarks') {
            // run benchmarks
        }
    
        stage('Process data') {
            // process data and send results to users
        }
    }
    finally {
        stage('Clean up') {
            sh 'sudo reboot'
        }
    }
    

    }



  • Quick answer:

    Changing sudo reboot to sudo shutdown -r +m5 or what ever time delay needed, would prevent the pipeline from failing. It forks the reboot process to the system and lets the Jenkins agent finish this pipeline cleanly.

    Longer (preventing potential conflicts answer):

    Depending on how the system is configured, it should prevent new logins to the server while the reboot is scheduled.

    If the agents are configured to launch via ssh,

    launch agents via ssh

    and configured to be brought offline if not in use,

    take offline when idle

    Provided the agent times out, Jenkins should be unable to start a new pipeline on that agent until the reboot completes.

    From the man page for shutdown (on one of my servers)

    If the time argument is used, 5 minutes before the system goes down the
    /run/nologin file is created to ensure that further logins shall not be
    allowed.
    



Suggested Topics

  • 2
  • 2
  • 2
  • 2
  • 2
  • 2
  • 2
  • 2
  • 2
  • 2
  • 2
  • 2
  • 2
  • 2
  • 2