14

If you ever been bitten by a plugin upgrade that broke some functionality you must have given some thought about this problem: What should be the Jenkins plugins upgrade policy? How do you test changes before deploying them?

Has anyone gone as far as having a test instance running dummy jobs to test the new versions or do you just pray that upgrading the versions won't break anything?

Pierre.Vriens
  • 7,205
  • 14
  • 37
  • 84
Michael Pereira
  • 651
  • 4
  • 12
  • Do you mean Jeankins' team policy or your (organisation's) policy? – Dan Cornilescu Mar 05 '17 at 04:22
  • I would take a snapshot of the Jenkins node before upgrading and just test it out. In my experience Jenkins has never been a mission critical component. If it's "down" for 15 minutes because some plugin upgrade broke it, it's usually not blocking production in any way, so manual intervention is acceptable. Of course, if that's not the case for you (and Jenkins must be 100% HA), it's not the right approach. – Assaf Lavie Mar 05 '17 at 08:24
  • @DanCornilescu my organisation policy as this is for our internal Jenkins server – Michael Pereira Mar 06 '17 at 19:58
  • @AssafLavie That's highly dependent on the way Jenkins is running: Standalone server, VM, docker container, kubernetes pods (our case). It may not be easy to take a snapshot of the current state to restore it as is.

    In our case we can clone the EBS volume holding the Jenkins data but it's a manual and time-consuming process to restore both the container and data volume to a specific state.

     – Michael Pereira Mar 06 '17 at 20:01
  • Hi @MichaelPereira if either of the two answers below have solved your question please consider accepting it by clicking the check-mark. This indicates to the wider community that you've found a solution and gives some reputation to both the answerer and yourself. There is no obligation to do this. If you don't feel that your question has been answered please do feel free to engage with the authors in the comments. – Richard Slater Apr 04 '17 at 14:15

3 Answers3

4

According to the policies of the company where I work we have dev, preprod and prod environments (on some service dev can be missing). And the path of new version preprod->tests->validation->prod.

In our case jobs in preprod are heavy and complex enough to be sure we do not need to pray when implement in prod :)

Note: We use svn to maintain and deliver configuration. We do not make changes inplace.

Pierre.Vriens
  • 7,205
  • 14
  • 37
  • 84
Romeo Ninov
  • 431
  • 5
  • 16
2

We needed a 100% HA Jenkins environment. we often upgrade plugins / Jenkins itself.

This causes a big headache if the build breaks after the upgrade.

The safest way to sort this is really getting a Demo Jenkins setup. Maybe on the same machine using multiple Tomcat application, you can achieve this cheaper.

What we did is create a separate (Demo)VM and replicated the prod set-up on the Demo VM. Before changing / upgrading anything we would take a snapshot of both VMs. Then we would test the upgrades on Demo VM. If it works good, change it on Prod.

I guess you can look up the community (like SE/SO) if anyone has faced any problems with the plugin you are planning for.

Mihir Deshpande
  • 21
  • 2
0

I'd always manually trigger a re-run or two on at least one recent green (or almost green) label on every relevant project/branch that uses the respective plugin and check that I get the same results. Just to be on the safe side.

Any result discrepancy would need to be investigated to determine it they're caused by the plugin update or not. Maybe a few more re-runs with both the old and new plugins?

Dan Cornilescu
  • 6,730
  • 2
  • 19
  • 44
  • Sure, no problem. – Dan Cornilescu Mar 15 '17 at 20:08
  • From past experience my opinions often aren't that popular, so in general I tend to avoid the spotlight if possible :) I'm also unfamiliar with the mod tools. But I don't mind helping, especially if it is needed - I have high hopes for this site. – Dan Cornilescu Mar 16 '17 at 00:04