How to keep track of broken releases?

Question

I spent considerable time googling around, but found nothing that would answer my question, so here it is.

If you do a canary release, and you learn the the release is somehow broken: how to you denote that a certain tag is broken and should not be deployed anywhere else, but removed from your productions ASAP?

I had several ideas:

Using GitLab, I could add something like **broken** to tag release note. Deploy jobs would call GitLab API to ensure that the tag I want to deploy is not annotated as broken.
I could have a text file in repo, where broken tags would be listed. In this case deploy job checks if the tag you want to deploy is not listed in the file.
Do no save information about release problems: just deploy a new stable, patched version to production and never look back.

How do/would you handle this? Any tips or hyperlinks are welcome.

Alexander Pushkarev · Answer 1 · 2020-04-26T17:28:53.730

2

I am not sure about other approaches, but the one that I use is a modification of #3.

There's usually a classification of releases (for example think of Ubuntu LTS and regular release).

For the regular release, I would just release a new version and add a new tag.
For LTS/Stable releases I would delete a tag pointing to a faulty release or change a commit this tag points to

edited Apr 26 '20 at 17:28

answered Apr 26 '20 at 13:38

Alexander Pushkarev

171
5

is understandable and seems like it resolves the problem without introducing complexity. As for 2: I heard from some senior people I should never remove tags, because changing/deleting history usually does more harm than good.

pjboro

Apr 26 '20 at 14:24

@pjboro I would not consider tag a history - it is just a pointer to a commit. You may also consider changing the commit a tag points to – Alexander Pushkarev Apr 26 '20 at 17:28

score 2 · Answer 2 · answered Apr 26 '20 at 15:53

2

I'm building a product to solve this problem among others - https://relizahub.com (note it's pretty early stage at the moment).

Idea goes as following - for all your releases you stream their details to Reliza Hub. You would then have a client that would pick up releases you need based on combination of tags, approvals, etc. So it is somewhat similar to your #1 approach. For broken releases you either do not approve them or mark them as 'REJECTED' and this would stop them from propagating in your pipeline.

answered Apr 26 '20 at 15:53

taleodor

867
6
10

GitLab seems like enough right now, but... I'll read on the docs on relizahub :) – pjboro Apr 26 '20 at 22:14
@pjboro Sounds good :) Reliza Hub is at a very early stage right now and I'm very open to talk personally to solve specific use cases - feel free to connect via my linkedin: https://www.linkedin.com/in/pshukhman/ – taleodor Apr 26 '20 at 22:42

score 1 · Answer 3 · answered May 09 '20 at 13:11

I think that it might help to distinguish between a "release" and a "release candidate". If you really have shipped a version of software to other people and it is 'totally broken' then something went really very badly wrong. To avoid this opensource projects have milestone release and snapshot release (aka "nighly releases" or "freshly releases"). These try to flush out critical failures through pre-release public testing. Once the risk of shipping something totally broken has passed then the software is tagged as the final release. After that, if there are bugs, and there always are bugs, then it is a matter of issuing advisories and warning folks about bad old versions within release notes.

For example, spring.io releases are versioned something like "5.2.5.RELEASE". We might expect that they have a "5.2" branch and that they had many release candidate builds on that branch. In fact they have actually have two special repos for milestone releases and snapshot releases as described here. These approximate to beta releases and alpha releases.

If you are not releasing software to other people to run, but are running the software as a service yourself, then your option 3 may work well:

(3) Do no save information about release problems: just deploy a new stable, patched version to production and never look back.

This approach can work well if you are a building a web-app or mobile backend based product. The internal software at large firms like global banks will typically use this approach. In fact, I have worked on many systems at both banks and startups over two decades and we only ever used this approach in practice.

I should mention that your Git server might have a "latest release feature". Both GitHub and GitHub Enterprise have this. Even if you release via a build pipeline to some artefact repository it can still be helpful to make a "release" within the Git server so that an API call to something like GET /repos/:owner/:repo/releases/latest API will return details of the latest version including the git tag that should match the release version. Your deployment piplien can then pull the corresponding prebuilt code from the artefact server to deploy.

How to keep track of broken releases?

3 Answers3