What is the purpose of a dedicated build machine?

Question

Due to a number of circumstances leading to a poor deployment last build cycle, I campaigned in our office to perform all future deployments with a dedicated build machine, and my boss accepted this proposal.

However, instead of getting an actual machine in our office to use, we're having to share a single machine with several other groups - and the hassle of having to leave my office with all the necessary information and then walk down a flight of stairs to another office just to perform a simple build is making me wonder why I ever proposed this in the first place.

The idea of having a separate build machine was, originally, to separate my own locally-written code from the code of several other developers, and to separate any hijacked files I had on my machine from deployment. It was also to resolve a growing concern I've had with our ClearCase file management system, which often refuses to let me deploy certain build activities unless I've also included another activity for which it 'has dependencies'.

Now that I'm actually going forward with this process, I'm wondering if I misunderstood the entire purpose of using a build machine - and since we're only using this machine for code deployment to our Test, Staging and Production environments, and not for our personal Developer testing deployments, I'm not sure it serves any purpose at all.

So, what is the actual reason for using a build machine, and have I even come close to using it correctly?

"the hassle of having to leave my office with all the necessary information and then walk down a flight of stairs to another office just to perform a simple build [...]" What do you mean exactly? You physically access that machine to create a build? — Vincent Savard, Mar 03 '17 at 14:07
@Vincent_Savard Admittedly I could remote access the machine, but I'm still a bit confused as to the purpose of using a separate machine at all. — Zibbobz, Mar 03 '17 at 14:18
The real WTF is Clearcase, which is worse than all the modern open source alternatives. What language(s) is this project and how large/complex is the build? — pjc50, Mar 04 '17 at 00:38
Are you using tools? Git/SVN and Jenkins/Team City/Octopus/TFS, etc? Or are you just logging into another computer, loading Visual Studio or what have you... copying project, loading, compiling... Are you using professional tools or doing it manually? — WernerCD, Mar 04 '17 at 00:58
My build machine is somewhere in South Carolina and I am in Seattle. I assure you I do not walk down any set of stairs to use it. I think the last time I had physical access to a build machine was when I was the intern in charge of build machines for Microsoft compilers in 1994, when they fit into a small closet; they are now an entire data center somewhere. Get the machine on your network; better yet, get it in the cloud and make someone else take care of it. — Eric Lippert, Mar 04 '17 at 08:07
And don't let developers have access to it - only the dedicated Jenkins guy — Mawg says reinstate Monica, Mar 06 '17 at 09:35
@VincentSavard The necessary information resides on a stack of 5-1/4 inch floppies that must be fed into the build machine one at a time. — shoover, Mar 06 '17 at 17:18
At my workplace we have a dedicated build machine. This builds every proposed change before it can be accepted. It also builds the complete project from scratch every night. This is all automated through Jenkins and Gitlab. It's shared with two other groups, but so far this hasn't been an issue. One of the main purposes of this machine is as an official source of truth - if it doesn't build on the build machine then someone broke the build, but if it does, then the problem must be on your machine. — user253751, Mar 07 '17 at 07:34
Plus, anything that's automated like that has to have a server - it's hard to imagine a distributed system for automatic scheduled builds that wouldn't duplicate a bunch of work. — user253751, Mar 07 '17 at 07:36

score 136 · Accepted Answer · answered Mar 03 '17 at 14:17

136

Normally you wouldn't just have a dedicated build machine, but also run a build server on that dedicated machine. A dedicated build machine merely offers the advantage of never blocking the work of a developer and deploying from a centralized machine.

A build server offers much more. A build sever allows for CI (continuous integration), meaning that it will automatically build on every push to your VCS (like git), might even execute unit tests if you have them and allows for "one click deployment". Build servers can notify you per mail if builds or tests fail. They offer historic data and trends on what happened.

Build servers can generally be accessed by multiple users or teams at once, by using a web gui that runs in a browser.

In the Java world one of the most used build servers is Jenkins. Jenkins works perfectly fine with C++ builds as well (Since you seem to use those two languages). Jenkins calls itself automation server, since it can run all kinds of tasks that don't have to be related to programming and building.

answered Mar 03 '17 at 14:17

ASA

1,279

3

I actually haven't touched c++ since college, but I can understand the usefulness. Though in this case, I think what we're actually doing might be far removed from the intended purpose. – Zibbobz Mar 03 '17 at 14:22
22

For some languages (C++ in particular) having a dedicated box with more processing power can be useful, too, given it's a relatively slow compilation. – enderland Mar 03 '17 at 15:16
32

Continuous Integration means more than just having a build server -- it also means everyone integrates each others changes as frequently as possible, to avoid "big bang" merge problems. Otherwise, spot-on. – Rob Crawford Mar 03 '17 at 18:00
While I like the power that an example gives to the point being made here, I still think the last paragraph is totally unnecessary in this answer. – Pierre Arlaud Mar 04 '17 at 15:09
3

"A dedicated build machine merely offers the advantage of never blocking the work of a developer and deploying from a centralized machine." Is not totally true. The asker pointed out that it is very easy to have an unclean environment on a developers machine. Included libraries and other environment variables can all change the build result. A dedicated machine should have a well documented environment to allow for easy build recreation. – TafT Mar 06 '17 at 16:45

score 107 · Answer 2 · answered Mar 03 '17 at 14:44

107

In addition to Traubenfuchs' answer, you have hinted at another reason for a build machine in your question.

Just because the software builds on your machine, it doesn't mean that it will build on anyone else's. You may be relying on some random files that just happen to be on your machine (and may not even be under version control). You may be relying on some forgotten application or library that is called up from an obscure build script.

If you have a dedicated build machine, you should know what is installed on it. This should be well documented. If there is ever a need to rebuild the software, perhaps years later, it should only be necessary to create a new build machine with the documented stuff installed on it.

answered Mar 03 '17 at 14:44

Simon B

9,621

37
1 The amount of times something works on a developer's machine, and not the rest of his team's...

user2259716

Mar 03 '17 at 15:31

46

This. The main purpose of building on another machine is to have reproducible builds; notably by eliminating non-committed stuff, disparate environment variables, etc... from the equation. – Matthieu M. Mar 03 '17 at 16:01

9

Extend on that: use a new, clean container image from which you start each build. Then have a bootstrap script setup the system. That truly guarantees reproducible builds. – Matthias Kuhn Mar 03 '17 at 20:12

9

@MatthiasKuhn: truly (byte for byte) reproducible builds require much more, cf: https://reproducible-builds.org/ https://wiki.debian.org/ReproducibleBuilds – ninjalj Mar 05 '17 at 12:04

1

Also, just because the Linux version of the product builds and passes tests on your Linux desktop does not mean that the Windows version or the Mac version will build. – Solomon Slow Mar 06 '17 at 21:59

Truly referred to guarantee and not reproducible /nitpick off. but concerning the topic, the main message is: scripts are more reliable than docs. that counts for build envs as well as for truly reproducible builds ;) – Matthias Kuhn Mar 06 '17 at 23:14

JimmyJames · Answer 3 · 2017-03-03T15:17:17.793

The main reason for having a dedicated build machine is to get consistent builds regardless of who is doing the build. Developer workstations are rarely (read: never) identical. It's hard to know that each build is using the same exact versions of dependencies and compilers etc. One of the worst problems with dev workstation builds is that developers can build from code that is not checked into version control.

It's not clear what platform/language(s) you are using but you ideally should have a build server that pulls directly from source control. That is, when a build is required, it will retrieve the source from a given version of the repository and compile it automatically. This requires the use of automated build tools to script the build. If you don't have this, that should be step #1.

Keep in mind that there is nothing wrong with building locally for development. You should definitely being working locally run unit tests, code quality analysis and for honing build scripts. Otherwise you will be wasting a lot of time. The output of the build server is for anything that you want to potentially move to production. All QA activities such as integration and acceptance testing should be done only with builds from the build server.

@jpmc26: The build machine gets far less software installed on it, needs less remote access points, and nobody is opening web browsers to random internet sites on it. — Joshua, Mar 05 '17 at 16:16

score 19 · Answer 4 · answered Mar 03 '17 at 18:25

The other answers quite correctly noted that you should automate the build, which means it isn't necessary to walk to another office. However, let me propose a certain number of steps you could take to improve your build process:

Firstly, set up remote access to the build server! If you build manually by typing the command "make", this means you don't have to walk to another office anymore to type "make", you can just SSH into the build server and type "make". If you don't use make or a similar build system yet, take such a build system into use.
Secondly, install a continuous integration environment that automatically pulls latest changes from the version control system (you have a version control system, right? if not, that would be an additional step) and builds them. I recommend Jenkins. Set up Jenkins to run your unit tests and system-level integration tests as well (you do have both, right? if not, create them starting from unit tests and then ending into system-level integration tests).
Thirdly, if you find sharing the same machine with other teams problematic (such as if you have different opinions on what operating system and which version and bittiness you should use), consider using virtualization. A good server today can run a huge number of virtual machines. Perhaps you could set yourself up a 32-bit machine and a 64-bit machine so that you know the build works on both architectures.
Lastly, this may not be necessary: if you absolutely must have a dedicated machine, such as if your application performance is of great importance and other builds / test runs running at the same time affect your results too much, install a dedicated hardware server than only you use. However, on recent servers that may have as many as 40 virtual CPU cores or even more, it is relatively straightforward to create a few virtual machines that don't share access to the same CPU cores.

I would consider a shared machine much better than manual builds. My current project uses a virtual machine now, but due to the need for system-level integration performance tests, we're moving to a dedicated server with 40 virtual CPU cores of which the performance tests requires 17.

score 16 · Answer 5 · answered Mar 03 '17 at 15:24

16

... instead of getting an actual machine in our office to use, we're having to share a single machine with several other groups ...

You say that like it's a bad thing.

You now have a common build server that all of your builds - yours and the other teams' - are built through. Consistency of build? Check.

... the hassle of having to leave my office with all the necessary information and then walk down a flight of stairs to another office just to perform a simple build is making me wonder why I ever proposed this in the first place.

You're still performing the build manually and that's not good.

You need a server process to which you submit/ queue requests for builds to be done on your behalf and have that process send you back the result(s).

answered Mar 03 '17 at 15:24

Phill W.

12,181

5

"You say that like it's a bad thing." It could be if they have free reign to install whatever junk they want. If they're remoting in or physically accessing it, I don't see how that could be prevented. – jpmc26 Mar 03 '17 at 21:46
7

"You say that like it's a bad thing. You now have a common build server that all of your builds - yours and the other teams' - are built through. Consistency of build? Check". That's the complete opposite of a consistent build! If any other team decides to update their compiler or any other tool you're suddenly dealing with a completely different build environment. That's pretty much a worst case scenario (apart from not having a build machine to begin with). – Voo Mar 04 '17 at 00:20
1

Agreed on wanting to have a automatic process for creating new builds, but at the same time you also want to virtualize your build agents to make sure your build environment is under your own control. VMs are an amazing tool for dev ops and you should make the most of it. – Voo Mar 04 '17 at 00:23
@Voo It's hardly a worst case scenario, it's a middle case scenario. What the asker currently has is a worst case scenario. (Also note that if you never reproduce builds, random compiler upgrades aren't too much of a problem) – user253751 Mar 07 '17 at 07:37
@immibis You did read the part in parens right after the part you quoted? It is a worst case scenario if a build server is involved. The problem with non reproducible bugs is that you can't really do Hotfixes if your whole build environment changed in the meantime. That isn't much of a problem if you only have a single released version that's kept close to trunk, but in all other cases it's pretty bad. – Voo Mar 07 '17 at 08:43

score 1 · Answer 6 · answered Mar 06 '17 at 14:18

In addition to other relevant answers, it also sounds like you're running your builds directly on the machine in question.

For a reliable build system, especially when sharing the build machine with other users, it's normal to run your builds within a virtual machine. This ensures other users cannot change the behaviour of your builds by installing their own versions of applications or libraries which your code depends on. A great advantage of this is that the VM can easily be backed up, and it can also easily be cloned to any other PC (including your own development machine).

score 1 · Answer 7 · answered Mar 06 '17 at 15:45

It provides a centralized, neutral location to perform builds, independent of the IDE, OS, library configurations of individual developers.

With a dedicated build machine, you can have it rebuild every time there is a code push to the repository. When someone breaks the build, the process can immediately send out an alert so that the problem can be rectified immediately.

In addition to making everything more repeatable and reliable and ensuring the repository isn't full of broken garbage with lurking dependency issues, it makes the lives of developers easier because all they have to do to get the build working on their machine is to copy whatever is being done on the build machine.

this doesn't seem to offer anything substantial over points made and explained in prior 6 answers — gnat, Mar 06 '17 at 16:46

What is the purpose of a dedicated build machine?

7 Answers7