11

My boss is planning to use the metrics from our continuous build (builds and runs tests on every commit) as part of our performance reviews (in our 'quality' metric). This seems like a REALLY bad idea to me, but I'd like to know if anyone has studied this or seen this tried before.

My thought is that it's going to have our developers not put in as many tests as they otherwise would, for fear that the tests are going to fail. I feel that he's turning a valuable developer tool into a stick to beat the developers with.

The obvious counter argument is that it will promote people being more careful before they commit, and therefore leading to higher quality.

Am I off base here? Please leave aside the question of whether we should be doing performance reviews at all or not - that's been answered elsewhere.

durron597
  • 7,600
Michael Kohne
  • 10,046
  • 8
    Any system that can be gamed is a horrible input for a performance eval. – Steve Jackson Aug 10 '11 at 18:20
  • Everyone has the option not to test? – JeffO Aug 10 '11 at 18:56
  • 1
    @Steve, And "systems" that can't be gamed give you a tiny narrow view of the bigger picture. To truly accurately track performance would require leg work. – maple_shaft Aug 10 '11 at 19:22
  • 2
    Note that some things work well on developers machines, but fail on the build server (an accidental dependency on an external jar, wrong way to use / and \ on Linux boxes etc). The primary reason for the build server is to catch these things, not to harass anybody for not testing for them first. In other words, this is a bad idea. –  Aug 10 '11 at 19:27
  • @Jeff O - it's not that they won't write tests, just that I'm afraid they won't write as good a test. – Michael Kohne Aug 11 '11 at 03:30
  • @Thorbjørn - that's part of the issue. I've also seen builds fail due to SAN issues on the build server, and we've previously had tests that would fail occasionally due to timing issues in the test code. It really stinks to be the guy whose commit failed the test because a month-old test decided to take a powder that day. – Michael Kohne Aug 11 '11 at 03:33
  • What does your boss say when you mention all these things for him? –  Aug 11 '11 at 06:36
  • @Thorbjørn - For the moment we're computing the stats manually, and ignoring the fails that weren't caused by the commit in question. – Michael Kohne Aug 11 '11 at 22:06
  • @Michael, perhaps if you could twist it so you measure which tests break the most instean of which persons that dó. In this way you Can find the brittle tests and focus on fixing Them. –  Aug 13 '11 at 10:16
  • 1
    Follow up: After we started doing this I found that the biggest problem had nothing to do with the other engineers and willingness to write proper tests, but rather with the fact that our existing tests were REALLY unstable, so every commit had a pretty large chance of breaking through no fault of the person doing the commit. This factor deadened everyone's enthusiasm for testing far more than any effects of the performance reviewing. – Michael Kohne Dec 11 '11 at 03:07

2 Answers2

7

Performance reviews are fine but how about useful metrics like:

  • Percentage of Unit Test coverage on features
  • Ability to meet deadlines
  • Clear and concise documentation
  • Follow proper coding conventions
  • Communicates well with others
  • Ability to turn requirements and user stories into tasks

These are all good ways to measure performance but the problems that management seem to have with these are that they actually require... ummm.. well ya know... ACTUAL WORK on their part.

Unfortunately most management has the attitude, "To hell with that, I want to judge my employees on metrics that don't actually require me to keep up with what they are doing."

maple_shaft
  • 26,511
  • 11
  • 57
  • 132
3

Gaming the system here is quite likely, in my opinion, and your boss has to find ways to prevent that from being the reality. The other case you didn't mention is where developers commit tons of times so that there is this flood of check-ins where the number of modifications is relatively low as if there is some part of the review where the count of builds is used this is where this becomes a new tool that could be misused rather easily. I'm thinking of check-ins where something is renamed or white space is changed is a check-in and counts as some form of productivity would be the pedantic view.

JB King
  • 16,795