What are the disadvantages of automated testing?

Question

There are a number of questions on this site that give plenty of information about the benefits that can be gained from automated testing. But I didn't see anything that represented the other side of the coin: what are the disadvantages? Everything in life is a tradeoff and there are no silver bullets, so surely there must be some valid reasons not to do automated testing. What are they?

Here's a few that I've come up with:

Requires more initial developer time for a given feature
Requires a higher skill level of team members
Increase tooling needs (test runners, frameworks, etc.)
Complex analysis required when a failed test in encountered - is this test obsolete due to my change or is it telling me I made a mistake?

Edit
I should say that I am a huge proponent of automated testing, and I'm not looking to be convinced to do it. I'm looking to understand what the disadvantages are so when I go to my company to make a case for it I don't look like I'm throwing around the next imaginary silver bullet.

Also, I'm explicity not looking for someone to dispute my examples above. I am taking as true that there must be some disadvantages (everything has trade-offs) and I want to understand what those are.

"Complex analysis required..." the test isn't the cause of the failure, it's an indicator. Saying having no tests means no complex failure analysis required is no better than sticking your head in the mud. — P.Brian.Mackey, Oct 27 '11 at 13:04

score 28 · Answer 1 · answered Oct 27 '11 at 14:38

28

Having just started trying automated tests in our team, the biggest disadvantage I've seen is that it's very difficult to apply to legacy code that wasn't designed with automated testing in mind. It would undoubtedly improve our code in the long term, but the level of refactoring needed for automated testing while retaining our sanity is a very high barrier for entry in the short term, meaning we're going to have to be very selective about introducing automated unit testing in order to meet our short term commitments. In other words, it's a lot harder to pay off the credit cards when you're already deep in technical debt.

answered Oct 27 '11 at 14:38

Karl Bielefeldt

147,435

2

As someone who works 80% of a very large legacy code base, I couldn't agree more. However, to mitigate this, I've used some of this in [link]http://www.amazon.com/Working-Effectively-Legacy-Michael-Feathers/dp/0131177052 are worth the look. – DevSolo Apr 06 '12 at 18:27
1

That's a really good book, I got a lot out of it. Three main points, have a go, a bit at a time. Some good tests are better than no tests. Stay in scope, don't refactor everything that needs refactoring at once. Be very clear where the boundaries between testable and untestable code are. Make Sure everyone else knows as well. – Tony Hopkinson Mar 03 '13 at 23:06

score 27 · Accepted Answer · answered Oct 27 '11 at 13:15

You pretty much nailed the most important ones. I have a few minor additions, plus the disadvantage of tests actually succeeding - when you don't really want them to (see below).

Development time: With test-driven development this is already calculated in for unit tests, but you still need integration and system tests, which may need automation code as well. Code written once is usually tested on several later stages.
Skill level: of course, the tools have to be supported. But it's not only your own team. In larger project you may have a separate testing team that writes tests for checking the interfaces between your team's product and other's. So many more people have to have more knowledge.
Tooling needs: you're spot on there. Not much to add to this.
Failed tests: This is the real bugger (for me anyways). There's a bunch of different reasons, each of which can be seen as a disadvantage. And the biggest disadvantage is the time required to decide which of these reasons actually applies to your failed test.
- failed, because of an actual bug. (just for completeness, as this is of course advantageous)
- failed, because your test code has been written with a traditional bug.
- failed, because your test code has been written for an older version of your product and is no longer compatible
- failed, because the requirements have changed and the tested behavior is now deemed 'correct'
Non-failed tests: These are a disadvantage too and can be quite bad. It happens mostly, when you change things and comes close to what Adam answered. If you change something in your product's code, but the test doesn't account for it at all, then it gives you this "false sense of security".

An important aspect of non-failed tests is that a change of requirements can lead earlier behavior to become invalid. If you have decent traceability, the requirement change should be able to be matched to your testcode and you know you can no longer trust that test. Of course, maintaining this traceability is yet another disadvantages. And if you don't, you end up with a test that does not fail, but actually verifies that your product works wrongly. Somewhere down the road this will hit you.. usually when/where you least expect it.
Additional deployment costs: You do not just run unit-tests as a developer on your own machine. With automated tests, you want to execute them on commits from others at some central place to find out when someone broke your work. This is nice, but also needs to be set up and maintained.

On failed tests, if the requirements change causing the current tests to fail, the test passes because the previous implementation is no longer valid, if it didn't fail it would mean the implementation doesnt fit the requirements... — CS01, Oct 27 '11 at 13:18
Case 4 (b) is what test-driven development is all about: you write a failing test, then you extend the product, then you verify that this change makes the test succeed. This protects you against faultily written test that always succeed, or always fail. — Kilian Foth, Oct 27 '11 at 13:39
@Frank thanks for the answer. There's a lot of insight there. I especially appreciated the distinctions of different causes of failed tests. Additional deployment costs is another excellent point. — RationalGeek, Oct 27 '11 at 14:17
The amusing thing I've found is that my bug per LOC ratio is far worse for tests than it is real code. I spend more time finding and fixing testing bugs than real ones. :-) — Brian Knoblauch, Dec 22 '11 at 15:09
failed, because your test code has been written for an older version of your product and is no longer compatible - if your tests are breaking because of this then likely your tests are testing implantation details rather than behavior. CalculateHighestNumber v1 should still return the same result as CalculateHighestNumber v2 — Tom Squires, Mar 03 '13 at 22:46

score 21 · Answer 3 · answered Oct 27 '11 at 19:10

21

Perhaps the most important disadvantage is ... tests are production code. Every test you write adds code to your codebase that needs to be maintained and supported. Failing to do so leads to tests you don't believe the results of, so you have no other choice. Don't get me wrong - I'm a big advocate of automated testing. But everything has a cost, and this is a big one.

answered Oct 27 '11 at 19:10

Ross Patterson

10,395

Good point Ross that is an interesting way of putting it. – RationalGeek Oct 28 '11 at 13:22
Agreed, although in my experience anyway, the time saved due to unit tests instantly finding potential bugs in newly written code (i.e. regression tests) has outweighed this additional maintenance time. – dodgy_coder Oct 31 '11 at 05:22

Adam Houldsworth · Answer 4 · 2011-11-02T10:00:54.160

16

I wouldn't say these are entirely applicable disadvantages, but the few I hit most are:

Time taken to set up the test in a complex enterprise application.
Handling old tests that fail incorrectly, in other words, the system has evolved and now the tests are wrong.
False confidence from patchy or unknown test coverage.

Test coverage that is patchy can lead to a false sense of security. If you do a refactor and use the tests to prove it's validity, what has proven your tests are able to prove this?

The time it takes to create the test is sometimes a problem for us. Our automated testing doesn't just include unit tests, but also use case tests. These tend to be broader and require context.

Of course, my perspective is from an application that is older than it's unit tests.

edited Nov 02 '11 at 10:00

answered Oct 27 '11 at 13:05

Adam Houldsworth

281

The OP already covered time and obsolete code in the question. – P.Brian.Mackey Oct 27 '11 at 13:07
@P.Brian.Mackey actually the time element is subjective. Time taken to code the test is different from time taken to understand what the test requires and code the test correctly. – Adam Houldsworth Oct 27 '11 at 13:07
@AdamHouldsworth Thank you those are some good examples of disadvantages. I hadn't really considered the false confidence angle. – RationalGeek Oct 27 '11 at 14:13

score 10 · Answer 5 · answered Oct 28 '11 at 14:22

10

I'd say the main problem with them is that they can provide a false sense of security. Just because you have unit tests it doesn't mean they're actually doing anything and that includes properly testing the requirements.

In addition automated tests can also include bugs themselves, so that brings the question over whether unit tests need testing themselves so not necessarily achieving anything.

answered Oct 28 '11 at 14:22

billy.bob

6,549
4
30
46

Test Driven Development helps with the first by requiring a failing test before writing the feature code. Now you know that if the feature breaks, the test will break. For the second, complicated test code is a code smell. Again, by writing the test first you can strive to make it simple and put the difficult work into the feature code that fixes the test. – David Harkness Mar 03 '13 at 23:17
1

Code being hard to test is not a code smell. The easiest code to test is a giant chain of function calls masquerading as classes. – Erik Reppen Jun 14 '13 at 23:31

score 5 · Answer 6 · edited Oct 28 '11 at 13:23

Though automation testing has many advantages, it has its own disadvantages too. Some of the disadvantages are:

Proficiency is required to write the automation test scripts.
Debugging the test script is major issue. If any error is present in the test script, sometimes it may lead to deadly consequences.
Test maintenance is costly in case of playback methods. Even though a minor change occurs in the GUI, the test script has to be rerecorded or replaced by a new test script.
Maintenance of test data files is difficult, if the test script tests more screens.

Some of the above disadvantages often subtract from the benefit gained from the automated scripts. Though the automation testing has pros and corns, it is adapted widely all over the world.

thanks. Good points. I edited your post for grammar and formatting. Hope you don't mind. — RationalGeek, Oct 28 '11 at 13:24

score 3 · Answer 7 · edited Apr 13 '17 at 12:18

I recently asked a question about tests in game development - this is BTW how I knew about this one. The answers there pointed some curious, specific disadvantages:

It is costly to do when your code should be highly coupled.
It is difficult to do when you have to be aware of the various hardware platforms, when you should analyze output to the user and the code result only makes sense in a broader context.
UI and UX testing is very hard.
And notably, automated tests can be more expensive and less effective than a bunch of very low-cost (or free) beta testers.

The 4th point makes me remember of some experience of mine. I worked on a very lean, XP-oriented, Scrum managed company where unit tests were highly recommended. However, in its path to a leaner, less bureaucratic style, the company just neglected the construction of a QA team - we had no testers. So frequently the customers found trivial bugs using some systems, even with test coverage of >95%. So I would add another point:

Automated tests may make you feel that QA and testing are not important.

Also, I was thinking those days about documentation and cogitated a hypothesis that may be valid (to a lesser extend) to tests two. I just felt that code evolves so quickly that it is pretty hard to make documentation that follows such a velocity, so it is more valuable to spend time making code readable than writing heavy, easily outdated documentation. (Of course, this does not apply to APIs, but only to internal implementation.) The test suffers a bit from the same problem: it may be too slow to write when compared with the tested code. OTOH, it is a lesser problem because the tests warn they are outdated, while your documentation will stay silent as long as you do not reread it very, very carefully.

Finally, a problem I find sometimes: automated testing may depend upon tools, and those tools may be poorly written. I started a project using XUL some time ago and, man, this is just painful to write unit tests for such platform. I started another application using Objective-C, Cocoa and Xcode 3 and the testing model on it was basically a bunch of workarounds.

I have other experiences about disadvantages of automated testing, but most of them are listed in other answers. Nonetheless, I am a vehement advocate of automated testing. This saved an awful lot of work and headache and I always recommend it by default. I judge those disadvantages are just mere details when compared to the benefices of automated testing. (It is important to always proclaim your faith after you comment heresies to avoid the auto da fé.)

score 3 · Answer 8 · edited Apr 06 '12 at 17:49

3

I want to add one more, a false sense of security.

Beyond very small well defined problem sets, it is not possible to create comprehensive tests. There may and often will still be bugs in our software that the automated tests simply don't test for. When the automated tests pass we all too often assume there are no bugs in the code.

edited Apr 06 '12 at 17:49

gnat

21,213
29
113
291

answered Dec 22 '11 at 14:37

Jim C

893

score 3 · Answer 9 · answered Dec 22 '11 at 19:08

Two that aren't mentioned are:

Length of clock time it may take to run a large test suite

I've been a part of automated QA efforts where it took half a day to run the tests, because the tests were slow. If you're not careful with writing your tests, your test suite could turn out this way too. This doesn't sound like a big deal until your now managing that time, "Oh, I just committed a fix, but it's going to take 4 hours to prove correctness".

Frailty of some test writing methods

Some testing methods (such as automating the UI) seem to break every time you turn around. Especially painful if your script, say, hangs the testing process because it's waiting for a button to appear - but the button has been renamed.

These are both things you can work around, with good testing practice: find ways to keep your test suite fast (even if you have to do tricks like distributing test runs across machines/CPUs). Likewise, some care can be taken to try to avoid frail methods of writing tests.

score 0 · Answer 10 · answered Apr 06 '12 at 17:27

It is hard to convince the management/venture capitalist

testautomation increases upfront cost.
testautomation increases time to market.
the benefit of testautomation comes in the mid- and logterm. fierce competition focuses more on shortterm benefits.

see Market Driven Unit Testing for details.

score -1 · Answer 11 · answered Mar 03 '13 at 15:53

One of the main disadvantages can be overcome by using self-learning tests. In this situation the expected results are all stored as data and updateable with minimal review by test suite user in self-learning mode (show diffs between old expected result and new actual result - update expected if press y). This testsuite learning mode needs to be used with caution so buggy behaviour is not learned as being acceptable.

What are the disadvantages of automated testing?

11 Answers11

Linked