13

I've seen the terms brittle and fragile used to describe poorly written unit tests. But I haven't been able to find a clear definition for what brittle or fragile actually mean in this context.

  • Is it unit tests that break for more than one reason?
  • Is it when a test fails because something in the code base is changed that is supposedly unrelated to the test?
  • Is it tests that test functionality in an inappropriate manner, such as testing how instead of what a function does?
  • Something else entirely?

Or are these terms simply a cover-all for all unit-test smells?

Ayb4btu
  • 303

1 Answers1

23

The terms "brittle" and "fragile" do not have a clear and specific definition within the context of software development. Rather, they are ordinary English words that are sometimes used in software development to describe certain kinds of code.

In the context of software development, these words mean exactly the same thing as they do everywhere else: "brittle" and "fragile" things are things that are easy to break.

That said, there is this "personal reflection essay" from Wikipedia. If you're looking for an "official" definition, this one is as good as any:

Software brittleness is the increased difficulty in fixing older software that may appear reliable, but fails badly when presented with unusual data or altered in a seemingly minor way.

Things that make unit tests brittle:

  1. Asserting against elements in a UI.
  2. Asserting against large result strings instead of discrete values.
  3. Static state or improper state sharing between threads.
  4. Unrealistic or otherwise inadequate test data.
  5. Elaborate mocking schemes.
  6. Brittle code under test.
  7. Too many edge cases or unreasonably high cyclomatic complexity.
  8. Too many responsibilities.
  9. Leaky abstractions.
  10. Code under test is not idempotent between test sessions.
  11. Tests whose results can vary based on the environment, such as performance timings.
  12. Tests that can't run independently of each other.

As you can see, what makes unit tests fragile is often not the fault of the test, but the fault of the code being tested.

One way to have better tests (and by extension, better code), is to practice Test-First Development. Writing your tests first forces you to think about the surface area of your code's API, and strongly encourages you to write that surface area in a way that makes it simpler, more robust, more deterministic and easier to test.

Further Reading
Some Good Advice for Newcomers to Unit Testing

Robert Harvey
  • 199,517
  • What if a class has a static initialized property that throws after second initialization? – Washington Guedes May 10 '21 at 22:09
  • it sounds like a commonality of your examples is that they are tests that fail for reasons other than having identified something broken that needs fixing. so that the failing test itself is a flaw when it's supposed to be helpful. – Dave Cousineau Jul 30 '23 at 23:15
  • @DaveCousineau: Most of the examples have to do with code design. A bad code design can make tests brittle. Only items 1, 4 and 12 have to do specifically with bad tests. – Robert Harvey Jul 31 '23 at 00:02