Is there a reason that tests aren't written inline with the code that they test?

Question

I've been reading a bit about Literate Programming recently, and it got me thinking... Well-written tests, especially BDD-style specs can do a better job at explaining what code does than prose does, and have the big advantage of verifying their own accuracy.

I've never seen tests written inline with the code that they test. Is this just because languages don't tend to make it simple to separate application and test code when written in the same source file (and nobody's made it easy), or is there a more principled reason that people separate test code from application code?

Some programming languages like python with doctest allows you to do that. — Simon Bergot, Feb 25 '13 at 13:40
You may feel the BDD-style specs are better than prose at explaining the code, but that doesn't mean the combination of the two isn't better. — JeffO, Feb 25 '13 at 13:49
Half of the arguments here apply to inline documentation as well. — CodesInChaos, Feb 25 '13 at 14:23
@Simon doctests are too simplistic for serious testing, mostly because they aren't designed for it. They were intended for, and excel at, having code examples in the documentation that can be verified automatically. Now, some people use them for unit testing as well, but lately (as in, for the past years) this took a lot of flak, because it tends to end in fragile messes, overly verbose "documentation", and other messes. — , Feb 25 '13 at 14:34
Take into consideration that most IDEs/editors have shortcuts to quickly jump to the test file, if you follow a naming convention. — Hakan Deryal, Feb 25 '13 at 16:51
Design by Contract allows for inline specifications that make testing straightforward. — Fuhrmanator, Feb 25 '13 at 17:20
Determining code coverage of your tests would be at least a little harder, I would think, if the test code was in the same place as the code it was testing. — Mark Allen, Feb 25 '13 at 21:16
I think your premise is somewhat flawed. Other than doctest which Simon already mentioned, most of C++ code has tons of asserts scattered in it. These are, essentially, tests. Unit testing etc. is used for separation of concerns and code reuse (the setup / teardown parts). — TC1, Feb 26 '13 at 07:31
On the earlier days of junit testing the most common pattern was to have a static inner class Test<ClassUnderTest> extends junit.framework.TestCase for tests. The only advantage was to have access to private fields and methods. — duros, Feb 26 '13 at 07:35
Here is another language with an idea similar to doctest: D. It has unit tests as a built-in feature. — Elias Zamaria, Feb 26 '13 at 17:38
The language being used to test may be different from the language used in the application. e.g. It is natural to test Java with Scala or Groovy. — Michael Easter, May 03 '13 at 18:59
Most of the answers are pretty negative, but this is still a neat idea. I'm thinking about what it would take to do something like this in Ruby. — Chris Pitman, May 03 '13 at 21:13
@ChrisPitman Agreed. I suspect the negative responses are partially caused by resistance to change :) The only way to effectively test this idea would be to try it and see its actual pros and cons, instead of arguing from a purely theoretical (and dogmatic) point of view. — Andres F., May 05 '13 at 15:38
I put my tests inline. The main advantage is that unit tests are less likely to go stale. — Pete Alvin, May 28 '18 at 10:45

score 87 · Accepted Answer · edited May 03 '13 at 18:47

87

The only advantage I can think of for inline tests would be reducing the number of files to be written. With modern IDEs this really isn't that big a deal.

There are, however, a number of obvious drawbacks to inline testing:

It violates separation of concerns. This may be debatable, but to me testing functionality is a different responsibility than implementing it.
You'd either have to introduce new language features to distinguish between tests/implementation, or you'd risk blurring the line between the two.
Larger source files are harder to work with: harder to read, harder to understand, you're more likely to have to deal with source control conflicts.
I think it would make it harder to put your "tester" hat on, so to speak. If you're looking at the implementation details, you'll be more tempted to skip implementing certain tests.

edited May 03 '13 at 18:47

samthebrand

368
2
12
27

answered Feb 25 '13 at 14:15

vaughandroid

7,599

9

That's interesting. I guess the advantage I can see is that when you've got your "coder" hat on, you want to be thinking about tests, but it's a good point that the reverse is not true. – Chris Devereux Feb 25 '13 at 14:24
2

Along these lines, it's possible (and perhaps desirable) to have one person creating the tests and a second actually implementing the code. Putting the tests inline makes this more difficult. – Jim Nutt Feb 25 '13 at 16:07
Agree with the "hats" point especially... TDD using red/green (stop and go) approach, usually means toggling between units on purpose, so that you can only see the interface (not the implementation) that you are testing or implementing, not both. Putting both in one file would be a Mess. – Warren P Feb 25 '13 at 18:13
7

would downvote if i could. How is this an answer in any way at all? Implementers not writing tests? People skipping tests if they look at implementation details? "Just too hard" Conflicts on big files?? ANd how in any way could a test be confused with an implementation detail??? – bharal Feb 26 '13 at 11:15
@bharal erm... because if you're using something like the gtest system, your test is some well-written c++ code, and your implementation is some well-written c++ code. You have to drop down a level when scanning to tell which is which. – deworde Feb 26 '13 at 11:32
7

@bharal Also, wrt to "Just too hard", masochism is a fool's virtue. I want everything to be easy except for the problem I'm actually trying to solve. – deworde Feb 26 '13 at 11:33
1

@bharal I've given a list of reasons why I think it's a bad idea; I don't think it's impossible but deworde has hit the nail on the head when he says "masochism is a fool's virtue". If you disagree, I encourage you to post your own answer! – vaughandroid Feb 26 '13 at 14:49
1

The answer i'd probably give would be code clutter, which isn't much of an answer. I'm not sure about the machismo thing - i'm not arguing for build conflicts. On the other hand, i honestly don't think that keeping files small because of a fear of code conflicts is reason enough. – bharal Feb 26 '13 at 15:36
1

On the other hand, while i agree with your "separation of concerns" point, i agree from the point of view of minimising code clutter, not wrt having a developer write code but another test it. I still don't think that point 4. is valid - people look at implementation details all the time to write tests. That is, in fact, the heart of TDD. – bharal Feb 26 '13 at 15:39
@bharal Smaller files is something you're always looking for, Baqueta mentioned only one benefit of smaller files, there are many others. You'd be better off pointing to why tests should be on the same file instead of complaining about how big files are no big deal... – Ruan Mendes May 03 '13 at 21:31
1

The problem with "separation of concerns" is that it has blurry limits. What about annotations in Java? Modern Java development has moved away from xml config files and tends to put config as annotations. Annotations are in practice "a separate language". Isn't that breaking this principle? And if you accept it, why not inline tests? – Andres F. May 04 '13 at 22:44
1

Very strong disagree. Testing code is the only way you know if what you've written actually works -- especially when you're modifying existing code. If you don't have an existing base of tests, it's virtually impossible in a non-trivial code base to know if what you just did broke anything. – xaxxon May 04 '13 at 22:46
3

Unit test can be considered documentation. That suggests unit tests should be included in the code for the same reason as comments - to improve readability. However, the problem with that is there tend to be a lot of unit tests, and a lot of test-implementation overhead that doesn't specify expected results. Even comments within code should be kept succinct, with larger explanations moved out of the way - to a comment block outside the function, to a separate file, or maybe to a design document. Unit tests are IMO rarely if ever short enough to keep in the tested code like comments. – May 05 '13 at 01:11
1

@ChrisDevereux the opposiite might be true, too. While you code, tests flow out naturally out of your mind, like: "oh, this one here could turn out wrong this way". Getting the "tester" cap on, instead means doing a lot of backtracking. – ZJR May 05 '13 at 14:21
2

@JuanMendes i disagree that "smaller files are something you're always looking for". What you're really looking for is "clear, easily maintainable code". The size of the file, then, is traditionally linked to this goal. BUT note that whether in one file or in 5000, you have the same amount of lines of code. There are times when i'd rather have one large file (spring config files, anybody?) and in the case of embedding test with prod, it might even make sense. But as others have noted, my main gripe is divvying coding from testing. – bharal May 08 '13 at 14:07
Another reason for separating tests is that you often use additional or even different libraries for testing than for the actual implementation. If you mix tests and implementation, accidential usage of test libraries in the implementation cannot be caught by the compiler. Also, tests tend to have way more code lines than the implementation parts they test, so you'll have trouble finding the implementation between the tests. :-) – Dr. Hans-Peter Störr Jun 04 '13 at 08:50
+1 for just mentioning separation of concerns, enough said. – Songo Jun 04 '13 at 09:32
I happen to use this answer as an example of 1) not thinking outside the box, 2) confirmation bias, 3) breaking the rule of "5 whys" and not least of all 4) doesn't pass the negation test. – zumalifeguard Dec 11 '15 at 03:43

Stephen C · Answer 2 · 2013-02-26T00:11:19.497

37

I can think of some:

Readability. Interspersing "real" code and tests will make it harder to read the real code.
Code bloat. Mixing "real" code and test code into the same files / classes / whatever is likely to result in larger compiled files, etc. This is particularly important for languages with late binding.
You may not want your customers / clients to see your test code. (I don't like this reason ... but if you are working on a closed source project, the test code is unlikely to help the customer anyway.)

Now there are possible workarounds for each of these issue. But IMO, it is simpler not to go there in the first place.

It is worth observing that in the early days, Java programmers used to do this kind of thing; e.g. including a main(...) method in a class to facilitate testing. This idea has almost completely disappeared. It is industry practice to implement tests separately using a test framework of some kind.

It is also worth observing that Literate Programming (as conceived by Knuth) has never caught on in the software engineering industry.

edited Feb 26 '13 at 00:11

answered Feb 25 '13 at 13:38

Stephen C

25,178

4

+1 Readability issues - test code might be proportionally greater to the implementation code, especially in OO designs. – Fuhrmanator Feb 25 '13 at 17:22
2

+1 for pointing out using test frameworks. I can't imagine using a good test framework concurrently with production code. – joshin4colours Feb 25 '13 at 17:23
1

RE: You may not want your customers / clients to see your test code. (I don't like this reason ... but if you are working on a closed source project, the test code is unlikely to help the customer anyway.) -- It may be desirable to run the tests on the clients machine. Running the tests may help to quickly identify what the issue is and to id differences in the clients env.. – sixtyfootersdude Feb 25 '13 at 18:24
1

@sixtyfootersdude - that is a pretty unusual situation. And assuming that you were developing closed source, you wouldn't want to include your tests in your standard binary distro just in case. (You would create a separate bundle containing the tests that you want the customer to run.) – Stephen C Feb 26 '13 at 00:02
I really don't like these types of answers because it's basically saying, "don't do it because it's not done". What everyone happens to be wrong and this is the right way? Presume for a second that that is the case and now apply critical thought to justify what that cannot be the case. – zumalifeguard Dec 11 '15 at 03:47
1
Did you miss the first part of my answer where I gave three actual reasons? There was some "critical thought" involved there .... 2) Did you miss the second part where I said that Java programmers used to do this, but they don't now? And the obvious implication that programmers stopped doing this ... for good reason?

Stephen C

Dec 11 '15 at 12:39

Caleb · Answer 3 · 2013-02-25T19:46:19.087

14

For many of the same reasons that you try to avoid tight coupling between classes in your code, it's also a good idea to avoid unnecessary coupling between tests and code.

Creation: Tests and code may be written at different times, by different people.

Control: If tests are used to specify requirements, you'd certainly want them to be subject to different rules about who can change them and when than the actual code is.

Reusability: If you put the tests inline, you can't use them with another piece of code.

Imagine that you've got a chunk of code that does the job correctly, but leaves a lot to be desired in terms of performance, maintainability, whatever. You decide to replace that code with new and improved code. Using the same set of tests can help you verify that the new code produces the same results as the old code.

Selectability: Keeping the tests separate from the code makes it easier to choose which tests you want to run.

For example, you might have a small suite of tests that relate just to the code you're currently working on, and a larger suite that tests the entire project.

edited Feb 25 '13 at 19:46

answered Feb 25 '13 at 19:03

Caleb

39,147
8
95
152

I'm puzzled about your reasons: TDD already says test creation happens before (or at the same time) as production code, and must be done by the same coder! They also hint that tests are pretty much like requirements. Of course, these objections don't apply if you don't subscribe to the TDD dogma (which would be acceptable, but you must make it clear!). Also, what exactly is a "reusable" test? Aren't tests, by definition, specific to the code they test? – Andres F. May 05 '13 at 15:32
1

@AndresF. No, tests are not specific to the code they test; they're specific to the behavior that they test for. So, say you've got a Widget module complete with a set of tests that verify that Widget is behaving correctly. Your colleague comes up with BetterWidget, which purports to do the same thing as Widget but three times faster. If the tests for Widget are embedded in the Widget's source code in the same way that Literate Programming embeds documentation within source code, you can't very well apply those tests to BetterWidget to verify that it behaves the same was that Widget does. – Caleb May 06 '13 at 03:54
@AndresF. no need to specify you do not follow TDD. it's not a cosmic default. As for the reuse point. When testing a system you care about the inputs and outputs, not the internals. When you then need to create a new system that behaves just like it but is implemented differently it's great to have tests that you can run on both the old and the new system. this happened to me more than once, sometimes you need to work on the new system while the old is still in prod or even run them side by side. look at the way Facebook was testing 'react fiber' with the react tests to get to parity. – user1852503 Aug 19 '17 at 22:32

score 14 · Answer 4 · answered Mar 21 '13 at 18:43

Actually, you can think of Design By Contract as doing this. The problem is most programming languages don't let you write code like this :( It's very easy to test for preconditions by hand, but the post conditions are a real challenge without changing the way you write code (a huge negative IMO).

Michael Feathers has a presentation about this and this is one of the many ways he mentions you can improve code quality.

score 11 · Answer 5 · answered Feb 25 '13 at 14:14

Here are some additional reasons I can think of:

having tests in a separate library makes it easier to link only that library against your testing framework, and not your production code (this could be avoided by some preprocessor, but why to build such a thing when the easier solution is to write the tests in a separate place)
tests of a function, a class, a library are typically written from a "users" point of view (a user of that function/class/library). Such "using code" is typically written in a separate file or library, and a test may be clearer or "more realistic" if it mimics that situation.

score 6 · Answer 6 · edited May 04 '13 at 21:06

This is in response to a large number of comments suggesting that inline tests are not done because it's difficult to impossible to remove the test code from release builds. This is untrue. Almost all compilers and assemblers support this already, with compiled languages, such as C, C++, C#, this is done with what are called compiler directives.

In the case of c# ( I believe c++ as well, the syntax might've slightly different depending on what compiler you are using) this is how you can do it.

#define DEBUG //  = true if c++ code
#define TEST /* can also be defined in the make file for c++ or project file for c# and applies to all associated .cs/.cpp files */

//somewhere in your code
#if DEBUG
// debug only code
#elif TEST
// test only code
#endif

Because this uses compiler directives, the code will not exist in the executable files that are built if the flags are not set. This is also how you make "write once , compile twice" programs for multiple platforms/hardware.

commenter8 · Answer 7 · 2013-05-04T17:06:42.063

This idea simply amounts to a "Self_Test" method within the context of an object-based or object-oriented design. If using a compiled object-based language like Ada, all the self-test code will be marked by the compiler as unused (never invoked) during the production compilation, and therefore it will all be optimized away - none of it will appear in the resulting executable.

Using a "Self_Test" method is an extremely good idea, and if programmers were really concerned with quality they would all be be doing it. One important issue, though, is that the "Self_Test" method needs to have intense discipline, in that it cannot access any of the implementation details and must instead rely only on all the other published methods within the object's specification. Obviously, if the self test fails, the implementation will need to change. The self test should be rigorously testing all the published properties of the object's methods, but never ever relying in any way upon any details of any specific implementation.

Object-based and object-oriented languages frequently do provide exactly that type of discipline with respect to methods external to the tested object (they enforce the object's specification, preventing any access to its implementation details and raising a compilation error if any such attempt is detected). But the object's own internal methods are all given complete access to every implementation detail. So the self test method is in a unique situation: it needs to be an internal method because of its nature (self test is obviously a method of the object being tested), yet it needs to receive all the compiler discipline of an external method (it has to be independent of the object's implementation details). Few if any programming languages provide the capability to discipline an object's internal method as if it were an external method. So this is an important programming language design issue.

In the absence of proper programming language support, the best way to do it is to create a companion object. In other words, for every object you code (let's call it "Big_Object"), you also create a second, companion object whose name consists of a standard suffix concatenated with the name of the "real" object (in this case, "Big_Object_Self_Test"), and whose specification consists of a single method ("Big_Object_Self_Test.Self_Test (This_Big_Object : Big_Object) return Boolean;"). The companion object will then depend upon the specification of the main object, and the compiler will fully enforce all of the discipline of that specification against the companion object's implementation.

score 4 · Answer 8 · edited Feb 25 '13 at 17:26

4

If tests were inline, it would be necessary to remove the code you need for testing when you ship the product to your customer. So an extra place where you store your tests simply separates between the code you need and the code your customer needs.

edited Feb 25 '13 at 17:26

Fuhrmanator

1,435

answered Feb 25 '13 at 13:38

mhr

660
6
18

9

Not impossible. It would require an additional preprocessing phase, just as LP does. It could be done easily in C, or a compile-to-js language, for example. – Chris Devereux Feb 25 '13 at 13:44
+1 for pointing that out to me. I've edited my answer to represent that. – mhr Feb 25 '13 at 15:23
There's also an assumption that code size matters in every case. Just because it matters in some cases doesn't mean it matters in all cases. There are plenty of environments where programmers are not driven to optimize source code size. If that were the case, they wouldn't be creating so many classes. – zumalifeguard Dec 11 '15 at 03:49

score 3 · Answer 9 · edited May 05 '13 at 13:52

We use inline tests with our Perl code. There's a module, Test::Inline, that generates test files from the inline code.

I'm not particularly good at organizing my tests, and have found them easier and more likely to be maintained when inlined.

Responding to a couple of the concerns raised:

The inlined tests are written in POD sections, so they're not part of the actual code. They are ignored by the interpreter, so there's no code bloat.
We use Vim folding to hide the test sections. The only thing you see is a single line above each method being tested like +-- 33 lines: #test----. When you want to work with the test, you just expand it.
The Test::Inline module "compiles" the tests to normal TAP-compatible files, so they can co-exist with traditional tests.

For reference:

POD
TAP

score 2 · Answer 10 · answered May 04 '13 at 19:38

2

Erlang 2 does actually support inline tests. Any boolean expression in the code which is not used (e.g. assigned to a variable or passed) is automatically treated as a test and evaluated by the compiler; if the expression is false, the code doesn't compile.

answered May 04 '13 at 19:38

Mark Rendle

201

rust too! https://doc.rust-lang.org/1.4.0/book/testing.html and you can move the tests into its own test-block and keep it at the bottom of your source file too, like this https://doc.rust-lang.org/stable/rust-by-example/testing/unit_testing.html – Simon B. May 05 '20 at 01:36

score 1 · Answer 11 · answered Jun 04 '13 at 08:52

Another reason for separating tests is that you often use additional or even different libraries for testing than for the actual implementation. If you mix tests and implementation, accidential usage of test libraries in the implementation cannot be caught by the compiler.

Also, tests tend to have way more code lines than the implementation parts they test, so you'll have trouble finding the implementation between all the tests. :-)

score 0 · Answer 12 · answered May 06 '16 at 21:07

This isn't true. It's much better to place your unit-tests along side the production code when the production code especially when the production routine is pure.

If you're developing under .NET for instance, you could put your test code in the production assembly, and then use Scalpel to remove them before you ship.

Is there a reason that tests aren't written inline with the code that they test?

12 Answers12