Can I run an XCTest suite multiple times?

Question

Is it possible to have Xcode run your unit tests multiple times?

I had an issue in several unit tests that caused intermittent failures. Now that I think I've fixed it, my only option appears to mash ⌘ + U until I'm 95% confident the bug is gone.

I know other unit testing frameworks make it quite easy to run a single test, test case, or test suite multiple times. Do we have this luxury in XCTest yet?

score 9 · Answer 1 · edited Apr 13 '21 at 11:37

9

For me it works in swift

override func invokeTest() {
    for time in 0...15 {
        print("this test is being invoked: \(time) times")
        super.invokeTest()
    }
}

edited Apr 13 '21 at 11:37

Parth Tamane

2,284
1
13
30

answered Dec 21 '18 at 07:49

Sultan Ali

2,347
26
23

This worked great for me in Xcode 10, but now in 11 not having any luck with it. – David H Feb 02 '20 at 20:28
well, i have tested it and it is working for me in xcode 11 as well – Sultan Ali Mar 04 '20 at 12:12
I tried with 11.3. On 11.4 beta now, will try again. I’ll try something trivial... – David H Mar 04 '20 at 12:28

score 7 · Answer 2 · answered Jun 28 '18 at 20:55

7

Try overriding invoke test: https://developer.apple.com/documentation/xctest/xctestcase/1496282-invoketest?language=objc

- (void)invokeTest
{
    for (int i=0; i<100; i++) {
        [super invokeTest];
    }
}

answered Jun 28 '18 at 20:55

electronix384128

6,377
11
43
65

score 4 · Accepted Answer · answered Jul 03 '15 at 08:00

4

It might help you to use

func testMultiple() {
    self.measureBlock() {
            ...
            XCTAssert(errMessage == nil, "no error expected")        
    }
}

This runs the code inside self.measureBlock() multiple times to measure the average time.

It is work to change the code, but you might want to know the execution time anyways.

This answer might be close enough to what you want and it is easy to do.

answered Jul 03 '15 at 08:00

Gerd Castan

5,601
3
39
78

1

That's true; I could wrap every test in a call to `measureBlock`, but does anyone know how many times the block is executed in `measureBlock`? – T Blank Jul 06 '15 at 16:35
The unit test tells you in the console when you run it. Example: measured [Time, seconds] average: 0.002, relative standard deviation: 154.282%, values: [0.011297, 0.001023, 0.000956, 0.000969, 0.000952, 0.000935, 0.000979, 0.000950, 0.000968, 0.001042], – Gerd Castan Jul 06 '15 at 18:51
What is bad in my answer is that the tests are not mixed randomly. If you have errors that occur from time to time, the sequence of tests might play a role. – Gerd Castan Jul 06 '15 at 18:55
I suppose I could just wrap each individual test in a `measureBlock` block, so they'd still run in a random order. Thanks for looking into this with me; I think this is the best we can do with XCTest. – T Blank Jul 08 '15 at 00:52
Does this call tearDown / setup after each run, I'm assuming not. – gran_profaci Jul 22 '16 at 22:44
Not while you're inside the method – Gerd Castan Jul 22 '16 at 22:52

score 3 · Answer 4 · edited Apr 25 '18 at 05:29

One alternative is to do this via the command line. You can run a single test using the -only-testing argument, and avoid building using test-without-building i.e. (new lines added for clarity)

for i in {1..10}; \
  do xcodebuild \
    test-without-building \
    -workspace MyApp.xcworkspace \
    -scheme Debug \
    -destination 'platform=iOS Simulator,OS=11.2,name=iPhone 8' \
    -only-testing:MyApp.Tests/TestFile/myTest;
done

score 0 · Answer 5 · answered Feb 01 '17 at 21:32

Try using a for loop:

func testMultiple() {
    for _ in 0...100 {
            ...
            XCTAssert(errMessage == nil, "no error expected")        
    }
}

Note this doesn't work within a self.measureBlock(). You'll get an NSInternalConsistencyException: Cannot measure metrics while already measuring metrics

However, you can CALL this within a measureBlock():

func testMultiple() {
    for _ in 0...100 {
            ...
            XCTAssert(errMessage == nil, "no error expected")        
    }
}

func testPerformance() {
    self.measureBlock() {
        self.testMultiple()
    }
}

Xcode 8 runs the measureBlock code 10 times.

score 0 · Answer 6 · answered Feb 03 '20 at 13:41

I had used the invokeTest() override in the past (Xcode 10) with great success. But now in Xcode 11 its not working (for me at least). What I ended up doing was:

func test99Loop() {
    for i in 0..<LOOP_COUNT {
        if i > 0 { tearDown(); sleep(1); setUp() }
        test3NineUrls()
        do { tearDown(); sleep(1) }
        setUp()
        test6NineCombine()

        print("Finished Loop \(i)")
    }
}

I obviously use setup/teardown, and this is the proper way to do those multiple times (since the first and last are called by Xcode).

Can I run an XCTest suite multiple times?

6 Answers6

Linked