Spock Framework executes test methods (features) in a single class (specification) in the declaration order. There is nothing wrong in this default behavior - we should write tests with their isolation in mind. However, in some cases, we would like to randomize test methods execution. Today we are going to learn how to do it.

Introduction

Let’s start with a reasonably simple specification that prints a number to the console.

Listing 1. src/test/groovy/com/github/wololock/RandomSpockSpec.groovy
package com.github.wololock

import spock.lang.Specification

class RandomSpockSpec extends Specification {

    def "test 1"() {
        when:
        def number = 1

        then:
        println "[${new Date().format("HH:mm:ss.SSS")}] number ${number}"
    }

    def "test 2"() {
        when:
        def number = 2

        then:
        println "[${new Date().format("HH:mm:ss.SSS")}] number ${number}"
    }

    def "test 3"() {
        when:
        def number = 3

        then:
        println "[${new Date().format("HH:mm:ss.SSS")}] number ${number}"
    }

    def "test 4"() {
        when:
        def number = 4

        then:
        println "[${new Date().format("HH:mm:ss.SSS")}] number ${number}"
    }

    def "test 5"() {
        when:
        def number = 5

        then:
        println "[${new Date().format("HH:mm:ss.SSS")}] number ${number}"
    }
}

When we execute this specification, we get all numbers printed in the ascending order.

spock random tests 01
You can find the source code of this example in the following repository.

Forcing random order

Now let’s try to randomize the execution order. One way to do it is to use Spock’s extensions - an annotation-driven local extensions in this case. Let’s create a new annotation called @RandomizedOrder with the following content.

Listing 2. src/test/groovy/com/github/wololock/RandomizedOrder.groovy
package com.github.wololock

import org.spockframework.runtime.extension.ExtensionAnnotation

import java.lang.annotation.ElementType
import java.lang.annotation.Retention
import java.lang.annotation.RetentionPolicy
import java.lang.annotation.Target

@Retention(RetentionPolicy.RUNTIME)
@Target(ElementType.TYPE)
@ExtensionAnnotation(RandomizedOrderExtension)
@interface RandomizedOrder {}

After creating annotation interface, we can create the extension class. This class implements visitSpecAnnotation interceptor method that gets executed right before specification executes any feature.

Listing 3. src/test/groovy/com/github/wololock/RandomizedOrderExtension.groovy
package com.github.wololock

import org.spockframework.runtime.extension.AbstractAnnotationDrivenExtension
import org.spockframework.runtime.model.SpecInfo

final class RandomizedOrderExtension extends AbstractAnnotationDrivenExtension<RandomizedOrder> {

    public static final String SPOCK_RANDOM_ORDER_SEED = "spock.random.order.seed"

    private static final long seed = System.getProperty(SPOCK_RANDOM_ORDER_SEED)?.toLong() ?: System.nanoTime()

    static {
        println "Random seed used: ${seed}\nYou can re-run the test with predefined seed by passing -D${SPOCK_RANDOM_ORDER_SEED}=${seed}\n\n"
    }

    @Override
    void visitSpecAnnotation(RandomizedOrder annotation, SpecInfo spec) {
        final Random random = new Random(seed) (1)

        final List<Integer> order = (0..(spec.features.size())) as ArrayList (2)

        Collections.shuffle(order, random) (3)

        spec.features.each { feature ->
            feature.executionOrder = order.pop() (4)
        }
    }
}
1We want to be able to reproduce issues, so we support -Dspock.random.order.seed parameter which allows us to provide a predefined seed value. For instance, running the test with the parameter -Dspock.random.order.seed=42 will always produce the same methods orders permutation.
2A list of all possible orders (0..n).
3Here we shuffle the list to get its random permutation.
4For each feature method iterated in the declaration order we assign a unique order popped from the shuffled list.

This way we override the default execution order of each feature. By default, every feature uses execution order set based on the declaration order. (The first method gets executionOrder == 0, the second one gets executionOrder == 1 and so on.)

The last thing we need to do is to add @RandomizedOrder annotation to our specification class.

Listing 4. @RandomizedOrder annotationed specification class
package com.github.wololock

import spock.lang.Specification

@RandomizedOrder
class RandomSpockSpec extends Specification {

    def "test 1"() {
        when:
        def number = 1

        then:
        println "[${new Date().format("HH:mm:ss.SSS")}] number ${number}"
    }

    def "test 2"() {
        when:
        def number = 2

        then:
        println "[${new Date().format("HH:mm:ss.SSS")}] number ${number}"
    }

    def "test 3"() {
        when:
        def number = 3

        then:
        println "[${new Date().format("HH:mm:ss.SSS")}] number ${number}"
    }

    def "test 4"() {
        when:
        def number = 4

        then:
        println "[${new Date().format("HH:mm:ss.SSS")}] number ${number}"
    }

    def "test 5"() {
        when:
        def number = 5

        then:
        println "[${new Date().format("HH:mm:ss.SSS")}] number ${number}"
    }
}

We are ready to run the test now. Let’s see if the execution order has changed.

spock random tests 02

It worked! We can see that in the above example the execution order was: Test 4, Test 3, Test 5, Test 1, and Test 2. And what’s even more important - the solution is simple and clean.

Why the random execution?

Is there any specific reason to run tests in the random order? It depends. In general, every feature in the specification should live in isolation. It means that it should not depend on any side effects or any state, and should not cause any side effects either. (If we need to rely on specific state and order, Spock’s @Stepwise [1] and @Shared [2] annotations are our best friends.) If we follow this rule, it doesn’t matter in which order the specification executes all features. However, sometimes we have to jump into the ongoing project, and we have to deal with existing unit tests we didn’t see before. Switching to a random order execution in the unit tests might help us verifying if they are correctly written. (We can also use Spock’s Global Extension mechanism to add the new extension without annotating classes - might be useful if we have tons of test classes to deal with.) In other cases, we might also benefit from the random execution order as a safeguard that always forces us (and our teammates) to write tests that are isolated and atomic.


Thank you!

If you read up to this point - leave me a comment below. It means a lot to me to hear your opinion. Share this blog post on Twitter, Facebook, or LinkedIn, if you found it helpful.

I will be very grateful if you consider supporting my work and effort. I run this blog to share my experience, inspire, and help other people to become better problem solvers.

Szymon Stepniak

25,833●9●58●88

Groovista, Upwork's Top Rated freelancer, Toruń Java User Group founder, open source contributor, Stack Overflow addict, bedroom guitar player. I walk through e.printStackTrace() so you don't have to.

You may also like

Ratpack: mocking Session object in GroovyRequestFixture test

Ratpack allows you unit test handlers using GroovyRequestFixture class. The good thing about this approach is that it does not require running the whole application and you can quickly test if the ...

How to avoid "No tests were found" when using JUnit 5 with Groovy?

In this short blog post I would like to explain how to avoid popular mistake when you write your first JUnit 5 test case in Groovy.

Ratpack: register SessionModule in handler unit test

Unit testing Ratpack handlers has many benefits. In the previous post we have learned how to mock Session object to use it with GroovyRequestFixture. Today instead of mocking we will register Sessi...