Spock random order of tests

Spock Framework executes test methods (features) in a single class (specification) in the declaration order. There is nothing wrong in this default behavior - we should write tests with their isolation in mind. However, in some cases, we would like to randomize test methods execution. Today we are going to learn how to do it.

Introduction

Let’s start with a reasonably simple specification that prints a number to the console.

Listing 1. src/test/groovy/com/github/wololock/RandomSpockSpec.groovy

package com.github.wololock

import spock.lang.Specification

class RandomSpockSpec extends Specification {

    def "test 1"() {
        when:
        def number = 1

        then:
        println "[${new Date().format("HH:mm:ss.SSS")}] number ${number}"
    }

    def "test 2"() {
        when:
        def number = 2

        then:
        println "[${new Date().format("HH:mm:ss.SSS")}] number ${number}"
    }

    def "test 3"() {
        when:
        def number = 3

        then:
        println "[${new Date().format("HH:mm:ss.SSS")}] number ${number}"
    }

    def "test 4"() {
        when:
        def number = 4

        then:
        println "[${new Date().format("HH:mm:ss.SSS")}] number ${number}"
    }

    def "test 5"() {
        when:
        def number = 5

        then:
        println "[${new Date().format("HH:mm:ss.SSS")}] number ${number}"
    }
}

When we execute this specification, we get all numbers printed in the ascending order.

You can find the source code of this example in the following repository.

Forcing random order

Now let’s try to randomize the execution order. One way to do it is to use Spock’s extensions - an annotation-driven local extensions in this case. Let’s create a new annotation called @RandomizedOrder with the following content.

Listing 2. src/test/groovy/com/github/wololock/RandomizedOrder.groovy

package com.github.wololock

import org.spockframework.runtime.extension.ExtensionAnnotation

import java.lang.annotation.ElementType
import java.lang.annotation.Retention
import java.lang.annotation.RetentionPolicy
import java.lang.annotation.Target

@Retention(RetentionPolicy.RUNTIME)
@Target(ElementType.TYPE)
@ExtensionAnnotation(RandomizedOrderExtension)
@interface RandomizedOrder {}

After creating annotation interface, we can create the extension class. This class implements visitSpecAnnotation interceptor method that gets executed right before specification executes any feature.

Listing 3. src/test/groovy/com/github/wololock/RandomizedOrderExtension.groovy

package com.github.wololock

import org.spockframework.runtime.extension.AbstractAnnotationDrivenExtension
import org.spockframework.runtime.model.SpecInfo

final class RandomizedOrderExtension extends AbstractAnnotationDrivenExtension<RandomizedOrder> {

    public static final String SPOCK_RANDOM_ORDER_SEED = "spock.random.order.seed"

    private static final long seed = System.getProperty(SPOCK_RANDOM_ORDER_SEED)?.toLong() ?: System.nanoTime()

    static {
        println "Random seed used: ${seed}\nYou can re-run the test with predefined seed by passing -D${SPOCK_RANDOM_ORDER_SEED}=${seed}\n\n"
    }

    @Override
    void visitSpecAnnotation(RandomizedOrder annotation, SpecInfo spec) {
        final Random random = new Random(seed) (1)

        final List<Integer> order = (0..(spec.features.size())) as ArrayList (2)

        Collections.shuffle(order, random) (3)

        spec.features.each { feature ->
            feature.executionOrder = order.pop() (4)
        }
    }
}

1	We want to be able to reproduce issues, so we support `-Dspock.random.order.seed` parameter which allows us to provide a predefined seed value. For instance, running the test with the parameter `-Dspock.random.order.seed=42` will always produce the same methods orders permutation.
2	A list of all possible orders (0..n).
3	Here we shuffle the list to get its random permutation.
4	For each feature method iterated in the declaration order we assign a unique order popped from the shuffled list.

This way we override the default execution order of each feature. By default, every feature uses execution order set based on the declaration order. (The first method gets executionOrder == 0, the second one gets executionOrder == 1 and so on.)

The last thing we need to do is to add @RandomizedOrder annotation to our specification class.

Listing 4. @RandomizedOrder annotationed specification class

package com.github.wololock

import spock.lang.Specification

@RandomizedOrder
class RandomSpockSpec extends Specification {

    def "test 1"() {
        when:
        def number = 1

        then:
        println "[${new Date().format("HH:mm:ss.SSS")}] number ${number}"
    }

    def "test 2"() {
        when:
        def number = 2

        then:
        println "[${new Date().format("HH:mm:ss.SSS")}] number ${number}"
    }

    def "test 3"() {
        when:
        def number = 3

        then:
        println "[${new Date().format("HH:mm:ss.SSS")}] number ${number}"
    }

    def "test 4"() {
        when:
        def number = 4

        then:
        println "[${new Date().format("HH:mm:ss.SSS")}] number ${number}"
    }

    def "test 5"() {
        when:
        def number = 5

        then:
        println "[${new Date().format("HH:mm:ss.SSS")}] number ${number}"
    }
}

We are ready to run the test now. Let’s see if the execution order has changed.

It worked! We can see that in the above example the execution order was: Test 4, Test 3, Test 5, Test 1, and Test 2. And what’s even more important - the solution is simple and clean.

Why the random execution?

Is there any specific reason to run tests in the random order? It depends. In general, every feature in the specification should live in isolation. It means that it should not depend on any side effects or any state, and should not cause any side effects either. (If we need to rely on specific state and order, Spock’s @Stepwise ^[1] and @Shared ^[2] annotations are our best friends.) If we follow this rule, it doesn’t matter in which order the specification executes all features. However, sometimes we have to jump into the ongoing project, and we have to deal with existing unit tests we didn’t see before. Switching to a random order execution in the unit tests might help us verifying if they are correctly written. (We can also use Spock’s Global Extension mechanism to add the new extension without annotating classes - might be useful if we have tons of test classes to deal with.) In other cases, we might also benefit from the random execution order as a safeguard that always forces us (and our teammates) to write tests that are isolated and atomic.