HomeAboutArchivesTools & ResourcesSupport me
Hi! 👋 I'm Szymon

I help you become a better software developer.

  • 1.15kFollowers
  • 2.36kSubscribers
  • 110Followers
  • 30.2kReputation

Latest posts from the Groovy Cookbook

Groovy Ecosystem Usage Report (2020)
How to merge two maps in Groovy?
Groovy dynamic Maps, generic type erasure, and raw types - an...
Groovy 3 @NullCheck annotation - less code and less NPE
Groovy 3 String GDK improvements - takeRight, takeBetween, an...
Three Groovy String methods that will make your life Groovier!
Quicksort in Groovy - can it be as fast as implemented in Java?
More

Popular categories

  • Groovy Cookbook28
  • Jenkins Pipeline Cookbook9
  • Programmer's Bookshelf6
  • Micronaut Cookbook4
  • Ratpack Cookbook4
  • Learning Java4
  • jq cookbook3
  • How to3
  • Blog Reports2

Don't miss anything - join my newsletter

Additional materials and updates for subscribers. No spam guaranteed.
Unsubscribe at any time.

Did you like this article?

Spread the !
  • ☕️
  1. e.printstacktrace.blog
  2. Groovy Cookbook

Spock random order of tests - how to?

  • April 6, 2019
  • 5 min. read
  • 0 Comments

Spock Framework executes test methods (features) in a single class (specification) in the declaration order. There is nothing wrong in this default behavior - we should write tests with their isolation in mind. However, in some cases, we would like to randomize test methods execution. Today we are going to learn how to do it.

Introduction

Let’s start with a reasonably simple specification that prints a number to the console.

Listing 1. src/test/groovy/com/github/wololock/RandomSpockSpec.groovy
package com.github.wololock

import spock.lang.Specification

class RandomSpockSpec extends Specification {

    def "test 1"() {
        when:
        def number = 1

        then:
        println "[${new Date().format("HH:mm:ss.SSS")}] number ${number}"
    }

    def "test 2"() {
        when:
        def number = 2

        then:
        println "[${new Date().format("HH:mm:ss.SSS")}] number ${number}"
    }

    def "test 3"() {
        when:
        def number = 3

        then:
        println "[${new Date().format("HH:mm:ss.SSS")}] number ${number}"
    }

    def "test 4"() {
        when:
        def number = 4

        then:
        println "[${new Date().format("HH:mm:ss.SSS")}] number ${number}"
    }

    def "test 5"() {
        when:
        def number = 5

        then:
        println "[${new Date().format("HH:mm:ss.SSS")}] number ${number}"
    }
}

When we execute this specification, we get all numbers printed in the ascending order.

You can find the source code of this example in the following repository.

Forcing random order

Now let’s try to randomize the execution order. One way to do it is to use Spock’s extensions - an annotation-driven local extensions in this case. Let’s create a new annotation called @RandomizedOrder with the following content.

Listing 2. src/test/groovy/com/github/wololock/RandomizedOrder.groovy
package com.github.wololock

import org.spockframework.runtime.extension.ExtensionAnnotation

import java.lang.annotation.ElementType
import java.lang.annotation.Retention
import java.lang.annotation.RetentionPolicy
import java.lang.annotation.Target

@Retention(RetentionPolicy.RUNTIME)
@Target(ElementType.TYPE)
@ExtensionAnnotation(RandomizedOrderExtension)
@interface RandomizedOrder {}

After creating annotation interface, we can create the extension class. This class implements visitSpecAnnotation interceptor method that gets executed right before specification executes any feature.

Listing 3. src/test/groovy/com/github/wololock/RandomizedOrderExtension.groovy
package com.github.wololock

import org.spockframework.runtime.extension.AbstractAnnotationDrivenExtension
import org.spockframework.runtime.model.SpecInfo

final class RandomizedOrderExtension extends AbstractAnnotationDrivenExtension<RandomizedOrder> {

    public static final String SPOCK_RANDOM_ORDER_SEED = "spock.random.order.seed"

    private static final long seed = System.getProperty(SPOCK_RANDOM_ORDER_SEED)?.toLong() ?: System.nanoTime()

    static {
        println "Random seed used: ${seed}\nYou can re-run the test with predefined seed by passing -D${SPOCK_RANDOM_ORDER_SEED}=${seed}\n\n"
    }

    @Override
    void visitSpecAnnotation(RandomizedOrder annotation, SpecInfo spec) {
        final Random random = new Random(seed) (1)

        final List<Integer> order = (0..(spec.features.size())) as ArrayList (2)

        Collections.shuffle(order, random) (3)

        spec.features.each { feature ->
            feature.executionOrder = order.pop() (4)
        }
    }
}
1We want to be able to reproduce issues, so we support -Dspock.random.order.seed parameter which allows us to provide a predefined seed value. For instance, running the test with the parameter -Dspock.random.order.seed=42 will always produce the same methods orders permutation.
2A list of all possible orders (0..n).
3Here we shuffle the list to get its random permutation.
4For each feature method iterated in the declaration order we assign a unique order popped from the shuffled list.

This way we override the default execution order of each feature. By default, every feature uses execution order set based on the declaration order. (The first method gets executionOrder == 0, the second one gets executionOrder == 1 and so on.)

The last thing we need to do is to add @RandomizedOrder annotation to our specification class.

Listing 4. @RandomizedOrder annotationed specification class
package com.github.wololock

import spock.lang.Specification

@RandomizedOrder
class RandomSpockSpec extends Specification {

    def "test 1"() {
        when:
        def number = 1

        then:
        println "[${new Date().format("HH:mm:ss.SSS")}] number ${number}"
    }

    def "test 2"() {
        when:
        def number = 2

        then:
        println "[${new Date().format("HH:mm:ss.SSS")}] number ${number}"
    }

    def "test 3"() {
        when:
        def number = 3

        then:
        println "[${new Date().format("HH:mm:ss.SSS")}] number ${number}"
    }

    def "test 4"() {
        when:
        def number = 4

        then:
        println "[${new Date().format("HH:mm:ss.SSS")}] number ${number}"
    }

    def "test 5"() {
        when:
        def number = 5

        then:
        println "[${new Date().format("HH:mm:ss.SSS")}] number ${number}"
    }
}

We are ready to run the test now. Let’s see if the execution order has changed.

It worked! We can see that in the above example the execution order was: Test 4, Test 3, Test 5, Test 1, and Test 2. And what’s even more important - the solution is simple and clean.

Why the random execution?

Is there any specific reason to run tests in the random order? It depends. In general, every feature in the specification should live in isolation. It means that it should not depend on any side effects or any state, and should not cause any side effects either. (If we need to rely on specific state and order, Spock’s @Stepwise [1] and @Shared [2] annotations are our best friends.) If we follow this rule, it doesn’t matter in which order the specification executes all features. However, sometimes we have to jump into the ongoing project, and we have to deal with existing unit tests we didn’t see before. Switching to a random order execution in the unit tests might help us verifying if they are correctly written. (We can also use Spock’s Global Extension mechanism to add the new extension without annotating classes - might be useful if we have tons of test classes to deal with.) In other cases, we might also benefit from the random execution order as a safeguard that always forces us (and our teammates) to write tests that are isolated and atomic.


1. http://spockframework.org/spock/javadoc/1.1/spock/lang/Stepwise.html
2. http://spockframework.org/spock/javadoc/1.1/spock/lang/Shared.html
  • groovy
  • spock
  • unit-test
  • junit
  • random

Programmer's Bookshelf

Debugging Teams - book review

  • April 16, 2019
  • 5 min. read
  • 0 Comments

"Geniuses still make mistakes and having brilliant ideas and elite programming skills don’t guarantee that your software will be a hit. What’s going to make or break your career is how well you collaborate with others", say Brian Fitzpatrick and Ben Collins-Sussman in their book "Debugging Teams. Better productivity through collaboration". It shows that crafting the ability to collaborate is equally important as learning new programming languages and mastering the ones we already know. Let’s take a quick look at how this book can help us to become a better team player and a better team leader.

Ratpack Cookbook

Ratpack: mocking Session object in GroovyRequestFixture test

  • June 24, 2018
  • 5 min. read
  • 0 Comments

Ratpack allows you unit test handlers using GroovyRequestFixture class. The good thing about thi...

Ratpack Cookbook

Ratpack: register SessionModule in handler unit test

  • June 26, 2018
  • 5 min. read
  • 0 Comments

Unit testing Ratpack handlers has many benefits. In the previous post we have learned how to moc...

Groovy Cookbook

How to avoid "No tests were found" when using JUnit 5 with Groovy?

  • October 24, 2018
  • 5 min. read
  • 0 Comments

In this short blog post I would like to explain how to avoid popular mistake when you write your...

Any thoughts or ideas?

Let's talk in the comment's section 💬

Want to put a code sample in the comment? Read the Syntax highlighting guide for more information.
Empty
Latest blog posts
  • Merging JSON files recursively in the command-line
  • Jenkins Declarative Pipeline with the dynamic agent - how to configure it?
  • Groovy Ecosystem Usage Report (2020)
  • How to convert JSON to CSV from the command-line?
  • 5 Common Jenkins Pipeline Mistakes
  • How to merge two maps in Groovy?
  • Building stackoverflow-cli with Java 11, Micronaut, Picocli, and GraalVM
  • How to catch curl response in Jenkins Pipeline?
  • Atomic Habits - book review
  • Parsing JSON in command-line with jq: basic filters and functions (part 1)
Trending videos
5 Common Jenkins Pipeline Mistakes 🔥

In this Jenkins Pipeline tutorial video, I reveal five common Jenkins Pipeline mistakes and I explain how those mistakes can ...

Using GraalVM native-image with a Groovy script - from 2.1s to 0.013s startup time 🚀

GraalVM native-image compiler uses ahead-of-time compilation to produce a highly optimized standalone executable file that ca...

Useful links
  • Start here
  • About
  • Archives
  • Resources
  • Privacy Policy
  • Merchandise
  • My Kit
  • RSS
  • Support the blog
Popular categories
  • Groovy Cookbook28
  • Jenkins Pipeline Cookbook9
  • Programmer's Bookshelf6
  • Micronaut Cookbook4
  • Ratpack Cookbook4
  • Learning Java4
  • jq cookbook3
  • How to3
  • Blog Reports2
Popular tags
affiliate async benchmark blogging book career cicd continuous-integration curl devops docker git github graalvm gradle grails groovy haskell hexo java java-8 jenkins jenkins-pipeline jenkinsfile jmh jq json junit learning maven metaprogramming micronaut native-image non-blocking progress ratpack reactive-programming reading recursion review rxjava sdkman session split spock stackoverflow string tail-call tail-recursion unit-test
  • Designed by @wololock
  • Created with Hexo
  • Hosted on GitHub
  • Deployed with Circle CI
  • License CC BY-NC-SA 4.0