Testing Spock 2 extensions

This is the follow-up of Testing JUnit 5 extensions article (assume you read it first). This article shows how to test Spock 2 extensions the same way.

Like JUnit's Jupiter, Spock 2 is implemented as JUnit platform engine and so TestKit might be used for running Spock tests too. But there are a few differences.

First of all, Spock already provides a shortcut for TestKit tests: spock.util.EmbeddedSpecRunner.

As with JUnit, suppose we want to check extension test method fail handling (and extension should stop this exception type):

@MySpockExtension
class extends Specification {

    void "test method fail"() {

        when:
        throw new IllegalStateException("Ups")

        then:
        true
    }
}

def specRunner = new EmbeddedSpecRunner()
specRunner.throwFailure = false
specRunner.runClass(FailingTestCase)
          .testEvents()
          .debug() // optional 
          .assertStatistics(stats -> stats.failed(1));

By default, EmbeddedSpecRunner re-throws exceptions which is not desirable (have to disable it for the same behavior as in pure JUnit kit tests).

Also, runner supports direct test compilation so the test itself could be described as a string instead of a separate class:

def specRunner = new EmbeddedSpecRunner()
specRunner.throwFailure = false
specRunner.run '''
package com.foo

import spock.lang.Specification
import spock.util.EmbeddedSpecRunner

@MySpockExtension
class extends Specification {

    void "test method fail"() {

        when:
        throw new IllegalStateException("Ups")

        then:
        true
    }
}
'''
          .testEvents()
          .debug() // optional 
          .assertStatistics(stats -> stats.failed(1));

This may look weird, but, probably, could be useful in some cases. At least, this solves the "failing tests problem" because the test in a string would never be recognized and executed directly. Still, I think separate test classes usage is handier.

You can find more EmbeddedSpecRunner usage examples in spock sources.

Failing tests problem

As with JUnit tests, failing test cases (used for TestKit tests) would be found and executed directly and fail the build. Spock does not support JUnit constraints and so @Disabled can't be used to skip "test scenarios" from separate execution. Spock @Ignore extension does not provide a way to disable for TestKit execution and that's why I have to use the following hack:

static Boolean ACTIVE = false with @Requires spock extension:

@Requires({ AbstractTest.ACTIVE })
class TestCase extends Specification { ...

Where AbstractTest.ACTIVE is set to true before each TestKit test (full source below) and reverted to false after.

You can evolve this approach for your case if required.

Testing parallel execution

As in JUnit, concurrent execution of test methods might be configured with @Execution(ExecutionMode.CONCURRENT) class annotation. IMPORTANT this is Spock annotation, not JUnit one (classes are the same just in different packages).

Executing several tests in parallel:

def specRunner = new EmbeddedSpecRunner()
specRunner.throwFailure = false
specRunner.configurationScript {
         runner {
             parallel {
                 enabled true
                 defaultExecutionMode ExecutionMode.CONCURRENT
                 defaultSpecificationExecutionMode ExecutionMode.CONCURRENT
             }
         }
}
specRunner.runClasses([Test1, Test2, Test3, Test4])
      .testEvents()
      .debug() // optional
      .assertStatistics(stats -> stats.succeeded(4));

WARNING: there should not be variables called the same as sections inside configurationScript (for example, runner, parallel, etc.) otherwise you'll see very non-intuitive errors.

In the example above both specs and spec methods would run in parallel, but you can configure differently. Also, note that for data-driven tests (with where section) each iteration is a different execution and so iterations could run in parallel too.

Simple testing technique

The base test from JUnit article would look like this for Spock:

abstract class AbstractTest extends Specification {

    // used for activating tests only for TestKit run
    static Boolean ACTIVE = false

    List<String> runTest(Class test) {
        ActionHolder.cleanup()
        ACTIVE = true
        try {
            def specRunner = new EmbeddedSpecRunner()
            // do not rethrow exception - all errors will remain in holder
            specRunner.throwFailure = false
            specRunner.runClass(test)
                    .allEvents().failed().stream()
                    // exceptions appended to events log
                    .forEach(event -> {
                        Throwable err = event.getPayload(TestExecutionResult.class).get().getThrowable().get();
                        ActionHolder.add("Error: (" + err.getClass().getSimpleName() + ") " + err.getMessage())
                    })
            return ActionHolder.getState()
        } finally {
            ACTIVE = false
            ActionHolder.cleanup()
        }
    }
}

Here, as in JUnit example, all execution errors are collected as string messages, so when anything goes wrong the list of the resulting strings would be different.

Complete example case for testing parameters injection (assuming extension injects required parameter):

@Requires({ AbstractTest.ACTIVE })
@MySpockExtension
class ExceptionCase extends Specification {

    def "Sample test"(Integer arg) {

        when:
        ActionHolder.add("test.body $arg")
        throw new IllegalStateException('Ups')

        then:
        true
    }
}

And the main TestKit test:

class SpockTest extends AbstractTest {

    def "Check parameter"() {

        expect: 'test fails'
        runTest(ExceptionCase) == [
                                  "test.body 11",
                                  "Error: (IllegalStateException) Ups"]
    }
}