Testing extensions always requires testing parallel/non-parallel execution and post-test callbacks validation. It is impossible to do with regular tests, but, thankfully, JUnit 5 provides a TestKit for running and recording test execution (so you can assert entire test engine execution).

TestKit dependency will be required: org.junit.platform:junit-platform-testkit

All required dependencies (gradle example):

testImplementation 'org.junit.jupiter:junit-jupiter-api:5.8.2'
testImplementation 'org.junit.platform:junit-platform-testkit:5.8.2'
testRuntimeOnly 'org.junit.jupiter:junit-jupiter:5.8.2'

Suppose we want to test situation when test throws an exception (because our extension implements TestExecutionExceptionHandler and we need to test its behaviour).

Writing test implementing required scenario:

@ExtendWith(MyExtension.class)
public class FailingTestCase {

    @Test
    public void test() {
        throw new IllegalStateException("Ups");
    }
}

And now the main (TestKit) test:

import static org.junit.platform.engine.discovery.DiscoverySelectors.selectClass;

public class MyExtensionFailureBypassTest {

    @Test
    public void testExceptionBypass() {
       EngineTestKit
             .engine("junit-jupiter")
             .selectors(selectClass(FailingTestCase.class))
             .execute()
             .testEvents()
             .debug() // optional 
             .assertStatistics(stats -> stats.failed(1));
    }
}

Here we test that exception was thrown (suppose extension should not handle this type of exceptions). More assertion technics could be found in junit docs - I just want to show the base idea here.

Note that there are different scopes of events: .testEvents(), .containerEvents() and .allEvents().

Failing tests problem

There is one problem: test scenario is also a valid junit test and so it would be found and executed during all tests execution. To avoid it annotating test case as disabled:

@Disabled
public class FailingTestCase { ...

And in TestKit test ignoring this annotation:

EngineTestKit
       .engine("junit-jupiter")
       .configurationParameter("junit.jupiter.conditions.deactivate", "org.junit.*DisabledCondition")

Now only TestKit tests will be executed and all "scenario" tests would be ignored.

Testing parallel execution

Parallel execution for several tests could be configured only on engine level so all tests execution is either parallel or not (on test class level parallel methods execution could be activated with @Execution(ExecutionMode.CONCURRENT) annotation). With TestKit you can run just a few tests in parallel and validate behavior:

EngineTestKit
      .engine("junit-jupiter")
      .configurationParameter("junit.jupiter.conditions.deactivate", "org.junit.*DisabledCondition")
       // enable parallel execution
      .configurationParameter("junit.jupiter.execution.parallel.enabled", "true")
      .configurationParameter("junit.jupiter.execution.parallel.mode.default", "concurrent")
       // use 4 tests to run in parallel
      .selectors(selectClass(Test1.class),
          selectClass(Test2.class),
          selectClass(Test3.class),
          selectClass(Test4.class))
      .execute()
      .testEvents()
      .debug()
      .assertStatistics(stats -> stats.succeeded(4));

Simple testing technique

OK/KO situations testing is often not enough: often you need to keep some state from the test in order to validate it in the TestKit test. The simplest way is to introduce shared thread local state:

public class ActionHolder {

    private static ThreadLocal<List<String>> STATE = new ThreadLocal<>();

    public static void cleanup() {
        STATE.remove();
    }

    public static void add(String state) {
        List<String> st = STATE.get();
        if (st == null) {
            st = new ArrayList<>();
            STATE.set(st);
        }
        st.add(state);
    }

    public static List<String> getState() {
        return STATE.get() == null ? Collections.emptyList() : new ArrayList<>(STATE.get());
    }
}

During the test, we would add strings to it and after TestKit execution would verify the results.

Such storage requires careful management, so it's better to move all test boilerplate into base test:

public abstract class AbstractTest {

    public List<String> runTest(Class test) {
        ActionHolder.cleanup();
        try {
            EngineTestKit
                  .engine("junit-jupiter")
                  .configurationParameter("junit.jupiter.conditions.deactivate", "org.junit.*DisabledCondition")
                  .selectors(selectClass(test))
                  .execute()
                  .allEvents().failed().stream()
                  // exceptions appended to events log
                  .forEach(event -> {
                      Throwable err = event.getPayload(TestExecutionResult.class).get().getThrowable().get();
                      ActionHolder.add("Error: (" + err.getClass().getSimpleName() + ") " + err.getMessage());
                  });
            return ActionHolder.getState();
        } finally {
            ActionHolder.cleanup();
        }
    }
}

Note that all exceptions are also added to the holder after execution, so if something goes wrong you'll know about it (no false-positive tests).

Let's take the initial case and assume that it also implements ParameterResolver and so we need to check the correctness of injected parameters:

@Disabled
@ExtendWith(MyExtension.class)
public class FailingTestCase {

    @Test
    public void test(Integer arg) {
        ActionHolder.add("test.arg: " + arg)
        throw new IllegalStateException("Ups")
    }
}

TestKit test:

public class StateTest extends AbstractTest {
    @Test
    public void testParams() {
        Assertions.assertEquals(Arrays.asList(
                "test.arg: 11",
                "Error: (IllegalStateException) Ups"

                ), runTest(FailingTestCase.class))
    }
}

Here we can validate that correct parameter value was injected and the exception. Note that in any case, all errors would be at the end of the list.

Such tests would be safe to execute in parallel because the state is managed under a single test method only.

As an example, I used this technique to validate (confirm) junit lifecycle sequence.

---

Follow up: How to do the same in Spock 2