Gradle configuration cache by example. Part 1: behavior

Gradle configuration cache by example. Part 1: behavior

Gradle configuration cache becomaing mandatory in gradle 9. This affects all plugin authors as configuration cache must be supported by plugin explicitly (actually, plugin just should not do disallowed things).

Article split into two parts (kind of theory and practice):

Part 1 shows how configuration cache works (by examples). It is the best way to learn it because, almost certainly, it works not like you think (if you never deal with it yourself).

Part 2 shows common problems (with example cases) and potential solutions.

Preprequisites

I prepared a github repository with samples. These samples include all described cases, but not exactly follow article samples (article describes cases separately for simplicity, whereas samples check multiple cases at once). You could clone this repo and experiment with samples yourself!

As with any other cache, the configuration cache could be tested by two runs: the first run prepares cache record and the second run validates execution under the cache.

Repository samples use gradle TestKit to verify cache behaviour:

// projectDir - temprorary directory (created per test)
// build.gradle file is created manually (with required test project configuration)

// run to create cache record
BuildResult result = GradleRunner.create()
        .withProjectDir(projectDir)
        .withArguments(List.of("myTask", "--configuration-cache"))
        .withPluginClasspath()
        .forwardOutput()
        .build();

// validation
result.getOutput().contains("Calculating task graph as no cached configuration is available for tasks");

// run again to check cached behaviour
result = GradleRunner.create()
        .withProjectDir(projectDir)
        .withArguments(List.of("myTask", "--configuration-cache"))
        .withPluginClasspath()
        .forwardOutput()
        .build();

// validation
result.getOutput().contains("Reusing configuration cache");

On first execution gradle inidicates cache recording:

Calculating task graph as no cached configuration is available for tasks: sample1Task
…
Configuration cache entry stored.

On second run, gradle would indicate cache usage:

Reusing configuration cache.
…
Configuration cache entry reused.

The executed code

The Configuration Cache builds on this idea of work avoidance and parallelization. When enabled, the Configuration Cache allows Gradle to skip the configuration phase entirely if nothing that affects the build configuration (such as build scripts) has changed. Additionally, Gradle applies performance optimizations to task execution.

gradle guide

This means that under configuration cache, plugin’s code from configuration phase would not be executed.

public abstract class Sample1Plugin implements Plugin<Project> {

    @Override
    public void apply(Project project) {
        System.out.println("[configuration] Plugin applied");

        // register custom task
        project.getTasks().register("sampleTask", task -> {
            System.out.println("[configuration] Task configured");

            // the only line that works also under the configuration cache
            task.doFirst(task1 -> System.out.println("[run] Before task"));
        });

        // afterEvaluate often used by plugins as the first point where user configuration applied
        project.afterEvaluate(p -> System.out.println("[configuration] Project evaluated"));
    }
}

First execution: sampleTask —configuration-cache

Calculating task graph as no cached configuration is available for tasks: sampleTask

> Configure project :
[configuration] Plugin applied
[configuration] Project evaluated
[configuration] Task configured

> Task :sampleTask
[run] Before task

BUILD SUCCESSFUL
1 actionable task: 1 executed
Configuration cache entry stored.

Run from cache: sampleTask —configuration-cache

Reusing configuration cache.

> Task :sampleTask
[run] Before task

BUILD SUCCESSFUL
1 actionable task: 1 executed
Configuration cache entry reused.

As you can see, the code, related to configuration phase, is not called at all. Paying attention: neither plugin, nor any task constructors would not be called under the cache (there is no configuration phase)!

The state

The configuration cache serialize configuration from the first execution and use it for later executions.

Now we will use a simple extension to access user configuration in task:

public class Sample1Extension {
    public String message = "Default";

    public Sample1Extension() {
        System.out.println("[configuration] Extension created")
    }

    public String getMessage() {
        // no prefix because it could be called in both phases
        System.out.println("Extension get message: " + message);
        return message;
    }
}

The plugin will become:

public abstract class Sample1Plugin implements Plugin<Project> {

    @Override
    public void apply(Project project) {
        System.out.println("[configuration] Plugin executed");
        Sample1Extension ext = project.getExtensions()
                    .create("sample1", Sample1Extension.class);

        project.getTasks().register("sampleTask", task -> {
            task.doFirst(task1 -> System.out.println("[run] User message: " + ext.message));
        });
    }
}

Project build.gradle would contain:

sample1 {
    message = "hello user!"
}

First execution: sampleTask —configuration-cache

Calculating task graph as no cached configuration is available for tasks: sampleTask

> Configure project :
[configuration] Plugin executed
[configuration] Extension created

> Task :sampleTask
Extension get message: hello user!
[run] User message: hello user!

BUILD SUCCESSFUL
1 actionable task: 1 executed
Configuration cache entry stored.

Run from cache: sampleTask —configuration-cache

Reusing configuration cache.

> Task :sampleTask
Extension get message: hello user!
[run] User message: hello user!

BUILD SUCCESSFUL
1 actionable task: 1 executed
Configuration cache entry reused.

Pay attention: on the second run extension constructor was not called, but extension getter was!

What happend: on first execution, gradle tracked that ext.message value is used at runtime and so the entire ext object was serialized. On second run, the state was deserialized and the task used it for execution.

Now, what would happen if user configuration would change? Let’s run task the third time, but with updated build.gradle

sample1 {
    message = "changed message!"
}

Run (configuration cache record already exists): sampleTask —configuration-cache

Calculating task graph as configuration cache cannot be reused because file 'build.gradle' has changed.

> Configure project :
[configuration] Plugin executed
[configuration] Extension created

> Task :sampleTask
Extension get message: changed message!
[run] User message: changed message!

BUILD SUCCESSFUL
1 actionable task: 1 executed
Configuration cache entry stored.

Cache was invalidated, full configuration performed and a new cache record stored.

Plugin state

Let’s see what would happen with plugin’s own fields:

public abstract class Sample1Plugin implements Plugin<Project> {

    private String pluginField;

    public Sample1Plugin() {
        System.out.println("[configuration] Plugin created");
    }

    @Override
    public void apply(Project project) {
        project.afterEvaluate(p -> {
            pluginField = "assigned value";
            System.out.println("[configuration] Project evaluated");
        });

        project.getTasks().register("sampleTask", task -> {
            task.doFirst(task1 -> System.out.println("[run] Before task: " + pluginField));
        });
    }
}

Run sampleTask —configuration-cache:

Calculating task graph as no cached configuration is available for tasks: sample1Task

> Configure project :
[configuration] Plugin created
[configuration] Project evaluated

> Task :sampleTask
[run] Before task: assigned value

BUILD SUCCESSFUL
1 actionable task: 1 executed
Configuration cache entry stored.

Run from cache sampleTask —configuration-cache:

Reusing configuration cache.

> Task :sample1Task
[run] Before task: assigned value

BUILD SUCCESSFUL in 61ms
1 actionable task: 1 executed
Configuration cache entry reused.

As you can see, plugin fields are serialized.

Task state

Now, let’s see how task fields survive serialization. Use a separate task class:

public abstract class Sample1Task extends DefaultTask {

    @Input
    abstract Property<String> getMessage();
    @Input
    abstract Property<String> getMessage2();

    public String field;
    private String privateField;

    public Sample1Task() {
        System.out.println("[configuration] Task created");
        privateField = "set";
    }

    @TaskAction
    public void run() {
        System.out.println("[run] Task executed: message=" + getMessage().get()
                + ", message2=" + getMessage2().get()
                + ", public field=" + field
                + ", private field=" + privateField);
    }
}

Here we have 2 gradle properties and private and public fields. Plugin would become:

public abstract class Sample1Plugin implements Plugin<Project> {

    @Override
    public void apply(Project project) {
        System.out.println("[configuration] Plugin applied");
        final Sample1Extension ext = project.getExtensions()
                                .create("sample1", Sample1Extension.class);

        // register custom task
        project.getTasks().register("sample1Task", Sample1Task.class, task -> {
            task.getMessage().convention(ext.message);
            task.getMessage2().convention("Default");
            task.field = "assigned value";
        });

        // custom (lazy) task configuration
        project.getTasks().withType(Sample1Task.class).configureEach(task -> {
            task.getMessage2().set("Custom");
        });
    }
}

Plugin:

Set message to extension value
Set message2 to “Custom” value (in lazy configuration block)
Public task field assigned to assigned value

The extension class remains the same as above. Configuration file is:

sample1 {
    message = "hello user!"
}

Run it two times sample1Task —configuration-cache (only run from cache is interesting):

Reusing configuration cache.

> Task :sample1Task
Extension get message: hello user!
[run] Task executed: message=hello user!, message2=Custom, public field=assigned value, private field=set

BUILD SUCCESSFUL
1 actionable task: 1 executed
Configuration cache entry reused.

As you can see:

Task properties values are correct
Public and private field values are also preserved

So, it is OK to rely on task properties under the configuration cache.

Broken uniquness

One caveat with state serialization is broken uniquness: same object, used at multiple places would become a different objects after deserialization.

We will use a custom object to share state between tasks:

public class SharedState {

    // show how externally assigned values survive
    public String direct;
    public List<String> list = new ArrayList<>();

    public SharedState() {
        System.out.println("[configuration] Shared state created: " + System.identityHashCode(this));
    }

    @Override
    public String toString() {
        return System.identityHashCode(this) + "@" + list.toString() + ", direct=" + direct;
    }
}

And the plugin:

public abstract class Sample2Plugin implements Plugin<Project> {

    @Override
    public void apply(Project project) {
        // some object, common for two tasks
        final SharedState state = new SharedState();
        // some value directly assigned
        state.direct = "Custom";

        project.getTasks().register("task1").configure(task ->
                task.doLast(task1 -> {
                    state.list.add("Task 1");
                    System.out.println("[run] Task 1 shared object: " + state);
                }));

        project.getTasks().register("task2").configure(task ->
                task.doLast(task1 -> {
                    state.list.add("Task 2");
                    System.out.println("[run] Task 2 shared object: " + state);
                }));
    }
}

Now run it without cache first task1 task2:

> Configure project :
[configuration] Shared state created: 1353516483

> Task :task1
[run] Task 1 shared object: 1353516483@[Task 1], direct=Custom

> Task :task2
[run] Task 2 shared object: 1353516483@[Task 1, Task 2], direct=Custom

BUILD SUCCESSFUL
2 actionable tasks: 2 executed

The same SharedState object instance is used in tasks.

Run with cache enabled: task1 task2 —configuration-cache

Calculating task graph as no cached configuration is available for tasks: task1 task2

> Configure project :
[configuration] Shared state created: 1202973804

> Task :task2
[run] Task 2 shared object: 1650372210@[Task 2], direct=Custom

> Task :task1
[run] Task 1 shared object: 724264550@[Task 1], direct=Custom

BUILD SUCCESSFUL
2 actionable tasks: 2 executed
Configuration cache entry stored.

We could already see different instances in tasks (its just cache recording phase, but we already see side effects!): obviously, object was serialized and deserialized (because object constructor was called just once)

Running from cache task1 task2 —configuration-cache:

Reusing configuration cache.

> Task :task1
[run] Task 1 shared object: 1796870189@[Task 1], direct=Custom

> Task :task2
[run] Task 2 shared object: 1139607728@[Task 2], direct=Custom

BUILD SUCCESSFUL
2 actionable tasks: 2 executed
Configuration cache entry reused.

Again, two different instances used.

Paying attention: configuration phase execution with the configuration cache enabled is already different from usual gradle execution. But it’s for good: performing configuration state serialization and deserialization even on the first run, gradle reveals many cache-related problems (which otherwise would happen in the second execution (from cache)).

Build services

The only way to workaround uniquness problem is build services.

Ideally, there should be no need to share any data between tasks (in the majority of cases configuration cache should cache required data). But, sometimes, tasks communication is required at runtime.

We will use a service to keep shared state between tasks:

public abstract class SharedService implements BuildService<SharedService.Params>, AutoCloseable {

    public String extParam;
    // tasks might be executed in parallel (this simply avoids ConcurrentModificationException)
    public List<String> list = new CopyOnWriteArrayList<>();

    public SharedService() {
        // could appear both in configuration and execution time
        System.out.println("Shared service created " + System.identityHashCode(this) + "@");
    }

    public interface Params extends BuildServiceParameters {
        Property<String> getExtParam();
    }

    @Override
    public String toString() {
        return System.identityHashCode(this) + "@" + list.toString()
                + ", param: " + getParameters().getExtParam().getOrNull()
                + ", field: " + extParam;
    }

    // IMPORTANT: gradle could close service at any time and start a new instance!
    @Override
    public void close() throws Exception {
        System.out.println("Shared service closed: " + System.identityHashCode(this));
    }
}

The plugin would declare service and two simple tasks:

public abstract class Sample3Plugin implements Plugin<Project> {

    @Override
    public void apply(Project project) {
        // IMPORTANT: service not created at this moment!
        final Provider<SharedService> service = project.getGradle().getSharedServices()
                .registerIfAbsent("service", SharedService.class, spec -> {
                        // configuration value set with parameter
                        spec.getParameters().getExtParam().convention("Default");
                    });

        // configuration value set DIRECTLY (after service creation)
        project.afterEvaluate(p -> {
            service.get().extParam = "Custom";
            System.out.println("[configuration] Project evaluated");
        });

        project.getTasks().register("task1").configure(task ->
                task.doLast(task1 -> {
                    final SharedService sharedService = service.get();
                    sharedService.list.add("Task 1");
                    System.out.println("[run] Task 1 shared object: " + sharedService);
                }));

        project.getTasks().register("task2").configure(task -> {
            // just to disable tasks concurrency (simplier to verify output)
            task.mustRunAfter("task1");

            task.doLast(task1 -> {
                final SharedService sharedService = service.get();
                sharedService.list.add("Task 2");
                System.out.println("[run] Task 2 shared object: " + sharedService);
            });
        });
    }
}

First run without cache task1 task2:

> Configure project :
Shared service created 1202901166@
[configuration] Project evaluated

> Task :task1
[run] Task 1 shared object: 1202901166@[Task 1], param: Default, field: Custom

> Task :task2
[run] Task 2 shared object: 1202901166@[Task 1, Task 2], param: Default, field: Custom
Shared service closed: 1202901166

BUILD SUCCESSFUL
2 actionable tasks: 2 executed

Shared service is created in afterEvaluate block (the first time service.get() was called) where its extParam field set to custom value.

The same service instance is used in both tasks. Service is closed after tasks execution.

Run with enabled cache task1 task2 —configuration-cache:

Calculating task graph as no cached configuration is available for tasks: task1 task2

> Configure project :
Shared service created 644777498@
[configuration] Project evaluated
Shared service closed: 644777498

> Task :task1
Shared service created 1665614489@
[run] Task 1 shared object: 1665614489@[Task 1], param: Default, field: null

> Task :task2
[run] Task 2 shared object: 1665614489@[Task 1, Task 2], param: Default, field: null
Shared service closed: 1665614489

BUILD SUCCESSFUL
2 actionable tasks: 2 executed
Configuration cache entry stored.

Behavior is different:

Service created in afterEvaluate block where its field is initialied
After configuration phase service is closed! This is intentional gradle behavior under the cache (gradle advice using services only at runtime)!
A new service instance is created before the first task execution. Direct field value is lost (service fields are not serialized!). Only parameter value preserved (it would be a value applied in service initialization block)

Overall, uniquness preserved: the service instance is the same between tasks, but it is a different instance from configuration phase.

Run from cache task1 task2 —configuration-cache:

Reusing configuration cache.

> Task :task1
Shared service created 1651282902@
[run] Task 1 shared object: 1651282902@[Task 1], param: Default, field: null

> Task :task2
[run] Task 2 shared object: 1651282902@[Task 1, Task 2], param: Default, field: null
Shared service closed: 1651282902

BUILD SUCCESSFUL
2 actionable tasks: 2 executed
Configuration cache entry reused.

The same service instance used, but directly initialized service field value was lost (same as in previous run).

So under configuration cache, the same service instance would be used in run phase. Service, used on configuration phase, would be closed befor run phase.

Prevent service closing

There is a way to prevent service closing before a run phase: service should implement OperationCompletionListener (pretend it would listen for tasks execution):

public abstract class SharedService implements BuildService<SharedServiceSingleton.Params>, AutoCloseable,
        OperationCompletionListener {

    ...

    @Override
    public void onFinish(FinishEvent finishEvent) {
        System.out.println("Finish event: " + finishEvent.getDescriptor().getName() + " caught on service " + this);
    }
}

(the rest is [the same](#build-services))

Plugin should now register service as a listener:

public abstract class Sample3Plugin implements Plugin<Project> {

    @Inject
    public abstract BuildEventsListenerRegistry getEventsListenerRegistry();

    @Override
    public void apply(Project project) {
        ...
        final Provider<SharedServiceSingleton> service = project.getGradle().getSharedServices().registerIfAbsent(
                "service", SharedServiceSingleton.class, spec -> {
                    spec.getParameters().getExtParam().convention("Default");
                });
        // service listens for tasks completion, which prevents gradle from stopping it in 
        // after configuration phase
        getEventsListenerRegistry().onTaskCompletion(service);

        ...
    }
}

Creating cache record task1 task2 --configuration-cache:

Calculating task graph as no cached configuration is available for tasks: task1 task2

> Configure project :
Shared service created 331815390@
[configuration] Project evaluated

> Task :task1
[run] Task 1 shared object: 331815390@[Task 1], param: Default, field: Custom
Finish event: :task1 caught on service 331815390@[Task 1], param: Default, field: Custom

> Task :task2
[run] Task 2 shared object: 331815390@[Task 1, Task 2], param: Default, field: Custom
Finish event: :task2 caught on service 331815390@[Task 1, Task 2], param: Default, field: Custom
Shared service closed: 331815390

BUILD SUCCESSFUL
2 actionable tasks: 2 executed
Configuration cache entry stored.

This time the service was not closed and the same instance is used in configuration and run phases. As a result, service field value is preserved.

But, when it run from cache task1 task2 --configuration-cache:

Reusing configuration cache.

> Task :task1
Shared service created 1976530883@
[run] Task 1 shared object: 1976530883@[Task 1], param: Default, field: null
Finish event: :task1 caught on service 1976530883@[Task 1], param: Default, field: null

> Task :task2
[run] Task 2 shared object: 1976530883@[Task 1, Task 2], param: Default, field: null
Finish event: :task2 caught on service 1976530883@[Task 1, Task 2], param: Default, field: null
Shared service closed: 1976530883

BUILD SUCCESSFUL in 53ms
2 actionable tasks: 2 executed
Configuration cache entry reused.

Service field will again be null because its value was assigned in plugin under configuration phase, which was missed here.

Usually, there is not much sense to use this trick (forcing singleton) because service state is not serialized, only parameters are. But, it make sense when the same instance as in configuration phase is required at runtime.

Moreover, gradle could close the service during runtime too: it tries to guess when service is not required anymore and close it. In complex cases, gradle could guess incorrectly and close service too early. Using the “listener trick” workarounds such cases.

Method calls

Here we will see if method calls are possible from runtime blocks:

public class Sample4Plugin implements Plugin<Project> {

    @Override
    public void apply(Project project) {
        project.getTasks().register("task1").configure(task -> {            
            task.doLast(task1 -> {
                System.out.println("[run] Task exec: " + computeMessage("static"));
            });
        });
    }

    private String computeMessage(String source) {
        System.out.println("called computeMessage('" + source + "')");
        return "computed message " + source;
    }
}

Creating cache entry task1 —configuration-cache:

Calculating task graph as no cached configuration is available for tasks: task1

> Task :task1
called computeMessage('static')
[run] Task exec: computed message static

BUILD SUCCESSFUL
1 actionable task: 1 executed
Configuration cache entry stored.

Run from cache task1 —configuration-cache:

Reusing configuration cache.

> Task :task1
called computeMessage('static')
[run] Task exec: computed message static

BUILD SUCCESSFUL
1 actionable task: 1 executed
Configuration cache entry reused.

As you can see, the method is called under the cache. Note that method was not static - it is a plugin’s instance method.

I will not show it, but there could problems with calling method in groovy plugins due to its dynamic nature. So in groovy plugins, its better to call public static methods.

Providers

Provider is “a magic wand” for dealing with configuration cache problems: it workarounds runtime objects restriction (we will see it in part 2). Here we will only see how provider being cached.

public class Sample4Plugin implements Plugin<Project> {

    @Override
    public void apply(Project project) {
        project.getTasks().register("task1").configure(task -> {
            Provider<String> provider = project.provider(() -> {
                String res = String.valueOf(project.findProperty("param"));
                System.out.println("Provider called: " + res);
                return res;
            });
            task.doLast(task1 -> {
                System.out.println("[run] Task exec: " + provider.get());
            });
        });
    }
}

Provider use configuration file property just to show how cache invalidates in this case.

First, run without cache task1 -Pparam=1:

> Task :task1
Provider called: 1
Task exec: 1

BUILD SUCCESSFUL
1 actionable task: 1 executed

As expected, provider was called at runtime.

Run with cache enabled task1 -Pparam=1 —configuration-cache

Calculating task graph as no cached configuration is available for tasks: task1
Provider called: 1

> Task :task1
[run] Task exec: 1

BUILD SUCCESSFUL
1 actionable task: 1 executed
Configuration cache entry stored.

Pay attention that provider was called at configuration time!

Running from cache: task1 -Pparam=1 —configuration-cache

Reusing configuration cache.

> Task :task1
[run] Task exec: 1

Provider not called! It’s value is already stored in cache. As a consequence, you can use any object inside provider because it is executed under configuration time!

And a bonus, what will happen if property value would change task1 -Pparam=2 —configuration-cache:

Calculating task graph as configuration cache cannot be reused because the set of Gradle properties has changed: the value of 'param' was changed.
Provider called: 2

> Task :task1
[run] Task exec: 2

BUILD SUCCESSFUL
1 actionable task: 1 executed
Configuration cache entry stored.

Gradle invalidated the cache.

Note that for system properties and environment variables special providers should be used (gradle would not be able to detect change in this case).

Always called providers

Gradle provides a special kind of providers: ValueSource (it’s mentioned in docs). In contrast to usual providers, value source implementations are always called (even under the cache).

Example source implementation:

public abstract class NonCacheableValue implements ValueSource<String, ValueSourceParameters.None> {

    @Override
    public @Nullable String obtain() {
        String val = System.getProperty("foo");
        System.out.println("NonCacheableValue: " + val);
        return val;
    }
}

The plugin:

public class Sample4ValuePlugin implements Plugin<Project> {

    @Override
    public void apply(Project project) {
        project.getTasks().register("task1").configure(task -> {
            final Provider<String> provider = project.getProviders()
                    .of(NonCacheableValue.class, spec -> {});
            task.doLast(task1 -> {
                System.out.println("[run] Task exec: " + provider.get());
            });
        });
    }
}

Run with enabled cache task1 -Dfoo=1 —configuration-cache:

Calculating task graph as no cached configuration is available for tasks: task1

> Task :task1
NonCacheableValue: 1
[run] Task exec: 1

BUILD SUCCESSFUL
1 actionable task: 1 executed
Configuration cache entry stored.

ValueSource (provider) called at runtime (in contrast to usual providers, which are called under configuration phase when cache is enabled).

Run from cache task1 -Dfoo=1 —configuration-cache:

Reusing configuration cache.

> Task :task1
NonCacheableValue: 1
[run] Task exec: 1

BUILD SUCCESSFUL in 73ms
1 actionable task: 1 executed
Configuration cache entry reused.

ValueSource (provider) is called.

Now changing property value task1 -Dfoo=2 —configuration-cache:

Reusing configuration cache.

> Task :task1
NonCacheableValue: 2
[run] Task exec: 2

BUILD SUCCESSFUL in 46ms
1 actionable task: 1 executed
Configuration cache entry reused.

No invalidation required - provider just provide the correct value.

Putting all together

When configuration cache is enabled, gradle would analyze runtime blocks and serialize related state.

There are no restrictions at configuration time - all restrictions applied at runtime blocks
Plugin itself, task objects and any other objects would not be created under cache (required objects are deserialized)
Task and plugin fields are serialized and so availble on cached executions (see part 2 for more details)
Values from user configuration would be serialized (and so it does not matter for configuration cache if value was directly assigned or taken from the extension)
Object uniquness (when the same instance used in multiple places) could be lost due to deserializarion
Build services:
- Is the only way to keep unique (shared) data between tasks
- Not serialized, so field values would be lost between executions
- Remember only parameter values in time of the first service creation
- Could be closed at any time
- If service listen for tasks it would not be closed
It is safe to call methods from runtime blocks (in groovy plugins better call public static methods)
Provider is called at configuration time (when cache is enabled) and so could use any gradle objects
ValueSource objects are never cached
Gradle tracks state change (like user configuration changes) and invalidate cache (but some edge cases are possible)

Read next the part 2 - problems