Improving Spring Boot Test efficiency

Problem statement

Spring test framework creates an application context according to test class configuration. The context is cached and reused for all subsequent tests. If there is an existing context with the same configuration, it will be reused. Otherwise, the new context will be created. This is a very efficient and flexible approach, but it has a drawback: eventually this may lead to out of memory errors if the number of unique configurations is too high and context has a lot of heavyweight beans like TestContainers. In many cases simple static bean definition can help, but this project suggests another approach: reordering test classes and eager context cleanup.

Consider a sample test suite of 8 classes that use 4 different configurations, classes that have the same configuration are marked with the same colour:

In a large test suites as well as in shared CI/CD environments with lots of test pipelines working simultaneously this may eventually lead to out of memory errors in Java process or Docker host.

Proposed solution

It's recommended to use statically-defined TestContainers beans, optimize reusing same configuration between tests e.g. via common test super-classes. But additionally this library makes two optimizations:

• test class execution is reordered to group tests with the same context configuration so they can be executed sequentially
• the order of tests is known, so if current test class is last per current configuration, the spring context will be automatically closed (it's called Smart DirtiesContext) and the beans will be disposed releasing resources

As a result, in a suite of single module there will always be not more than 1 active spring contexts:

This chart is done via calculating the number of active docker containers while executing a suite of 120 integration test classes that actively uses TestContainers for databases (several datasources simultaneously) and other services (as spring-managed Beans):

As shown on the chart, the suite just fails with OOM without the optimization. As an advantage, the total test execution time will also become less, because resource consumption (especially memory) will be reduced, hence tests are executed faster.

References

• This idea was submitted to the Spring Framework team as a feature request: spring-framework#32289
• Public presentation at Spring I/O: recording slides
• Public presentation with AtomicJar (TestContainers creators): recording

Limitations

At the moment only single thread test execution per module is supported. Parallel test execution is work in progress. Also there can be problems with Jupiter Nested test classes if they declare own @ContextConfiguration or @Import of spring beans.

Supported versions

Java 8+ (Java 17+ for spring-boot 3.x projects)

Spring Boot 2.4+, 3.x, 4.x as well as bare Spring framework

Supported test frameworks:

• JUnit 4 (via JUnit junit-vintage-engine) removed; for JUnit 4 support use old 0.15 version
• JUnit 5/6 Jupiter
• TestNG (both bare TestNG and JUnit platform testng-engine)
• Kotest

Known test execution caching solutions correctly support changed behaviour.

How to use

Add maven dependency (available in maven central):

<dependency>
    <groupId>com.github.seregamorph</groupId>
    <artifactId>spring-test-smart-context</artifactId>
    <version>0.15</version>
    <scope>test</scope>
</dependency>

Or Gradle dependency:

testImplementation("com.github.seregamorph:spring-test-smart-context:0.15")

Then remove usages of standard @DirtiesContext annotation in your project on root test classes to use smart auto-close context implemented by this library instead. Keep the annotations for tests that really "dirty" context changing the shared context state in a way that may lead to conflicts with other tests.

Also it's recommended to configure "INFO" level for com.github.seregamorph.testsmartcontext logger.

Check the Demo projects for examples.

How it works

JUnit 5/6 Jupiter

Add the library dependency in test scope, it will automatically setup SmartDirtiesClassOrderer which will reorder test classes before execution and prepare the list of last test class per context configuration. Then this test execution listener SmartDirtiesContextTestExecutionListener will be auto-discovered via spring.factories. Alternatively it can be defined explicitly

@TestExecutionListeners(listeners = {
        SmartDirtiesContextTestExecutionListener.class
}, mergeMode = TestExecutionListeners.MergeMode.MERGE_WITH_DEFAULTS)

or even inherited from AbstractJUnitSpringIntegrationTest

TestNG

Add the library dependency in test scope, it will automatically setup SmartDirtiesSuiteListener which will reorder test classes before execution and prepare the list of last test class per context configuration. The integration test classes should add SmartDirtiesContextTestExecutionListener

@TestExecutionListeners(listeners = {
        SmartDirtiesContextTestExecutionListener.class
}, mergeMode = TestExecutionListeners.MergeMode.MERGE_WITH_DEFAULTS)

Note: the annotation is inherited, so it makes sense to annotate the base test class or use AbstractTestNGSpringIntegrationTest parent.

Kotest

Add the library dependency in test scope, it will automatically setup SmartDirtiesSpecExecutionOrderExtension which relies on enabled Kotest autoscan. It reorders Spec classes before execution and prepares the list of test class per context configuration.

JUnit 4

Note: support of JUnit 4 is removed in version 1.0. In case you need it, check the 0.15 README.

Known projects using library

Miro is using this approach to optimize huge integration test suites and it saved a lot of resource for CI/CD pipelines.

Related projects

The Spring Test Profiler is a Spring Test utility that provides visualization and insights for Spring Test execution, with a focus on Spring context caching. It helps you identify optimization opportunities in your Spring Test suite to speed up your builds. This project is not alternative, but can be complementary to spring-test-smart-context library.