Introduction

Approval Testing

Approval testing is a technique that allows you to compare the output of your code with a known good/previously approved output.

An approval test case will only succeed, if the actually received output is not equal to the previously approved output.

If the received output is different from the approved output, the test will fail and leave it to human reviewer to approve received output or to fix the code.

Approval testing is especially useful for testing complex objects or large data sets, where it is impractical to write individual assertions for each property.

ApproveJ

ApproveJ is a Java implementation of approval testing with a builder-based fluent API, several built-in tools, and optional extension points.

To review the code, file issues or suggest changes, please visit the project’s GitHub page.

Getting Started

Requirements

In oder to use ApproveJ you need a JDK 21 or higher.

Dependencies

To use ApproveJ in your own project, you need to add it as a dependency.

Gradle
testImplementation 'org.approvej:core:0.5.1'
Gradle.kts
testImplementation("org.approvej:core:0.5.1")
Maven
<dependency>
  <groupId>org.approvej</groupId>
  <artifactId>core</artifactId>
  <version>0.5.1</version>
  <scope>test</scope>
</dependency>

Bill of Materials (BOM)

If you want to use more than one module in the same project, you can use ApproveJ’s bill of materials (BOM) and omit the explicit version for the other modules.

Gradle
implementation platform('org.approvej:bom:0.5.1')
implementation 'org.approvej:json-jackson'
Gradle.kts
implementation(platform("org.approvej:bom:0.5.1"))
implementation("org.approvej:json-jackson")
Maven
<project>
  <!--…-->
  <dependencyManagement>
    <dependencies>
      <dependency>
        <groupId>org.approvej</groupId>
        <artifactId>bom</artifactId>
        <version>0.5.1</version>
        <type>pom</type>
        <scope>import</scope>
      </dependency>
    </dependencies>
  </dependencyManagement>
  <!-- … -->
  <dependencies>
    <dependency>
      <groupId>org.approvej</groupId>
      <artifactId>json-jackson</artifactId>
    </dependency>
  </dependencies>
  <!-- … -->
</project>

Basics

The general entry point to start an approval is the static initializer approve of the ApprovalBuilder. It takes the object which you want to approve an argument and returns a builder to configure the approval with a fluent API.

Approve Strings

If you have anything that returns an arbitrary string, you can simply build an approval like this

Java
String result = hello("World");

approve(result) (1)
    .verify(); (2)
Kotlin
val result = hello("World")

approve(result) (1)
  .verify() (2)
1 creates an ApprovalBuilder<String>
2 compares result to a previously approved value stored in a file next to the test and fails the test if the result differs

Executing such a test, will create two files next to the test code file named like <TestClass>-<testMethod>-received.txt and <TestClass>-<testMethod>-approved.txt.

The received file will always contain a string representation of the object you want to approve at the last execution.

The approved file will be empty at first. You can use a diff tool of your choice to compare the two files and merge values that you want to approve. If the received value equals the content of the approved file, the received file will be deleted automatically.

Approved file
Hello, World!

You can adjust various details of this process or use no file at all. See Verifying — adjust the verification for more details on how to do that.

Approve POJOs

Of course, you might want to approve more complex objects than just strings.

For example a simple POJO like this

Person POJO
public record Person(String name, LocalDate birthDate) {}

By default, ApproveJ will simply call the object’s toString method to turn the object into a string just before verifying.

Java
Person person = createPerson("John Doe", LocalDate.of(1990, 1, 1));

approve(person) (1)
    .verify(); (2)
Kotlin
val person = createPerson("John Doe", LocalDate.of(1990, 1, 1))

approve(person) (1)
  .verify() (2)
1 creates an ApprovalBuilder<Person> approve the person
2 compares the person.toString() to a previously approved version and fails if they are not equal
Approved file
Person[name=John Doe, birthDate=1990-01-01]

See Printing — customize how values are turned into Strings if need a more sophisticated way of printing.

Printing — customize how values are turned into Strings

While some toString implementations already are quite good, they typically return a one-liner. This is fine as long as you only have a few properties. However, if you have a lot of properties, it is much easier to read the result if it is formatted nicely.

In order to change the way objects are being transformed to strings, you can use the ApprovalBuilder's printWith method.

Generic Object Printer

ApproveJ provides a generic ObjectPrinter that will print the object with each of its properties on a new line to make comparing the result easier. You can use this printer by calling the printWith method on the builder.

Java
Person person = createPerson("John Doe", LocalDate.of(1990, 1, 1));

approve(person)
    .printWith(objectPrinter()) (1)
    .verify();
Kotlin
val person = createPerson("John Doe", LocalDate.of(1990, 1, 1))

approve(person)
  .printWith(objectPrinter()) (1)
  .verify()
1 applies the ObjectPrinter and returns a new ApprovalBuilder<String>

Now the approved file will look like this

Approved file
Person [
  birthDate=1990-01-01,
  name=John Doe
]

Custom Printer

You can also provide a custom Printer/Function<T, String> implementation to the builder’s printWith method.

Java
Person person = createPerson("John Doe", LocalDate.of(1990, 1, 1));

approve(person)
    .printWith(it -> String.format("%s, born %s", it.name(), it.birthDate())) (1)
    .verify();
Kotlin
val person = createPerson("John Doe", LocalDate.of(1990, 1, 1))

approve(person)
  .printWith { "%s, born %s".format(it.name, it.birthDate) } (1)
  .verify()
1 applies the given Printer and returns a new ApprovalBuilder<String>
Approved file
John Doe, born 1990-01-01

Additionally, a custom Printer can override the filenameExtension method. If you use a FileVerifier (see Verifying — adjust the verification), the returned String is used as the files' filename extension. This is useful, if your Printer creates a certain format (e.g. JSON, XML, YAML, …).

Scrubbing — make random parts static

Sometimes you might not be able to control the exact output of the object you want to approve. For example, if the result object contains a timestamp or a generated ID, you might want to ignore these for the approval.

You can do this by using the scrubbedOf method of the ApprovalBuilder and provide a Scrubber/UnaryOperator<T> implementation.

For instance, in the following BlogPost POJO there are two generated fields:

Java
class BlogPost {
  private final UUID id;
  private final String title;
  private final String content;
  private final LocalDateTime published;

  public BlogPost(String title, String content) {
    this.id = UUID.randomUUID(); (1)
    this.title = title;
    this.content = content;
    this.published = LocalDateTime.now(); (2)
  }

  public String title() {
    return title;
  }

  public String content() {
    return content;
  }

  public LocalDateTime published() {
    return published;
  }

  public UUID id() {
    return id;
  }

  @Override
  public String toString() {
    return "BlogPost[title=%s, content=%s, published=%s, id=%s]"
        .formatted(title, content, published, id);
  }
}
1 the id is a UUID that’s being generated randomly, and
2 the published is a LocalDateTime set to now.

In the flowing example, the two dynamic properties are scrubbed with the built-in Scrubbers for uuids and instants.

Java
BlogPost blogPost =
    createBlogPost("Latest News", "Lorem ipsum dolor sit amet, consectetur adipiscing elit.");

approve(blogPost)
    .printWith(objectPrinter())
    .scrubbedOf(instants(ISO_LOCAL_DATE_TIME)) (1)
    .scrubbedOf(uuids()) (2)
    .verify(); (3)
Kotlin
val blogPost = createBlogPost("Latest News", "Lorem ipsum dolor sit amet, consectetur adipiscing elit.")

approve(blogPost)
  .printWith(objectPrinter())
  .scrubbedOf(instants(ISO_LOCAL_DATE_TIME)) (1)
  .scrubbedOf(uuids()) (2)
  .verify()
1 replaces the published date with a numbered placeholder
2 replaces the id UUID with a numbered placeholder
3 so that the verified result looks like this
Approved file
BlogPost [
  content=Lorem ipsum dolor sit amet, consectetur adipiscing elit.,
  id=[uuid 1],
  published=[instant 1],
  title=Latest News
]

Generally a built-in Scrubber uses a replacement function that replaces all matches with a numbered placeholder in the form of [<label> <counter>] (e.g. [uuid 1], [date 2], …). Note that two matches of the same (e.g. the same UUID in two places) will be replaced with the same placeholder, so you can still see that two scrubbed values were equal.

Custom Scrubber

The RegexScrubber already allows for a lot of special custom cases. In case this isn’t enough, you can also provide a custom Scrubber<T>/UnaryOperator<T> implementation to the builder’s scrubbedOf method.

Java
Contact contact = createContact("Jane Doe", "jane@approvej.org", "+1 123 456 7890");
approve(contact)
    .scrubbedOf(it -> new Contact(-1, it.name(), it.email(), it.phoneNumber())) (1)
    .printWith(objectPrinter())
    .verify(); (2)
Kotlin
val contact = createContact("Jane Doe", "jane@approvej.org", "+1 123 456 7890")

approve(contact)
  .scrubbedOf { Contact(-1, it.name, it.email, it.phoneNumber) } (1)
  .printWith(objectPrinter())
  .verify() (2)
1 this custom Scrubber specifically the number property of the Contact with a constant
2 so that the verified result looks like this
Approved file
Contact [
  email=jane@approvej.org,
  name=Jane Doe,
  number=-1,
  phoneNumber=+1 123 456 7890
]

Note that this Scrubber is a Scrubber<Contact> and not a Scrubber<String>. Hence, it is necessary to apply it before the Printer is applied.

Of course the Scrubber can also be defined in a separate class, instead of a lambda.

Built-In Scrubbers

All built-in Scrubber implementations are available via the Scrubbers utility class.

Verifying — adjust the verification

InplaceVerifier

The InplaceVerifier will verify the result with a directly provided previously approved value. That way, the approved value is plainly visible in the test code. However, this might not be practical for large objects. It also does not allow to use a diff tool to compare the result with the previously approved value.

Java
Person person = createPerson("John Doe", LocalDate.of(1990, 1, 1));

approve(person).verify("Person[name=John Doe, birthDate=1990-01-01]");
Kotlin
val person = createPerson("John Doe", LocalDate.of(1990, 1, 1))

approve(person)
  .verify("Person[name=John Doe, birthDate=1990-01-01]")

FileVerifier

The FileVerifier will verify the result with a previously approved value stored in a file. It is used in most of the examples above.

If no approved file exists, it will be created as an empty approved file. The received value will be written to another received file.

If the approved file exists, it will be compared with the received value. If they are equal, the test will pass. Any existing received file will be deleted automatically in that case.

If the files are not equal, the test will fail. The received value will be persisted in a received file. Any existing received value will be overwritten by this.

You can use a diff tool of your choice to compare the two files and merge values that you want to approve.

Next to Test

By default, the FileVerifier will use a NextToTestPathProvider, which will put the received and approved files next to the test class and name them like the test case method.

You can make this explicit by using the inFile method.

Java
Person person = createPerson("John Doe", LocalDate.of(1990, 1, 1));

approve(person).verify(inFile()); (1)
Kotlin
val person = createPerson("John Doe", LocalDate.of(1990, 1, 1))

approve(person)
  .verify(inFile()) (1)
1 defines the NextToTestPathProvider explicitly, same as just calling verify()
File structure
.
└── 📁src/test/java/…
    ├── 📄 <TestClass>.java
    ├── 📄 <TestClass>-<testMethod>-approved.txt
    └── 📄 <TestClass>-<testMethod>-received.txt
Custom Filename Extension

The filenameExtension method gives you the opportunity to use a different file extension for the approved and received files.

Java
Person person = createPerson("John Doe", LocalDate.of(1990, 1, 1));

approve(person)
    .printWith(personYamlPrinter()) (1)
    .verify(inFile(nextToTest().filenameExtension("yaml"))); (2)
Kotlin
val person = createPerson("John Doe", LocalDate.of(1990, 1, 1))

approve(person)
  .printWith(personYamlPrinter()) (1)
  .verify(inFile(nextToTest().filenameExtension("yaml"))) (2)
1 this printer will create a YAML version of the object
2 so it makes sense to change the filename extension, so your IDE will apply appropriate syntax highlighting
File structure with custom filename extension
.
└── 📁src/test/java/…
    ├── 📄 <TestClass>.java
    ├── 📄 <TestClass>-<testMethod>-approved.<filenameExtension>
    └── 📄 <TestClass>-<testMethod>-received.<filenameExtension>
In a Subdirectory

If you have test classes with a lot of approval tests, there a quite a lot of files created next to the test class. In that case, you can use the inSubdirectory to instruct the NextToTestPathProvider, which will create a directory next to the test class and put the approved and received files in there.

Java
Person person = createPerson("John Doe", LocalDate.of(1990, 1, 1));

approve(person)
    .printWith(personYamlPrinter())
    .verify(inFile(nextToTest().inSubdirectory().filenameExtension("yaml")));
Kotlin
val person = createPerson("John Doe", LocalDate.of(1990, 1, 1))

approve(person)
  .printWith(personYamlPrinter())
  .verify(inFile(nextToTest().inSubdirectory().filenameExtension("yaml")))
File structure with subdirectory
.
└── 📁src/test/java/…
    ├── 📁 <TestClass>
    │   ├── 📄 <testMethod>-approved.<filenameExtension>
    │   └── 📄 <testMethod>-received.<filenameExtension>
    └── 📄 <TestClass>.java
Given Path

Alternatively, you can simply specify the path of the approved file. If the given approved file path contains the word approved just before the filename extension, it will be replaced with received in the to determine the received file path. Otherwise, the word received will be added at the end of the filename.

For example

  • src/test/resources/BasicsDocTest-verify_file_approved_path-approved.yaml → src/test/resources/BasicsDocTest-verify_file_approved_path-received.yaml

  • src/test/resources/BasicsDocTest-verify_file_approved_path.yaml → src/test/resources/BasicsDocTest-verify_file_approved_path-received.yaml.

Java
Person person = createPerson("John Doe", LocalDate.of(1990, 1, 1));

approve(person)
    .printWith(personYamlPrinter())
    .verify(inFile("src/test/resources/BasicExamples-verify_file_approved_path.yaml")); (1)
Kotlin
val person = createPerson("John Doe", LocalDate.of(1990, 1, 1))

approve(person)
  .printWith(personYamlPrinter())
  .verify(inFile("src/test/resources/BasicExamples-verify_file_approved_path.yaml")) (1)
1 this will expect the approved file at this path, the received file will be created next to it at src/test/resources/BasicsDocTest-verify_file_approved_path-reveived.yaml
File structure with given path
.
└── 📁src/test/java/…
│   └── 📄 <TestClass>.java
└── 📁src/test/resources
    ├── 📄 src/test/resources/BasicExamples-verify_file_approved_path.yaml
    └── 📄 src/test/resources/BasicExamples-verify_file_approved_path-received.yaml
Custom PathProvider

You can also define your own PathProvider.

Built-In Verifiers and PathProviders

All built-in Verifier and PathProvider implementations are available via the Verifiers and PathProviders utility classes.

JSON with Jackson

The json-jackson module provides several JSON-related features impelemented with Jackson.

To use it, you need to add it as a dependency to your project.

Gradle
implementation 'org.approvej:json-jackson:0.5.1'
Gradle.kts
implementation("org.approvej:json-jackson:0.5.1")
Maven
<project>
  <!-- … -->
  <dependencies>
    <dependency>
      <groupId>org.approvej</groupId>
      <artifactId>json-jackson</artifactId>
      <version>0.5.1</version>
    </dependency>
  </dependencies>
  <!-- … -->
</project>

Scrub JSON

The JsonPointerScrubber can be used to scrub a JSON node identified by a JsonPointer.

Compared to generic Scrubber<String> implementations this is particularly useful when there are several values matching the same pattern, but only one of them needs to be scrubbed. For instance, if you have a JSON containing two UUIDs, one that was generated by the code (and hence needs to be scrubbed) and one that is a reference to another resource and should not be scrubbed.

Java
String createdBlogPostJson =
    createTaggedBlogPost(
        "Latest News",
        "Lorem ipsum dolor sit amet, consectetur adipiscing elit.",
        List.of(NEWS, ENTERTAINMENT));

approve(jsonMapper.readTree(createdBlogPostJson))
    .scrubbedOf(jsonPointer("/id").replacement("[scrubbed id]")) (1)
    .scrubbedOf(jsonPointer("/published").replacement("[scrubbed published]")) (2)
    .verify(inFile(nextToTest().filenameExtension("json"))); (3)
Kotlin
val createdBlogPostJson = createTaggedBlogPost(
  "Latest News",
  "Lorem ipsum dolor sit amet, consectetur adipiscing elit.",
  listOf(NEWS, ENTERTAINMENT)
)

approve(jsonMapper.readTree(createdBlogPostJson))
  .scrubbedOf(jsonPointer("/id").replacement("[scrubbed id]")) (1)
  .scrubbedOf(jsonPointer("/published").replacement("[scrubbed published]")) (2)
  .verify(inFile(nextToTest().filenameExtension("json"))) (3)
1 scrubs the dynamically assigned id node on the root level of the JSON and replaces it with "[scrubbed id]"
2 scrubs the published timestamp node on the root level of the JSON and replaces it with "[scrubbed published]"
3 stores the received data at a file next to the test with the file extension .json
Approved file
{"id":"[scrubbed id]","title":"Latest News","content":"Lorem ipsum dolor sit amet, consectetur adipiscing elit.","tagIds":["00000000-0000-0000-0000-000000000001","00000000-0000-0000-0000-000000000002"],"published":"[scrubbed published]"}

Pretty Print JSON

In the example above, the JSON persisted in a one-liner. Even for the simple example, this is not very readable, let alone easily comparable to a slightly different JSON. To improve this, you can use the JsonPrettyPrinter.

Java
String createdBlogPostJson =
    createTaggedBlogPost(
        "Latest News",
        "Lorem ipsum dolor sit amet, consectetur adipiscing elit.",
        List.of(NEWS, ENTERTAINMENT));

approve(jsonMapper.readTree(createdBlogPostJson))
    .scrubbedOf(jsonPointer("/id").replacement("[scrubbed id]"))
    .scrubbedOf(jsonPointer("/published").replacement("[scrubbed published]"))
    .printWith(jsonPrettyPrinter()) (1)
    .verify();
Kotlin
val createdBlogPostJson = createTaggedBlogPost(
  "Latest News",
  "Lorem ipsum dolor sit amet, consectetur adipiscing elit.",
  listOf(NEWS, ENTERTAINMENT)
)

approve(jsonMapper.readTree(createdBlogPostJson))
  .scrubbedOf(jsonPointer("/id").replacement("[scrubbed id]"))
  .scrubbedOf(jsonPointer("/published").replacement("[scrubbed published]"))
  .printWith(jsonPrettyPrinter()) (1)
  .verify()
1 applies the JsonPrettyPrinter to convert the JsonNode object to a string

Now the approved file is much more readable.

Approved file
{
  "id" : "[scrubbed id]",
  "title" : "Latest News",
  "content" : "Lorem ipsum dolor sit amet, consectetur adipiscing elit.",
  "tagIds" : [ "00000000-0000-0000-0000-000000000001", "00000000-0000-0000-0000-000000000002" ],
  "published" : "[scrubbed published]"
}

Pretty Print JSON Strings

Sometimes you might not need/want to parse a received JSON string into a JsonNode object. In this case, you can use the JsonStringPrettyPrinter to pretty print a JSON string directly.

Java
String createdBlogPostJson =
    createTaggedBlogPost(
        "Latest News",
        "Lorem ipsum dolor sit amet, consectetur adipiscing elit.",
        List.of(NEWS, ENTERTAINMENT));

approve(createdBlogPostJson)
    .scrubbedOf(uuids())
    .scrubbedOf(instants(DateTimeFormatter.ISO_INSTANT))
    .printWith(jsonStringPrettyPrinter()) (1)
    .verify();
Kotlin
val createdBlogPostJson = createTaggedBlogPost(
  "Latest News",
  "Lorem ipsum dolor sit amet, consectetur adipiscing elit.",
  listOf(NEWS, ENTERTAINMENT)
)

approve(createdBlogPostJson)
  .scrubbedOf(uuids())
  .scrubbedOf(instants(DateTimeFormatter.ISO_INSTANT))
  .printWith(jsonStringPrettyPrinter()) (1)
  .verify()
1 applies the JsonPrettyPrinter to convert the JsonNode object to a string
Approved file
{
  "id" : "[uuid 1]",
  "title" : "Latest News",
  "content" : "Lorem ipsum dolor sit amet, consectetur adipiscing elit.",
  "tagIds" : [ "[uuid 2]", "[uuid 3]" ],
  "published" : "[instant 1]"
}

Note that the applied scrubbers also replaced the tag IDs.