Introduction
Approval Testing
Approval testing is a technique that allows you to compare the output of your code with a known good/previously approved output.
An approval test case will only succeed, if the actually received output is equal to the previously approved output.
If the received output is different from the approved output, the test will fail and leave it to human reviewer to approve received output or to fix the code.
Approval testing is especially useful for testing complex objects or large data sets, where it is impractical to write individual assertions for each property.
ApproveJ
ApproveJ is a Java implementation of approval testing with a builder-based fluent API, several built-in tools, and optional extension points.
To review the code, file issues or suggest changes, please visit the project’s GitHub page.
Getting Started
Requirements
In oder to use ApproveJ you need a JDK 21 or higher.
Dependencies
To use ApproveJ in your own project, you need to add it as a dependency.
testImplementation 'org.approvej:core:0.9.2'
testImplementation("org.approvej:core:0.9.2")
<dependency>
<groupId>org.approvej</groupId>
<artifactId>core</artifactId>
<version>0.9.2</version>
<scope>test</scope>
</dependency>
Bill of Materials (BOM)
If you want to use more than one module in the same project, you can use ApproveJ’s bill of materials (BOM) and omit the explicit version for the other modules.
implementation platform('org.approvej:bom:0.9.2')
implementation 'org.approvej:json-jackson'
implementation(platform("org.approvej:bom:0.9.2"))
implementation("org.approvej:json-jackson")
<project>
<!--…-->
<dependencyManagement>
<dependencies>
<dependency>
<groupId>org.approvej</groupId>
<artifactId>bom</artifactId>
<version>0.9.2</version>
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies>
</dependencyManagement>
<!-- … -->
<dependencies>
<dependency>
<groupId>org.approvej</groupId>
<artifactId>json-jackson</artifactId>
</dependency>
</dependencies>
<!-- … -->
</project>
Basics
The general entry point to start an approval is the static initializer approve
of the ApprovalBuilder
.
It takes the object which you want to approve as an argument and returns a builder to configure the approval with a fluent API.
Approve Strings
If you have anything that returns an arbitrary string, you can simply build an approval like this
String result = hello("World");
approve(result) (1)
.byFile(); (2)
val result = hello("World")
approve(result) (1)
.byFile() (2)
1 | creates an ApprovalBuilder<String> |
2 | compares result to a previously approved value stored in a file next to the test and fails the test if the result differs |
Executing such a test, will create two files next to the test code file named like <TestClass>-<testMethod>-received.txt
and <TestClass>-<testMethod>-approved.txt
.
The received file will always contain a string representation of the object you want to approve at the last execution.
The approved file will be empty at first. You can use a diff tool of your choice to compare the two files and merge values that you want to approve. If the received value equals the content of the approved file, the received file will be deleted automatically.
Hello, World!
You can adjust various details of this process:
Approve POJOs
Of course, you might want to approve more complex objects than just strings.
For example a simple POJO like this
public record Person(String name, LocalDate birthDate) {}
By default, ApproveJ will simply call the object’s toString
method to turn the object into a string just before approving it.
Person person = createPerson("John Doe", LocalDate.of(1990, 1, 1));
approve(person) (1)
.byFile();
val person = createPerson("John Doe", LocalDate.of(1990, 1, 1))
approve(person) (1)
.byFile()
1 | creates an ApprovalBuilder<Person> approve the person |
Will approve the following value:
Person[name=John Doe, birthDate=1990-01-01]
See Printing — customize how values are turned into Strings if need a more sophisticated way of printing.
Printing — customize how values are turned into Strings
While some toString
implementations already are quite good, they typically return a one-liner.
This is fine as long as you only have a few properties.
However, if you have a lot of properties, it is much easier to read the result if it is formatted nicely.
In order to change the way objects are being transformed to strings, you can use the ApprovalBuilder
's printWith
method and provide a Printer
/Function<T, String>
instance as an argument.
Generic Object Printer
ApproveJ provides a generic ObjectPrinter
that will print the object with each of its properties on a new line to make comparing the result easier.
You can use this printer by calling the printWith
method on the builder.
Person person = createPerson("John Doe", LocalDate.of(1990, 1, 1));
approve(person)
.printWith(objectPrinter()) (1)
.byFile();
val person = createPerson("John Doe", LocalDate.of(1990, 1, 1))
approve(person)
.printWith(objectPrinter()) (1)
.byFile()
1 | applies the ObjectPrinter and returns a new ApprovalBuilder<String> |
Now the approved file will look like this
Person [
birthDate=1990-01-01,
name=John Doe
]
Custom Printer Function
You can provide a custom Function<T, String>
to the builder’s printWith
method.
Person person = createPerson("John Doe", LocalDate.of(1990, 1, 1));
approve(person)
.printWith(it -> String.format("%s, born %s", it.name(), it.birthDate())) (1)
.byFile();
val person = createPerson("John Doe", LocalDate.of(1990, 1, 1))
approve(person)
.printWith { "%s, born %s".format(it.name, it.birthDate) } (1)
.byFile()
1 | applies the given Function and returns a new ApprovalBuilder<String> |
So the content of the approved file will look like this
John Doe, born 1990-01-01
Custom Printer Implementation
For more complex cases, you can implement your own Printer
.
This will allow you to also override the filenameExtension
method.
If you use a FileApprover
(see Approving — adjust the verification), it will be used to determine the files' filename extension.
This is useful, if your Printer
creates a certain format (e.g. JSON, XML, YAML, …).
E.g. the following implementation will print a Person
as a YAML string.
public static class PersonYamlPrinter implements Printer<Person> {
@Override
public String apply(Person person) {
return """
person:
name: "%s"
birthDate: "%s"
"""
.formatted(person.name(), person.birthDate());
}
@Override
public String filenameExtension() {
return "yaml";
}
}
class PersonYamlPrinter : Printer<Person> {
override fun apply(person: Person) =
"""
person:
name: "${person.name}"
birthDate: "${person.birthDate}"
"""
.trimIndent()
override fun filenameExtension() = "yaml"
}
The resulting file will look like this
person:
name: "John Doe"
birthDate: "1990-01-01"
Scrubbing — make random parts static
Sometimes you might not be able to control the exact output of the object you want to approve. For example, if the result object contains a timestamp or a generated ID, you might want to ignore these for the approval.
You can do this by using the scrubbedOf
method of the ApprovalBuilder
and provide a Scrubber
/UnaryOperator<T>
implementation.
For instance, in the following BlogPost
POJO there are two generated fields:
class BlogPost {
private final UUID id;
private final String title;
private final String content;
private final Instant published;
public BlogPost(String title, String content) {
this.id = UUID.randomUUID(); (1)
this.title = title;
this.content = content;
this.published = Instant.now(); (2)
}
public String title() {
return title;
}
public String content() {
return content;
}
public Instant published() {
return published;
}
public UUID id() {
return id;
}
@Override
public String toString() {
return "BlogPost[title=%s, content=%s, published=%s, id=%s]"
.formatted(title, content, published, id);
}
}
1 | the id is a UUID that’s being generated randomly, and |
2 | the published is a LocalDateTime set to now. |
In the flowing example, the two dynamic properties are scrubbed with the built-in Scrubbers for uuids
and instants
.
BlogPost blogPost =
createBlogPost("Latest News", "Lorem ipsum dolor sit amet, consectetur adipiscing elit.");
approve(blogPost)
.printWith(objectPrinter())
.scrubbedOf(dateTimeFormat("yyyy-MM-dd'T'HH:mm:ss.SSSX")) (1)
.scrubbedOf(uuids()) (2)
.byFile(); (3)
val blogPost =
createBlogPost("Latest News", "Lorem ipsum dolor sit amet, consectetur adipiscing elit.")
approve(blogPost)
.printWith(objectPrinter())
.scrubbedOf(dateTimeFormat("yyyy-MM-dd'T'HH:mm:ss.SSSX")) (1)
.scrubbedOf(uuids()) (2)
.byFile()
1 | replaces the published date with a numbered placeholder |
2 | replaces the id UUID with a numbered placeholder |
3 | so that the approved result looks like this |
BlogPost [
content=Lorem ipsum dolor sit amet, consectetur adipiscing elit.,
id=[uuid 1],
published=[datetime 1],
title=Latest News
]
Generally a built-in Scrubber
uses a replacement function that replaces all matches with a numbered placeholder in the form of [<label> <counter>]
(e.g. [uuid 1]
, [date 2]
, …).
Note that two matches of the same (e.g. the same UUID in two places) will be replaced with the same placeholder, so you can still see that two scrubbed values were equal.
Custom Scrubber
The RegexScrubber
already allows for a lot of special custom cases.
In case this isn’t enough, you can also provide a custom Scrubber<T>
/UnaryOperator<T>
implementation to the builder’s scrubbedOf
method.
Contact contact = createContact("Jane Doe", "jane@approvej.org", "+1 123 456 7890");
approve(contact)
.scrubbedOf(it -> new Contact(-1, it.name(), it.email(), it.phoneNumber())) (1)
.printWith(objectPrinter())
.byFile();
val contact = createContact("Jane Doe", "jane@approvej.org", "+1 123 456 7890")
approve(contact)
.scrubbedOf { Contact(-1, it.name, it.email, it.phoneNumber) } (1)
.printWith(objectPrinter())
.byFile()
1 | this custom Scrubber specifically replaces the number property of the Contact with a constant |
Contact [
email=jane@approvej.org,
name=Jane Doe,
number=-1,
phoneNumber=+1 123 456 7890
]
Note that this Scrubber
is a Scrubber<Contact>
and not a Scrubber<String>
.
Hence, it is necessary to apply it before the Printer
is applied.
In case you want to reuse the Scrubber
, you can also define in a separate class implementing the Scrubber<T>
interface.
Built-In Scrubbers
All built-in Scrubber
implementations are available via the Scrubbers utility class.
Approving — adjust the verification
You conclude the ApprovalBuilder
by specifying by
which Approver
the received value should be approved.
Approve by Value
The ApprovalBuilder.byValue()
method will use an InplaceApprover
to approve the received value by comparison with a directly provided previously approved value.
That way, the approved value is plainly visible in the test code.
However, this might not be practical for large objects.
It also does not allow to use a diff tool to compare the result with the previously approved value.
Person person = createPerson("John Doe", LocalDate.of(1990, 1, 1));
approve(person).byValue("Person[name=John Doe, birthDate=1990-01-01]");
val person = createPerson("John Doe", LocalDate.of(1990, 1, 1))
approve(person).byValue("Person[name=John Doe, birthDate=1990-01-01]")
Approve by File
The ApprovalBuilder.byFile()
method will use a FileApprover
to approve the received value by comparison a previously approved value stored in a file.
It is used in most of the examples above.
If no approved file exists, it will be created as an empty approved file. The received value will be written to another received file.
If the approved file exists, it will be compared with the received value. If they are equal, the test will pass. Any existing received file will be deleted automatically in that case.
If the files are not equal, the test will fail. The received value will be persisted in a received file. Any existing received value will be overwritten by this.
You can use a diff tool of your choice to compare the two files and merge values that you want to approve.
Next to Test
By default, the FileApprover
will put the received and approved files next to the test class and name them like the test case method.
You can make this explicit by using the PathProviderBuilder.nextToTest()
method.
Person person = createPerson("John Doe", LocalDate.of(1990, 1, 1));
approve(person).byFile(nextToTest()); (1)
val person = createPerson("John Doe", LocalDate.of(1990, 1, 1))
approve(person).byFile(nextToTest()) (1)
1 | defines the PathProviderBuilder explicitly, same as just calling byFile() |
. └── 📁src/test/java/… ├── 📄 <TestClass>.java ├── 📄 <TestClass>-<testMethod>-approved.txt └── 📄 <TestClass>-<testMethod>-received.txt
Custom Filename Extension
The PathProviderBuilder.filenameExtension
method gives you the opportunity to use a different file extension for the approved and received files.
- NOTE
-
most of the time you probably want to do this because you’re using a special printer that creates a specific format (e.g. JSON, XML, YAML, …). In that case, you might want to provide a Custom Printer Implementation and override the
filenameExtension
method of thePrinter
instead of changing the filename extension here.
Person person = createPerson("John Doe", LocalDate.of(1990, 1, 1));
approve(person)
.printWith(
it ->
"""
person:
name: "%s"
birthDate: "%s"
"""
.formatted(it.name(), it.birthDate())) (1)
.byFile(nextToTest().filenameExtension("yml"));
val person = createPerson("John Doe", LocalDate.of(1990, 1, 1))
approve(person)
.printWith {
"""
person:
name: "${it.name}"
birthDate: "${it.birthDate}"
"""
.trimIndent()
} (1)
.byFile(nextToTest().filenameExtension("yml")) (2)
1 | this printer will create a YAML version of the object |
2 | so it makes sense to change the filename extension, so your IDE will apply appropriate syntax highlighting |
. └── 📁src/test/java/… ├── 📄 <TestClass>.java ├── 📄 <TestClass>-<testMethod>-approved.<filenameExtension> └── 📄 <TestClass>-<testMethod>-received.<filenameExtension>
In a Subdirectory
If you have test classes with a lot of approval tests, there a quite a lot of files created next to the test class.
In that case, you can use the PathProviderBuilder.nextToTestInSubdirectory
to put all the files in a subdirectory named after the test class.
Person person = createPerson("John Doe", LocalDate.of(1990, 1, 1));
approve(person).printWith(new PersonYamlPrinter()).byFile(nextToTestInSubdirectory());
val person = createPerson("John Doe", LocalDate.of(1990, 1, 1))
approve(person).printWith(PersonYamlPrinter()).byFile(nextToTestInSubdirectory())
. └── 📁src/test/java/… ├── 📁 <TestClass> │ ├── 📄 <testMethod>-approved.txt │ └── 📄 <testMethod>-received.txt └── 📄 <TestClass>.java
Given Path
Alternatively, you can simply specify the path of the approved file.
If the given approved file path contains the word approved
just before the filename extension, it will be replaced with received
in the to determine the received file path.
Otherwise, the word received
will be added at the end of the filename.
For example
-
src/test/resources/BasicsDocTest-approve_file_approved_path-approved.yaml
→src/test/resources/BasicsDocTest-approve_file_approved_path-received.yaml
-
src/test/resources/BasicsDocTest-approve_file_approved_path.yaml
→src/test/resources/BasicsDocTest-approve_file_approved_path-received.yaml
.
Person person = createPerson("John Doe", LocalDate.of(1990, 1, 1));
approve(person)
.printWith(new PersonYamlPrinter())
.byFile("src/test/resources/BasicExamples-approve_file_approved_path.yaml"); (1)
val person = createPerson("John Doe", LocalDate.of(1990, 1, 1))
approve(person)
.printWith(PersonYamlPrinter())
.byFile("src/test/resources/BasicExamples-approve_file_approved_path.yaml") (1)
1 | this will expect the approved file at this path, the received file will be created next to it at src/test/resources/BasicsDocTest-approve_file_approved_path-reveived.yaml |
. └── 📁src/test/java/… │ └── 📄 <TestClass>.java └── 📁src/test/resources ├── 📄 src/test/resources/BasicExamples-approve_file_approved_path.yaml └── 📄 src/test/resources/BasicExamples-approve_file_approved_path-received.yaml
Custom PathProvider
/PathProviderBuilder
You can also define your own PathProvider
and pass it to the byFile
method.
Or you can create a method that returns a PathProviderBuilder
and pass it to the byFile
method.
That way the filename extension of the used Printer
is set just before approval.
In that case, you might want to take advantage of the StackTraceTestFinderUtil
class to find the test source path or the current test method based on the current stack trace.
Reviewing — check differences
If the received value differs from the previously approved, ApproveJ will by default simply fail the test. You then need to review the differences and decide if these are to be approved or actually were not intended. This can simply be done by find the failing test and compare the received and approved files within you IDE or with an external tool. To help you with that process, ApproveJ allows to configure a script that will open such a tool automatically (see Configuration).
Blocking/Non-Blocking Review
Some FileReviewers
are blocking.
They trigger the diff/merge tool and wait for you to close it again.
This gives you the opportunity to merge the content of the two files before the test finishes, so the test does not fail due to your given approval.
Other implementations are non-blocking. These simply display the differences between the two files, but will fail the test immediately. So, after you merged the files or fixed the code, you need to run the test again.
Choose Reviewer in Test
You can choose the FileReviewer
to use in the test code by calling the reviewWith
method on the ApprovalBuilder
.
Person person = createPerson("John Doe", LocalDate.of(1990, 1, 1));
approve(person)
.printWith(new PersonYamlPrinter())
.reviewWith("idea diff {receivedFile} {approvedFile}") (1)
.byFile(); (2)
val person = createPerson("John Doe", LocalDate.of(1990, 1, 1))
approve(person)
.printWith(PersonYamlPrinter())
.reviewWith("idea diff {receivedFile} {approvedFile}") (1)
.byFile() (2)
1 | sets the given script to be executed to support the review |
2 | executes the review script if the received value differs from the approved value |
Configuration
Optionally you can configure the default behaviors of ApproveJ by putting an approvej.properties
file into your resource folder of your test suite (e.g. src/test/resources/approvej.properties
).
approvej.properties
defaultPrinter = org.approvej.print.ToStringPrinter defaultFileReviewerScript = diff {receivedFile} {approvedFile}
Supported properties are:
defaultPrinter
-
the default printer class if none is specified via the
printWith
method (see Printing — customize how values are turned into Strings)
JSON with Jackson
The json-jackson
module provides several JSON-related features implemented with Jackson.
To use it, you need to add it as a dependency to your project.
implementation 'org.approvej:json-jackson:0.9.2'
implementation("org.approvej:json-jackson:0.9.2")
<project>
<!-- … -->
<dependencies>
<dependency>
<groupId>org.approvej</groupId>
<artifactId>json-jackson</artifactId>
<version>0.9.2</version>
</dependency>
</dependencies>
<!-- … -->
</project>
Scrub JSON
The JsonPointerScrubber
can be used to scrub a JSON node identified by a JsonPointer
.
Compared to generic Scrubber<String>
implementations this is particularly useful when there are several values matching the same pattern, but only one of them needs to be scrubbed.
For instance, if you have a JSON containing two UUIDs, one that was generated by the code (and hence needs to be scrubbed) and one that is a reference to another resource and should not be scrubbed.
String createdBlogPostJson =
createTaggedBlogPost(
"Latest News",
"Lorem ipsum dolor sit amet, consectetur adipiscing elit.",
List.of(NEWS, ENTERTAINMENT));
approve(jsonMapper.readTree(createdBlogPostJson))
.scrubbedOf(jsonPointer("/id").replacement("[scrubbed id]")) (1)
.scrubbedOf(jsonPointer("/published").replacement("[scrubbed published]")) (2)
.by(file(nextToTest().filenameExtension("json"))); (3)
val createdBlogPostJson =
createTaggedBlogPost(
"Latest News",
"Lorem ipsum dolor sit amet, consectetur adipiscing elit.",
listOf(NEWS, ENTERTAINMENT),
)
approve(jsonMapper.readTree(createdBlogPostJson))
.scrubbedOf(jsonPointer("/id").replacement("[scrubbed id]")) (1)
.scrubbedOf(jsonPointer("/published").replacement("[scrubbed published]")) (2)
.by(file(nextToTest().filenameExtension("json"))) (3)
1 | scrubs the dynamically assigned id node on the root level of the JSON and replaces it with "[scrubbed id]" |
2 | scrubs the published timestamp node on the root level of the JSON and replaces it with "[scrubbed published]" |
3 | stores the received data at a file next to the test with the file extension .json |
{"id":"[scrubbed id]","title":"Latest News","content":"Lorem ipsum dolor sit amet, consectetur adipiscing elit.","tagIds":["00000000-0000-0000-0000-000000000001","00000000-0000-0000-0000-000000000002"],"published":"[scrubbed published]"}
Pretty Print JSON
In the example above, the JSON persisted in a one-liner.
Even for the simple example, this is not very readable, let alone easily comparable to a slightly different JSON.
To improve this, you can use the JsonPrettyPrinter
.
String createdBlogPostJson =
createTaggedBlogPost(
"Latest News",
"Lorem ipsum dolor sit amet, consectetur adipiscing elit.",
List.of(NEWS, ENTERTAINMENT));
approve(jsonMapper.readTree(createdBlogPostJson))
.scrubbedOf(jsonPointer("/id").replacement("[scrubbed id]"))
.scrubbedOf(jsonPointer("/published").replacement("[scrubbed published]"))
.printWith(jsonPrettyPrinter()) (1)
.byFile();
val createdBlogPostJson =
createTaggedBlogPost(
"Latest News",
"Lorem ipsum dolor sit amet, consectetur adipiscing elit.",
listOf(NEWS, ENTERTAINMENT),
)
approve(jsonMapper.readTree(createdBlogPostJson))
.scrubbedOf(jsonPointer("/id").replacement("[scrubbed id]"))
.scrubbedOf(jsonPointer("/published").replacement("[scrubbed published]"))
.printWith(jsonPrettyPrinter()) (1)
.byFile()
1 | applies the JsonPrettyPrinter to convert the JsonNode object to a string |
Now the approved file is much more readable.
{
"id" : "[scrubbed id]",
"title" : "Latest News",
"content" : "Lorem ipsum dolor sit amet, consectetur adipiscing elit.",
"tagIds" : [ "00000000-0000-0000-0000-000000000001", "00000000-0000-0000-0000-000000000002" ],
"published" : "[scrubbed published]"
}
Pretty Print JSON Strings
Sometimes you might not need/want to parse a received JSON string into a JsonNode
object.
In this case, you can use the JsonStringPrettyPrinter
to pretty print a JSON string directly.
String createdBlogPostJson =
createTaggedBlogPost(
"Latest News",
"Lorem ipsum dolor sit amet, consectetur adipiscing elit.",
List.of(NEWS, ENTERTAINMENT));
approve(createdBlogPostJson)
.scrubbedOf(uuids())
.scrubbedOf(dateTimeFormat("yyyy-MM-dd'T'HH:mm:ss.SSSX"))
.printWith(jsonStringPrettyPrinter()) (1)
.byFile();
val createdBlogPostJson =
createTaggedBlogPost(
"Latest News",
"Lorem ipsum dolor sit amet, consectetur adipiscing elit.",
listOf(NEWS, ENTERTAINMENT),
)
approve(createdBlogPostJson)
.scrubbedOf(uuids())
.scrubbedOf(dateTimeFormat("yyyy-MM-dd'T'HH:mm:ss.SSSX"))
.printWith(jsonStringPrettyPrinter()) (1)
.byFile()
1 | applies the JsonPrettyPrinter to convert the JsonNode object to a string |
{
"id" : "[uuid 1]",
"title" : "Latest News",
"content" : "Lorem ipsum dolor sit amet, consectetur adipiscing elit.",
"tagIds" : [ "[uuid 2]", "[uuid 3]" ],
"published" : "[datetime 1]"
}
Note that the applied scrubbers also replaced the tag IDs.
YAML with Jackson
The yaml-jackson
module provides several YAML-related features implemented with Jackson.
To use it, you need to add it as a dependency to your project.
implementation 'org.approvej:yaml-jackson:0.9.2'
implementation("org.approvej:yaml-jackson:0.9.2")
<project>
<!-- … -->
<dependencies>
<dependency>
<groupId>org.approvej</groupId>
<artifactId>yaml-jackson</artifactId>
<version>0.9.2</version>
</dependency>
</dependencies>
<!-- … -->
</project>
Print as YAML
The YamlPrinter
allows to print any object in YAML format.
Person person = createPerson("John Doe", LocalDate.of(1990, 1, 1));
approve(person)
.printWith(yamlPrinter()) (1)
.byFile();
val person = createPerson("John Doe", LocalDate.of(1990, 1, 1))
approve(person)
.printWith(yamlPrinter()) (1)
.byFile()
1 | applies the YamlPrinter to convert the Person object to a string |
Creates the following approved file:
---
name: "John Doe"
birthDate: "1990-01-01"