Have you ever been in a situation where your tests pass but the code has bugs?

It could be a matter of a missing scenario you forgot to test… But sometimes, this is due to differences in data between the data your test uses and the actual data the code runs against.

Coverage isn’t only about branch coverage, it’s also about covering different flows of data!

If your tests can’t find bugs due to data variations, you need to test more variations, but adding them manually isn’t always the best choice.

It can work well when the data is limited, but it’s hard to scale and requires a lot of maintenance to keep working when you need to refactor the tests because more variations have been added or the data model has changed.

There’s a much better way to test scenarios where data is dynamic:

Introducing - Property Based Testing

In today’s article I’m going to showcase Property Based Testing - a testing technique that can potentially replace hundreds of your unit tests with a single property based test.

The key in Property Based Testing is that instead of defining specific input and outputs, you define rules for generating these inputs and for asserting outputs.

For example, you can replace a test that checks whether an email is valid with a test that generates 2000 emails and checks that they’re valid.

I’m going to demonstrate Property Based Testing by replacing some unit tests that failed to find a bug, with a property based test that gives us much higher data flow coverage and more confidence our code works.

You will also learn more about what property based testing is and when to use it.

A Testing Scenario - Starting with A Unit Test

Let’s say you have an input which accepts a time string and returns a ms amount. You can use this to allow users to write time based inputs, and from it get a date of a time in the future:

Now let’s add some unit tests for it. We’re going to hardcode the inputs and the outputs, this is also called Example Based Testing:

This is a nice start, but because users could specify any time string they want, there are many more possible inputs that we didn’t test for.

The above code actually has a bug in it, let’s see what happens when we try to specify a time string with a negative sign in it:

There’s a bug in the code that the unit test didn’t catch.

This is because it’s hard to anticipate all the possible inputs users could provide, negative signs is one example to the kind of edge cases that are often not tested in Example Based Testing.

While we could add more unit tests to cover more scenarios, it will be hard to maintain, and we will likely only cover a small subset of the inputs from what we actually can with Property Based Testing.

So let’s see how we can use Property Based Testing to test this code, but first you’ll need to understand what it is, and how to write a Property Based Test in TypeScript.

What is Property Based Testing

To understand what property based testing is, let’s first understand what a property is:

In Property Based Testing, a property is a rule or invariant about how a function should behave, given a range of inputs.

A property describes a behavior - we can’t assert that 1d is equal to 86400000 milliseconds, but we can assert that we should get a valid ms result for any integers followed by d.

An example of more properties can be:

• Non-Negative Result - the function should always return a non-negative number of milliseconds
• Partial input - should work even with a subset time string, for example only 1d2h or just 1h.
• Negative Inputs - should reduce the time if input is negative, for example -1d should return -86400000 milliseconds

Property based testing is about testing that properties hold true over a range of inputs.

Or in simpler words, it’s about generating hundreds or thousands of variations in data and testing our code against it.

npm install --save-dev fast-check

function assert<Ts>(property: IProperty<Ts>, params?: Parameters<Ts>): void;

fc.property(...arbitraries, (...args) => {});

fc.assert(
        fc.property(fc.array(fc.integer()), (data) => {
            const sortedData = sortNumbersAscending(data);
            for (let i = 1; i < data.length; ++i) {
            expect(sortedData[i - 1]).toBeLessThanOrEqual(sortedData[i]);
        }
    }),
);

Property Based Tests Examples

Let’s see how a basic property based tests run, notice the console to look at all the generated values:

Now let’s modify it a bit, so it tests our use case with the time string:

As you can see the tst is failing as it found the bug we had in our code related to negative signs in the time string.

Let’s fix the code so the test passes:

We can also take it a step further by testing for partial inputs, and reversed order:

And that’s it! We took our simple unit tests and improved their coverage by testing them against a far more comprehensive dataset.

Tradeoffs

You can’t talk about property based testing without talking about some of the tradeoffs it has. The main three are:

1. Complexity - it is harder to construct a property based test, it will take more time and require more thought to think about what properties you’d like to test.
2. Execution time - property based tests will increase your test execution time. You can play with the number of runs to reduce the time it takes to run or add parallelism in your CI to handle the additional tests
3. Relying on programmatic logic - you’re always relying on some programmatic logic to assert your tests, this means that a mistake in the logic could cause false positives or false negatives.

In contrary to example based tests, where you can rely on hard-coded examples you know are right. It can definitely be beneficial to have this confidence in cases where it’s hard to verify correctness solely based on programmatic logic.

So feel free to mix Property Based Testing with Example Based Testing.

Summary

Property based testing can help you test code which can accept high variations in data.

By defining properties, which are like rules, you are able to generate thousands of datasets and test those rules on them to make sure your system works accordingly.

This increases the scope of what you’re testing and helps you find bugs and edge cases earlier, rather than waiting till a user with different data finds them.

With that being said, property based tests are more complex to write and take more time to execute - write them when you actually need to test a highly dynamic data set and when example based tests don’t suffice on their own.