System test helpers

[dainiusjocas]

We've created a set of helpers for the system tests written in Java that we believe is worth sharing.

We've wanted BDD style (i.e. given-when-then) and fluent API. A nice aspect is that the implementation by default exposes/requires very little Java specifics and is already powerful enough to allow for granular comparison between multiple responses.

Here I want to present how it looks and fish for feedback. For the presentation I'm using the same application that demonstrates the system tests. This allows to compare the current example with the alternative.

Pasting here the entire equivalent test class (source):

package ai.vespa.example.album;

import ai.vespa.example.album.systemtests.BaseSystemTest;
import ai.vespa.hosted.cd.SystemTest;
import org.junit.jupiter.api.Test;

import java.util.List;

import static org.hamcrest.Matchers.*;

@SystemTest
public class AlternativeFeedAndSearchSystemTest extends BaseSystemTest {

    @Test
    void feedAndSearch() {
        var yql = "SELECT * FROM music WHERE artist CONTAINS 'coldplay'";
        given("all documents are deleted from the cluster")
                .deleteAll("music")
                .expectDeleted();
        when("searching for music")
                .search("yql", yql, "timeout", "10s");
        then("no documents are found")
                .status(200)
                .body("/root/fields/totalCount", equalTo(0L));

        when("a document is fed and searched")
                .feed("music", "test1",
                        "artist", "Coldplay")
                .status(200)
                .and()
                .search("yql", yql, "timeout", "10s");
        then("1 document is found and the artist is Coldplay")
                .body("/root/fields/totalCount", equalTo(1L),
                        "/root/children/0/fields/artist", containsString("Coldplay"));
    }

    @Override
    protected String namespace() {
        return "mynamespace";
    }

    @Override
    protected List<String> contentClusters() {
        return List.of("music");
    }
}

Full application can be found here.

The proposed usage is that your system test class extends the BaseSystemTest which brings this fluent style helpers like given, etc. And the tests can be run everywhere from an IDE all the way to the Vespa Cloud integrated CI/CD.

For the matching we decided to use Hamcrest matchers. This is because it is not trivial to put Vespa responses into some class and then work with AssertJ.

For the selectors in the JsonNode the JsonPointer was picked because Jackson has it by default. Maybe it would be worthwhile to bring in a Jackson-JQ dependency and use that as it allows for transformations.

Happy to contribute the code.