EPIC: support multi-dimensional testing

[laurentsenta]

As a libp2p maintainer, I want the ability to define test suites that combine many implementations, many muxers, many transports, etc. Defining and running these test suites should be simple, and the outcome should be clear. It should be easy to trigger these test suites before a release. It should be easy to display these results in a readable form on a website.

eta: 2022Q4

Tasks

• Create a "simple" interop dashboard #55
  • this will add the tooling used to describe, build, and run multiple combinations of tests in Testground
• Implement new parameters (muxers, transports, etc)
  • we need to define what the matrix looks like
  • we need to update the ping test to support these parameters
    • ⚠️ we need to make sure we test the connection from A to B AND from B to A (we don't at the moment)
• add the "expected RTT" test
  • This requires the ability to "generate" the expected outcomes for every cell in this N-D matrix,
• [tracking issue] Interop test-plans for all existing/developing libp2p transports #61
• Generate a "nice" dashboard Canonical interop tests & dashboard #62
  • @BigLep shared some notes in Create a "simple" interop dashboard #55 (comment)
  • use case 1: As a maintainer, I want to check the status of our interop testing before making a release.
  • use case 2: As a user, I want to know more about libp2p interoperability capabilities.

Follow-up tasks

• "optimize" by skipping versions libraries that are not used anymore (see notes below),
• "optimize" using artifact caching,

Description

A high-level approach:

1. first, we use versions resources file to generate "complex" test matrixes,
2. then we use another resource (data or code) to produce the expected RTT matrix,
3. then we generate the relevant composition file (as shown below), with the expected RTT as a parameter,
4. then we iterate through these test cases and call testground run.

Configurations

Ideally, the libp2p team provides a resource file that contains the versions and their features:

[[groups]]
# go v0.42
GoVersion = '1.18'
Modfile = "go.v0.22.mod"
Selector = 'v0.42'
Implementation = 'go'
SupportedTransports = ["tcp", "quic", "webrtc"]
Muxer = ["yamux"]

[[groups]]
# go v0.22
GoVersion = '1.18'
Modfile = "go.v0.22.mod"
Selector = 'v0.22'
Implementation = 'go'
SupportedTransports = ["tcp", "quic"]
Muxer = ["mplex"]


[[groups]]
# rust v0.51
Libp2pVersion = 'v0.51.0'
Implementation = 'rust'
SupportedTransports = ["tcp", "webrtc"]

[[groups]]
# rust v0.47.0
Libp2pVersion = 'v0.47.0'
Implementation = 'rust'
SupportedTransports = ["tcp"]
Muxer = ["yamux", "mplex"]

And we'll create some way to have "meta-compositions" that can describe multiple tests and run many pairs together,

something like
(pseucode)

{ for every group }
    { if !group.SupportedTransports contains ENV.TESTED_TRANSPORT }
        { continue }
    { end }

    { if !group.SupportedTransports contains ENV.TESTED_MUXER }
        { continue }
    { end }

    [testground_instance]
    {}
{ endfor }

Called with

ENV.TESTED_TRANSPORT = "tcp"
ENV.TESTED_MUXER = "yamux"
testground run composition file

Related discussions and issues

• Adding a webrtc test #49 (comment)
• Organization: add PROCESS.md and ROADMAP.md #44 (comment)
• Create a "simple" interop dashboard #55

[laurentsenta]

cc @julian88110 @tinydb @mxinden @galargh

[laurentsenta]

note from sync w/ @marten-seemann (he was away for the chat yesterday and had related requests).

@mxinden @marten-seemann I have no strong feelings about this, but we might lose information between impromptu meetings. Do you feel the need to schedule some sort of "interop squad sync" where everyone joins at the same time?

Some more good ideas were brought up. A few notes:

Generating Matrix Parameters

The point of this epic is to let the libp2p team define a multi-dimensional test matrix,
and make sure it's maintainable. This is covered by the configuration above.

Using this configuration file, we can generate multiple tests, for example:

• every pair that can communicate over quic.
• every group of versions that can use TCP + yamux.

Implicitly this configuration represents compatible pairs. This means we have a matrix of implementation * versions * muxer * transports and each cell in this matrix is YES (interoperable) or NO (not interoperable).

An extension to this matrix is the RTT use case: @marten-seemann needs to add "expected RTTs" for each of these pairs. Each cell in the matrix will contain the number of RTTs an expected RTT in milliseconds.

A high-level approach:

• first, we use the versions resources file (above) to generate the test matrix,
• then we use another resource (data or code) to produce the expected RTT matrix,
• then we generate the relevant composition file (as shown above), with the expected RTT as a parameter,
• then we iterate through these test cases and call testground run.

I plan to iterate through solutions:

• first, implement an interop suite that runs in CI and test pairs of instances, and generate "some" dashboards.
• second, extend this feature to support the full matrix of muxer * transports * etc...
• third, extends this feature to support the RTT matrix.

Unknown: this is focused on generating test pairs, how to express "I want to create a group with every webrtc-compatible implementation"? Do we need it?

Supported versions

The size of the test matrix will explode quickly. This should not be a problem to create the configuration files, but this will be a problem for the time to build & run test suites.

I believe EKS will solve some of this problem (with caching out of the box and having enough resources to enable parallelization), but we'll have to pick and choose which versions and combinations to test eventually.

One option is to test the most used version, see the diagram:

http://162.55.187.75:3000/d/CSQsORs7k/nebula?orgId=1&viewPanel=11
We have ~10 versions that cover ~60% of the network + "others"

We can use these metrics to enable/disable versions, probably something like:

• X latest versions,
• X most used versions

This solution sounds reasonable enough to keep this problem for later.

Config Structure

keep in mind: muxer + security applies only to TCP, so it might be helpful to use a nested structure instead of:

conf:
  SupportedTransports = ["tcp", "quic", "webrtc"]
  Muxer = ["yamux"]

Something like this might represent our matrix better, and it might be easier to expand.

conf:
   transports:
       TCP:
          muxer: yamux
       quic: true

We don't have to decide on this now (data is easy to transform), I recommend we stick to the flat structure for now.

[marten-seemann]

An extension to this matrix is the RTT use case: @marten-seemann needs to add "expected RTT" for each of these pairs. Each cell in the matrix will contain an expected RTT in milliseconds.

Minor correction: It’s expected handshake duration, and it’s a dimensionless number: the number of RTTs. Calculating the actual duration will be done by the test plan, based on that number and the RTT that’s used for the run.

[mxinden]

keep in mind: muxer + security applies only to TCP, so it might be helpful to use a nested structure instead of:

We will face the same problem on the dimension of multiplexer-negotiation (via multistream-select or via security protocol) as this is only relevant for TCP.

I think expressing hierarchies within dimensions (e.g. tcp/noise/mplex, tcp/noise/yamux) is a valid option.

[mxinden]

@mxinden @marten-seemann I have no strong feelings about this, but we might lose information between impromptu meetings. Do you feel the need to schedule some sort of "interop squad sync" where everyone joins at the same time?

I would suggest to continue doing these ad-hoc. That said, this is not a strong opinion.

[mxinden]

Unknown: this is focused on generating test pairs, how to express "I want to create a group with every webrtc-compatible implementation"? Do we need it?

While long-term we will need this, I do think we should focus on point-to-point testing for now, i.e. two nodes (potentially 3 including a relay node for WebRTC browser-to-browser) instead of a group of nodes.

[julian88110]

I have created a google doc as a way to capture our test case requirements, filled in with the info I know, please take a look and if you can, fill in the part especially regarding JS, rust. We can migrate this doc to github once it is in a good shape. https://docs.google.com/document/d/1-akPPFW7kko9SkpedxXOV2foWJhnliELGd1WKn-0RFw/edit#

[John-LittleBearLabs]

I plan to iterate through solutions:

Does this mean that you're working on this, @laurentsenta ?

then we iterate through these test cases and call testground run.

... & ...

first, implement an interop suite that runs in CI and test pairs of instances, and generate "some" dashboards.

I had thought the matrix would be a single composition with some elaborate template structure. But it sounds like you're thinking of writing a higher-level script that orchestrates calls into testground?

If so, could/should this sort of matrix instead be a new feature added to testground itself, for reuse?