Create standardized CI helpers for community modules

[acoates-ms]

Introduction

As the community expands we want to ensure that community modules can provide the highest quality features in a reliable way. More repositories have a need to setup and maintain CI loops and testing infrastructure for community modules, additional platforms, and the core project itself to provide this reliability.

We need to develop some standard helpers so that these projects can benefit from shared work and all get the best testing without having everyone invest huge amounts of time investing and maintaining these solutions.

The Core of It

Various things that would be useful to have solutions for:

• A package, or set of packages that provide scripts that can test community JS modules
• A package, or set of packages that provide scripts that can test community native modules modules
• Solutions for testing iOS, Android and additional out of tree platforms such as Windows and MacOS.
• Tests auto-linking and E2E running of native modules and view managers on a new app
• Tests for bundle size, memory usage other performance metrics and trends
• Tests to target multiple versions of RN
• Tests to verify upgrade scenarios
• Documentation and best practices on what testing should be done and how to setup CI.
• Other tests that we work out over time that would be useful to use more broadly.

Discussion points

While additional helpers could be CI infra dependent the core scripts should be agnostic to running in GitHub Actions, CircleCI, AzureDevOps etc.

I recognize this is very light on details. For now lets get some input on what kinds of testing we should try to standardize, what tools to use. Whats working for people, whats not. And general thoughts on how best to achieve this.

[acoates-ms]

At Microsoft one of the things we are planning is to go to many of the popular community modules and add support for react-native-windows, and MacOS. While doing this we want to ensure that we are providing test infrastructure for Windows and MacOS to these packages. We'll be working out some min bar for our Windows / MacOS contributions. But right now things are vastly different in all the community modules. It would help everyone if there was some more consistency.

[elicwhite]

Here is some related art that might be interesting to you:

• https://github.com/react-native-community/react-native-circleci-orb "A CircleCI Orb to simplify testing your React Native app." cc @matt-oakes
• https://github.com/react-native-community/tester "facilitate testing of React Native libraries within a single easy to upgrade tester app" cc @bartolkaruza
• https://github.com/react-native-community/ci-sample "Showcase for using ci to build react native app." cc @gengjiawen
• https://github.com/react-native-community/react-native-dummy "'Dummy' unfunctional out-of-tree platform for unit testing RNPM + Metro" cc @empyrical

[gengjiawen]

react-native-community/ci-sample "Showcase for using ci to build react native app." cc @gengjiawen

Building for android is pretty simple for all kinds of CI when using docker. Pretty simple and clean. For iOS part I am not a expert through :)

[acoates-ms]

Microsoft is working on contributing to various existing community modules to contriubute macOS and windows implementations. As we work through that process we'll be trying to ensure that those community modules have a good set of validation CI processes. We'll be doing that using where they meet our needs, but also help flush out issues with the existing tools as needed. But will also likely be creating additional tools where we feel there are gaps.

Generally we view this as something we'll have more information and opinions on once we've worked in a few community repos and got a better sense of the current state of things.

[harinikmsft]

cc @tom-un, @chrisglein - you can chime in here with what you are discovering as part of contributing windows/mac support to some existing community modules.

[acdlite]

Hi everyone! @TheSavior asked me to chime in to share what the React core team does to make it easier for outside teams to test against the latest changes. I don't know how helpful this will be because React core/DOM releases are a bit simpler than React Native releases — everything is distributed via npm; there's no native code (i.e. releases are platform agnostic); the API surface area is smaller and managed completely by the React core team — but I figure it can't hurt to share:

We had a recent incident where a minor React release accidentally broke users of React Native Web because of an internal API change. There was a similar incident where one release included accidental support for an edge case behavior that Next.js came to rely on, and then broke when a subsequent release happened to remove it.

In response to these incidents, we came up with a strategy for sharing upcoming changes so we can receive bug reports and feedback before they reach stable.

We now have three separate release channels: Stable, Next, and Experimental:

• The Stable channel is what is sounds like: publicly available, semver-compliant releases, e.g. v16.12.0.
• The Next channel represents the next minor release. These releases do not follow semver; versions are assigned based on the source revision and track the the React repo's master branch (e.g. v0.0.0-1022ee0ec). (We don't publish on every commit, but theoretically we could. We're thinking we might move to a nightly release cadence via a cron job; right now we publish via a manually-initiated script about once a week.) But the idea is that any React project should be able to run on the Next release without breaking. And therefore, any React project's test suite should pass when running against Next.
• The Experimental channel is the same as the Next channel (e.g. v0.0.0-1022ee0ec-experimental) except with additional feature flags turned on. This channel is for sharing unfinished features, so unlike Next it may include intentional breaking changes.

Details about the release channels are available here: https://reactjs.org/docs/release-channels.html

Our recommendation is that outside teams can set up a cron job to periodically run their project's test suite against the most recent release in the Next channel. If something breaks, they should file a bug report. (Similarly, teams that want to try out unstable features can run against the Experimental channel, though occasional breakages are to be expected as features are added/removed/changed.) Several teams including Next.js have adopted this strategy.

This has worked pretty well for us so far. Other strategies we considered included running the test suites of important open source projects as part of our own CI pipeline. That would remove the burden from other teams needing to manage their own CI jobs, and being responsible for reporting failures. We may still do some version of this in the future; however, this has the downside of only working for open source projects, whereas the prerelease strategy works for closed source projects, too.

We've also discussed making the reporting process easier by, say, setting up an API endpoint that teams can ping when something goes wrong, and a dashboard to track the status of various projects. We haven't done anything like that yet, though.

[elicwhite]

Thank you so much for sharing @acdlite!

One of the things I like about the React model is that infrastructure could be built that helps support any team test and report changes back with their CI. That feels more reasonable to me than some group of people trying to maintain a test app that includes a bunch of common modules who might not have expertise on each individual part.

The incentives are also aligned by having the teams maintaining the projects wanting to keep their CI running against latest and reporting back feedback.