React Native Benchmark Suite

[tido64]

Introduction

We are currently blind to any regressions that don't directly translate to pixels on the screen. Performance regressions, in particular, isn't always straight forward to measure as different platforms and devices may yield wildly different results depending on how you do the measurements.

The Core of It

A suite of benchmarks and tools should be run on all PRs, and on every commit to master, to catch regressions (or improvements) compared to a baseline.

What we should measure:

• App Size
  • Measure the app size and working set of a clean react-native init app or RNTester.
• Performance Regressions
  • React initialization time
    • Time to interaction (TTI)
    • Native module initialization
    • JS parsing
    • JS execution
  • Number of thread jumps during initialization
  • Blocked threads, and how long they were blocked for
  • Memory usage
    • Working set
• Memory Leaks
  • Static analysis and sanitizers, e.g. run RNTester with MemorySanitizer
  • Linters, e.g. Clang-Tidy
  • Leverage online code analysis services such as LGTM
  • Essentially, look to tools and automation to help us find bugs, memory leaks, and potential security issues.

How do we report this?

• Make comments on PRs (and commits?) commenting on the delta change for all metrics.
• Every night (or every commit on master) there should be a benchmark run to keep track of performance over time. The report should be publicly available, similar to the Firefox Performance Dashboard.

Suggested Timeline

Task	Estimate	Assignee	Issue #
Implement PR bot for posting comments from CI agents	1 week	@tido64	1 2 3
Nightly CI builds	1 week	@sweggersen	1
Add reporting for performance regressions compared to master	1 week	@tido64	1 2
Add reporting for app bundle size	1 week	@tido64	1
Add app memory usage tests	1 week	@tido64	1
Add reporting for app memory usage	1 week
Implement performance markers for React TTI	2 weeks
Implement performance markers for thread jumps/blocks	2 weeks
RNTester run with MemorySanitizer	2 weeks	@HeyImChris
RNTester run with UndefinedBehaviorSanitizer	2 weeks	@HeyImChris
Add reporting of static analysis results	1 week
Implement perf dashboards	???
Figure out whether we need a device farm or alternate solutions	???

Discussion points

• Thoughts?
• Missing metrics?
• Milestones/timeline?

cc @sweggersen, @danrydholm

[kelset]

Hey @tido64 that sounds like a great idea!

I'll be working more closely on the topic of benchmarking & tooling from April but until then I think that starting this conversation is 🔥

I think that we should try to come up with at least a baseline of what is "acceptable" (ex. 200ms for a full-round from click to visual change), maybe on a range of different devices (ex. android+old, android+new, ios+old, ios+new).

The one benchmark that I can think of missing here is the "app bundling" benchmark, which could underline suboptimal (or improvements) of the native-side settings of the mobile app project; at least, this was an area I managed to find a few easy wins the last times I've done some performances improvements for a prod app I was working on -> https://gist.github.com/kelset/2eb61161a68d1ab35337d1eaa9a05e78

[kelset]

Was totally forgetting, but @Kudo did a lot of work in the benchmarking space -> https://github.com/Kudo/react-native-js-benchmark

[tido64]

@kelset That's great to hear. Do you have some sort of plan for what you want to do? Regarding the baseline, I like the idea of having goals. I was thinking more of having a diff against master but we should have both. Can you explain what you mean by "app bundling", and how it affects performance?

@Kudo How much work do you think it will take to bring this to iOS as well?

[oblador]

I'm interested in this problem space and can help out 👍

[kelset]

Do you have some sort of plan for what you want to do?

So the plan is a bit hand-wavey still but I'll be working 6 weeks full time on a react-native performances open source book (both from perspective of how to measure, finding the baselines, bottlenecks, etc and how to act upon those). So for your proposal I think that I'll be able to help you out with the sides of baseline & benchmarking (at least). But I'll only start in April AFAIK.

Regarding the baseline, I like the idea of having goals. I was thinking more of having a diff against master but we should have both.

Just for me to better understand - with diff you mean like a report of the "delta" between commits? Like, "this commit has made the TTI 30 ms slower"?

Can you explain what you mean by "app bundling", and how it affects performance?

So, by app bundling I mean the process that both iOS and Android need to perform to generate the .ipa/.apk. I think that by keeping an eye on this too we could make sure that whenever native-side configurations are touched, they are not leading to less optimal processes (ex. let's say a commit changes the compiler optimization, if we check the build time we could detect if it's actually improving the overall time VS making it more slower, etc).

Another side effect of this is that it would help detect easy wins in terms of the native-side project configuration: let's say that we never realised that we can use the App Bundling split feature of Android (can't recall the precise name of it atm, the one that only bundles for the precise architecture)(I may be missremembering completely btw, my brain is ded atm). By having a baseline (ex. a fresh new Android app takes 2 mins to generate the apk) we could detect that our config, because it was first created let's say 3 years ago, it's x10 slower because we didn't update that particular config to take advantage of the new native side tooling. And the side effect of this is that then the app will be built more efficiently for the device it will run against, leading to then better "in-app" perfs.

[tido64]

Just for me to better understand - with diff you mean like a report of the "delta" between commits? Like, "this commit has made the TTI 30 ms slower"?

Correct. The baseline should the branch you're merging against, most commonly master branch. That way, we catch perf regressions before they get merged.

So, by app bundling I mean the process that both iOS and Android need to perform to generate the .ipa/.apk. I think that by keeping an eye on this too we could make sure that whenever native-side configurations are touched, they are not leading to less optimal processes (ex. let's say a commit changes the compiler optimization, if we check the build time we could detect if it's actually improving the overall time VS making it more slower, etc).

I'm not sure if this makes sense. If we misconfigured React Native to be built with with -Og instead of -Os, we will improve build time but performance will be worse for sure. Maybe it makes more sense to open up the bundles and see what's in there? E.g. for Android, we can check whether you've created bundles for each supported architecture, and that all the .so files only include code for the target architecture. The checks are more explicit and there will be less confusion.

[kelset]

I see - makes sense!

Ok then let's leave the app bundling out of this... there's plenty already anyway ;)

[tido64]

I've added a table of tasks and estimates. Let me know if I missed anything. Also feel free to assign yourself to any of them 😉

[elicwhite]

Thanks for adding the table! The measurement that we get asked for a lot internally is what the “working set” is of React Native. All the people that ask for that are ex Microsoft :-). We aren’t fully sure how we are supposed to measure that, and there is no one better to help us get that set up than you! :-)

I’d like to make sure that is a part of this as that will help us get long awaited answers for our VPs asking for it.

[shergin]

BTW, we have a couple of synth performance test in new Fabric infra, and we (I) plan to add a couple more that will cover the most performance-critical (hot paths) aspects of the UI part of the framework (prop parsing, diffing, layout, nodes cloning and so on). We use Google Benchmark for that.

The challenging (unexplored) part of that is running that continuously on CI and running that on ARM CPUs to get actual results for architectures we care about (x86 is usually fast enough).

[tido64]

@shergin That's awesome! How much work is it to get them pushed to the open source repo? I think it's a good starting point regardless. We can figure out how to get non-x86 results after we have them running continuously. I've added a task for it.

@TheSavior Updated the table. I think we have a team internally who have solved this. I can hear with them.

[Kudo]

react-native-js-benchmark currently focused at JS engine comparison but I think it could support to change as CI regression monitoring.

For iOS support, the most difficult part is about running on real devices including the developer certificates and code signing.
Fortunately, we are focusing on CI regression, testing by simulators would be enough.
I could add iOS simulators support from my very limited free time, maybe 1.5 weeks estimation.

[danrydholm]

From the MS side, we're going to look into funding CI using AppCenter which has all the real devices we would need for iOS and Android.

[Kudo]

Please let me clarify if my understanding is correct.
Does the benchmark focus at regression across different builds?
What I did for react-native-js-benchmark is a fixed pre-configured project that is not for regression.

I had a similar work which to use RN-cli to generate a pure project and patch project to test my V8 integration.
These scripts are written in Python and things like that I should rewrite a JavaScript version.