Merge confidence value for OSS maintainers

[thibaudcolas]

Nice work on the merge confidence feature! This is very interesting and feels like there’s lots that could be done with this data like you show in your blog post.

From the perspective of a package maintainer – I think I would be very interested in having access to the merge confidence data for my packages’ updates. Is this something you’ve considered? For example, as a maintainer of draftjs_exporter, when I make a release I would be interested in knowing whether users’ test suites are passing with this new release, whether people have managed to merge the upgrade, etc. Projects like Prettier, or any and all linters generally, could also be projects where it’s very interesting to know how much breakage there is with each release, as the breakage is somewhat inevitable.

Currently there isn’t really a good way to do this at scale in the package management ecosystem. Some package repositories have quantitative info about adoption in the form of "download statistics" for packages, but that’s about it. Some projects publish alpha/beta/RC releases in the hopes of collecting feedback from users, but that’s all very ad-hoc and qualitative rather than quantitative. Merge confidence feels like it could automate this feedback loop.

[rarkins]

Hi @thibaudcolas, thanks very much for your feedback!

We definitely would love to weaponize this capability to help open source maintainers. I think you're already thinking along the same lines like this:

• use Merge Confidence as a filter to deliver open source project maintainers scheduled and grouped updates that "just work" and can be merged. This is not something exclusive for OSS projects but I think that the burden of having to maintain dependencies can be a disincentive to open source things but if we can put it mostly on autopilot then that changes things for the better.
• Create pages per-package so that we can show the historical performance of a package's releases, including drilling down into any low confidence releases and linking to any public repos available
• Allowing Renovate users to "opt in" to silent beta testing of new packages. For example you could publish a new alpha/beta/RC candidate and downstream packages or applications could let Renovate create a branch/commit to trigger testing. The results from this could be fed back to the maintainers of the upstream packages while for the downstream users it could be essentially invisible unless they go looking. The biggest advantage of Merge Confidence data here would be our access to private application test results, which sometimes is the only "true" test of a library. Once we gather this data, there's multiple ways we could close that feedback loop, including upstream issues or even email notifications if the maintainer opts in.

Re: lint packages like prettier, I created #4 to discuss. e.g. it's not necessarily a bad thing if a lint package release breaks things - if it was intended. I've been wanting to enable Renovate to "lint fix" such packages though so interestingly that would not show up as a failure if Renovate ran prettier --write as a post update command. Overall the users would be better with auto-updated source files after a Prettier update although it would obscure our data a little if so.

A related challenge is identifying packages where a low confidence score doesn't mean "don't merge". e.g. there are many cases where if a release has broken a lot of people but not you, then you should still be cautious. But there are cases like prettier where the advice may be "doesn't really matter if it breaks others as long as it doesn't break you".

[thibaudcolas]

Well that’s all very exciting, I’m looking forward to see what you’ll build from there!