Nearly all ECMA-402 APIs (and some in ECMA-262) have implementation- and locale-dependent ("ILD") behavior, and those details change over time (e.g. with new browser customizations or CLDR/ICU releases). Relying on "golden output" is therefore brittle, but we also want to achieve good coverage.
There are options such as looking at stable substrings or other parts of output (such as month names), comparative testing across related inputs (e.g. that each setting for an input option produces a distinct output), general metamorphic testing, and broader decisions to make such as the set of locales and locale options to utilize (and whether or not to always utilize all of them).
Guidance should be added to CONTRIBUTING.md in the form of both explanations and examples. A proposed first step in establishing such guidance is to start with a relatively simple ILD API, and Intl.RelativeTimeFormat seems like a good candidate because it supports a small number of options (just numberingSystem, style "long" vs. "short" vs. "narrow", and numeric "always" vs. "auto") and its formatted output includes both numeric and textual parts (the former of which depends upon and can therefore be compared against Intl.NumberFormat functionality). Let's take a look at the existing tests and see what conclusions can be drawn from them.
Nearly all ECMA-402 APIs (and some in ECMA-262) have implementation- and locale-dependent ("ILD") behavior, and those details change over time (e.g. with new browser customizations or CLDR/ICU releases). Relying on "golden output" is therefore brittle, but we also want to achieve good coverage.
There are options such as looking at stable substrings or other parts of output (such as month names), comparative testing across related inputs (e.g. that each setting for an input option produces a distinct output), general metamorphic testing, and broader decisions to make such as the set of locales and locale options to utilize (and whether or not to always utilize all of them).
Guidance should be added to CONTRIBUTING.md in the form of both explanations and examples. A proposed first step in establishing such guidance is to start with a relatively simple ILD API, and Intl.RelativeTimeFormat seems like a good candidate because it supports a small number of options (just numberingSystem, style "long" vs. "short" vs. "narrow", and numeric "always" vs. "auto") and its formatted output includes both numeric and textual parts (the former of which depends upon and can therefore be compared against Intl.NumberFormat functionality). Let's take a look at the existing tests and see what conclusions can be drawn from them.