Research software quality assessment combines insights from the two worlds of software engineering and research. Because of software's fundamental characteristics, many quality aspects centre on technical dimensions that software engineers and developers apply universally, regardless of whether they work in academia or industry. However, research software may place different emphasis on these dimensions compared to typical industry applications. Additionally, research contexts introduce specific quality requirements that extend beyond traditional software engineering concerns.
In EVERSE, we structure the discussion around research software quality with three main concepts: Research Quality Dimensions, Research Software Quality Indicators and Research Software quality tools. Research Software Quality Dimensions represent common criteria relevant for assessing software quality, at a conceptual level (e.g., documentation, performance, etc.). At a lower level of granularity, a software quality indicator represents a specific aspect of software that can be measured (e.g., FAIRness, test coverage, documentation coverage, etc.). Finally, a research software quality tool is the instrument that measures a software quality indicator against specific criteria or, alternatively, helps improve in one particular aspect (such as linters). Figure 1 shows an example of this division with an example: the "Maintainability" dimension, as defined in ISO/IEC 25010 may be associated with an indicator to check whether continuous integration tests exist in a source code repository (e.g., as defined by the OpenSSF community). Then an assessment tool like the OpenSSF Scorecard may implement the indicator by looking at whether the project runs tests before pull requests are merged in GitHub. However, a different quality assessment tool may check the same indicator against GitLab repositories (not supported by OpenSSF Scorecard).
{ width=90% }
The EVERSE Framework encompasses thirteen main dimensions of research software quality, derived from established standards and research software best practices. These consist of nine technical dimensions based on ISO/IEC standards, plus FAIRness, Open Source Software, Sustainability, and Community dimensions that reflect the unique requirements of research software. Each dimension is described in the sections that follow.