Improve SPARQL test coverage

[Tpt]

SPARQL test coverage has quite a few holes (please complete if you find other missing elements):

Syntax:

• no LANGDIR ("foo"@en--ltr) SPARQL 1.2: add a test with directional language string literals in the query #242
• no invalid LANGDIR ("foo"@en--foo) SPARQL: add negative syntax test for invalid language direction #247
• no DECIMAL_POSITIVE (+2.1) SPARQL 1.0: cover positive numeric literals #272
• no DOUBLE_POSITIVE (+2e1) SPARQL 1.0: cover positive numeric literals #272
• no StrReplaceExpression with 4 arguments (REPLACE("abab", "B.", "Z","i")) SPARQL 1.1: REPLACE: cover regex options #268
• no BuiltInCall with SHA384 (SHA384("foo")) SPARQL 1.1: cover SHA384 #257
• no ExprTripleTermSubject and ExprTripleTermObject with IRI, RDFLiteral, NumericLiteral or BooleanLiteral SPARQL 1.2: cover literals in triple terms #256
• no TripleTermDataObject with a RDFLiteral, NumericLiteral or BooleanLiteral SPARQL 1.2: cover literals in triple terms #256
• no TripleTermObject with RDFLiteral, NumericLiteral or BooleanLiteral SPARQL 1.2: cover literals in triple terms #256
• no ReifiedTripleObject with NumericLiteral or BooleanLiteral SPARQL 1.2: cover literals in triple terms #256
• no PathOneInPropertySet with a or ^a SPARQL 1.1: cover !a and !^a in property paths #270
• no HavingClause with multiple children (HAVING (COUNT(?s) > 1) (COUNT(?o) > 1)) SPARQL 1.1: cover GROUP BY with a function and HAVING with multiple clauses #259
• no GroupCondition with BuiltInCall (GROUP BY LANG(?s)) SPARQL 1.1: cover GROUP BY with a function and HAVING with multiple clauses #259
• no GroupCondition with FunctionCall (GROUP BY xsd:boolean(?value)) SPARQL 1.1: cover GROUP BY with a function and HAVING with multiple clauses #259

Evaluation:

• no DISTINCT aggregate SPARQL 1.1: aggregates with DISTINCT #262
• no xsd:float - xsd:float SPARQL 1.0: cover all "main" numerical datatypes with base + - * / operators #261
• no xsd:double - xsd:double SPARQL 1.0: cover all "main" numerical datatypes with base + - * / operators #261
• no xsd:decimal - xsd:decimal SPARQL 1.0: cover all "main" numerical datatypes with base + - * / operators #261
• no xsd:float * xsd:float SPARQL 1.0: cover all "main" numerical datatypes with base + - * / operators #261
• no xsd:double * xsd:double SPARQL 1.0: cover all "main" numerical datatypes with base + - * / operators #261
• no xsd:decimal * xsd:decimal SPARQL 1.0: cover all "main" numerical datatypes with base + - * / operators #261
• no division test SPARQL 1.0: cover all "main" numerical datatypes with base + - * / operators #261
• no + xsd:float SPARQL 1.0: cover all "main" numerical datatypes with base + - * / operators #261
• no + xsd:double SPARQL 1.0: cover all "main" numerical datatypes with base + - * / operators #261
• no + xsd:decimal SPARQL 1.0: cover all "main" numerical datatypes with base + - * / operators #261
• no - xsd:float SPARQL 1.0: cover all "main" numerical datatypes with base + - * / operators #261
• no - xsd:double SPARQL 1.0: cover all "main" numerical datatypes with base + - * / operators #261
• no - xsd:decimal SPARQL 1.0: cover all "main" numerical datatypes with base + - * / operators #261
• no COALESCE() (without arguments) SPARQL 1.1: cover COALESCE without arguments #258
• no DATATYPE("foo"@en--ltr) (on directional language string) SPARQL 1.2: rdf:dirLangString with DATATYPE, CONCAT and CONTAINS #273
• no IRI(<ex:foo>) (called on an IRI) SPARQL 1.1: test IRI()/URI() on an IRI #263
• no ABS on xsd:float and xsd:double SPARQL 1.1: test ABS, CEIL, FLOOR, ISNUMERIC and ROUND on xsd:float and xsd:double #269
• no CEIL on xsd:float and xsd:double SPARQL 1.1: test ABS, CEIL, FLOOR, ISNUMERIC and ROUND on xsd:float and xsd:double #269
• no FLOOR on xsd:float and xsd:double SPARQL 1.1: test ABS, CEIL, FLOOR, ISNUMERIC and ROUND on xsd:float and xsd:double #269
• no ROUND on xsd:float and xsd:double SPARQL 1.1: test ABS, CEIL, FLOOR, ISNUMERIC and ROUND on xsd:float and xsd:double #269
• no testing of float and double special values (NaN, +Inf and -Inf), especially on comparison and ordering)
• no deep testing of regex syntax
• no testing of rdf:dirLangString on string manipulation functions SPARQL 1.2: rdf:dirLangString with DATATYPE, CONCAT and CONTAINS #273
• no ORDER BY on xsd:dateTime
• Evaluation of SELECT ?s WHERE { ?s ex:p* ex:o } on the empty dataset (returns ex:o) SPARQL 1.1: property path: * and ? with a set start/end do not check if the term exists in the graph #264

[rubensworks]

This is a very useful list.
Did you produce this in some automatic manner?

[Tpt]

Thanks! I ran test coverage on Oxigraph SPARQL parser and evaluator and listed the not executed code paths that were not special paths for optimizations, error propagation from the underlying storage or features not in SPARQL 1.2.

[afs]

SPARQL test coverage has quite a few holes

A requirement of the tests is complete the W3C process requirement to demonstrate features can be implemented; that's not total test coverage. The goal is to get as many implementation reports as possible (min 2 per feature).

We can add all those tests over time. I'm not sure how we should organise in the 1.0 and 1.1 test areas - one way is as a mixture in the current structure, another is to keep a demarcation of original compliance tests + extra tests.

For SPARQL 1.2, we (all) have a clean slate.

[Tpt]

I'm not sure how we should organise in the 1.0 and 1.1 test areas

Good point. Currently it seems we have a mixture in the current structure. But extra tests do not use the dawgt:approval predicate, allowing to distinguish them from the original tests. I tend to think it's fine, the original test suites are still available somewhere if people want to run them and not the "live" version from this CG.

[RobinTF]

If I may propose some missing test cases:

1. To my knowledge the handling of nested graph clauses is not explicitly tested. So patterns like these GRAPH ?g { GRAPH ?g { ?s ?p ?o } } are not covered (including variants with fixed IRIs).
2. Also I don't think there's a test case that covers handling of the graph clause around something that doesn't return values from the graph, like VALUES or the empty pattern which doesn't bind to the graph variable.
3. A direct test for case insensitivity of language tags: SELECT ("a"@en = "a"@EN AS ?res) WHERE {} (There are a couple of tests that imply this, but I don't think there's an explicit one)
4. A test should also ensure that LIMIT/OFFSET within a graph clause should of course apply to each graph respectively and not to the cumulative result.

[afs]

@RobinTF - would you like to propose some PRs?

[RobinTF]

@afs I'll consider it (but unlikely to happen during the holidays).
I'd have no problems coming up with example datasets + queries + expected results, but I have no clue how this repository works, so that would be the biggest hurdle.
The primary motivation to write my comment was to document my thoughts somewhere, because I did run into these issues in the past (most of them are rather niche scenarios, but it makes a difference when you're actually trying to implement things) and didn't write them down anywhere.

[afs]

Tests are manged in RDF.

Find appropriate directories for a each test under sparql/ or start a new one if no logical place.

Example originally from sparql/sparql11/property-path/manifest.ttl (simplified to remove some original process foo stuff)
See list at the start of the manifest then:

:pp01 rdf:type mf:QueryEvaluationTest ;
    mf:name    "(pp01) Simple path" ;
    mf:action
         [ qt:query  <pp01.rq> ;
           qt:data   <pp01.ttl> ] ;
    mf:result  <pp01.srx>
    .

A test is an mf:action (here, of a query and a data file) and a mf:result (here, a result set).

[Tpt]

Also I don't think there's a test case that covers handling of the graph clause around something that doesn't return values from the graph, like VALUES or the empty pattern which doesn't bind to the graph variable.

This recent commit is covering part of this use case: f19aa15