Search traces with non-string tags

[idohalevi]

Describe the bug
When I search for traces with a tag that its value is not a string I get a 500 error. I found that the ES query contains a regex format query. It seems like this type of query can search for string values only and when I try to search for tags that have other values I get an error.
Maybe it's related to #2364 ?

To Reproduce
Steps to reproduce the behavior:

1. Run Jaeger with ES
2. Search for traces with tag error=true or http.status_code=200

Expected behavior
The expected behavior is to get all the traces with the requested tags

Screenshots

Version (please complete the following information):

• OS: Linux
• Jaeger version: 1.20
• Deployment: Kubernetes

What troubleshooting steps did you try?
When I change the field mapping from boolean to string the search works as expected

[albertteoh]

I can look into this if there are no takers.

[objectiser]

@albertteoh All yours.

[idohalevi]

Adding some more info. Looks like there was a PR for adding regex tag search for es and I believe that since then we use regex query for every tag we have.

This is the PR: #2049. (@annanay25 I saw you sent the PR, WDYT?)

The change from match to regex:

Maybe we should have something that indicates the tags contain a regex exp. and only if so use the regex query? or should we assume the es mapping for all tags is a string?

[albertteoh]

As discussed with @idohalevi offline, it looks like this is not a bug with Jaeger, but the root cause was because a custom index mapping was instead instead of the "vanilla" jaeger-span index definition.

This resulted in leaving ES to guess the best data types (dynamic mapping) for the incoming span tags which, in this case was bool for error and long for http.status_code, when they should be a keyword. This resulted in ES returning errors whenever we attempt to filter on error or http.status_code tags which resemble:

"Can only use regexp queries on keyword and text fields - not on [tags.error] which is of type [boolean]"

It also looks quite likely that #2049 was the cause of the issue we're seeing, as the last time a search on the error tag was attempted was around the time of that PR.

The tradeoff for #2049, IIUC, is that we can't leverage Kibana for visualisations (e.g. line charts) of span data based on aggregations of numeric field types as Kibana will simply not show that field as an option to aggregate on if it's not numeric.

@annanay25 is trade-off something you and the community were aware of when creating that PR?

[albertteoh]

Ah please ignore, the jaeger-span index definition already defined tags as keyword types, well before #2049; and I can see the benefits of regex searches are more useful than numeric aggregations.

[idohalevi]

@albertteoh thanks, I'll close the issue