[BUG] Query parsing discrepancies lead to runtime failures

[ohltyler]

There are some discrepancies between some input query, and the final input query received by search request processors at runtime. This is due to the internal query construction done in OpenSearch.

Example:
User writes below query:

{
  "query": {
    "term": {
      "my_field": "my_field_value"
    }
  }
}

which internally gets converted to

{
  "query": {
    "term": {
      "my_field": {
        "value": "my_field_value"  
      }
    }
  }
}

which is also a valid query, and what is finally read by the processor.

So, writing some transform like query.term.my_field will fail, as it actually needs to be query.term.my_field.value. It will succeed on the UI (since UI only has access to the initial query), but fail at runtime, with something like the following:

Error searching index: [illegal_argument_exception] cannot find field: query.term.my_field in query string: {"query":{"term":{"my_field":{"value":{"my_value"...

There may be other examples / edge cases that can trigger the same unexpected failures. Further investigation is needed on how/when the query gets parsed & transformed. If it is possible to fetch the transformed query on the UI, then we can display it somewhere, and use it as the input JSON to the search request processors to prevent the errors.

If not, at the least, adding some regex checking to proactively provide some dynamic warning messages may help.

[ohltyler]

Some OSD references that may solve the issue:

https://github.com/opensearch-project/OpenSearch-Dashboards/blob/main/src/plugins/data/common/opensearch_query/opensearch_query/from_filters.ts

https://github.com/opensearch-project/OpenSearch-Dashboards/blob/main/src/plugins/data/common/opensearch_query/opensearch_query/build_opensearch_query.ts

[ohltyler]

Another example of a discrepancy:

If match query is passed without the sub-query field, it fails. Similar to term not including the sub-value field.

Example:

{ "match": { "foo": "bar" } }

is translated into

{ "match": { "foo": { "query": "bar" } } }

which is what is actually received by the search pipelines.

[ohltyler]

Created opensearch-project/OpenSearch#17358 for further tracking.

[ohltyler]

In the meantime, we should do regex checks on the UI and update where we can detect paths defined on the non-expanded forms of different queries. I've done a deep check on all query types and how they are changed/translated when reaching the search request processors:

• term-level queries (term / prefix / fuzzy / wildcard / regexp fixed by Implicitly update input maps defined on non-expanded queries (common cases) #632
• full-text queries (match / match_bool_prefix / match_phrase / match_phrase_prefix) fixed by Implicitly update input maps defined on non-expanded queries (common cases) #632
• aggs renaming fixed by Implicitly update input maps defined on non-expanded queries (common cases) #632
• range query rewrite
• geo_bounding_box/geo_distance/geo_polygon query rewrite
• fields rewrite
• must/must_not/should/filter if written as single obj, turns into array containing single obj

[ohltyler]

An issue on multi-character substrings with the suffix added was causing bugs. Fixed by refactoring the regex in #649