[BUG] k-NN score discrepancy after update to 2.17

[tiagohconte]

Describe the bug

After updating from version 2.11 to 2.17 on AWS OpenSearch (I know it's a large step, but bear with me), we noticed a significant score variation for newly indexed documents. For documents already in the index, the kNN score remains consistent across OpenSearch versions. However, if you re-index the same document, the score drops dramatically (e.g., from 0.72 to 0.12).

Upon further investigation using local OpenSearch versions, we identified that the issue arises when updating from 2.15 to 2.17.

The root cause appears to be related to the index.knn.space_type setting, which is documented here as deprecated. When we use the new approach of setting space_type directly in the field, the issue is resolved. However, this solution requires re-indexing all our documents, which is a significant drawback.

I couldn't find any mention of the deprecation or removal of support for this setting in the release notes. Is this the expected behavior?

Related component

Indexing

To Reproduce

1. Using OpenSearch 2.15, create an index with index.knn.space_type setting, don't use the space_type option in the field and index some documents
2. Perform a search and check if the semantic score makes sense
3. Update OpenSearch to 2.17 and index the same documents
4. See different score from 2.15

Extra:

1. Using OpenSearch 2.17, create a new index using space_type option in the field instead of index.knn.space_type and index the same documents
2. See same score from 2.15

Expected behavior

It was expected that the minor version update would not change the behavior of index.knn.space_type

Additional Details

Plugins
All plugins enabled by default from AWS OpenSearch

Host/Environment:

• AWS OpenSearch
• Official OpenSearch containers

[navneet1v]

assigning it to @0ctopus13prime