[META] Breaking changes in 3.0

[reta]

This is a meta issue which lists down all breaking changes in OpenSearch 3.0

Major updates:

• Upgrade to Lucene 10 - PR
• JDK21 as minimum supported Java runtime - Meta Issue
• Remove all deprecated non-inclusive terms from API
• Fix split Java package problem (phase 0)

Breaking behavior in 3.0+:

• Add ThreadContextPermission for stashAndMergeHeaders and stashWithOrigin #15039
• Add ThreadContextPermission for markAsSystemContext and allow core to perform the method #15016
• Block access to system indices from REST APIs #7936
• Migrate client transports to Apache HttpClient / Core 5.x #4459
• [BUG] Indexing a document through the bulk api bypasses the 512 byte _id size limit #6595
• [BUG] OpenSearch app is not able to pick up empty node.roles= environment variable to run as coordinating node. #3412 (Treat Setting value with empty array string as empty array #10625)
• Ensure Jackson default maximums introduced in 2.16.0 do not conflict with OpenSearch settings #11811 - content limits in XContent / Jackson
• Introduce index.query.max_nested_depth. fix #3268 #11670
• Fix interchanged formats of total_indexing_buffer_in_bytes and total_indexing_buffer #17070

Removed code in 3.0+

• Cleanup deprecated thread pool settings [REMOVE] Cleanup deprecated thread pool settings #2595
• Replace 'whitelist' terminology in Java API Replace "blacklist/whitelist" terminology in Java APIs #1683
• Removed deprecated methods from JodaCompatibleZonedDateTime [Remove] Remove deprecated methods from JodaCompatibleZonedDateTime which are called by scripts #3346
• List of deprecated code removal in 3.0 #2773
• Remove mmap.extensions setting #9392
• remove the code marked for deprecation (https://github.com/search?q=repo%3Aopensearch-project%2FOpenSearch+%22%40Deprecated%22&type=code by @dbwiddis)
• Remove COMPAT locale provider Issue PR

Newly deprecated code in 3.0+

• Deprecate CamelCase PathHierarchy tokenizer name #10894

Misc

• Rename Class ending with Plugin to Module under modules dir Rename Class ending with Plugin to Module under modules dir  #4042
• [Discuss] Changed default settings for 3.0 #17170

[reta]

@saratvemulapalli fyi, meta for 3.0 breaking changes, thanks!

[saratvemulapalli]

@CEHENKLE could you help add this to 3.0 roadmap?

[CEHENKLE]

Done.

[saratvemulapalli]

Looks like this is getting stale.
@andrross /@reta do you think we can use Changelog for this ?

[reta]

@andrross /@reta do you think we can use Changelog for this ?

We probably would have to, but what are other breaking changes to mention? I think @nknize is doing some large refactorings but so far I think it was non breaking.

[andrross]

I did some searching and couldn't find any open issues, but we have deprecation warning about blocking direct access to system indices. @reta is implementing the block for direct access to system indices something we should track for 3.0?

This came up in discussion in #7778.

[reta]

@andrross I think we should track it, thanks!

[andrross]

Created issue #7936 and added to the list here. Thanks!

[msfroh]

@reta -- is this a good place to build a wishlist for breaking changes in 3.0? Do you know if we have an existing good place to do that?

(For example, #1029 is a longstanding issue that would be nice to address in a 3.0 release, since otherwise I think it would need to wait for 4.0. I don't want to clutter up this issue with more wishlist items, but that one would be near the top of my list.)

[reta]

@reta -- is this a good place to build a wishlist for breaking changes in 3.0? Do you know if we have an existing good place to do that?

@msfroh I think this is the designated place for breaking changes which are going to be in 3.0, regarding wishlist - I don't think we have any lists for that besides targeting issues / pull requests as v3.0.0

[jed326]

Another major update: apache/lucene#13337
It should be non-breaking though as it's just introducing a new interface method.

[reta]

Another major update: apache/lucene#13337

Thanks @jed326 ! #11415 would be a better place to mention it I think, thanks!

[Bukhtawar]

I think there are few decision points that needs a broader discussion like

1. Search semantics: While async search continues as a plugin, we wanted to unify this as a mode in the mainstream search API, which might cause search response to return just an id, to be subsequently polled for final results
2. Replication semantics : We support two replication modes docrep and segrep, going forward should we consider making segrep the default replication type?
3. Feature defaults : A lot of resiliency features that have been released, including back-pressure, cluster manager throttling are disabled by default, should we consider enabling them by defaults
4. Guardrails enforcements: We have quite a few knobs to control queries, shard count but don't have hard limits enforced which might cause cluster instability

[mgodwan]

In line with the themes highlighted above, I think there are a few issues which we can discuss:

1. [Feature Request] Evaluate removal of DATETIME_FORMATTER_CACHING_SETTING feature flag #13744 [The new way provides faster parsing and efficient way to pick parser during the document parsing step of indexing flow. Some users were relying on internal details due to which this was kept as an opt-in and experimental. We should benefit from removing this in 3.0]
2. [Feature Request] Disable field data on "_id" field #12185 [Field Data feature for _id field results in more problems than it is able to solve and it may be a good idea to remove this.
3. [Cleanup] TransportUpdateAction should extend TransportSingleItemBulkWriteAction #16980 (comment) [There are inconsistencies between using single update and update within a bulk operation which leads to difficulty for users as well as operators in debugging. With 3.0, we can choose one code flow which works same across both the APIs]

[vikasvb90]

With In-place Shard Split, parent shard ids will not longer exist and therefore, looping from 0 to number of shards in places such as this will no longer be valid.

This will be a breaking change because it is an assumption today that number of shards in an index cannot change dynamically. This assumption will no longer hold true after this.

Solution
An iterator from IndexMetadata holding and hiding context of which shards exist can be exposed to loop over shard ids. This will be adapted in OpenSearch, plugins within OpenSearch repo and external plugins. Plugins which work directly with shards may need to adapt to this change. CCR is one such plugin.