v0.51.6-public

fix: build error 'parameter 'descriptor' has been removed from the pl…

v0.51.5-public

update version to 0.51.5-public

v0.51.4-public

Changelog

[0.51.4-public] - 2025-01-14

Features

• MCQA2: Added several optimizations to improve the execution efficiency of MCQA2 jobs. MCQA2 jobs now use ExecuteMode.INTERACTIVE_V2 mode, distinguishing it from MCQA1's ExecuteMode.INTERACTIVE
• SQLExecutor: Added new getExecuteMode method to retrieve job execution mode

Changes

• UpsertStream: In version 0.51.0, the signature of the close method was modified (no longer throwing TunnelException). This version restores it to maintain API compatibility.
• ClusterInfo: The toString method was changed in version 0.51.0. This version restores it to maintain API compatibility.
• TunnelRetryStrategy and ConfigurationImpl classes: These were removed in version 0.48.6. This version restores them (though they won't have any effect!) to maintain API compatibility.

---

更新日志

[0.51.4-public] - 2025-01-14

功能

• MCQA2 增加若干项优化，提升了 MCQA2 作业的执行效率。MCQA2 作业的模式变为 ExecuteMode.INTERACTIVE_V2，与 MCQA1 的 ExecuteMode.INTERACTIVE 区别开
• SQLExecutor 新增 getExecuteMode 方法，用于获取作业执行模式

变更

• UpsertStream 在 0.51.0 版本，修改了 close 方法的函数签名（不再抛出 TunnelException），在本版本中恢复，以保证接口兼容性。
• ClusterInfo 在 0.51.0 版本，toString 方法有所变更，在本版本中恢复，以保证接口兼容性。
• TunnelRetryStrategy，ConfigurationImpl 类在 0.48.6 版本被移除，在本版本中恢复（但不会起到任何效果！），以保证接口兼容性。

v0.51.3-public

Changelog

[0.51.3-public] - 2025-01-07

Features

• MCQA2 SQLExecutorImpl adds a new setProject method to specify the default project used for submitting jobs.

Changes

• StreamTunnel when calling the append method, if the number of Record columns exceeds the number of Session Schema columns, it will now throw a SchemaMismatchException (extends IOException) instead of throwing an IOException, and the error message has been optimized.

---

更新日志

[0.51.3-public] - 2025-01-07

功能

• MCQA2 SQLExecutorImpl 新增 setProject 方法，用于指定提交作业使用的默认项目

变更

• StreamTunnel 在调用 append 方法时，当 Record 列数量大于 Session Schema 列数量，现在将抛出 SchemaMismatchException(extend IOExcption)，而不是抛出 IOException，并优化了错误信息

v0.51.2-public

[0.51.2-public] - 2024-12-20

Features

• Authorization Introduced the credential-java authorization package, now supporting authentication with AlibabaCloudCredentialsProvider.
• StreamUploadSession Added awareness for Slot updates and automatic retry logic.
• table-api Introduced the TableRetryHandler class, adding retry logic to the table-api.
• udf The InputSplitter now includes the method setLimit.

Changes

• TypeInfo The StructTypeInfo class now includes the method getTypeName(boolean quote). In version 0.51.0-public (rc0), StructTypeInfo defaulted to quoting field names with backticks. We suspect that this change may affect users, so we decided to revert to the original behavior (not quoting by default). Users can now call getTypeName(true) when quoting is needed.

Fixes

• TypeInfo Field names will now be correctly escaped when quoted with backticks.
• MCQA2 Fixed an issue where the getRawTaskResults interface call in MCQA2 jobs could not retrieve results.

---

[0.51.2-public] - 2024-12-20

功能

• Authorization 引入credential-java鉴权包，现在能够使用AlibabaCloudCredentialsProvider进行鉴权
• StreamUploadSession 新增对 Slot 更新的感知和自动重试逻辑
• table-api 引入TableRetryHandler类，为table-api添加重试逻辑
• udf 中InputSplitter新增方法 setLimit

变更

• TypeInfo StructTypeInfo 新增方法 getTypeName(boolean quote)，在 0.51.0-public (rc0) 版本，StructTypeInfo 默认会对字段名使用反引号进行 quote，我们怀疑这项变更对用户有影响，因此决定恢复原行为（默认不进行quote）
  而是当用户需要 quote 时，可以调用 getTypeName(true)

修复

• TypeInfo 当对字段使用反引号进行 quote 时，现在会正确对字段名进行转义
• MCQA2 修复了 MCQA2 作业调用 getRawTaskResults 接口取不到结果的问题

v0.51.0-public

Changelog

[0.51.0-public] - 2024-12-05

Features

• MapReduce Supports multi pipeline output.
• VolumeBuilder Added the accelerate method to speed up the download process using dragonfly when the external volume is too large.
• Table Introduced TableType OBJECT_TABLE and the method isObjectTable to verify it.
• Project The list method now includes a filter condition enableDr to filter projects based on whether data disaster recovery is enabled.
• Cluster New fields added: clusterRole, jobDataPath, and zoneId.

Changes

• TableBatchReadSession The predicate class variable is now set to transient.
• Attribute added escaping logic and will no longer double quote.
• SQLTask Restored the SQLTask.run(Odps odps, String project, String sql, String taskName, Map<String, String> hints, Map<String, String> aliases, int priority) method removed in version 0.49.0 to resolve potential interface conflicts when users' MR jobs depend on older versions of the SDK.

Fixes

• Table.changeOwner Fixed SQL spelling error.
• Instance.getTaskSummary Removed unreasonable debug logging introduced since version 0.50.2.
• TruncTime Uses backticks to quote columnName during table creation/toString.

Note: This version also includes all changes from "0.51.0-public.rc0" and "0.51.0-public.rc1".

---

更新日志

[0.51.0-public] - 2024-12-05

功能

• MapReduce 支持多重管道输出 (multi pipeline output)。
• VolumeBuilder 新增 accelerate 方法，用于在 external volume 过大时，使用 dragonfly 加速下载过程。
• Table 新增 TableType OBJECT_TABLE 和判断方法 isObjectTable。
• Project list 方法增加过滤条件 enableDr，用于过滤项目是否开启存储容灾。
• Cluster 新增字段 clusterRole、jobDataPath、zoneId。

变更

• TableBatchReadSession 类变量 predicate 现在设置为 transient。
• Attribute 增加转义逻辑，并不再会 double quote。
• SQLTask 恢复了在 0.49.0 版本移除的 SQLTask.run(Odps odps, String project, String sql, String taskName, Map<String, String> hints, Map<String, String> aliases, int priority) 方法，以解决用户的 MR 作业依赖老版本 SDK 时可能发生的接口冲突问题。

修复

• Table.changeOwner 修复 SQL 拼写错误。
• Instance.getTaskSummary 移除自 0.50.2 版本开始的不合理打印的 debug 日志。
• TruncTime 在建表/toString 时，使用反引号对 columnName 进行 quote。

注意： 此版本还包括“0.51.0-public.rc0”和“0.51.0-public.rc1”的所有更改。

v0.50.6-public

[0.50.6-public] - 2024-11-27

Features

• Logview Added support for Logview V2, details see November 14, 2024 (UTC+8): Notice on the security upgrade for MaxCompute LogView. It can be created using new Logview(odps, 2), and SQLExecutor specifies the version through the logviewVersion
  method.

---

[0.50.6-public] - 2024-11-27

功能

• Logview 新增对 Logview V2 的支持，V2 版本保障了数据安全，更多信息参考 2024年11月14日-MaxCompute Logview安全升级 。可以通过 new Logview(odps, 2) 创建，SQLExecutor 通过 logviewVersion 方法指定。

v0.51.0-public.rc1

Changelog

[0.51.0-public.rc1] - 2024-11-22

Features and Changes

• Column ColumnBuilder adds a new withGenerateExpression method for constructing auto-partition columns
• TableSchema
  • Added generatePartitionSpec method, used to generate partition information from Record
  • The setPartitionColumns method now accepts List<Column> instead of ArrayList<Column>
• TableCreator
  • Added support for GenerateExpression and introduced the method autoPartitionBy, which allows for the creation of AutoPartition tables.
  • Added support for ClusterInfo, enabling the creation of Hash/Range Cluster tables.
  • Added the option to specify TableFormat, allowing for the creation of tables in APPEND, TRANSACTION, DELTA, EXTERNAL, and VIEW formats.
  • Introduced the selectStatement parameter for create table as and create view as scenarios.
  • Added the getSql method to obtain the SQL statement for table creation.
  • Now quotes all Comment parameters to support those that contain special characters.
  • Integrated DataHub-related table creation parameters (hubLifecycle, shardNum) into DataHubInfo.
  • Renamed the withJars method to withResources to indicate it can use resources other than JAR files.
  • Renamed the withBucketNum method to withDeltaTableBucketNum to indicate this method is for Delta Tables only.
  • Modified the logic of withHints, withAlias, withTblProperties, and withSerdeProperties methods, now overwriting previous values instead of merging.
  • Removed the createExternal method; you can now use the create method instead.
• Table
  • Introduced the getSchemaVersion method, allowing users to retrieve the current schema version of the table. The version number is updated each time a Schema Evolution occurs, and this field is used primarily for specifying when creating a StreamTunnel.
  • Added setLifeCycle, changeOwner, changeComment, touch, changeClusterInfo, rename, addColumns, dropColumns methods to support modification of table structure.
• StreamTunnel Modified the initialization logic; if allowSchemaMismatch is set to false, it will automatically retry until the latest version of the table structure is used (with a timeout of 5 minutes).

Fixes

• GenerationExpression Fixed the issue where an exception would be thrown when the TruncTime was uppercase during table creation and reloading the table.
• TypeInfoParser Can now correctly handle Struct types, with fields quoted using backticks in TypeInfo.

---

更新日志

[0.51.0-public.rc1] - 2024-11-22

功能与变更

• Column ColumnBuilder 新增 withGenerateExpression 方法，用于构造 auto-partition 列
• TableSchema
  • 新增 generatePartitionSpec 方法，用于从Record中生成分区信息
  • setPartitionColumns 方法现在接收List<Column>，而不是ArrayList<Column>
• TableCreator
  • 新增对GenerateExpression的支持，新增方法autoPartitionBy，现在可以创建 AutoPartition 表了
  • 新增对ClusterInfo的支持，现在可以创建 Hash/Range Cluster 表了
  • 新增指定 TableFormat，现在可以指定创建APPEND,TRANSACTION,DELTA,EXTERNAL,VIEW格式的表
  • 新增selectStatement参数，用于create table as 和 create view as 场景
  • 新增getSql方法，用于获取创建表的 SQL 语句
  • 现在会对所有的 Comment 参数进行 quote，以支持包含特殊字符的 Comment 参数
  • 将 DataHub 相关的建表参数（hubLifecycle, shardNum) 整合为 DataHubInfo
  • 重命名withJars方法为withResources，以表示不仅可以使用JAR类型资源
  • 重命名withBucketNum方法为withDeltaTableBucketNum，以表示该方法仅用于 Delta Table
  • 修改了 withHints，withAlias，withTblProperties，withSerdeProperties 方法的逻辑，现在会覆盖之前设置的值，而不是合并
  • 移除了createExternal方法，现在使用create方法即可
• Table
  • 新增 getSchemaVersion 方法，用户获取当前表结构的版本，用户每次进行 SchemaEvolution 都会更新版本号，目前该字段仅用于在创建 StreamTunnel 时指定
  • 新增 setLifeCycle，changeOwner，changeComment，touch，changeClusterInfo，rename，addColumns，dropColumns方法，以支持对表结构进行修改
• StreamTunnel 修改初始化逻辑，当指定 allowSchemaMismatch 为 false 时，会自动重试直到使用最新版本的表结构(超时时间为5min)

修复

• GenerationExpression 修复了当建表时TruncTime为大写，reload table 会抛出异常的问题
• TypeInfoParser 能够正确处理 Struct 类型，字段被反引号quote的 TypeInfo了

v0.51.0-public.rc0

Changelog

[0.51.0-public.rc0] - 2024-11-18

Features

• GenerateExpression added support for generating expression lists for partition columns, along with the first generated expression TruncTime. For usage, please refer to Example
• UpsertStream supports writing values with primary keys of type TIMESTAMP_NTZ
• Table added new methods for querying CDC-related data: getCdcSize(), getCdcRecordNum(), getCdcLatestVersion(), getCdcLatestTimestamp()
• SQLExecutor MCQA 2.0 job supports retrieving InstanceProgress information

Changes

• Quote added backticks for quoting names in Struct type TypeInfo and other methods that assemble SQL
• AutoClosable to remind users to properly close resources, added corresponding close() methods to the following resource classes to prompt users to close resources correctly:
  • UpsertStream in the odps-sdk-core package,
  • LocalOutputStreamSet, ReduceDriver.ReduceContextImpl, MapDriver.DirectMapContextImpl, LocalRecordWriter in the odps-sdk-impl package
  • VectorizedOutputer, VectorizedExtractor, RecordWriter, RecordReader, Outputer, Extractor in the odps-sdk-udf package

---

更新日志

[0.51.0-public.rc0] - 2024-11-18

功能

• GenerateExpression 增加对分区列的生成列表达式功能的支持，和第一个生成列表达式TruncTime，使用方式请参考Example
• UpsertStream 支持写入主键为 TIMESTAMP_NTZ 类型的值
• Table 新增对 cdc 相关数据的查询，getCdcSize()，getCdcRecordNum()，getCdcLatestVersion()，getCdcLatestTimestamp()
• SQLExecutor MCQA 2.0 作业支持获取 InstanceProgress 信息

变更

• Quote 对 Struct 类型的 TypeInfo，和其他拼装 SQL 的方法，使用反引号对名字进行 quote
• AutoClosable 为了提醒用户正确关闭资源，对下列资源类，增加了相应的 close() 方法，以提醒用户正确关闭资源。
  • odps-sdk-core 包下的 UpsertStream，
  • odps-sdk-impl 包下的 LocalOutputStreamSet，ReduceDriver.ReduceContextImpl，MapDriver.DirectMapContextImpl，LocalRecordWriter
  • odps-sdk-udf 包下的 VectorizedOutputer，VectorizedExtractor，RecordWriter，RecordReader，Outputer，Extractor

v0.50.5-public

Changelog

[0.50.5-public] - 2024-11-13

Features

• TableAPI added retry logic for errors in network requests that can be safely retried, improving the stability of the interface. A new configuration option, retryWaitTimeInSeconds, has been added to RestOptions to specify the retry wait time.
• SQLTask added an overload of the run method that supports passing in the mcqaConnHeader parameter for submitting MCQA 2.0 jobs.
• SQLExecutor now supports specifying the odps.task.wlm.quota hint to set the interactive quota when submitting MCQA 2.0 jobs.
• RestClient introduced a new retryWaitTime parameter along with corresponding getter and setter methods to configure the retry wait time for network requests.
• Configuration added a new socketRetryTimes parameter with corresponding getter and setter methods to configure the retry wait time for Tunnel network requests. If not set, it will use the configuration in RestClient; otherwise, this configuration will be used.

Changes

• Instances removed the overloaded get method get(String projectName, String id, String quotaName, String regionId), which was added in version 0.50.2-public to retrieve MCQA 2.0 instances. Now, users do not need to distinguish whether a job is an MCQA 2.0 job when using the get method, so this method has been removed. Users can directly use the get(String projectName, String id) method to retrieve instances.

Fixes

• Table.read fixed an issue where the configured network-related parameters (such as timeout and retry logic) did not take effect correctly during data preview.
• Streams fixed an issue where specifying the version in the create method would cause an error. A default value of 1 has also been added for version, indicating the initial version of the table.

---

更新日志

[0.50.5-public] - 2024-11-13

功能

• TableAPI 为可以安全重试的网络请求类型的报错增加了相应的重试逻辑，从而提高了接口的稳定性。在 RestOptions 中增加了 retryWaitTimeInSeconds 配置项，用于设置重试等待时间。
• SQLTask 新增了 run 方法的重载，支持传入 mcqaConnHeader 参数，以便提交 MCQA 2.0 作业。
• SQLExecutor 支持通过指定 hints 中的 odps.task.wlm.quota 来设置提交 MCQA 2.0 作业时的 interactive quota。
• RestClient 新增了 retryWaitTime 参数，以及相应的 getter 和 setter 方法，以配置网络请求的重试等待时间。
• Configuration 新增了 socketRetryTimes 参数以及相应的 getter 和 setter 方法，用于配置 Tunnel 网络请求的重试等待时间。如果未设置，则使用 RestClient 中的配置，否则使用此配置。

变更

• Instances 移除了 get 的重载方法 get(String projectName, String id, String quotaName, String regionId) ，该方法在 0.50.2-public 版本中新增，用于获取 MCQA 2.0 实例。现在，用户在使用 get 方法时无须区分作业是否为 MCQA 2.0 作业，因此移除该方法可以直接使用 get(String projectName, String id) 方法来获取实例。

修复

• Table.read() 修复了在数据预览时，配置的网络相关参数（如超时时间、重试逻辑）无法正确生效的问题。
• Streams 修复了 create 方法中，如果指定了 version 会报错的问题。同时增加了 version 的默认值（1），表示表的初始版本。