**What's New**

BETA Iceberg conversion support for the STORAGE_MIGRATION data strategy. See [Iceberg Conversion](hms-mirror-iceberg_migration.md) for more details. **Bugs (Fixed)** [Add to connection init the ability to set the queue AND trigger an engine resource](cloudera-labs/hms-mirror#173) [cloudera-labs/hms-mirror#168 Validate SQL elements before making changes to the Cluster](cloudera-labs/hms-mirror#168) [Extend the HS2 connection validation with Tez task validation](cloudera-labs/hms-mirror#167) [The reset-to-default-location doesn't seem to be working in v2](cloudera-labs/hms-mirror#165)
dstreev · Jan 24, 2025 · 39af71d · 39af71d
1 parent b5209c0
commit 39af71d
Show file tree

Hide file tree

Showing 230 changed files with 2,572 additions and 1,158 deletions.
diff --git a/Writerside/images/iceberg_conversion.png b/Writerside/images/iceberg_conversion.png
diff --git a/Writerside/topics/Release-Notes.md b/Writerside/topics/Release-Notes.md
@@ -10,6 +10,20 @@ The latest set of enhancement requests can be found [here](https://github.com/cl
 
 If there is something you'd like to see, add a new issue [here](https://github.com/cloudera-labs/hms-mirror/issues)
 
+## 2.3.0.4
+
+**What's New**
+BETA Iceberg conversion support for the STORAGE_MIGRATION data strategy. See [Iceberg Conversion](hms-mirror-iceberg_migration.md) for more details.
+
+**Bugs (Fixed)**
+[Add to connection init the ability to set the queue AND trigger an engine resource](https://github.com/cloudera-labs/hms-mirror/issues/173)
+[https://github.com/cloudera-labs/hms-mirror/issues/168
+Validate SQL elements before making changes to the Cluster](https://github.com/cloudera-labs/hms-mirror/issues/168)
+[Extend the HS2 connection validation with Tez task validation](https://github.com/cloudera-labs/hms-mirror/issues/167)
+[The reset-to-default-location doesn't seem to be working in v2](https://github.com/cloudera-labs/hms-mirror/issues/165)
+
+
+
 ## 2.2.0.19.6 (pre-release)
 
 **What's New**

diff --git a/Writerside/topics/hms-mirror-iceberg_migration.md b/Writerside/topics/hms-mirror-iceberg_migration.md
@@ -2,13 +2,12 @@
 
 This process will look at Hive tables, evaluate if the table is a candidate for migration to Iceberg, and then migrate the table to Iceberg.
 
-Specify `-d ICEBERG_CONVERSION` as the DataStrategy to run the Iceberg Migration.
+Currently the conversion of Hive Table Format to Iceberg is supported through the [STORAGE_MIGRATION](storage_migration.md) data strategy.  Efforts are in place to extend this conversion to SCHEMA_ONLY and SQL data strategies.
 
-This process uses Hive SQL to build(and run) the conversion scripts.  These are "inplace" migrations.
+![iceberg_conversion.png](iceberg_conversion.png)
 
-The following options apply to the Iceberg Migration:
-- `-iv|--iceberg-version` - The Iceberg version to use.  Default is 1.  Can be 1 or 2.
-- `-itpo|--iceberg-table-property-overrides` - A comma separated list of table properties to override.  See the Iceberg Table Properties available [here](https://iceberg.apache.org/docs/latest/configuration/#configuration) and [CDP Properties](https://docs.cloudera.com/cdw-runtime/cloud/iceberg-how-to/topics/iceberg-table-properties.html)
+Simply 'enable' the conversion and all tables covered by the data strategy will be converted to Iceberg.  You can 
+also determine which iceberg table format to apply to the table.
 
 ## Requirements
 

diff --git a/bin/deploy.yaml b/bin/deploy.yaml
@@ -11,14 +11,18 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-
 - hosts: edge
   become: true
 
   vars:
 
 
   tasks:
+#    - name: Stop hms-mirror service
+#      become: true
+#      become_user: hms-mirror
+#      shell: "/usr/local/hms-mirror/bin/hms-mirror --stop"
+
     - name: Check Directories
       file:
         path: /usr/local/hms-mirror/bin
@@ -86,4 +90,6 @@
         mode: 0755
         state: link
 
+#    - name: Start hms-mirror service
+#      shell: "sudo -u hms-mirror /usr/local/hms-mirror/bin/hms-mirror --service"
 
diff --git a/docs/sample_reports/storage_migration/literal-dc/sm_orders_hms-mirror.yaml b/docs/sample_reports/storage_migration/literal-dc/sm_orders_hms-mirror.yaml
@@ -10,7 +10,7 @@ tableMirrors:
     dbName: "sm_orders"
     name: "order_item_orc"
     strategy: "STORAGE_MIGRATION"
-    phaseState: "SUCCESS"
+    phaseState: "CALCULATED_SQL"
     environments:
       SHADOW:
         name: null
@@ -133,7 +133,7 @@ tableMirrors:
     dbName: "sm_orders"
     name: "order_item_small_orc"
     strategy: "STORAGE_MIGRATION"
-    phaseState: "SUCCESS"
+    phaseState: "CALCULATED_SQL"
     environments:
       SHADOW:
         name: null
@@ -245,7 +245,7 @@ tableMirrors:
     dbName: "sm_orders"
     name: "order_orc"
     strategy: "STORAGE_MIGRATION"
-    phaseState: "SUCCESS"
+    phaseState: "CALCULATED_SQL"
     environments:
       SHADOW:
         name: null
@@ -354,7 +354,7 @@ tableMirrors:
     dbName: "sm_orders"
     name: "order_small_orc"
     strategy: "STORAGE_MIGRATION"
-    phaseState: "SUCCESS"
+    phaseState: "CALCULATED_SQL"
     environments:
       SHADOW:
         name: null
@@ -463,7 +463,7 @@ tableMirrors:
     dbName: "sm_orders"
     name: "order_src"
     strategy: "STORAGE_MIGRATION"
-    phaseState: "SUCCESS"
+    phaseState: "CALCULATED_SQL"
     environments:
       SHADOW:
         name: null

diff --git a/docs/sample_reports/storage_migration/literal-exec/sm_orders_hms-mirror.yaml b/docs/sample_reports/storage_migration/literal-exec/sm_orders_hms-mirror.yaml
@@ -6,7 +6,7 @@ tableMirrors:
     dbName: "sm_orders"
     name: "mngd_order_item_orc"
     strategy: "STORAGE_MIGRATION"
-    phaseState: "SUCCESS"
+    phaseState: "CALCULATED_SQL"
     environments:
       SHADOW:
         name: null
@@ -125,7 +125,7 @@ tableMirrors:
     dbName: "sm_orders"
     name: "mngd_order_item_small_orc"
     strategy: "STORAGE_MIGRATION"
-    phaseState: "SUCCESS"
+    phaseState: "CALCULATED_SQL"
     environments:
       SHADOW:
         name: null
@@ -244,7 +244,7 @@ tableMirrors:
     dbName: "sm_orders"
     name: "mngd_order_orc"
     strategy: "STORAGE_MIGRATION"
-    phaseState: "SUCCESS"
+    phaseState: "CALCULATED_SQL"
     environments:
       SHADOW:
         name: null
@@ -360,7 +360,7 @@ tableMirrors:
     dbName: "sm_orders"
     name: "mngd_order_small_orc"
     strategy: "STORAGE_MIGRATION"
-    phaseState: "SUCCESS"
+    phaseState: "CALCULATED_SQL"
     environments:
       SHADOW:
         name: null
@@ -476,7 +476,7 @@ tableMirrors:
     dbName: "sm_orders"
     name: "order_item_orc"
     strategy: "STORAGE_MIGRATION"
-    phaseState: "SUCCESS"
+    phaseState: "CALCULATED_SQL"
     environments:
       SHADOW:
         name: null
@@ -602,7 +602,7 @@ tableMirrors:
     dbName: "sm_orders"
     name: "order_item_small_orc"
     strategy: "STORAGE_MIGRATION"
-    phaseState: "SUCCESS"
+    phaseState: "CALCULATED_SQL"
     environments:
       SHADOW:
         name: null
@@ -717,7 +717,7 @@ tableMirrors:
     dbName: "sm_orders"
     name: "order_orc"
     strategy: "STORAGE_MIGRATION"
-    phaseState: "SUCCESS"
+    phaseState: "CALCULATED_SQL"
     environments:
       SHADOW:
         name: null
@@ -829,7 +829,7 @@ tableMirrors:
     dbName: "sm_orders"
     name: "order_small_orc"
     strategy: "STORAGE_MIGRATION"
-    phaseState: "SUCCESS"
+    phaseState: "CALCULATED_SQL"
     environments:
       SHADOW:
         name: null
@@ -941,7 +941,7 @@ tableMirrors:
     dbName: "sm_orders"
     name: "order_src"
     strategy: "STORAGE_MIGRATION"
-    phaseState: "SUCCESS"
+    phaseState: "CALCULATED_SQL"
     environments:
       SHADOW:
         name: null

diff --git a/docs/sample_reports/storage_migration/literal/sm_orders_hms-mirror.yaml b/docs/sample_reports/storage_migration/literal/sm_orders_hms-mirror.yaml
@@ -6,7 +6,7 @@ tableMirrors:
     dbName: "sm_orders"
     name: "mngd_order_item_orc"
     strategy: "STORAGE_MIGRATION"
-    phaseState: "SUCCESS"
+    phaseState: "CALCULATED_SQL"
     environments:
       SHADOW:
         name: null
@@ -125,7 +125,7 @@ tableMirrors:
     dbName: "sm_orders"
     name: "mngd_order_item_small_orc"
     strategy: "STORAGE_MIGRATION"
-    phaseState: "SUCCESS"
+    phaseState: "CALCULATED_SQL"
     environments:
       SHADOW:
         name: null
@@ -244,7 +244,7 @@ tableMirrors:
     dbName: "sm_orders"
     name: "mngd_order_orc"
     strategy: "STORAGE_MIGRATION"
-    phaseState: "SUCCESS"
+    phaseState: "CALCULATED_SQL"
     environments:
       SHADOW:
         name: null
@@ -360,7 +360,7 @@ tableMirrors:
     dbName: "sm_orders"
     name: "mngd_order_small_orc"
     strategy: "STORAGE_MIGRATION"
-    phaseState: "SUCCESS"
+    phaseState: "CALCULATED_SQL"
     environments:
       SHADOW:
         name: null
@@ -476,7 +476,7 @@ tableMirrors:
     dbName: "sm_orders"
     name: "order_item_orc"
     strategy: "STORAGE_MIGRATION"
-    phaseState: "SUCCESS"
+    phaseState: "CALCULATED_SQL"
     environments:
       SHADOW:
         name: null
@@ -602,7 +602,7 @@ tableMirrors:
     dbName: "sm_orders"
     name: "order_item_small_orc"
     strategy: "STORAGE_MIGRATION"
-    phaseState: "SUCCESS"
+    phaseState: "CALCULATED_SQL"
     environments:
       SHADOW:
         name: null
@@ -717,7 +717,7 @@ tableMirrors:
     dbName: "sm_orders"
     name: "order_orc"
     strategy: "STORAGE_MIGRATION"
-    phaseState: "SUCCESS"
+    phaseState: "CALCULATED_SQL"
     environments:
       SHADOW:
         name: null
@@ -829,7 +829,7 @@ tableMirrors:
     dbName: "sm_orders"
     name: "order_small_orc"
     strategy: "STORAGE_MIGRATION"
-    phaseState: "SUCCESS"
+    phaseState: "CALCULATED_SQL"
     environments:
       SHADOW:
         name: null
@@ -941,7 +941,7 @@ tableMirrors:
     dbName: "sm_orders"
     name: "order_src"
     strategy: "STORAGE_MIGRATION"
-    phaseState: "SUCCESS"
+    phaseState: "CALCULATED_SQL"
     environments:
       SHADOW:
         name: null

diff --git a/docs/sample_reports/storage_migration/rdl-exec/sm_orders_hms-mirror.yaml b/docs/sample_reports/storage_migration/rdl-exec/sm_orders_hms-mirror.yaml
@@ -6,7 +6,7 @@ tableMirrors:
     dbName: "sm_orders"
     name: "mngd_order_item_orc"
     strategy: "STORAGE_MIGRATION"
-    phaseState: "SUCCESS"
+    phaseState: "CALCULATED_SQL"
     environments:
       SHADOW:
         name: null
@@ -128,7 +128,7 @@ tableMirrors:
     dbName: "sm_orders"
     name: "mngd_order_item_small_orc"
     strategy: "STORAGE_MIGRATION"
-    phaseState: "SUCCESS"
+    phaseState: "CALCULATED_SQL"
     environments:
       SHADOW:
         name: null
@@ -250,7 +250,7 @@ tableMirrors:
     dbName: "sm_orders"
     name: "mngd_order_orc"
     strategy: "STORAGE_MIGRATION"
-    phaseState: "SUCCESS"
+    phaseState: "CALCULATED_SQL"
     environments:
       SHADOW:
         name: null
@@ -369,7 +369,7 @@ tableMirrors:
     dbName: "sm_orders"
     name: "mngd_order_small_orc"
     strategy: "STORAGE_MIGRATION"
-    phaseState: "SUCCESS"
+    phaseState: "CALCULATED_SQL"
     environments:
       SHADOW:
         name: null
@@ -488,7 +488,7 @@ tableMirrors:
     dbName: "sm_orders"
     name: "order_item_orc"
     strategy: "STORAGE_MIGRATION"
-    phaseState: "SUCCESS"
+    phaseState: "CALCULATED_SQL"
     environments:
       SHADOW:
         name: null
@@ -613,7 +613,7 @@ tableMirrors:
     dbName: "sm_orders"
     name: "order_item_small_orc"
     strategy: "STORAGE_MIGRATION"
-    phaseState: "SUCCESS"
+    phaseState: "CALCULATED_SQL"
     environments:
       SHADOW:
         name: null
@@ -727,7 +727,7 @@ tableMirrors:
     dbName: "sm_orders"
     name: "order_orc"
     strategy: "STORAGE_MIGRATION"
-    phaseState: "SUCCESS"
+    phaseState: "CALCULATED_SQL"
     environments:
       SHADOW:
         name: null
@@ -838,7 +838,7 @@ tableMirrors:
     dbName: "sm_orders"
     name: "order_small_orc"
     strategy: "STORAGE_MIGRATION"
-    phaseState: "SUCCESS"
+    phaseState: "CALCULATED_SQL"
     environments:
       SHADOW:
         name: null
@@ -949,7 +949,7 @@ tableMirrors:
     dbName: "sm_orders"
     name: "order_src"
     strategy: "STORAGE_MIGRATION"
-    phaseState: "SUCCESS"
+    phaseState: "CALCULATED_SQL"
     environments:
       SHADOW:
         name: null