Conflict Detection and Local Resolution in FedPlan

[min-guk]

Implementation Summary

• Added functionality to detect conflicts in OptimalFedPlan, where specific FedPlans (Hops) have different FedOutputTypes.
• The OptimalFedPlan is traversed from top to bottom using a BFS approach to detect conflicts.
• Conflicts are resolved locally, even when one child has multiple parents, with network costs calculated only once.

Limitations

The current implementation has the following limitations:

1. It cannot guarantee either an optimal cost plan or a suboptimal plan.
2. Changes to the local cost affect all nodes up to the root, but these nodes are not updated.
3. OptimalFedPlan must be traversed again to search for conflicts.

While this approach works, exploring better solutions to address these issues would be valuable.

Questions

1. Is it acceptable for the compute cost and memory access cost calculations for Federated Workers and the Local Coordinator to remain the same?
2. Could you provide more specific examples of scenarios where our OptimalFedPlan calculations are effective or where conflicts occur?

Future Work

1. Develop test code workloads to simulate scenarios where conflicts arise.
2. Write and validate test code in a federated environment.

[mboehm7]

Great - thanks for the contribution @min-guk. Please address the minor cleanups as discussed today and then we can merge it in.

[min-guk]

I wanted to share some minor updates that I've been working on:

1. Immediate Pruning: Each FederatedPlan is pruned immediately after enumeration to reduce memory usage.

2. Separation of MemoTablePrint Function: The MemoTablePrint function has been separated from MemoTable.java and is now managed in MemoTablePrinter.java.

3. Additional Cost Calculation: The enumerateFederatedPlanCost function now calculates the additionalTotalCost, which is then printed by MemoTablePrinter. If there's a need to print each additionalCost individually, feel free to let me know.

4. Conflict Detection and Resolution:

• Previously, the detectAndResolveConflictFedPlan function would enumerate a conflict fedPlan as both LOUT and FOUT, even after detecting a conflict. This could lead to incorrect, unnecessary, and duplicate detection and resolution of conflicts for the fedPlan and its child fedPlans.

• For example, it might mistakenly identify a non-conflict fedPlan as having conflicts due to shared HopID among parent FedPlans, or it might incorrectly flag a fedPlan with a single parent as having a conflict.

• To address this, we now store the fedPlans where conflicts occur and exclude these fedPlans and their child fedPlans from the BFS target. After BFS completes, we resolve the collected conflicts and then perform BFS again on the resolved fedPlans and their children.

5. Bug Fixes in resolveConflictFedPlan: Fixed bugs in the resolveConflictFedPlan implementation.

6. Test Enhancements: Added if-else DML test script for FederatedPlanCostEnumeratorTest.

[min-guk]

However, the current detect-and-resolve approach resolves conflicts from top to bottom, which poses a risk of producing inaccurate results because it resolves conflicts higher up while lower-level conflicts are still present.

I understand that this is not an ideal approach, but I am struggling to come up with an appropriate solution to address this issue. I would greatly appreciate any advice or suggestions you could provide.

[mboehm7]

Thanks for pointing out these challenges - then let's try the approach of marking FedPlans as conflicts in a first pass (materialized in FedPlan), and in a second bottom-up pass we then fix these issues.

[mboehm7]

LGTM - thanks for the patch @min-guk. During the merge I fixed the missing license header and incorrect formatting (tabs over spaces) of new files, unnecessary imports, and wrong javadoc.