Add Flow Scheduler

[ruizehung]

Summary

Implement a simplified version of Flow Scheduler presented in Quincy: fair scheduling for distributed computing clusters.

TL;lDR; The major files to review

• schedulers/flow_scheduler.py
• tests/test_flow_scheduler.py

Changes:

• Copy https://github.com/ICGog/Flowlessly into /schedulers. I was thinking of using git submodules. But since I make some custom patches to the origin repo, git submodule becomes very annoying to deal with. Note I separate the changes that is just copying Flowlessly to this repo in port-flowlessly to keep this PR clean.
  • What I changed:
    • Fix gtest external dependency git branch. I was pulling from master, which no longer exists. Now I have it pull from main.
    • Add https://pybind11.readthedocs.io/en/stable/installing.html as a submodule
    • Add python binding in schedulers/Flowlessly/src/python/Flowlessly.cpp using pybind11
• Implement Flow scheduler in schedulers/flow_scheduler.py
• Add unit test for some Flow scheduler specific helper function in tests/test_flow_scheduler.py
• Add configs/simple_workload_flow.conf
• Set up devcontainer so that people (including myself) using Apple Silicon Mac can run this repo in Github codespace. [cplex](https://pypi.org/project/cplex/) isn't available on arm 64 architecture.

Test Plan

Pass all unit test

@ruizehung ➜ /workspaces/erdos-scheduling-simulator (flow-scheduler) $ pytest tests
================================================================== test session starts ==================================================================
platform linux -- Python 3.10.12, pytest-7.4.2, pluggy-1.3.0
rootdir: /workspaces/erdos-scheduling-simulator
plugins: cov-4.1.0
collected 206 items / 1 skipped

tests/test_clockwork_scheduler.py ...........                                                                                                     [  5%]
tests/test_flow_scheduler.py ..                                                                                                                   [  6%]
tests/test_ilp_scheduler.py ...................................                                                                                   [ 23%]
tests/test_jobs.py ........................                                                                                                       [ 34%]
tests/test_loader.py ......                                                                                                                       [ 37%]
tests/test_profile.py .                                                                                                                           [ 38%]
tests/test_resources.py ............................                                                                                              [ 51%]
tests/test_schedulers.py .......                                                                                                                  [ 55%]
tests/test_simulator.py ............                                                                                                              [ 61%]
tests/test_tasks.py ...........................................                                                                                   [ 82%]
tests/test_time.py ........                                                                                                                       [ 85%]
tests/test_workers.py ..................                                                                                                          [ 94%]
tests/test_z3_scheduler.py .s....s..s.                                                                                                            [100%]

============================================================ 203 passed, 4 skipped in 5.13s =============================================================
@ruizehung ➜ /workspaces/erdos-scheduling-simulator (flow-scheduler) $

Can run flow scheduler against simple_workload

@ruizehung ➜ /workspaces/erdos-scheduling-simulator (flow-scheduler) $ python3 main.py --flagfile=configs/simple_workload_flow.conf
@ruizehung ➜ /workspaces/erdos-scheduling-simulator (flow-scheduler) $

simple_workload_flow.log: https://gist.github.com/ruizehung/af43e79cbfff7e39f19d416db002761e
simple_workload_flow.csv: https://gist.github.com/ruizehung/725c529232734d88a78f373a9541942c

And also I'm able to plot based on the output csv. E.g.

There are still some issues with plotting though since we hardcode what markers, colors mapping in analyze.py. I think we can fix it when we need it.

@ruizehung ➜ /workspaces/erdos-scheduling-simulator (flow-scheduler) $ python3 analyze.py \
  --csv_files=simple_workload_flow.csv \
  --csv_labels=FlowScheduler \
  --inter_task_time \
  --task_placement \
  --task_placement_delay \
  --task_slack \
  --plot
scheduler_labels=['FlowScheduler']
2023-10-24,16:55:50.459 plotting DEBUG: Simulation end time for simple_workload_flow.csv: 2405
2023-10-24,16:55:50.459 plotting DEBUG: ================== Num placed tasks ==================
2023-10-24,16:55:50.461 plotting DEBUG:
+----------+-----------+----------+-----------+-----------+----------------------+--------------------+---------------------+---------------------+---------------------+---------------------+---------------------+---------------------+-----------------------+
|   Length |   Average |   Median |   Minimum |   Maximum |   Standard Deviation |   Percentile (1st) |   Percentile (10th) |   Percentile (25th) |   Percentile (50th) |   Percentile (75th) |   Percentile (90th) |   Percentile (99th) |   Percentile (99.9th) |
+==========+===========+==========+===========+===========+======================+====================+=====================+=====================+=====================+=====================+=====================+=====================+=======================+
|       16 |       0.5 |      0.5 |         0 |         1 |                  0.5 |                  0 |                   0 |                   0 |                 0.5 |                   1 |                   1 |                   1 |                     1 |
+----------+-----------+----------+-----------+-----------+----------------------+--------------------+---------------------+---------------------+---------------------+---------------------+---------------------+---------------------+-----------------------+
2023-10-24,16:55:50.461 plotting DEBUG: ================== Num unplaced tasks ==================
2023-10-24,16:55:50.462 plotting DEBUG:
+----------+-----------+----------+-----------+-----------+----------------------+--------------------+---------------------+---------------------+---------------------+---------------------+---------------------+---------------------+-----------------------+
|   Length |   Average |   Median |   Minimum |   Maximum |   Standard Deviation |   Percentile (1st) |   Percentile (10th) |   Percentile (25th) |   Percentile (50th) |   Percentile (75th) |   Percentile (90th) |   Percentile (99th) |   Percentile (99.9th) |
+==========+===========+==========+===========+===========+======================+====================+=====================+=====================+=====================+=====================+=====================+=====================+=======================+
|       16 |         0 |        0 |         0 |         0 |                    0 |                  0 |                   0 |                   0 |                   0 |                   0 |                   0 |                   0 |                     0 |
+----------+-----------+----------+-----------+-----------+----------------------+--------------------+---------------------+---------------------+---------------------+---------------------+---------------------+---------------------+-----------------------+
2023-10-24,16:55:50.892 plotting DEBUG: ================== Actual task completion slack [ms] ===============
2023-10-24,16:55:50.892 plotting DEBUG: Tasks that match the regex: .*
2023-10-24,16:55:50.894 plotting DEBUG:
+----------+-----------+----------+-----------+-----------+----------------------+--------------------+---------------------+---------------------+---------------------+---------------------+---------------------+---------------------+-----------------------+
|   Length |   Average |   Median |   Minimum |   Maximum |   Standard Deviation |   Percentile (1st) |   Percentile (10th) |   Percentile (25th) |   Percentile (50th) |   Percentile (75th) |   Percentile (90th) |   Percentile (99th) |   Percentile (99.9th) |
+==========+===========+==========+===========+===========+======================+====================+=====================+=====================+=====================+=====================+=====================+=====================+=======================+
|        4 |   -0.8015 |  -0.7515 |    -1.703 |         0 |             0.629604 |           -1.68197 |             -1.4927 |            -1.17725 |             -0.7515 |            -0.37575 |             -0.1503 |            -0.01503 |             -0.001503 |
+----------+-----------+----------+-----------+-----------+----------------------+--------------------+---------------------+---------------------+---------------------+---------------------+---------------------+---------------------+-----------------------+
2023-10-24,16:55:50.894 plotting DEBUG: ================== Intended task completion slack [ms] =============
2023-10-24,16:55:50.894 plotting DEBUG: Tasks that match the regex: .*
2023-10-24,16:55:50.895 plotting DEBUG:
+----------+-----------+----------+-----------+-----------+----------------------+--------------------+---------------------+---------------------+---------------------+---------------------+---------------------+---------------------+-----------------------+
|   Length |   Average |   Median |   Minimum |   Maximum |   Standard Deviation |   Percentile (1st) |   Percentile (10th) |   Percentile (25th) |   Percentile (50th) |   Percentile (75th) |   Percentile (90th) |   Percentile (99th) |   Percentile (99.9th) |
+==========+===========+==========+===========+===========+======================+====================+=====================+=====================+=====================+=====================+=====================+=====================+=======================+
|        4 |         0 |        0 |         0 |         0 |                    0 |                  0 |                   0 |                   0 |                   0 |                   0 |                   0 |                   0 |                     0 |
+----------+-----------+----------+-----------+-----------+----------------------+--------------------+---------------------+---------------------+---------------------+---------------------+---------------------+---------------------+-----------------------+
2023-10-24,16:55:51.607 plotting DEBUG: ================== Inter-Task Time [ms] ==================
2023-10-24,16:55:51.607 plotting DEBUG: Tasks that match the regex: .*
2023-10-24,16:55:51.607 plotting DEBUG: Statistics for Task_One:
2023-10-24,16:55:51.609 plotting DEBUG:
+----------+-----------+----------+-----------+-----------+----------------------+--------------------+---------------------+---------------------+---------------------+---------------------+---------------------+---------------------+-----------------------+
|   Length |   Average |   Median |   Minimum |   Maximum |   Standard Deviation |   Percentile (1st) |   Percentile (10th) |   Percentile (25th) |   Percentile (50th) |   Percentile (75th) |   Percentile (90th) |   Percentile (99th) |   Percentile (99.9th) |
+==========+===========+==========+===========+===========+======================+====================+=====================+=====================+=====================+=====================+=====================+=====================+=======================+
|        3 |         0 |        0 |         0 |         0 |                    0 |                  0 |                   0 |                   0 |                   0 |                   0 |                   0 |                   0 |                     0 |
+----------+-----------+----------+-----------+-----------+----------------------+--------------------+---------------------+---------------------+---------------------+---------------------+---------------------+---------------------+-----------------------+
2023-10-24,16:55:52.144 plotting DEBUG: ================ Task placement delay [ms] ================
Traceback (most recent call last):
  File "/workspaces/erdos-scheduling-simulator/analyze.py", line 1346, in <module>
    app.run(main)
  File "/home/vscode/.local/lib/python3.10/site-packages/absl/app.py", line 308, in run
    _run_main(main, args)
  File "/home/vscode/.local/lib/python3.10/site-packages/absl/app.py", line 254, in _run_main
    sys.exit(main(argv))
  File "/workspaces/erdos-scheduling-simulator/analyze.py", line 1320, in main
    analyze_task_placement_delay(
  File "/workspaces/erdos-scheduling-simulator/analyze.py", line 702, in analyze_task_placement_delay
    plot_colors = [colors[label] for label in scheduler_labels]
  File "/workspaces/erdos-scheduling-simulator/analyze.py", line 702, in <listcomp>
    plot_colors = [colors[label] for label in scheduler_labels]
KeyError: 'FlowScheduler'
@ruizehung ➜ /workspaces/erdos-scheduling-simulator (flow-scheduler) $

[sukritkalra]

Closing because too old.