Code

WIP: Service rewrite#99

Code

Draft

1ntEgr8 wants to merge 117 commits intoerdos-project:mainerdos-project/erdos-scheduling-simulator:mainfrom 1ntEgr8:service-rewrite1ntEgr8/erdos-scheduling-simulator:service-rewriteCopy head branch name to clipboard

+4,749-916

Commits

Commits on Sep 23, 2024

• Implement TPC-H data loader

  1ntEgr8

  committed
  956afcd

  Copy full SHA for 956afcd

• Bug fix: convert job graph to task graph

  1ntEgr8

  committed
  a25fbe8

  Copy full SHA for a25fbe8

• Make loop_timeout configurable

  1ntEgr8

  committed
  4d06a95

  Copy full SHA for 4d06a95

• Make profile path configurable

  1ntEgr8

  committed
  c756d17

  Copy full SHA for c756d17

• release time handling on workload

  1ntEgr8

  committed
  9f6aa8b

  Copy full SHA for 9f6aa8b

• Wrap up tpch loader implementation

  1ntEgr8

  committed
  be66704

  Copy full SHA for be66704

Commits on Oct 1, 2024

• scale runtime based on max number of tasks

  1ntEgr8

  committed
  a4d0ded

  Copy full SHA for a4d0ded

Commits on Oct 2, 2024

• fix bug in runtime calc

  1ntEgr8

  committed
  8111ba6

  Copy full SHA for 8111ba6

Commits on Nov 4, 2024

• rename optimization_passes flag to opt_passes

  1ntEgr8

  committed
  e172b56

  Copy full SHA for e172b56

• add cloudlab support, fix runtime rounding bug, make rng gen match service

  1ntEgr8

  committed
  90e696c

  Copy full SHA for 90e696c

• restore tpch_utils to main version

  1ntEgr8

  committed
  06cf4f7

  Copy full SHA for 06cf4f7

• split tpch_replay config files

  1ntEgr8

  committed
  fcb0180

  Copy full SHA for fcb0180

• remove opt_passes flag

  1ntEgr8

  committed
  9014090

  Copy full SHA for 9014090

• Merge branch 'opt-passes-flag-fix' of github.com:1ntEgr8/erdos-scheduling-simulator into tpch-loader

  1ntEgr8

  committed
  fbce571

  Copy full SHA for fbce571

• update tpch_utils.py

  1ntEgr8

  committed
  86420f3

  Copy full SHA for 86420f3

• setup new service.py

  1ntEgr8

  committed
  2f09d5e

  Copy full SHA for 2f09d5e

• update rpc proto dir hierarchy to resolve module import issue with sys.path hack

  1ntEgr8

  committed
  eba9a45

  Copy full SHA for eba9a45

• checkout dhruv's version of service.py

  1ntEgr8

  committed
  e6a364a

  Copy full SHA for e6a364a

• make workload_loader optional, make step and get_time_until_next_event public

  1ntEgr8

  committed
  69876ef

  Copy full SHA for 69876ef

• implement register/deregister framework

  1ntEgr8

  committed
  d68772d

  Copy full SHA for d68772d

• refactor sim time calculation

  1ntEgr8

  committed
  320cb34

  Copy full SHA for 320cb34

• implement RegisterWorker

  1ntEgr8

  committed
  43b8331

  Copy full SHA for 43b8331

• factor out __get_worker_pool

  1ntEgr8

  committed
  eccddc5

  Copy full SHA for eccddc5

Commits on Nov 7, 2024

• refactor tpch loader

  1ntEgr8

  committed
  e94b1f1

  Copy full SHA for e94b1f1

• implement register task graph

  1ntEgr8

  committed
  ed510ab

  Copy full SHA for ed510ab

Commits on Nov 8, 2024

• add testing for service

  1ntEgr8

  committed
  81c4307

  Copy full SHA for 81c4307

• implement register environment ready

  1ntEgr8

  committed
  e1faa7f

  Copy full SHA for e1faa7f

• init impl for get placements, readme with spark-erdos setup

  

  Dhruv Garg

  committed
  0025f3c

  Copy full SHA for 0025f3c

Commits on Nov 12, 2024

• WIP: service changes to handle first tpch taskgraph

  

  Dhruv Garg

  committed
  a7f18e3

  Copy full SHA for a7f18e3

Commits on Nov 13, 2024

• Fix the tick() function to dequeue all events upto n, pass runtime_unit in tpch_loader

  

  Dhruv Garg

  committed
  942ead7

  Copy full SHA for 942ead7

Commits on Nov 20, 2024

• refactor runtime unit setting

  1ntEgr8

  committed
  2b0c895

  Copy full SHA for 2b0c895

• reomve microsecond granularity for now, will add support later

  1ntEgr8

  committed
  7a41bfe

  Copy full SHA for 7a41bfe

• rename sim_time to stime to make it less confusing

  1ntEgr8

  committed
  6170819

  Copy full SHA for 6170819

• refactor tick

  1ntEgr8

  committed
  d76453a

  Copy full SHA for d76453a

• update rpc service to invoke new tick, update test

  1ntEgr8

  committed
  35df499

  Copy full SHA for 35df499

• add comment explaining scheduler runtime zero setting

  1ntEgr8

  committed
  acf6a4e

  Copy full SHA for acf6a4e

• oops forget to add comment on update workload

  1ntEgr8

  committed
  08aa855

  Copy full SHA for 08aa855

• refactor naming in tpch loader

  1ntEgr8

  committed
  106945f

  Copy full SHA for 106945f

• fix documentation error in workload/tasks.py

  1ntEgr8

  committed
  52828b7

  Copy full SHA for 52828b7

• add support for returning current task graph placements from simulator

  1ntEgr8

  committed
  2dac435

  Copy full SHA for 2dac435

• fix logging

  1ntEgr8

  committed
  023805c

  Copy full SHA for 023805c

• construct and return placements response

  1ntEgr8

  committed
  0dfd34e

  Copy full SHA for 0dfd34e

• do not return placements if task graph is complete

  1ntEgr8

  committed
  05b8f47

  Copy full SHA for 05b8f47

• fix placement time bug in simulator

  1ntEgr8

  committed
  14ec521

  Copy full SHA for 14ec521

• implement orchestrated mode

  1ntEgr8

  committed
  062aa5c

  Copy full SHA for 062aa5c

• remove redundant checks

  1ntEgr8

  committed
  dfabf71

  Copy full SHA for dfabf71

Commits on Nov 21, 2024

• enqueue scheduler start event in register task graph

  1ntEgr8

  committed
  76117c6

  Copy full SHA for 76117c6

• bug fixes

  1ntEgr8

  committed
  fcf2142

  Copy full SHA for fcf2142

• add a lock, haven't checked all places

  1ntEgr8

  committed
  a6d72e4

  Copy full SHA for a6d72e4

• Add tests for notify task completion

  

  Dhruv Garg

  committed
  2ec812e

  Copy full SHA for 2ec812e

• add cancelled field to proto

  

  1ntEgr8

  authored and

  Dhruv Garg

  committed
  42eee94

  Copy full SHA for 42eee94

• populate cancelled field in rpc response

  

  1ntEgr8

  authored and

  Dhruv Garg

  committed
  972a020

  Copy full SHA for 972a020

• Updates to spark erdos service documentation

  

  Dhruv Garg

  committed
  1a7e7dd

  Copy full SHA for 1a7e7dd

• implement register driver

  1ntEgr8

  committed
  b244b7e

  Copy full SHA for b244b7e

• quick fixes in regsister and de-register driver

  

  Dhruv Garg

  committed
  0afc85a

  Copy full SHA for 0afc85a

• Updates to spark erdos documentation

  

  Dhruv Garg

  committed
  fd389ea

  Copy full SHA for fd389ea

• register driver bug fix

  1ntEgr8

  committed
  8e313a9

  Copy full SHA for 8e313a9

• Add override_worker_cpu_count flag

  

  Dhruv Garg

  committed
  ab56afb

  Copy full SHA for ab56afb

Commits on Nov 22, 2024

• More documentation: tpch-spark and spark-mirror related

  

  Dhruv Garg

  committed
  b6fbab2

  Copy full SHA for b6fbab2

• add delay in test_service for register env ready

  

  Dhruv Garg

  committed
  977f36a

  Copy full SHA for 977f36a

• doc update for tpch spark

  

  Dhruv Garg

  committed
  d793b25

  Copy full SHA for d793b25

• Add test to invoke getPlacements before task registration

  

  Dhruv Garg

  committed
  6264772

  Copy full SHA for 6264772

• create task graph after environment is ready

  1ntEgr8

  committed
  bf658fe

  Copy full SHA for bf658fe

• update test

  1ntEgr8

  committed
  7873e73

  Copy full SHA for 7873e73

Commits on Nov 23, 2024

• Separate out internal state mgmt for driver and application

  

  Dhruv Garg

  committed
  e76c3fc

  Copy full SHA for e76c3fc

• re-add impl for override_worker_cpu_count

  

  Dhruv Garg

  committed
  5891dc3

  Copy full SHA for 5891dc3

• fix: correct enqueue of task_finished and sched_start in notifyTaskCompletion

  

  Dhruv Garg

  committed
  25f3acd

  Copy full SHA for 25f3acd

• enable task cancellations to be sent back to backend

  

  Dhruv Garg

  committed
  7ad75ec

  Copy full SHA for 7ad75ec

• update test script to include cancel_task scenario (needs to be sped up)

  

  Dhruv Garg

  committed
  83d63d8

  Copy full SHA for 83d63d8

Commits on Nov 25, 2024

• allow service to use different schedulers based on args

  

  Dhruv Garg

  committed
  66ec393

  Copy full SHA for 66ec393

• format service

  1ntEgr8

  committed
  400ec44

  Copy full SHA for 400ec44

• add enforce deadlines flag

  1ntEgr8

  committed
  f69c0a9

  Copy full SHA for f69c0a9

Commits on Nov 26, 2024

• step workers during a tick

  1ntEgr8

  committed
  9e8ab1e

  Copy full SHA for 9e8ab1e

• update placement time only if it is in the past

  1ntEgr8

  committed
  84d89d2

  Copy full SHA for 84d89d2

• refactor file

  1ntEgr8

  committed
  bf301aa

  Copy full SHA for bf301aa

• document refactored simulate_f

  1ntEgr8

  committed
  0499541

  Copy full SHA for 0499541

• handle case where task graph cannot be accommodated in worker pool

  1ntEgr8

  committed
  9539069

  Copy full SHA for 9539069

Commits on Nov 27, 2024

• Potpourri of bug fixes and improvements

  Show description for 89ee06d
  1ntEgr8

  committed
  89ee06d

  Copy full SHA for 89ee06d

• Add log stats method to simulator

  1ntEgr8

  committed
  98a5247

  Copy full SHA for 98a5247

• check for worker registration

  1ntEgr8

  committed
  618348a

  Copy full SHA for 618348a

• update documentation in service

  1ntEgr8

  committed
  15bf380

  Copy full SHA for 15bf380

• updating the service to operate in US instead of S time unit

  

  Dhruv Garg

  committed
  0b3a661

  Copy full SHA for 0b3a661

• misc improvements

  1ntEgr8

  committed
  cca24d8

  Copy full SHA for cca24d8

• override flags in service

  1ntEgr8

  committed
  7387b06

  Copy full SHA for 7387b06

• fix workload release bug

  1ntEgr8

  committed
  1461a3c

  Copy full SHA for 1461a3c

Commits on Nov 28, 2024

• update docs about sched time discretization

  

  Dhruv Garg

  committed
  209a4a3

  Copy full SHA for 209a4a3

• added comments about setting deadlines in generate_task_graph

  

  Dhruv Garg

  committed
  9104f7e

  Copy full SHA for 9104f7e

• updates to test script (verified across edf, fifo, dsched)

  

  Dhruv Garg

  committed
  8fd7f2a

  Copy full SHA for 8fd7f2a

• update documentation for the service

  

  Dhruv Garg

  committed
  9cf701a

  Copy full SHA for 9cf701a

Commits on Nov 29, 2024

• correctly handle task cancellations

  1ntEgr8

  committed
  25f4a3d

  Copy full SHA for 25f4a3d

• simulator bug fixes and improvements

  Show description for 4d979c1
  1ntEgr8

  committed
  4d979c1

  Copy full SHA for 4d979c1

Commits on Dec 2, 2024

• add launch script

  1ntEgr8

  committed
  48a9711

  Copy full SHA for 48a9711

Commits on Dec 3, 2024

• add shutdown rpc method

  1ntEgr8

  committed
  b1942ea

  Copy full SHA for b1942ea

• add service experiment runner

  1ntEgr8

  committed
  2e70864

  Copy full SHA for 2e70864

• sleep for a bit before launching queries

  1ntEgr8

  committed
  aeb4d6d

  Copy full SHA for aeb4d6d

• remove extraneous sleep

  1ntEgr8

  committed
  4094e01

  Copy full SHA for 4094e01

• set start_time of task to its placement time

  

  Dhruv Garg

  committed
  167ba49

  Copy full SHA for 167ba49

• add testcase to verify new taskgraph termination approach

  

  Dhruv Garg

  committed
  53cc735

  Copy full SHA for 53cc735

• nit documentation in erdos-spark setup

  

  Dhruv Garg

  committed
  29e306f

  Copy full SHA for 29e306f

• more explainable msgs from GetPlacements

  

  Dhruv Garg

  committed
  14190ad

  Copy full SHA for 14190ad

• reorder event queue priority to process scheduler events before task placement

  

  Dhruv Garg

  committed
  e2636f6

  Copy full SHA for e2636f6

• improvements to experiment runner

  1ntEgr8

  committed
  eeb4854

  Copy full SHA for eeb4854

• fix profile path in tpch loader

  1ntEgr8

  committed
  cb96c3e

  Copy full SHA for cb96c3e

• add spark-master-ip flag

  1ntEgr8

  committed
  7036fcf

  Copy full SHA for 7036fcf

Commits on Dec 4, 2024

• reinstate previous eventQueue priority order (task_placement before scheduer)

  

  Dhruv Garg

  committed
  bd16310

  Copy full SHA for bd16310

• [simulator] Unschedule subtree rooted at task if task is unable to run at timestep

  

  Dhruv Garg

  committed
  b4aceeb

  Copy full SHA for b4aceeb

• [service] log line to track tasks that get delayed in execution

  

  Dhruv Garg

  committed
  ce582e0

  Copy full SHA for ce582e0

• sleep for some time before signalling shutdown

  1ntEgr8

  committed
  3831f44

  Copy full SHA for 3831f44

• hack analyze pipeline to work with tpch output

  1ntEgr8

  committed
  2a74838

  Copy full SHA for 2a74838

• [simulator] check task state before invoking unschedule on it

  

  Dhruv Garg

  committed
  f4bbe6a

  Copy full SHA for f4bbe6a

• add support for tpch query partitioning

  1ntEgr8

  committed
  b958a8a

  Copy full SHA for b958a8a

Commits on Dec 14, 2024

• run_service_experiments: Log service stdout/stderr

  rohanbafna

  committed
  038cc7f

  Copy full SHA for 038cc7f

• run_service_experiments.py: Fix --dry_run

  rohanbafna

  committed
  6f84681

  Copy full SHA for 6f84681

• run_service_experiments: Timestamp results folder

  rohanbafna

  committed
  968b158

  Copy full SHA for 968b158

Commits on Dec 15, 2024

• run_service_experiments: Fix hang on exception

  Show description for 51d6f46
  rohanbafna

  committed
  51d6f46

  Copy full SHA for 51d6f46

• Spark service: Correctly log stats on shutdown

  Show description for 3491d59
  rohanbafna

  committed
  3491d59

  Copy full SHA for 3491d59

• Set correct completion time for finishing Tasks

  Show description for 7275702
  rohanbafna

  committed
  7275702

  Copy full SHA for 7275702