New Spark Heuristics in Dr. Elephant which are supported by custom Spark History Server changes.

This release includes the below commits since v2.1.7. Please note that this release runs with custom Spark History Server changes made internally in LinkedIn.

We are running this with MapReduceFSFetcherHadoop2 for MapReduce and SparkFetcher for Spark Jobs.

7c1e88b making rest calls sequential
4ebd4b9 adding failedTasks value (#363)
638eb77 Removing blocking keyword so as to prevent a large number of threads being spawned (#362)
25c07bb Spark Heuristic Fixes for Dr. Elephant (#324)
019a9f4 Changing GC thresholds and calculation in spill heuristic (#319)
8e193a2 Fixed resources used/wasted computation for spark jobs - (Depends on Custom SHS - Requires peakJvmUsedMemory metric) (#287)
78dd699 Peak Unified Memory Heuristic - (Depends on Custom SHS - Requires peakUnifiedMemory metric) (#281)
7ca8706 Spark Peak jvm memory Heuristic - (Depends on Custom SHS - Requires peakJvmUsedMemory metric) (#318)
a40d251 Spark Stages with Failed tasks Heuristic - (Depends on Custom SHS - Requires stages/failedTasks Rest API) (#288)
6b4a3cf Spark Executor Spill Heuristic - (Depends on Custom SHS - Requires totalMemoryBytesSpilled metric) (#310)
50a7409 Removing blocking keyword (#361)
a0470a3 Dr. Elephant Tez Support working patch (#313)
d5a6897 added connection timeout for REST Calls. (#359)
fe7bfea changed async for LogClient (#354)
977623d Changed async to future/blocking and changed the error to warn (#353)
c89bafe Reducing timeout of spark fetcher from 60 to 5 seconds (#345)
fe076f7 Bug fix: Auto tuning disable model unit test failure (#343)
79bb59f Added support for multiple Azkaban Host URL (#342)
8ea2850 Bug fix: Delay computation of MR application (#340)
b2c24b8 Adding Auto Tuning Feature (#338)
c182c98 added a function to check if the script's required programs exist or exit the program with an indicative message (#326)
5500aad Revert "Peak JVM used memory heuristic - (Depends on Custom SHS - Requires peakJvmUsedMemory metric) (#283)" (#317)
6b2f7e8 Peak JVM used memory heuristic - (Depends on Custom SHS - Requires peakJvmUsedMemory metric) (#283)
7a27a3f Secondary Sort suggestion to reduce memory footprint at reducer (#316)
a208c31 Spark Configuration Threshold Heuristic (#286)
8c99625 Spark Executor GC Heuristic (#311)
a384fcc Added a Second Retry Queue - Useful while fetching Spark Metrics (#314)
8b46933 Add httpcore dependency to solve classpath issues (#308)
35d06d9 Dr. Elephant should check for finished directory before listing
e756226 TUNING Updating default MR fetcher for performance (#300)
9c8915c BUGFIX Updating java_args to elephant.conf for resolving argument conflicts (#299)
53fd50c BUGFIX Updating AnalyticJobGeneratorHadoop2.java to resolve the Job listing Conflict (#302)
83c1ef3 Fix MapReduceFSFetcherHadoop2 Fetcher filesystem to pick the configured URI (#292)
37ad77f BUGFIX: Fixes NullPointerException in AnalyticJobGeneratorHadoop2 (#294)