enhancement: make MAX_ATTEMPTS configurable

[dhpikolo]

Resolves: #2276

User can set METAFLOW_MAX_ATTEMPTS to configure MAX_ATTEMPTS internal envvar.

I think the hard limitation is not okay, would be great if the user can configure max times a task can be retried. In this PR to keep the default value to 6, but add MAX_ATTEMPTS to list of metaflow envars

Current Behaviour

import pandas as pd
from metaflow import (
    FlowSpec,
    Parameter,
    card,
    project,
    step,
    retry
)


@project(name="dummy_project")
class HelloWorld(FlowSpec):
    force_error = Parameter("force-error", type=bool, default=False)

    @card
    @step
    def start(self):
        print("something")
        self.next(self.end)

    @card
    @retry(times=10)
    @step
    def end(self):
        if self.force_error:
            raise Exception("Testing errors in metaflow")
        print(f"the data artifact is: {self.my_var}")


if __name__ == "__main__":
    HelloWorld()

• Running the above flow locally via python hello_world.py run throws the following exception

Metaflow 2.14.0 executing HelloWorld for user:j.kollipara
Project: dummy_project, Branch: user.j.kollipara
Validating your flow...
    The graph looks good!
Running pylint...
    Pylint is happy!
    Flow failed:
    The maximum number of retries is @retry(times=4).

error: Recipe `_poetry-run` failed with exit code 1

Source code:

metaflow/metaflow/plugins/retry_decorator.py

Lines 30 to 37 in 5c960ea

	def step_init(self, flow, graph, step, decos, environment, flow_datastore, logger):
	# The total number of attempts must not exceed MAX_ATTEMPTS.
	# attempts = normal task (1) + retries (N) + @catch fallback (1)
	if int(self.attributes["times"]) + 2 > MAX_ATTEMPTS:
	raise MetaflowException(
	"The maximum number of retries is "
	"@retry(times=%d)." % (MAX_ATTEMPTS - 2)
	)

Proposed Behaviour

Applying this change and setting METAFLOW_MAX_ATTEMPTS=12 would allow users to run the above flow.