feat(health_check.py): set upperbound for api when making health chec…

…k call (#7865)

* feat(health_check.py): set upperbound for api when making health check call

prevent bad model from health check to hang and cause pod restarts

* fix(health_check.py): cleanup task once completed

* fix(constants.py): bump default health check timeout to 1min

* docs(health.md): add 'health_check_timeout' to health docs on litellm

* build(proxy_server_config.yaml): add bad model to health check

docs/my-website/docs/proxy/health.md

@@ -245,6 +245,22 @@ general_settings:
   health_check_details: False
 ```
 
+## Health Check Timeout
+
+The health check timeout is set in `litellm/constants.py` and defaults to 60 seconds.
+
+This can be overridden in the config.yaml by setting `health_check_timeout` in the model_info section.
+
+```yaml
+model_list:
+  - model_name: openai/gpt-4o
+    litellm_params:
+      model: openai/gpt-4o
+      api_key: os.environ/OPENAI_API_KEY
+    model_info:
+      health_check_timeout: 10 # 👈 OVERRIDE HEALTH CHECK TIMEOUT
+```
+
 ## `/health/readiness`
 
 Unprotected endpoint for checking if proxy is ready to accept requests

litellm/constants.py

@@ -140,3 +140,5 @@
 
 BATCH_STATUS_POLL_INTERVAL_SECONDS = 3600  # 1 hour
 BATCH_STATUS_POLL_MAX_ATTEMPTS = 24  # for 24 hours
+
+HEALTH_CHECK_TIMEOUT_SECONDS = 60  # 60 seconds

litellm/proxy/_new_secret_config.yaml

@@ -6,8 +6,8 @@ model_list:
       api_base: https://exampleopenaiendpoint-production.up.railway.app
   - model_name: openai-o1
     litellm_params:
-      model: openai/o1
+      model: openai/random_sleep
       api_key: sk-1234
-      api_base: https://exampleopenaiendpoint-production.up.railway.app
-      mock_timeout: true
-      timeout: 3
+      api_base: http://0.0.0.0:8090
+    model_info:
+      health_check_timeout: 1