Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Use LiteLLM in place of ChatOpenAI #84

Open
wants to merge 19 commits into
base: main
Choose a base branch
from
Open

Use LiteLLM in place of ChatOpenAI #84

wants to merge 19 commits into from

Conversation

morganmcg1
Copy link
Member

@morganmcg1 morganmcg1 commented Dec 26, 2024 •
edited
Loading

This PR adds a model-agnostic fallback system that allows using any model as a fallback for another model, with comprehensive error handling and testing.

Changes

  • Add model-agnostic fallback system in base ChatModel
  • Implement error handling with retryable errors
  • Add model tracking in responses
  • Add comprehensive test suite for fallback behavior
  • Update Gemini model to use new error handling system

Features

  • Support for cross-model fallbacks (e.g., Gemini -> OpenAI)
  • Retryable vs non-retryable error handling
  • Model tracking in responses
  • Fallback chains (multiple fallbacks)
  • Comprehensive test coverage

Example Usage

# Create models with fallback chain
final_fallback = OpenAIChatModel("gpt-3.5-turbo")
middle_fallback = GeminiChatModel("gemini-1.0-pro", fallback_model=final_fallback)
primary_model = GeminiChatModel("gemini-pro", fallback_model=middle_fallback)

# Response includes which model was used
response = primary_model.generate_response(messages)
print(f"Response from {response["model_used"]}: {response["content"]}")

@openhands-agent
Add cross-model fallback support and error handling
0b36103 
- Add model-agnostic fallback system in base ChatModel
- Implement error handling with retryable errors
- Add model tracking in responses
- Add comprehensive test suite for fallback behavior
- Update Gemini model to use new error handling system
@socket-security Socket Security
Copy link

socket-security bot commented Dec 26, 2024 •
edited
Loading

New and removed dependencies detected. Learn more about Socket for GitHub ↗︎

Package New capabilities Transitives Size Publisher
pypi/[email protected] Transitive: environment, eval, filesystem, network, shell, unsafe +438 3.58 GB

View full report↗︎

@openhands-agent
Update PR:
0c3352d 
- Remove temp files
- Update OpenAI model to use 'developer' role
- Keep existing prompt logic
@openhands-agent
Simplify chat model using LiteLLM
c42deb6 
- Use LiteLLM for provider-agnostic interface
- Handle OpenAI 'developer' role conversion
- Simple error handling with retryable errors
- Add comprehensive tests
@openhands-agent
Remove extra test file
ee0683f
@openhands-agent
Update chat model implementation:
f50be06 
- Use LiteLLM for provider-agnostic interface
- Support latest models:
  * OpenAI GPT-4 Turbo Mini (128K)
  * Gemini 2.0 Flash (1M)
  * Claude 3 Haiku (200K)
- Add comprehensive tests for:
  * Context window limits
  * Message format validation
  * Temperature validation
  * Error handling
  * Provider-specific message formats
@openhands-agent
Use LiteLLM's built-in fallback mechanism:
6902e9c 
- Remove custom fallback/retry logic
- Use litellm.completion fallbacks parameter
- Configure fallbacks in constructor
- Update tests to match LiteLLM behavior
@openhands-agent
Simplify ChatModel to use LiteLLM's built-in features:
e06257d 
- Remove custom message format handling (LiteLLM handles it)
- Remove custom retry/fallback logic (use LiteLLM's fallbacks param)
- Update tests to match LiteLLM behavior
@openhands-agent
Simplify error handling:
ef7f29b 
- Remove custom error type handling
- Remove retryable flag (LiteLLM handles retries)
- Update tests to match simpler error handling
@openhands-agent
Further simplify ChatModel:
9171818 
- Remove provider-specific model files (LiteLLM handles it)
- Remove token count tracking (not needed)
- Update tests to match simpler response format
@openhands-agent
Make ChatModel a descriptor that wraps LiteLLM:
84d188c 
- Match original ChatModel descriptor pattern
- Keep LiteLLM's provider-agnostic features
- Update tests to match descriptor pattern
@openhands-agent
Keep response_synthesis.py API unchanged:
99a62d2 
- Keep RESPONSE_SYNTHESIS_SYSTEM_PROMPT and RESPONSE_SYNTHESIS_PROMPT_MESSAGES
- Update imports to use BaseChatModel
- Keep LangChain interface compatibility
@openhands-agent
Use ChatModel type hint in response_synthesis.py:
ff93f5d 
- Remove BaseChatModel import (not needed)
- Update _load_chain to use ChatModel type hint
@openhands-agent
Remove warning filters from conftest.py:
3d5c0c5 
- Remove protobuf warning filters (not needed)
- Keep pytest import
@openhands-agent
Remove tests/mock_models.py:
a0cc8cc 
- No longer needed since we're using MagicMock
- Provider-specific models have been removed
- Mock functionality handled in test files
@openhands-agent
Restore original poetry.lock
d6f81a2
@openhands-agent
Remove tests/conftest.py:
cc87aaf 
- No longer needed after removing warning filters
- Only contained import pytest
- No fixtures or test configuration needed
@openhands-agent
Update chat.py to use ChatModel:
6b38d4e 
- Replace OpenAI client with ChatModel
- Keep same model name (gpt-4o-2024-08-06)
- Keep same temperature (0)
- Add retries for reliability
@openhands-agent
Move ChatModel to utils.py:
c87e35a 
- Move LiteLLM-based ChatModel to utils.py
- Remove old ChatOpenAI-based ChatModel
- Remove chat_model.py (no longer needed)
- Update imports to use ChatModel from utils.py
@morganmcg1 morganmcg1 changed the title Add cross-model fallback support and error handling Use LiteLLM in place of ChatOpenAI Dec 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@morganmcg1 @openhands-agent