Autogen latency v2 [do not merge]

[areibman]

🔍 Review Summary

Purpose:
Streamline module loading and improve HTTP request performance.

Key Changes:

• Enhancement: Relocated autogen check within initialization process for streamlined module loading.
• Enhancement: Implemented connection pooling in HTTP client to optimize request performance.
• New Feature: Added format_duration function for duration calculation and formatting.
• New Feature: Added /autogen_completion endpoint to server with detailed logging and time tracking.

Impact:
Improves system efficiency and functionality by reducing request time, optimizing module loading, and providing detailed logging and time tracking for session operations.

Original Description

No existing description found

[entelligence-ai-pr-reviews]

Walkthrough

This update introduces several enhancements and optimizations across the AgentOps project. The autogen check has been relocated within the initialization process to streamline module loading. Connection pooling has been implemented in the HTTP client to improve performance, reducing request time by 0.5 seconds. A new utility function for formatting durations has been added. Additionally, the server has been updated to include a new endpoint for autogen completion, featuring detailed logging and time tracking for session operations. These changes collectively enhance the efficiency and functionality of the system.

Changes

File(s)	Summary
agentops/__init__.py	Moved the autogen check into the initialization function to ensure it is executed during module setup.
agentops/http_client.py	Implemented connection pooling by introducing a class-level session object, optimizing HTTP requests by reusing connections. Adjusted connection pool settings for better performance.
agentops/helpers.py	Added a new function format_duration to calculate and format the duration between two timestamps.
tests/core_manual_tests/api_server/server.py	Updated the server to include a new /autogen_completion endpoint. Added detailed logging for session initialization, LLM calls, and session termination, with time tracking for each operation.

Files selected (5)

• agentops/init.py
  - agentops/http_client.py
  - agentops/helpers.py
  - agentops/http_client.py
  - tests/core_manual_tests/api_server/server.py

Files ignored (0)

Instructions

Emoji Descriptions:

• ⚠️ Potential Issue - May require further investigation.
• 🔒 Security Vulnerability - Fix to ensure system safety.
• 💻 Code Improvement - Suggestions to enhance code quality.
• 🔨 Refactor Suggestion - Recommendations for restructuring code.
• ℹ️ Others - General comments and information.

Interact with the Bot:

• Send a message or request using the format:

  @Entelligence.AI + *your message*

  • Example: @Entelligence.AI Can you suggest improvements for this code?

• Execute a command using the format:

  @Entelligence.AI + *"/command"*

Available Commands:

• /updateCommit: Apply the suggested changes and commit them.
• /updateGuideline: Modify an existing guideline.
• /addGuideline: Introduce a new guideline.

Tips for Using @Entelligence.AI Effectively:

• Specific Queries: For the best results, be specific with your requests. Example: @Entelligence.AI summarize the changes in this PR.
• Focused Discussions: Tag @Entelligence.AI directly on specific code lines or files for detailed feedback.
• Managing Reviews: Use review comments for targeted discussions on code snippets, and PR comments for broader queries about the entire PR.

Need More Help?

• Visit our documentation for detailed guides on using Entelligence.AI.
• Join our community to connect with others, request features, and share feedback.
• Follow us for updates on new features and improvements.

[entelligence-ai-pr-reviews]

ℹ️ Error Handling

Enhanced Error Handling for HTTP Requests

The error handling for HTTP requests has been improved by catching specific exceptions like Timeout, HTTPError, and RequestException. This ensures that the application can gracefully handle network issues and provide meaningful error messages.

[entelligence-ai-pr-reviews]

ℹ️ Performance Improvement

Implement Connection Pooling for HTTP Requests

The introduction of connection pooling in the HTTP client significantly reduces the time taken for HTTP requests by reusing existing connections. This change enhances performance by minimizing the overhead of establishing new connections for each request, saving approximately 0.5 seconds per request.

[entelligence-ai-pr-reviews]

🔒 Security Suggestion

Secure Handling of Sensitive Headers

Ensure that sensitive headers such as API keys and JWT tokens are not logged or exposed in error messages. Consider using environment variables or secure vaults to manage these credentials.

[entelligence-ai-pr-reviews]

ℹ️ Performance Improvement

Re-evaluate Connection Pool Configuration for Scalability

The change reduces the number of connection pools from 10 to 1, assuming that api.agentops.ai is the only host. While this might be suitable for a single-host scenario, it could lead to performance bottlenecks if multiple hosts are accessed in the future. Consider maintaining a higher number of connection pools to accommodate potential future scalability needs.

Commitable Code Suggestion:

Suggested change

	# Configure session defaults
	adapter = HTTPAdapter(
	max_retries=retry_config,
	pool_connections=1, # Assuming api.agentops.ai is the only host
	pool_maxsize=100, # Maximum number of connections to save in the pool
	)
	cls._session.mount('http://', adapter)
	- pool_connections=1, # Assuming api.agentops.ai is the only host
	+ pool_connections=10, # Maintain higher pool for scalability

[entelligence-ai-pr-reviews]

ℹ️ Performance Improvement

Implement Connection Pooling for Improved Performance

The introduction of a class-level session object for connection pooling significantly improves performance by reusing connections, reducing the overhead of establishing new connections for each request. This change is crucial for high-throughput applications.

Commitable Code Suggestion:

Suggested change

	class HttpClient:
	@staticmethod
	_session = None # Class-level session object
	@classmethod
	def get_session(cls) -> requests.Session:
	if cls._session is None:
	cls._session = requests.Session()
	# Configure session defaults
	adapter = HTTPAdapter(
	max_retries=retry_config,
	pool_connections=1, # Assuming api.agentops.ai is the only host
	pool_maxsize=100, # Maximum number of connections to save in the pool
	)
	cls._session.mount('http://', adapter)
	cls._session.mount('https://', adapter)
	cls._session.headers.update({
	'Content-Type': 'application/json; charset=UTF-8',
	'Accept': '*/*',
	'User-Agent': 'AgentOps-Client'
	})
	return cls._session
	@classmethod
	def post(
	cls,
	url: str,
	payload: bytes,
	api_key: Optional[str] = None,
	class HttpClient:
	_session = None # Class-level session object
	@classmethod
	def get_session(cls) -> requests.Session:
	if cls._session is None:
	cls._session = requests.Session()
	# Configure session defaults
	adapter = HTTPAdapter(
	max_retries=retry_config,
	pool_connections=10, # Number of connection pools to cache
	pool_maxsize=100, # Maximum number of connections to save in the pool
	)
	cls._session.mount('http://', adapter)
	cls._session.mount('https://', adapter)
	cls._session.headers.update({
	'Content-Type': 'application/json; charset=UTF-8',
	'Accept': '*/*',
	'User-Agent': 'AgentOps-Client'
	})
	return cls._session
	@classmethod
	def post(
	cls,
	url: str,
	payload: bytes,
	api_key: Optional[str] = None,

[entelligence-ai-pr-reviews]

💻 Code Refactoring

Refactor HTTP Methods to Use Shared Session

Refactoring the post and get methods to use a shared session object reduces code duplication and improves maintainability. This change also ensures consistent configuration across all HTTP requests.

Commitable Code Suggestion:

Suggested change

	return result
	@staticmethod
	@classmethod
	def get(
	cls,
	url: str,
	api_key: Optional[str] = None,
	jwt: Optional[str] = None,
	header=None,
	) -> Response:
	result = Response()
	try:
	# Create request session with retries configured
	request_session = requests.Session()
	request_session.mount(url, HTTPAdapter(max_retries=retry_config))
	if api_key is not None:
	JSON_HEADER["X-Agentops-Api-Key"] = api_key
	if jwt is not None:
	JSON_HEADER["Authorization"] = f"Bearer {jwt}"
	session = cls.get_session()
	res = request_session.get(url, headers=JSON_HEADER, timeout=20)
	# Update headers for this request
	headers = dict(session.headers)
	if api_key:
	headers["X-Agentops-Api-Key"] = api_key
	if jwt:
	headers["Authorization"] = f"Bearer {jwt}"
	try:
	res = session.get(url, headers=headers, timeout=20)
	result.parse(res)
	except requests.exceptions.Timeout:
	result.code = 408
	result.status = HttpStatus.TIMEOUT
	raise ApiServerException(
	"Could not reach API server - connection timed out"
	)
	raise ApiServerException("Could not reach API server - connection timed out")
	except requests.exceptions.HTTPError as e:
	try:
	result.parse(e.response)
	def post(
	return result
	@classmethod
	def get(
	cls,
	url: str,
	api_key: Optional[str] = None,
	jwt: Optional[str] = None,
	header=None,
	) -> Response:
	result = Response()
	session = cls.get_session()
	# Update headers for this request
	headers = dict(session.headers)
	if api_key:
	headers["X-Agentops-Api-Key"] = api_key
	if jwt:
	headers["Authorization"] = f"Bearer {jwt}"
	try:
	res = session.get(url, headers=headers, timeout=20)
	result.parse(res)
	except requests.exceptions.Timeout:
	result.code = 408
	result.status = HttpStatus.TIMEOUT
	raise ApiServerException("Could not reach API server - connection timed out")
	except requests.exceptions.HTTPError as e:
	try:
	result.parse(e.response)

[entelligence-ai-pr-reviews]

🔒 Security Suggestion

Handle Division by Zero in Calculator Function

The calculator function currently performs division without handling division by zero, which could lead to runtime errors. Implement error handling to manage this scenario gracefully.

     def calculator(a: int, b: int, operator: Annotated[Operator, "operator"]) -> int:
         if operator == "+":
             return a + b
         elif operator == "-":
             return a - b
         elif operator == "*":
             return a * b
         elif operator == "/":
+            if b == 0:
+                raise ValueError("Division by zero is not allowed")
             return int(a / b)
         else:
             raise ValueError("Invalid operator")

Commitable Code Suggestion:

Suggested change

	returns=str(response.choices[0].message.content),
	),
	)
	record_time = time.time() - record_start
	print(f"Recording event took {record_time:.2f} seconds")
	end_start = time.time()
	session.end_session(end_state="Success")
	end_time = time.time() - end_start
	print(f"Ending session took {end_time:.2f} seconds")
	total_time = time.time() - start_time
	print(f"Total completion time: {total_time:.2f} seconds")
	return {"response": response}
	uvicorn.run(app, host="0.0.0.0", port=9696)
	@app.get("/autogen_completion")
	def autogen_completion():
	# Add time tracking
	start_time = time.time()
	# Record start time for session initialization
	session_start = time.time()
	session = agentops.start_session(tags=["api-server-test", "fastapi", "autogen"])
	session_time = time.time() - session_start
	print(f"Session initialization took {session_time:.2f} seconds")
	# Define model, openai api key, tags, etc in the agent configuration
	config_list = [
	{
	"model": "gpt-4o-mini",
	"api_key": os.getenv("OPENAI_API_KEY"),
	"tags": ["mathagent-example", "tool"],
	}
	]
	Operator = Literal["+", "-", "*", "/"]
	def calculator(a: int, b: int, operator: Annotated[Operator, "operator"]) -> int:
	if operator == "+":
	return a + b
	elif operator == "-":
	return a - b
	elif operator == "*":
	return a * b
	elif operator == "/":
	return int(a / b)
	else:
	raise ValueError("Invalid operator")
	# Create the agent that uses the LLM
	assistant = ConversableAgent(
	name="Assistant",
	system_message="You are a helpful AI assistant. "
	"You can help with simple calculations. "
	"Return 'TERMINATE' when the task is done.",
	llm_config={"config_list": config_list},
	)
	# Create the user proxy agent
	user_proxy = ConversableAgent(
	name="User",
	llm_config=False,
	is_termination_msg=lambda msg: msg.get("content") is not None
	and "TERMINATE" in msg["content"],
	human_input_mode="NEVER",
	)
	# Register the calculator function
	assistant.register_for_llm(name="calculator", description="A simple calculator")(calculator)
	user_proxy.register_for_execution(name="calculator")(calculator)
	register_function(
	calculator,
	caller=assistant,
	executor=user_proxy,
	name="calculator",
	description="A simple calculator",
	)
	response = None
	try:
	response = user_proxy.initiate_chat(
	assistant, message="What is (1423 - 123) / 3 + (32 + 23) * 5?"
	)
	except StdinNotImplementedError:
	print("Stdin not implemented. Skipping initiate_chat")
	session.end_session("Indeterminate")
	return {"response": "Chat initiation skipped - stdin not implemented"}
	# Close your AgentOps session to indicate that it completed
	end_start = time.time()
	session.end_session("Success")
	end_duration = time.time() - end_start
	print(f"Session ended in {end_duration:.2f}s. Visit your AgentOps dashboard to see the replay")
	# Calculate duration at the end
	duration = time.time() - start_time
	print(f"Total duration: {duration:.2f} seconds")
	return {"response": response, "duration": duration}
	if __name__ == "__main__":
	uvicorn.run("server:app", host="0.0.0.0", port=9696, reload=True)
	def calculator(a: int, b: int, operator: Annotated[Operator, "operator"]) -> int:
	if operator == "+":
	return a + b
	elif operator == "-":
	return a - b
	elif operator == "*":
	return a * b
	elif operator == "/":
	if b == 0:
	raise ValueError("Division by zero is not allowed")
	return int(a / b)
	else:
	raise ValueError("Invalid operator")

[entelligence-ai-pr-reviews]

ℹ️ Logic Error

Handle Durations Exceeding 24 Hours

The format_duration function currently assumes that the duration will not exceed 24 hours. If the duration can be longer, consider adding logic to handle days as well. This will ensure accurate representation of longer durations.

+def format_duration(start_time, end_time):
+    start = datetime.fromisoformat(start_time.replace("Z", "+00:00"))
+    end = datetime.fromisoformat(end_time.replace("Z", "+00:00"))
+    duration = end - start
+
+    days = duration.days
+    hours, remainder = divmod(duration.seconds, 3600)
+    minutes, seconds = divmod(remainder, 60)
+
+    parts = []
+    if days > 0:
+        parts.append(f"{days}d")
+    if hours > 0:
+        parts.append(f"{int(hours)}h")
+    if minutes > 0:
+        parts.append(f"{int(minutes)}m")
+    parts.append(f"{seconds:.1f}s")
+
+    return " ".join(parts)

Commitable Code Suggestion:

Suggested change

	return datetime.now(timezone.utc).isoformat()
	def format_duration(start_time, end_time):
	start = datetime.fromisoformat(start_time.replace("Z", "+00:00"))
	end = datetime.fromisoformat(end_time.replace("Z", "+00:00"))
	duration = end - start
	hours, remainder = divmod(duration.total_seconds(), 3600)
	minutes, seconds = divmod(remainder, 60)
	parts = []
	if hours > 0:
	parts.append(f"{int(hours)}h")
	if minutes > 0:
	parts.append(f"{int(minutes)}m")
	parts.append(f"{seconds:.1f}s")
	return " ".join(parts)
	def is_jsonable(x):
	try:
	json.dumps(x)
	def format_duration(start_time, end_time):
	start = datetime.fromisoformat(start_time.replace("Z", "+00:00"))
	end = datetime.fromisoformat(end_time.replace("Z", "+00:00"))
	duration = end - start
	days = duration.days
	hours, remainder = divmod(duration.seconds, 3600)
	minutes, seconds = divmod(remainder, 60)
	parts = []
	if days > 0:
	parts.append(f"{days}d")
	if hours > 0:
	parts.append(f"{int(hours)}h")
	if minutes > 0:
	parts.append(f"{int(minutes)}m")
	parts.append(f"{seconds:.1f}s")
	return " ".join(parts)

[entelligence-ai-pr-reviews]

ℹ️ Performance Improvement

Use Logging Library for Time Tracking

The introduction of time tracking for various operations (session initialization, LLM call, recording, and session ending) is a valuable addition for performance monitoring. However, consider using a dedicated logging library instead of print statements for better control over log levels and outputs.

+    import logging
+    logging.basicConfig(level=logging.INFO)

     start_time = time.time()
     session_start = time.time()
     session = agentops.start_session(tags=["api-server-test", "fastapi"])
     session_time = time.time() - session_start
-    print(f"Session initialization took {session_time:.2f} seconds")
+    logging.info(f"Session initialization took {session_time:.2f} seconds")

     messages = [{"role": "user", "content": "Hello"}]
     llm_start = time.time()
     response = litellm.completion(
         model="gpt-4o",
         messages=messages,
         temperature=0.5,
     )
-    print(response)
     llm_time = time.time() - llm_start
-    print(f"LLM call took {llm_time:.2f} seconds")
+    logging.info(f"LLM call took {llm_time:.2f} seconds")

     record_start = time.time()
     session.record(
         ActionEvent(
             action_type="Agent says hello",
             returns=str(response.choices[0].message.content),
         ),
     )
     record_time = time.time() - record_start
-    print(f"Recording event took {record_time:.2f} seconds")
+    logging.info(f"Recording event took {record_time:.2f} seconds")

     end_start = time.time()
     session.end_session(end_state="Success")
     end_time = time.time() - end_start
-    print(f"Ending session took {end_time:.2f} seconds")
+    logging.info(f"Ending session took {end_time:.2f} seconds")

     total_time = time.time() - start_time
-    print(f"Total completion time: {total_time:.2f} seconds")
+    logging.info(f"Total completion time: {total_time:.2f} seconds")

Commitable Code Suggestion:

Suggested change

	import autogen
	import agentops
	from fastapi import FastAPI
	import uvicorn
	from dotenv import load_dotenv
	from agentops import ActionEvent
	from openai import OpenAI
	import os
	from autogen import ConversableAgent, UserProxyAgent
	import os
	from dotenv import load_dotenv
	from IPython.core.error import (
	StdinNotImplementedError,
	) # only needed by AgentOps testing automation
	from typing import Annotated, Literal
	from autogen import register_function
	import time
	import litellm
	load_dotenv()
	openai = OpenAI()
	agentops.init()
	agentops.init(auto_start_session=False)
	client = OpenAI()
	app = FastAPI()
	@app.get("/completion")
	def completion():
	start_time = time.time()
	session = agentops.start_session(tags=["api-server-test"])
	session_start = time.time()
	session = agentops.start_session(tags=["api-server-test", "fastapi"])
	session_time = time.time() - session_start
	print(f"Session initialization took {session_time:.2f} seconds")
	messages = [{"role": "user", "content": "Hello"}]
	response = session.patch(openai.chat.completions.create)(
	model="gpt-3.5-turbo",
	llm_start = time.time()
	response = litellm.completion(
	model="gpt-4o",
	messages=messages,
	temperature=0.5,
	)
	print(response)
	llm_time = time.time() - llm_start
	print(f"LLM call took {llm_time:.2f} seconds")
	record_start = time.time()
	session.record(
	ActionEvent(
	action_type="Agent says hello",
	import logging
	logging.basicConfig(level=logging.INFO)
	start_time = time.time()
	session_start = time.time()
	session = agentops.start_session(tags=["api-server-test", "fastapi"])
	session_time = time.time() - session_start
	logging.info(f"Session initialization took {session_time:.2f} seconds")
	messages = [{"role": "user", "content": "Hello"}]
	llm_start = time.time()
	response = litellm.completion(
	model="gpt-4o",
	messages=messages,
	temperature=0.5,
	)
	llm_time = time.time() - llm_start
	logging.info(f"LLM call took {llm_time:.2f} seconds")
	record_start = time.time()
	session.record(
	ActionEvent(
	action_type="Agent says hello",
	returns=str(response.choices[0].message.content),
	),
	)
	record_time = time.time() - record_start
	logging.info(f"Recording event took {record_time:.2f} seconds")
	end_start = time.time()
	session.end_session(end_state="Success")
	end_time = time.time() - end_start
	logging.info(f"Ending session took {end_time:.2f} seconds")
	total_time = time.time() - start_time
	logging.info(f"Total completion time: {total_time:.2f} seconds")

[entelligence-ai-pr-reviews]

Files selected (1)

• agentops/http_client.py

Files ignored (0)

Instructions

Emoji Descriptions:

• ⚠️ Potential Issue - May require further investigation.
• 🔒 Security Vulnerability - Fix to ensure system safety.
• 💻 Code Improvement - Suggestions to enhance code quality.
• 🔨 Refactor Suggestion - Recommendations for restructuring code.
• ℹ️ Others - General comments and information.

Interact with the Bot:

• Send a message or request using the format:

  @Entelligence.AI + *your message*

  • Example: @Entelligence.AI Can you suggest improvements for this code?

• Execute a command using the format:

  @Entelligence.AI + *"/command"*

Available Commands:

• /updateCommit: Apply the suggested changes and commit them.
• /updateGuideline: Modify an existing guideline.
• /addGuideline: Introduce a new guideline.

Tips for Using @Entelligence.AI Effectively:

• Specific Queries: For the best results, be specific with your requests. Example: @Entelligence.AI summarize the changes in this PR.
• Focused Discussions: Tag @Entelligence.AI directly on specific code lines or files for detailed feedback.
• Managing Reviews: Use review comments for targeted discussions on code snippets, and PR comments for broader queries about the entire PR.

Need More Help?

• Visit our documentation for detailed guides on using Entelligence.AI.
• Join our community to connect with others, request features, and share feedback.
• Follow us for updates on new features and improvements.

[entelligence-ai-pr-reviews]

ℹ️ Performance Improvement

Ensure Consistent Protocol Handling with HTTP Adapter

The refactoring of the HTTP adapter configuration ensures that the session mounts for both HTTP and HTTPS protocols are correctly set up. This change improves the reliability of the HTTP client by ensuring that all requests are handled consistently, regardless of the protocol used.

Commitable Code Suggestion:

Suggested change

	adapter = HTTPAdapter(
	max_retries=retry_config,
	pool_connections=1, # Assuming api.agentops.ai is the only host
	pool_maxsize=100, # Maximum number of connections to save in the pool
	)
	cls._session.mount("http://", adapter)
	cls._session.mount("https://", adapter)
	cls._session.headers.update(
	{
	"Content-Type": "application/json; charset=UTF-8",
	"Accept": "*/*",
	"User-Agent": "AgentOps-Client",
	}
	)
	return cls._session
	@staticmethod
	@classmethod
	def get_session(cls) -> requests.Session:
	adapter = HTTPAdapter(
	max_retries=retry_config,
	pool_connections=1, # Assuming api.agentops.ai is the only host
	pool_maxsize=100, # Maximum number of connections to save in the pool
	)
	cls._session.mount("http://", adapter)
	cls._session.mount("https://", adapter)
	cls._session.headers.update(
	{
	"Content-Type": "application/json; charset=UTF-8",
	"Accept": "*/*",
	"User-Agent": "AgentOps-Client",
	}
	)
	return cls._session

[entelligence-ai-pr-reviews]

Files selected (1)

• tests/core_manual_tests/api_server/server.py

Files ignored (0)

Instructions

Emoji Descriptions:

• ⚠️ Potential Issue - May require further investigation.
• 🔒 Security Vulnerability - Fix to ensure system safety.
• 💻 Code Improvement - Suggestions to enhance code quality.
• 🔨 Refactor Suggestion - Recommendations for restructuring code.
• ℹ️ Others - General comments and information.

Interact with the Bot:

• Send a message or request using the format:

  @Entelligence.AI + *your message*

  • Example: @Entelligence.AI Can you suggest improvements for this code?

• Execute a command using the format:

  @Entelligence.AI + *"/command"*

Available Commands:

• /updateCommit: Apply the suggested changes and commit them.
• /updateGuideline: Modify an existing guideline.
• /addGuideline: Introduce a new guideline.

Tips for Using @Entelligence.AI Effectively:

• Specific Queries: For the best results, be specific with your requests. Example: @Entelligence.AI summarize the changes in this PR.
• Focused Discussions: Tag @Entelligence.AI directly on specific code lines or files for detailed feedback.
• Managing Reviews: Use review comments for targeted discussions on code snippets, and PR comments for broader queries about the entire PR.

Need More Help?

• Visit our documentation for detailed guides on using Entelligence.AI.
• Join our community to connect with others, request features, and share feedback.
• Follow us for updates on new features and improvements.

[areibman]

Closing until I get this under control