Dev (#11)

* Resolving definitions references in swagger files * Fix NL2API modal height responsiveness * Change 'Choose File' to 'Upload the API Spec File' * Change 'Choose File' to 'Upload the API Spec File' * No reload button added * Small fixes * Changes by anukrati:- Upload merged into 1 * Added continuous upload + interaction flow * No reload button added * Small fixes * NL2AI --> Natural Language (NL) to API * [UI Changes] Remove the Start Crew Button, and disable Upload button while files are processing * [UI Changes] Remove the Start Crew Button, and disable Upload button while files are processing * Change Task Matcher Agent to Input Matcher Agent * System messages and UI display messages * swagger to API Spec * Updated images * Made small UI changes and added avatars * remove comments * Revert "swagger to API Spec" This reverts commit 711080d. * reverting commit 711080d * adding footer * Removing remaining comments and code cleanup * Removed dev creds --------- Co-authored-by: ksarkar-cloudera <[email protected]> Co-authored-by: anukrati15 <[email protected]> Co-authored-by: Abhishek <[email protected]> Co-authored-by: Kousani Sarkar <[email protected]>
cloudera · Dec 3, 2024 · 6e2f246 · 6e2f246
1 parent 9b402c0
commit 6e2f246
Show file tree

Hide file tree

Showing 31 changed files with 550 additions and 458 deletions.
diff --git a/.DS_Store b/.DS_Store
diff --git a/README.md b/README.md
@@ -1,6 +1,24 @@
-# aiagents
+# Multi-Agent API Orchestrator using CrewAI
 
-## Structure and Components
+### Welcome to the Multi-Agent API Orchestrator
+
+Get started by configuring your model and API integrations. We support both **Azure OpenAI** and **OpenAI** services. Please follow the steps below:
+
+#### Configuration Steps
+- **Azure OpenAI**: Provide the deployment name, endpoint URL, embedding deployment name, and API key.
+- **OpenAI**: Simply provide your OpenAI API key to enable model integration.
+
+#### Natural Language (NL) to API Integration
+- **Upload API Specification**: Upload a valid API Specification file.
+- **API Endpoint**: Enter the API endpoint for the integration.
+- **API Bearer Token**: Provide the bearer token to enable secure API authentication.
+
+Once all details are entered, click **Upload** to validate your inputs and initiate the orchestration process. **Streamline API orchestration** with CrewAI—handle complexities effortlessly and focus on results.""",
+
+
+The framework is segregated into 2 crews as described below
+
+## Structure and Components of the Interaction Crew
 
 ### Tools Used
 1. **swagger_directory_lister** = list all the files in the '{configuration.swagger_files_directory}' directory
@@ -21,7 +39,7 @@
 ### Agents in Action
 1. **Swagger Splitter Agent** = splits large Swagger JSON files into smaller, manageable files, ensuring efficient processing.
 2. **Human Input Agent** = gathers required information from the user and relaying it back to the delegating agent.
-3. **Task Matcher Agent** = identifies the best matching Swagger metadata file for a given task using metadata summaries.
+3. **Input Matcher Agent** = identifies the best matching Swagger metadata file for a given task using metadata summaries.
 4. **Metadata Summarizer Agent** = generates concise, descriptive summaries of API capabilities for all Swagger JSON files.
 5. **API Selector Agent** = selects the most appropriate API endpoint and method from a metadata file to fulfill a task.
 6. **Decision Validator Agent** = validates whether the actions of the API Selector Agent will satisfactorily fulfill the<br>
@@ -65,4 +83,4 @@ errors or missing information.
 5. **Manager Task** [<b style="color:#ffc482">API Selector Agent</b>] = Identifies and return the most suitable Swagger file based on the metadata summary and user context.
 6. **Decision Validator Task** [<b style="color:#c5813d">Decision Validator Agent</b>] = Validates the API Selector Agent's proposed answer against the original query, concluding whether it meets the user's intent and expectations.
 7. **API Calling Task** [<b style="color:#cc9999">API Caller Agent</b>] = Makes the API call, ensuring all parameters are identified, requested, and the payload is confirmed before execution.
-8. Stop execution. Need to click on 'Reload Crew' to start a new session.
+8. Stop execution. Need to click on 'Restart Crew' to start a new session.
diff --git a/aiagents/cml_agents/agents.py b/aiagents/cml_agents/agents.py
@@ -10,54 +10,6 @@
 
 class Agents:
     def __init__(self, configuration: Initialize) -> None:
-        # self.api_caller_agent = Agent(
-        #     role="API Caller Agent",
-        #     goal=dedent(
-        #         f"""
-        #         Based on the information provided by the manager task, fetch the user query, as well as the file containing
-        #         the API call information. If you cannot find the generated folder, that means the swagger splitter has not
-        #         run yet. After that you will need to read the file and determine the exact details of the API call. Identify 
-        #         which parameters are required and which are optional. 
-
-        #         Based on all the information gathered, incase you don't already have the information in hand, using the 
-        #         human input tool ask the user to provide details about the fields they would like to set while making the
-        #         call. Make no assumptions here. Make sure to clarify any doubts a user might have about the API call.
-
-        #         If the user doesn't have some information on hand, or they ask you a question you aren't sure of, make sure
-        #         to ask the manager agent to fetch the information for you. Once the manager agent has fetched the information, 
-        #         provide it back to the user so they can tell you what information they need.
-
-        #         Once the user has confirmed all the details, make the API call and return the results. If you run into errors
-        #         while making the API call, make sure to try and deal with the error to the best of your abilities. If you 
-        #         determine that user inputs are needed to complete the call, provide the error message back to the user, 
-        #         and ask for clarification about the same. Once the user has provided all the required information, make the
-        #         call again.
-
-        #         After successfully making the API call, give the user the results through the human input tool. Post this, ask 
-        #         the user using the human input tool to Reload the Crew if they have any further queries, and end the execution.
-        #         """
-        #     ),
-        #     backstory=dedent(
-        #         """
-        #         You are an intelligent, helpful API caller who is very experienced with understanding the importance of 
-        #         varied parameters. You are patient and can help users with their queries. You are also very good at
-        #         figuring out what action to take based on the situation, and if the user asks for information that you
-        #         aren't aware of, you can ask your boss, the API selector to help fetch the information so you can,
-        #         provide it to the user. Make sure you clearly state which parameters are optional and which are required.
-        #         Make no assumptions."
-        #         """
-        #     ),
-        #     allow_delegation=True,
-        #     verbose=True,
-        #     tools=[
-        #         get_human_input,
-        #         FileReadTool(),
-        #         DirectoryReadTool(),
-        #         api_caller,
-        #     ],
-        #     llm=configuration.llm,
-        #     callbacks=configuration.customInteractionCallbacks,
-        # )
 
         self.human_input_agent = Agent(
             role="Human Input Agent",

diff --git a/aiagents/cml_agents/callback_utils.py b/aiagents/cml_agents/callback_utils.py
@@ -10,21 +10,18 @@ def custom_agent_callback(output, *args, **kwargs):
         agent_match = re.search(r"return_values='([^']+)'", str(args))
         agent_name = agent_match.group(1) if agent_match else ""
         if agent_name != "Swagger API Description Summarizer":
-            # print("agent name =", agent_name)
             configuration.current_agent = agent_name
-            # CustomPanelCallbackHandler.on_chain_end()
 
     pprint(f"The callback kwargs are {kwargs}")
-    #pprint(f"The callback received from the agent is {formatted_answer}")
 
 def custom_callback(*args, **kwargs):
-    # pprint(f"The callback args are {args}")
-    # pprint(f"The callback kwargs are {kwargs}")
+    pprint(f"The callback args are {args}")
+    pprint(f"The callback kwargs are {kwargs}")
     pass
     #pprint(f"The callback received from the agent is {formatted_answer}")
 
 def custom_initialization_callback(*args, **kwargs):
-    # pprint(f"The callback args are {args}")
-    # pprint(f"The callback kwargs are {kwargs}")
+    pprint(f"The callback args are {args}")
+    pprint(f"The callback kwargs are {kwargs}")
     pass
     #pprint(f"The callback received from the agent is {formatted_answer}")
diff --git a/aiagents/cml_agents/manager_agents.py b/aiagents/cml_agents/manager_agents.py
@@ -18,7 +18,7 @@
 class ManagerAgents:
     def __init__(self, configuration: Initialize) -> None:
         self.task_matching_agent = Agent(
-            role="Task Matcher",
+            role="Input Matcher",
             goal="""Match the tasks to the best matching API based on the metadata summaries."""
             """ Fetch the metadata summary using the metadata_summary_fetcher tool. If there"""
             """ is no metadata summary available, make sure that you return a meesage to the user with appropriate message that"""
@@ -33,7 +33,7 @@ def __init__(self, configuration: Initialize) -> None:
             llm=configuration.llm,
             callbacks=configuration.customInteractionCallbacks,
             step_callback=custom_agent_callback,
-            step_kwargs={"agent": "Task Matcher"}
+            step_kwargs={"agent": "Input Matcher"}
         )
 
         self.manager_agent = Agent(
@@ -66,7 +66,7 @@ def __init__(self, configuration: Initialize) -> None:
             ),
             backstory=dedent(
                 """
-                You are an API selector and caller where based on the context received by the task matcher, you need to choose the exact 
+                You are an API selector and caller where based on the context received by the input matcher, you need to choose the exact 
                 metadata file needed to use to service a user query. Make sure to generate full complete answers, and make no assumptions.
 
                 You are also very experienced with understanding the importance of varied parameters. You are patient and can help 

diff --git a/aiagents/cml_agents/parse_for_manager.py b/aiagents/cml_agents/parse_for_manager.py
@@ -5,13 +5,23 @@
 
 class CustomJSONEncoder(json.JSONEncoder):
     """Custom JSON encoder to handle circular references."""
+    def __init__(self, *args, **kwargs):
+        super().__init__(*args, **kwargs)
+        self.seen = set()
+
     def default(self, obj):
         # Handle specific types that may cause issues
+        if isinstance(obj, (dict, list)):
+            if id(obj) in self.seen:
+                return "CircularReferenceDetected"
+            self.seen.add(id(obj))
+
         if isinstance(obj, dict):
             return {key: self.default(value) for key, value in obj.items()}
         elif isinstance(obj, list):
             return [self.default(element) for element in obj]
-        return super().default(obj)
+        else:
+            return super().default(obj)
 
 def read_swagger_file(swagger_file_location):
     """
@@ -25,7 +35,7 @@ def read_swagger_file(swagger_file_location):
             with open(swagger_file_location, 'r') as file:
                 return jsonref.load(file, lazy_load=False, proxies=False, merge_props=True)
         except (IOError, jsonref.JsonRefError) as e:
-            print(f"Error loading Swagger file: {e}")
+            print(f"Error loading API Specification file: {e}")
     elif swagger_file_location.endswith('.yaml') or swagger_file_location.endswith('.yml'):
         with open(swagger_file_location, 'r') as file:
             return yaml.safe_load(file)
@@ -76,8 +86,11 @@ def swagger_parser(swagger_file_name: str, swagger_file_root: str, generated_fol
         sanitized_key = sanitize_file_name(path)
         chunk_file_name = os.path.join(output_dir, f"{sanitized_key}.json")
 
-        with open(chunk_file_name, 'w') as file:
-            json.dump(chunk, file, cls=CustomJSONEncoder, indent=2)
+        try:
+            with open(chunk_file_name, 'w') as file:
+                json.dump(chunk, file, cls=CustomJSONEncoder, indent=2)
+        except Exception as e:
+            print(f"Error loading API Specification file chunk: {e}")
 
         # Populate the metadata for mapping paths to chunk files
         methods_metadata = {}

diff --git a/aiagents/cml_agents/swagger_splitter.py b/aiagents/cml_agents/swagger_splitter.py
@@ -16,35 +16,6 @@
 
 class SwaggerSplitterAgents:
     def __init__(self, configuration: Initialize) -> None:
-        # Define the Swagger Splitter Agent
-        # self.swagger_splitter_agent = Agent(
-        #     role="swagger_splitter",
-        #     goal=dedent(
-        #         """
-        #         Follow the following steps:
-        #             1. Use the directory read tool to find all Swagger JSON files in the specified directory.
-        #             2. If the generated folder doesn't exist, for each Swagger JSON file found run the 'swagger_splitter'
-        #             tool on the file. Upon running the tool, mark your execution as finished.
-        #             3. Using the previous output, check if the generated folder exists. If it does, then that means that
-        #             the tool has already run, and we can consider that the 'swagger_splitter' tool has run. Note that this
-        #             does not mean that the metadata summaries have been generated. 
-
-        #         The 'swagger_splitter' tool processes the file and generates smaller files. The swagger_splitter tool does
-        #         not generate the metadata summaries, and the check whether or not the metadata summary files have to be generated
-        #         needs to be performed by the Swagger API Description Summarizer agent.
-
-        #         Make no assumptions whatsoever. You are not to interact with the user for any purpose.
-        #         """
-        #     ),
-        #     backstory="""You are an expert in processing Swagger files. Your primary responsibility is to """
-        #     """split large Swagger JSON files into smaller, more manageable files to facilitate efficient """
-        #     """processing and distribution across multiple services.""",
-        #     verbose=True,
-        #     allow_delegation=False,
-        #     tools=[swagger_splitter, swagger_directory_lister],
-        #     llm=configuration.llm,
-        #     callbacks=configuration.customInteractionCallbacks,
-        # )
 
         # Define the Summary Agent
         self.metadata_summarizer_agent = Agent(