Merge pull request #248 from tigergraph/GML-1824-regression-test-with…

…-out-cypher-support-only

Gml 1824 regression test with cypher support and no hallucination and usefulness check

common/prompts/aws_bedrock_claude3haiku/generate_function.txt

@@ -1,8 +1,8 @@
 Use the vertex types, edge types, and their attributes and IDs below to write the pyTigerGraph function call to answer the question using a pyTigerGraph connection.
-When the question asks for "How many", make sure to select a function that contains "Count" in the description/function call. Make sure never to generate a function that is not listed below. 
+When the question asks for "How many", make sure to always select a function that contains "Count" in the description/function call. Make sure never to generate a function that is not listed below. 
 When certain entities are mapped to vertex attributes, may consider to generate a WHERE clause.
 If a WHERE clause is generated, please follow the instruction with proper quoting. To construct a WHERE clause string. Ensure that string attribute values are properly quoted. 
-For example, if the generated function is "conn.getVertices('Person', where='name=William Torres')", Expected Output: "conn.getVertices('Person', where='name="William Torres"')", This rule applies to all types of attributes. e.g., name, email, address and so on.
+For example, if the generated function contains "('Person', where='name=William Torres')", Expected Output: "('Person', where='name="William Torres"')", This rule applies to all types of attributes. e.g., name, email, address and so on.
 Documentation contains helpful Python docstrings for the various functions. Use this knowledge to construct the proper function call. Choose one function to execute.
 Don't generate target_vertex_ids if there is no the term 'id' explicitly mentioned in the question.
 Vertex Types: {vertex_types}

common/prompts/aws_bedrock_claude3haiku/map_question_to_schema.txt

@@ -10,4 +10,5 @@ Don't generate target_vertex_ids if there is no the term 'id' explicitly mention
 
 Respond in JSON (and only JSON). Follow the format instructions below:
 {format_instructions}
-QUESTION: {question}
+question: {question}
+conversation: {conversation}

common/prompts/aws_bedrock_titan/generate_function.txt

@@ -1,7 +1,8 @@
 Use the vertex types, edge types, and their attributes and IDs to write the pyTigerGraph function call to answer the question using a pyTigerGraph connection.
+When the question asks for "How many", make sure to always select a function that contains "Count" in the description/function call. Make sure never to generate a function that is not listed below. 
 When certain entities are mapped to vertex attributes, may consider to generate a WHERE clause.
 If a WHERE clause is generated, please follow the instruction with proper quoting. To construct a WHERE clause string. Ensure that string attribute values are properly quoted. 
-For example, if the generated function is "conn.getVertices('Person', where='name=William Torres')", Expected Output: "conn.getVertices('Person', where='name="William Torres"')", This rule applies to all types of attributes. e.g., name, email, address and so on.
+For example, if the generated function contains "('Person', where='name=William Torres')", Expected Output: "('Person', where='name="William Torres"')", This rule applies to all types of attributes. e.g., name, email, address and so on.
 Documentation contains helpful Python docstrings for the various functions. Use this knowledge to construct the proper function call. Choose one function to execute.
 Don't generate target_vertex_ids if there is no the term 'id' explicitly mentioned in the question.
 Vertex Types: {vertices}

common/prompts/aws_bedrock_titan/map_question_to_schema.txt

@@ -15,4 +15,5 @@ Response: What is the schema?
 Example: How many transactions are there?
 Response: How many TRANSACTION Edges are there?
 {format_instructions}
-QUESTION: {question}
+question: {question}
+conversation: {conversation}

common/prompts/azure_open_ai_gpt35_turbo_instruct/generate_function.txt

@@ -1,8 +1,8 @@
 Use the vertex types, edge types, and their attributes and IDs below to write the pyTigerGraph function call to answer the question using a pyTigerGraph connection.
-When the question asks for "How many", make sure to select a function that contains "Count" in the description/function call. Make sure never to generate a function that is not listed below. 
+When the question asks for "How many", make sure to always select a function that contains "Count" in the description/function call. Make sure never to generate a function that is not listed below. 
 When certain entities are mapped to vertex attributes, may consider to generate a WHERE clause.
 If a WHERE clause is generated, please follow the instruction with proper quoting. To construct a WHERE clause string. Ensure that string attribute values are properly quoted. 
-For example, if the generated function is "conn.getVertices('Person', where='name=William Torres')", Expected Output: "conn.getVertices('Person', where='name="William Torres"')", This rule applies to all types of attributes. e.g., name, email, address and so on.
+For example, if the generated function contains "('Person', where='name=William Torres')", Expected Output: "('Person', where='name="William Torres"')", This rule applies to all types of attributes. e.g., name, email, address and so on.
 Documentation contains helpful Python docstrings for the various functions. Use this knowledge to construct the proper function call. Choose one function to execute.
 Don't generate target_vertex_ids if there is no the term 'id' explicitly mentioned in the question.
 

common/prompts/azure_open_ai_gpt35_turbo_instruct/map_question_to_schema.txt

@@ -12,4 +12,6 @@ Generate the complete question with the appropriate replacements. Keep the case
 Format your response following the directions below.
 Don't generate target_vertex_ids if there is no the term 'id' explicitly mentioned in the question.
 
-{format_instructions}
+{format_instructions}
+question: {question}
+conversation: {conversation}

common/prompts/gcp_vertexai_palm/generate_function.txt

@@ -1,8 +1,8 @@
 Use the vertex types, edge types, and their attributes and IDs below to write the pyTigerGraph function call to answer the question using a pyTigerGraph connection.
-When the question asks for "How many", make sure to select a function that contains "Count" in the description/function call. Make sure never to generate a function that is not listed below. 
+When the question asks for "How many", make sure to always select a function that contains "Count" in the description/function call. Make sure never to generate a function that is not listed below. 
 When certain entities are mapped to vertex attributes, may consider to generate a WHERE clause.
 If a WHERE clause is generated, please follow the instruction with proper quoting. To construct a WHERE clause string. Ensure that string attribute values are properly quoted. 
-For example, if the generated function is "conn.getVertices('Person', where='name=William Torres')", Expected Output: "conn.getVertices('Person', where='name="William Torres"')", This rule applies to all types of attributes. e.g., name, email, address and so on.
+For example, if the generated function contains "('Person', where='name=William Torres')", Expected Output: "('Person', where='name="William Torres"')", This rule applies to all types of attributes. e.g., name, email, address and so on.
 Documentation contains helpful Python docstrings for the various functions. Use this knowledge to construct the proper function call. Choose one function to execute.
 Don't generate target_vertex_ids if there is no the term 'id' explicitly mentioned in the question.
 

common/prompts/gcp_vertexai_palm/map_question_to_schema.txt

@@ -10,4 +10,5 @@ Generate the complete question with the appropriate replacements. Keep the case
 Don't generate target_vertex_ids if there is no the term 'id' explicitly mentioned in the question.
 
 {format_instructions}
-QUESTION: {question}
+question: {question}
+conversation: {conversation}

common/prompts/llama_70b/generate_function.txt

@@ -1,8 +1,8 @@
 Use the vertex types, edge types, and their attributes and IDs below to write the pyTigerGraph function call to answer the question using a pyTigerGraph connection.
-When the question asks for "How many", make sure to select a function that contains "Count" in the description/function call. Make sure never to generate a function that is not listed below. 
+When the question asks for "How many", make sure to always select a function that contains "Count" in the description/function call. Make sure never to generate a function that is not listed below. 
 When certain entities are mapped to vertex attributes, may consider to generate a WHERE clause.
 If a WHERE clause is generated, please follow the instruction with proper quoting. To construct a WHERE clause string. Ensure that string attribute values are properly quoted. 
-For example, if the generated function is "conn.getVertices('Person', where='name=William Torres')", Expected Output: "conn.getVertices('Person', where='name="William Torres"')", This rule applies to all types of attributes. e.g., name, email, address and so on.
+For example, if the generated function contains "('Person', where='name=William Torres')", Expected Output: "('Person', where='name="William Torres"')", This rule applies to all types of attributes. e.g., name, email, address and so on.
 Documentation contains helpful Python docstrings for the various functions. Use this knowledge to construct the proper function call. Choose one function to execute.
 Don't generate target_vertex_ids if there is no the term 'id' explicitly mentioned in the question.
 Vertex Types: {vertex_types}

common/prompts/llama_70b/map_question_to_schema.txt

@@ -10,4 +10,6 @@ Generate the complete question with the appropriate replacements. Keep the case
 Don't generate target_vertex_ids if there is no the term 'id' explicitly mentioned in the question.
 
 {format_instructions}
-QUESTION: {question}
+question: {question}
+conversation: {conversation}
+

common/prompts/openai_gpt4/generate_function.txt

@@ -1,8 +1,8 @@
 Use the vertex types, edge types, and their attributes and IDs below to write the pyTigerGraph function call to answer the question using a pyTigerGraph connection.
-When the question asks for "How many", make sure to select a function that contains "Count" in the description/function call. Make sure never to generate a function that is not listed below. 
+When the question asks for "How many", make sure to always select a function that contains "Count" in the description/function call. Make sure never to generate a function that is not listed below. 
 When certain entities are mapped to vertex attributes, may consider to generate a WHERE clause.
 If a WHERE clause is generated, please follow the instruction with proper quoting. To construct a WHERE clause string. Ensure that string attribute values are properly quoted. 
-For example, if the generated function is "conn.getVertices('Person', where='name=William Torres')", Expected Output: "conn.getVertices('Person', where='name="William Torres"')", This rule applies to all types of attributes. e.g., name, email, address and so on.
+For example, if the generated function contains "('Person', where='name=William Torres')", Expected Output: "('Person', where='name="William Torres"')", This rule applies to all types of attributes. e.g., name, email, address and so on.
 Documentation contains helpful Python docstrings for the various functions. Use this knowledge to construct the proper function call. Choose one function to execute.
 Don't generate target_vertex_ids if there is no the term 'id' explicitly mentioned in the question.
 Vertex Types: {vertex_types}

common/prompts/openai_gpt4/map_question_to_schema.txt

@@ -10,5 +10,6 @@ Generate the complete question with the appropriate replacements. Keep the case
 Don't generate target_vertex_ids if there is no the term 'id' explicitly mentioned in the question.
 
 {format_instructions}
-QUESTION: {question}
+question: {question}
+conversation: {conversation}
 

copilot/app/agent/agent.py

@@ -1,4 +1,5 @@
 import logging
+import json
 import time
 from typing import Dict, List
 
@@ -119,20 +120,28 @@ def question_for_agent(
 
             input_data = {}
             input_data["input"] = question
-            logger.info(f"conversation: {conversation}")
 
             if conversation is not None:
                 input_data["conversation"] = [
                     {"query": chat["query"], "response": chat["response"]}
                     for chat in conversation
                 ]
+
             else:
                 # Handle the case where conversation is None or empty
                 input_data["conversation"] = []
-            logger.info(f"input_data: {input_data}")
 
-            for output in self.agent.stream({"question": str(input_data)}):
+            # Validate and convert input_data to JSON string
+            try:
+                input_data_str = json.dumps(input_data)
+            except (TypeError, ValueError) as e:
+                logger.error(f"Failed to serialize input_data to JSON: {e}")
+                raise ValueError("Invalid input data format. Unable to convert to JSON.")
+
+            for output in self.agent.stream({"question": input_data["input"], "conversation": input_data["conversation"]}):
+
                 for key, value in output.items():
+                    logger.info(f"testing steps {key}: {value}")
                     LogWriter.info(f"request_id={req_id_cv.get()} executed node {key}")
 
             LogWriter.info(f"request_id={req_id_cv.get()} EXIT question_for_agent")

copilot/app/agent/agent_generation.py

@@ -1,4 +1,3 @@
-
 import logging
 from langchain.prompts import PromptTemplate
 from langchain_core.output_parsers import PydanticOutputParser
@@ -9,7 +8,7 @@
 logger = logging.getLogger(__name__)
 
 class CoPilotAnswerOutput(BaseModel):
-    generated_answer: str = Field(description="The generated answer to the question. Make sure maintain a professional tone and keep the answer consice.")
+    generated_answer: str = Field(description="The generated answer to the question. Make sure maintain a professional tone.")
     citation: list[str] = Field(description="The citation for the answer. List the information used.")
 
 class TigerGraphAgentGenerator:
@@ -29,19 +28,28 @@ def generate_answer(self, question: str, context: str) -> str:
         answer_parser = PydanticOutputParser(pydantic_object=CoPilotAnswerOutput)
 
         prompt = PromptTemplate(
-            template="""Given the question and the context, generate an answer. \n
-                        Make sure to answer the question in a friendly and informative way. \n
+            template="""Given the answer context in JSON format, rephrase it to answer the question. \n
+                        Use only the provided information in context without adding any reasoning or additional logic. \n
+                        Make sure all information in the answer are covered in the generated answer.\n
                         Question: {question} \n
-                        Context: {context}
+                        Answer: {context} \n
                         Format: {format_instructions}""",
             input_variables=["question", "context"],
             partial_variables={
                 "format_instructions": answer_parser.get_format_instructions()
             }
         )
 
+        full_prompt = prompt.format(
+            question=question,
+            context=context,
+            format_instructions=answer_parser.get_format_instructions()
+        )
+
         # Chain
         rag_chain = prompt | self.llm.model | answer_parser
-        generation = rag_chain.invoke({"context": context, "question": question})
+        generation = rag_chain.invoke({"question": question, "context": context})
+
         LogWriter.info(f"request_id={req_id_cv.get()} EXIT generate_answer")
+
         return generation