AI Project client says "Invalid tool value(s): azure_function"

[kuljotSB]

Hi all,
I have been trying to create an agent with an azure function (queue trigger) deployed into azure with the azure ai foundry SDK. The only problem is when i create an agent with tool as the azure function, it fails saying: "Invalid tool value(s): azure_function"

Pasting the code for reference:

import os
from azure.ai.projects import AIProjectClient
from azure.identity import DefaultAzureCredential
from dotenv import load_dotenv

load_dotenv()

project_connection_string = os.getenv("PROJECT_CONNECTION_STRING")
model=os.getenv("MODEL_DEPLOYMENT_NAME")
storage_connection_string = os.getenv("STORAGE_CONNECTION_queueServiceUri")
input_queue_name = os.getenv("INPUT_QUEUE_NAME")
output_queue_name = os.getenv("OUTPUT_QUEUE_NAME")

project_client = AIProjectClient.from_connection_string(
        credential=DefaultAzureCredential(),
        conn_str=project_connection_string
    )



agent = project_client.agents.create_agent(
        model=model,
        name="azure-function-agent",
        instructions="You are a helpful support agent. Answer the user's questions to the best of your ability.",
        headers={"x-ms-enable-preview": "true"},
        tools=[
            {
                "type": "azure_function",
                "azure_function": {
                    "function": {
                        "name": "queue_trigger1",
                        "description": "Deliver a DeathNote to the user in Batman's style.",
                        "parameters": {
                            "type": "object",
                            "properties": {
                                "name": {"type": "string", "description": "the name of the user or the name of the person to which the deathnote is to be delivered."},
                            },
                            "required": ["name"]
                        }
                    },
                    "input_binding": {
                        "type": "storage_queue",
                        "storage_queue": {
                            "queue_service_uri": storage_connection_string,
                            "queue_name": input_queue_name
                        }
                    },
                    "output_binding": {
                        "type": "storage_queue",
                        "storage_queue": {
                            "queue_service_uri": storage_connection_string,
                            "queue_name": output_queue_name
                        }
                    }
                }
            }
        ],
    )

thread = project_client.agents.create_thread()

message = project_client.agents.create_message(
        thread_id=thread.id,
        role="user",
        content="deliver a deathnote by batman to Kuljot",
)
print(f"Created message, ID: {message.id}")

# Create and process agent run in thread with azure function supplied as a tool
run = project_client.agents.create_and_process_run(thread_id=thread.id, assistant_id=agent.id)
print(f"Run finished with status: {run.status}")

if run.status == "failed":
        print(f"Run failed: {run.last_error}")


# Fetch and log all messages
messages = project_client.agents.list_messages(thread_id=thread.id)
#Displaying the assistant response
print(messages.data[0].content[0].text.value)
    

Any help or resolution would be highly appreciated!

[github-actions]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @achauhan-scc @kingernupur @luigiw @needuv @paulshealy1 @singankit.

[nick863]

The reason of this error is because the Azure AI Foundry and hub are missing the capability host. The easiest way is to create the new Azur AI Foundry project as outlined here.
There are several steps needed to make it work with the old workspace:

1. Create Search service resource.
2. In its Settings > Keys set API Access control to Both.
3. Connect the AI search resource to the Azure Foundry Hub using API Key.
4. [Need to be a group Owner!] Assign roles Search Index Data Contributor and Search Service Contributor for Azure AI Foundry.
5. Create capability host for Hub and project (https://learn.microsoft.com/en-us/cli/azure/ml/capability-host?view=azure-cli-latest)

pip install -U azure-cli
# Create the capability host for hub
az config set extension.dynamic_install_allow_preview=true
az ml capability-host create -g my-resource-group -w my-hub  
# Create capabilityhost for the project, but now provide necessary connections to the data store and AI search resource:
az ml capability-host create  create -g my-resource-group -w my-project -s my-project/workspaceblobstore -v vector_store_connection_name_from_step_3 -a ai_services_connection_used_by_a_project

Please also follow the Readme steps here to deploy Azure function.

[kuljotSB]

Hi @nick863 I am having trouble with Azure Function as tool calling to my Azure AI Agent. Azure AI Search works fine.

[nick863]

What is the Azure AI Foundry region?

[kuljotSB]

What is the Azure AI Foundry region?

@nick863 Sweden Central

[kuljotSB]

Hi,
Any updates on this?

[jhakulin]

@kuljotSB swedencentral is supported region, https://learn.microsoft.com/en-us/azure/ai-services/agents/concepts/model-region-support#azure-openai-models, which model you were using?

Also did you choose agents standard template when deployed resources? That is required for azure functions tool to work.

https://learn.microsoft.com/en-us/azure/ai-services/agents/quickstart?pivots=programming-language-python-azure#choose-basic-or-standard-agent-setup

[kuljotSB]

@jhakulin I believe the model was gpt-4o......tried doing it with other models like gpt-4, gpt-35-turbo and in east us region too but doesn't work....keep receiving this in the terminal -

"Traceback (most recent call last):
  File "C:\Users\HP VICTUS\Downloads\AzureAIAgentService-3\Lab_Azure_Functions\agent.py", line 21, in <module>
    agent = project_client.agents.create_agent(
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\HP VICTUS\AppData\Local\Programs\Python\Python311\Lib\site-packages\azure\core\tracing\decorator.py", line 105, in wrapper_use_tracer
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\HP VICTUS\AppData\Local\Programs\Python\Python311\Lib\site-packages\azure\ai\projects\operations\_patch.py", line 1062, in create_agent
    new_agent = super().create_agent(
                ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\HP VICTUS\AppData\Local\Programs\Python\Python311\Lib\site-packages\azure\core\tracing\decorator.py", line 105, in wrapper_use_tracer
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\HP VICTUS\AppData\Local\Programs\Python\Python311\Lib\site-packages\azure\ai\projects\operations\_operations.py", line 1757, in create_agent    
    raise HttpResponseError(response=response)
azure.core.exceptions.HttpResponseError: (None) Invalid tool value(s): azure_function
Code: None
Message: Invalid tool value(s): azure_function"

[jhakulin]

@kuljotSB Which deployment template you used? Basic or Standard?
https://learn.microsoft.com/en-us/azure/ai-services/agents/quickstart?pivots=programming-language-python-azure#choose-basic-or-standard-agent-setup

Or did you create resources somehow else?

[kuljotSB]

@jhakulin I must have followed the "Standard Agent Setup"

[kuljotSB]

@jhakulin I had already configured an Azure AI Foundry project and created an agent via code-first environment (VSCode IDE) so I presume it follows the "Standard Agent Setup"

[jhakulin]

@kuljotSB Did you create project in Azure AI Foundry UI? If so, it is probable that the needed capabilities are not be set for Azure Functions.

Thus currently, what I can suggest is to use the Standard Agent deployment template below to deploye resources (incl. project) again and then test with the project resource created there.
https://learn.microsoft.com/en-us/azure/ai-services/agents/quickstart?pivots=programming-language-python-azure#choose-basic-or-standard-agent-setup

[kuljotSB]

@jhakulin Yes upon deploying the standard agent template I am able to create an azure function tooled agent..........although the reason for this still remains unknown and strange, thanks for your help, appreciate it!

[jhakulin]

@kuljotSB Thanks for information! The reason is probably that required capability hosts were not set.

Would be great if you remember still how did you earlier created the resources, was it using Azure AI Foundry UI or something else?

[kuljotSB]

@jhakulin I did it via the Foundry UI earlier........wondering if there's a way to fix it via the UI without launching the template again?

[nick863]

Generally, we have these issues arise from the capbility host being missing. Have you tried to add the capability host?

[github-actions]

Hi @kuljotSB. Thank you for opening this issue and giving us the opportunity to assist. To help our team better understand your issue and the details of your scenario please provide a response to the question asked above or the information requested above. This will help us more accurately address your issue.

[jhakulin]

@jhakulin Yes upon deploying the standard agent template I am able to create an azure function tooled agent..........although the reason for this still remains unknown and strange, thanks for your help, appreciate it!

@kuljotSB Were you able to get azure function working with an agent?

[kuljotSB]

@jhakulin so sorry for replying late, was busy with my college work.

Coming to the question, I went a step ahead and surprisingly, the function app integration with ai agent doesn't work.....gives an "internal server error".

Function app code (app.py):

import azure.functions as func
import json
import logging
import os
import time
from azure.ai.projects import AIProjectClient
from azure.identity import DefaultAzureCredential
from azure.storage.queue import QueueClient, BinaryBase64EncodePolicy, BinaryBase64DecodePolicy

app = func.FunctionApp()

# Name of the queues to get and send the function call messages
input_queue_name = "inputqueue"
output_queue_name = "outputqueue"

# Function to initialize the agent client and the tools Azure Functions that the agent can use
def initialize_client():
    # Create a project client using the connection string from local.settings.json
    project_client = AIProjectClient.from_connection_string(
        credential=DefaultAzureCredential(),
        conn_str=os.environ["PROJECT_CONNECTION_STRING"]
    )

    # Get the connection string from local.settings.json
    storage_connection_string = os.environ["STORAGE_CONNECTION__queueServiceUri"]
    
    agent = project_client.agents.create_agent(
            model="gpt-4o-mini",
            name="GreetingByBatman",
            instructions="You are a helpful support agent. Answer the user's questions to the best of your ability.",
            headers={"x-ms-enable-preview": "true"},
            tools=[
                {
                    "type": "azure_function",
                    "azure_function": {
                        "function": {
                            "name": "GreetingByBatman",
                            "description": "Deliver a Greeting to the user in Batman's style.",
                            "parameters": {
                                "type": "object",
                                "properties": {
                                    "name": {"type": "string", "description": "the name of the user or the name of the person to which the greeting is to be delivered."},
                                },
                                "required": ["name"]
                            }
                        },
                        "input_binding": {
                            "type": "storage_queue",
                            "storage_queue": {
                                "queue_service_uri": storage_connection_string,
                                "queue_name": input_queue_name
                            }
                        },
                        "output_binding": {
                            "type": "storage_queue",
                            "storage_queue": {
                                "queue_service_uri": storage_connection_string,
                                "queue_name": output_queue_name
                            }
                        }
                    }
                }
            ],
        )
    logging.info(f"Created agent, agent ID: {agent.id}")

    # Create a thread
    thread = project_client.agents.create_thread()
    logging.info(f"Created thread, thread ID: {thread.id}")

    return project_client, thread, agent

@app.route(route="prompt", auth_level=func.AuthLevel.FUNCTION)
def prompt(req: func.HttpRequest) -> func.HttpResponse:
    logging.info('Python HTTP trigger function processed a request.')

    # Get the prompt from the request body
    req_body = req.get_json()
    prompt = req_body.get('Prompt')

    # Initialize the agent client
    project_client, thread, agent = initialize_client()

    # Send the prompt to the agent
    message = project_client.agents.create_message(
        thread_id=thread.id,
        role="user",
        content=prompt,
    )
    logging.info(f"Created message, message ID: {message.id}")

    # Run the agent
    run = project_client.agents.create_run(thread_id=thread.id, assistant_id=agent.id)
    # Monitor and process the run status
    while run.status in ["queued", "in_progress", "requires_action"]:
        time.sleep(1)
        run = project_client.agents.get_run(thread_id=thread.id, run_id=run.id)

        if run.status not in ["queued", "in_progress", "requires_action"]:
            break

    logging.info(f"Run finished with status: {run.status}")

    if run.status == "failed":
        logging.error(f"Run failed: {run.last_error}")
        logging.error(run)

    # Get messages from the assistant thread
    messages = project_client.agents.list_messages(thread_id=thread.id)
    #Displaying the assistant response
    logging.info(messages.data[0].content[0].text.value)
    last_msg = messages.data[0].content[0].text.value

    # Delete the agent once done
    project_client.agents.delete_agent(agent.id)
    print("Deleted agent")

    return func.HttpResponse(last_msg)

@app.function_name(name="GreetingByBatman")
@app.queue_trigger(arg_name="msg", queue_name="inputqueue",
                               connection="STORAGE_CONNECTION") 
def GreetingByBatman(msg: func.QueueMessage) -> None:
    logging.info('Python queue trigger function processed a queue item')

    # Queue to send message to
    queue_client = QueueClient(
        os.environ["STORAGE_CONNECTION__queueServiceUri"],
        queue_name="outputqueue",
        credential=DefaultAzureCredential(),
        message_encode_policy=BinaryBase64EncodePolicy(),
        message_decode_policy=BinaryBase64DecodePolicy()
    )

    messagepayload = json.loads(msg.get_body().decode('utf-8'))
    name=messagepayload["name"]

    greeting_note = f""" It's not who you are underneath, {name}, but what you do that defines you. Stay strong, stay vigilant… and happy to have you here """ 

    result_message = {
        'Value': greeting_note
    }

    queue_client.send_message(json.dumps(result_message).encode('utf-8'))

    logging.info(f"Sent message to queue with message {result_message}")

local.settings.json file:

{
    "IsEncrypted": false,
    "Values": {
      "FUNCTIONS_WORKER_RUNTIME": "python",
      "STORAGE_CONNECTION__queueServiceUri": "https://YOUR_STORAGE_ACCOUNT_NAME.queue.core.windows.net",
      "PROJECT_CONNECTION_STRING": "YOUR_AI_PROJECT_CONNECTION_STRING",
      "AzureWebJobsStorage": "STORAGE_CONNECTION_STRING"
      }
  }

Observations:
1)The queue triggered function works when invoked manually on cloud (where in I manually put a message block in the input queue) but the implementation fails when tried with the azure ai agent.

2)When a request is send to the HTTP triggered function which invokes the agent (example: hello, how are you?) - the agent responds properly but when invoked with a request that triggers the function tool (example - send a greeting message to kuljot in Batman's voice) - the agent gives internal server error.

3)I don't see any message body in the input queue when triggered by agent which suggests the agent is not able to push the message body into the input queue in the very first place.

4. I also don't see a poison queue created which means, when triggered by the agent, the queue triggered function didn't reaally have a message body in the input queue to begin with.

[nick863]

One thing that we need to add is the correlation ID. It is used by the application to correlate its input messages to the output messages, used by the app.
Another note is that we need to make sure that the function resource have the System Identity enabled:

And it is allowed to write to the Queue resource of the Azure Blob you have created the output query in (Click Asure role Assignment and set Storage Queue Data Contributor role):

Please also check that the
Let us try to change the function:

@app.function_name(name="GreetingByBatman")
@app.queue_trigger(arg_name="msg", queue_name="inputqueue",
                               connection="STORAGE_CONNECTION") 
def GreetingByBatman(msg: func.QueueMessage) -> None:
    logging.info('Python queue trigger function processed a queue item')

    # Queue to send message to
    queue_client = QueueClient(
        os.environ["STORAGE_CONNECTION__queueServiceUri"],
        queue_name="outputqueue",
        credential=DefaultAzureCredential(),
        message_encode_policy=BinaryBase64EncodePolicy(),
        message_decode_policy=BinaryBase64DecodePolicy()
    )

    messagepayload = json.loads(msg.get_body().decode('utf-8'))
    name=messagepayload["name"]
    correlation_id=messagepayload['CorrelationId']

    greeting_note = f""" It's not who you are underneath, {name}, but what you do that defines you. Stay strong, stay vigilant… and happy to have you here """ 

    result_message = {
        'Value': greeting_note,
        'CorrelationId': correlation_id
    }

    queue_client.send_message(json.dumps(result_message).encode('utf-8'))

    logging.info(f"Sent message to queue with message {result_message}")

Please make sure, that if you send the input (in Azure portal) as

{
  "name": "Test",
  "CorrelationId": "42"
}

You get

{
  "Value": "It's not who you are underneath, Test, but what you do that defines you. Stay strong, stay vigilant… and happy to have you here",
  "CorrelationId": "42"
}

[github-actions]

Hi @kuljotSB. Thank you for opening this issue and giving us the opportunity to assist. To help our team better understand your issue and the details of your scenario please provide a response to the question asked above or the information requested above. This will help us more accurately address your issue.

[kuljotSB]

@nick863 I modified the code to make up for the missing "CorrelationId" payload but it doesn't solve the problem. Moreover I have made sure that I have the necessary RBAC permissions - for My user (as I am running this setup locally but not with Azurite - instead a storage account on cloud - which means the function app that I am running locally will take my user identity), AI Project and AI Hub.

I modified the code by adding logging statements to go deeper inside to see how all of this is behaving (Attaching the modified code below):

import azure.functions as func
import json
import logging
import os
import time
from azure.ai.projects import AIProjectClient
from azure.identity import DefaultAzureCredential
from azure.storage.queue import QueueClient, BinaryBase64EncodePolicy, BinaryBase64DecodePolicy

app = func.FunctionApp()

# Name of the queues to get and send the function call messages
input_queue_name = "inputqueue"
output_queue_name = "outputqueue"

# Function to initialize the agent client and the tools Azure Functions that the agent can use
def initialize_client():
    # Create a project client using the connection string from local.settings.json
    project_client = AIProjectClient.from_connection_string(
        credential=DefaultAzureCredential(),
        conn_str=os.environ["PROJECT_CONNECTION_STRING"]
    )

    # Get the connection string from local.settings.json
    storage_connection_string = os.environ["STORAGE_CONNECTION__queueServiceUri"]
    
    agent = project_client.agents.create_agent(
            model="gpt-4o-mini",
            name="GreetingByBatman",
            instructions="You are a helpful support agent. Answer the user's questions to the best of your ability.",
            headers={"x-ms-enable-preview": "true"},
            tools=[
                {
                    "type": "azure_function",
                    "azure_function": {
                        "function": {
                            "name": "GreetingByBatman",
                            "description": "Deliver a Greeting to the user in Batman's style.",
                            "parameters": {
                                "type": "object",
                                "properties": {
                                    "name": {"type": "string", "description": "the name of the user or the name of the person to which the greeting is to be delivered."},
                                },
                                "required": ["name"]
                            }
                        },
                        "input_binding": {
                            "type": "storage_queue",
                            "storage_queue": {
                                "queue_service_uri": storage_connection_string,
                                "queue_name": input_queue_name
                            }
                        },
                        "output_binding": {
                            "type": "storage_queue",
                            "storage_queue": {
                                "queue_service_uri": storage_connection_string,
                                "queue_name": output_queue_name
                            }
                        }
                    }
                }
            ],
        )
    logging.info(f"Created agent, agent ID: {agent.id}")

    # Create a thread
    thread = project_client.agents.create_thread()
    logging.info(f"Created thread, thread ID: {thread.id}")

    return project_client, thread, agent

@app.route(route="prompt", auth_level=func.AuthLevel.FUNCTION)
def prompt(req: func.HttpRequest) -> func.HttpResponse:
    logging.info('Python HTTP trigger function processed a request.')

    # Get the prompt from the request body
    req_body = req.get_json()
    prompt = req_body.get('Prompt')

    # Initialize the agent client
    project_client, thread, agent = initialize_client()

    # Send the prompt to the agent
    message = project_client.agents.create_message(
    thread_id=thread.id,
    role="user",
    content=prompt,
    )
    logging.info(f"Created message, message ID: {message.id}")

    # Run the agent
    run = project_client.agents.create_run(thread_id=thread.id, assistant_id=agent.id)
    # Monitor and process the run status
    while run.status in ["queued", "in_progress", "requires_action"]:
        time.sleep(1)
        run = project_client.agents.get_run(thread_id=thread.id, run_id=run.id)

        if run.status not in ["queued", "in_progress", "requires_action"]:
            break

    logging.info(f"Run finished with status: {run.status}")

    if run.status == "failed":
        logging.error(f"Run failed: {run.last_error}")
        logging.error(run)

    # Get messages from the assistant thread
    messages = project_client.agents.list_messages(thread_id=thread.id)
    # Displaying the assistant response
    logging.info(messages.data[0].content[0].text.value)
    last_msg = messages.data[0].content[0].text.value

    # Check if the agent wrote to the input queue
    try:
        queue_client = QueueClient(
            os.environ["STORAGE_CONNECTION__queueServiceUri"],
            queue_name=input_queue_name,
            credential=DefaultAzureCredential(),
            message_encode_policy=BinaryBase64EncodePolicy(),
            message_decode_policy=BinaryBase64DecodePolicy()
        )
        messages = queue_client.receive_messages()
        for message in messages:
            messagepayload = json.loads(message.content)
            logging.info(f"Received message payload from input queue: {messagepayload}")
            if 'CorrelationId' in messagepayload:
                logging.info(f"CorrelationId found in message payload: {messagepayload['CorrelationId']}")
            else:
                logging.error("CorrelationId not found in message payload")
    except Exception as e:
        logging.error(f"Failed to read from the input queue: {e}")

    # Delete the agent once done
    project_client.agents.delete_agent(agent.id)
    print("Deleted agent")

    return func.HttpResponse(last_msg)

@app.queue_trigger(arg_name="msg", queue_name="inputqueue",
                               connection="STORAGE_CONNECTION") 
def GreetingByBatman(msg: func.QueueMessage) -> None:
    logging.info('Python queue trigger function processed a queue item')

    try:
        # Queue to send message to
        queue_client = QueueClient(
            os.environ["STORAGE_CONNECTION__queueServiceUri"],
            queue_name="outputqueue",
            credential=DefaultAzureCredential(),
            message_encode_policy=BinaryBase64EncodePolicy(),
            message_decode_policy=BinaryBase64DecodePolicy()
        )

        messagepayload = json.loads(msg.get_body().decode('utf-8'))
        logging.info(f"Received message payload: {messagepayload}")
        name = messagepayload["name"]
        correlation_id = messagepayload['CorrelationId']

        greeting_note = f""" It's not who you are underneath, {name}, but what you do that defines you. Stay strong, stay vigilant… and happy to have you here """

        result_message = {
            'Value': greeting_note,
            'CorrelationId': correlation_id
        }

        queue_client.send_message(json.dumps(result_message).encode('utf-8'))
        logging.info(f"Sent message to queue with message {result_message}")

    except KeyError as e:
        logging.error(f"Missing key in message payload: {e}")
    except Exception as e:
        logging.error(f"Error processing queue message: {e}")

A point to be noted is that the following logging statement isn't visible anywhere in the terminal:
logging.info(f"Sent message to queue with message {result_message}")

This suggests that the queue triggered function doesn't have the input payload in the first place so it doesn't write anything to the output queue. This suggests that there is something wrong with the agent that isn't letting the agent submit the payload to the inputqueue

[nick863]

Do you see the first logging statement logging.info('Python queue trigger function processed a queue item')?
Do you see the failures of the function, running locally? The user needs to be Azure AI Developer, Storage Blob Data Contributor, Storage File Data Privileged Contributor, Storage Queue Data Contributor and Storage Table Data Contributor. Is the function being called but fails?

[github-actions]

Hi @kuljotSB. Thank you for opening this issue and giving us the opportunity to assist. To help our team better understand your issue and the details of your scenario please provide a response to the question asked above or the information requested above. This will help us more accurately address your issue.