In this post we are going to see how we can use OpenAI’s Function Calling feature with Microsoft Semantic Kernel.
Context
To explain the concepts in this post, let’s set the context. Let’s say that you are building an AI application that helps users answer basic questions about Maths (e.g. what is 20% of 100). For the sake of argument, let’s assume that our AI model is not capable of answering such questions.
So what we are doing to do is that we are going to write some functions which does mathematical operations like Add, Subtract, Multiply, and Division etc. In Semantic Kernel lingo, we are creating some native functions. In OpenAI lingo, we are creating some tools.
Now the challenge is how do we invoke these tools or functions based on the user prompts (which are in natural language, BTW). This is where OpenAI Function Calling comes in handy and we will see how we can use Semantic Kernel for that purpose.
What is OpenAI Function Calling?
Let’s first briefly talk about OpenAI Function Calling. This is a feature which was recently released by OpenAI.
The way I understand it is that you give a prompt in natural language and a list of functions (tools) to OpenAI. OpenAI then tries to find the best functions (tools) suited to fulfill user’s request and it then returns those function(s) back to the calling program. Not only it returns the function(s), it also tries to extract the arguments from the prompt that are needed to execute that function.
Considering our context, let’s say the user asks “what is 2 + 12” in our AI application. We pass this prompt and the list of functions (Add, Subtract, Multiply, and Division etc.) to OpenAI and it returns Add back to the calling application. If our Add function has 2 arguments say number1 and number2, OpenAI will also return back these arguments with value for number1 argument as 2 and the value for number2 argument as 12. Your application can then execute that function and return the result (14) to the user.
The way it works (I think) is that OpenAI extracts the intent of the prompt and then semantically compares it with the description of the functions.
Here’s the code for our Add function:
[KernelFunction, Description("Add two numbers")]
public static double Add(
[Description("The first number to add")] double number1,
[Description("The second number to add")] double number2
)
{
return number1 + number2;
}
Here, the description of the function is very important. You must be concise yet very clear about what a function does for this whole thing to work properly.
Now when OpenAI sees the prompt (what is 2 + 12), it somehow infers that the user wants to add 2 numbers and then compares that with the description of the functions and determines that Add function is the most suited function to answer user’s prompt. It also then maps “2” with “number1” argument and “12” with “number2” argument and returns this information.
You can read more about OpenAI Function Calling here: https://platform.openai.com/docs/guides/function-calling.
What is Semantic Kernel?
Simply put, Semantic Kernel is an open-source SDK that helps you build AI applications. You can use C#, Python or Java to write AI applications using Semantic Kernel.
I wrote a post on overview Microsoft Semantic Kernel a few months back when it was first released that you can read here: https://gauravmantri.com/2023/09/03/microsoft-semantic-kernel-an-overview/.
You can read more about Semantic Kernel here: https://learn.microsoft.com/en-us/semantic-kernel/overview/.
Semantic Kernel makes it super easy to make use of OpenAI Function Calling and that’s what we will see in this post.
Code
Let’s look at the code! I have built a simple console application.
For explanation, I will break the code in small chunks and then we will put the entire code together in the end.
Plugin
Let’s first write our plugin that will contain our functions (tools). Since our AI app deals with Maths, let’s call it MathPlugin. Here’s the code for that which I took from here:
using System.ComponentModel;
using Microsoft.SemanticKernel;
namespace FunctionCallingWithSemanticKernel.Plugins;
public class MathPlugin
{
[KernelFunction, Description("Take the square root of a number")]
public static double Sqrt(
[Description("The number to take a square root of")] double number1
)
{
return Math.Sqrt(number1);
}
[KernelFunction, Description("Add two numbers")]
public static double Add(
[Description("The first number to add")] double number1,
[Description("The second number to add")] double number2
)
{
return number1 + number2;
}
[KernelFunction, Description("Subtract two numbers")]
public static double Subtract(
[Description("The first number to subtract from")] double number1,
[Description("The second number to subtract away")] double number2
)
{
return number1 - number2;
}
[KernelFunction, Description("Multiply two numbers. When increasing by a percentage, don't forget to add 1 to the percentage.")]
public static double Multiply(
[Description("The first number to multiply")] double number1,
[Description("The second number to multiply")] double number2
)
{
return number1 * number2;
}
[KernelFunction, Description("Divide two numbers")]
public static double Divide(
[Description("The first number to divide from")] double number1,
[Description("The second number to divide by")] double number2
)
{
return number1 / number2;
}
[KernelFunction, Description("Raise a number to a power")]
public static double Power(
[Description("The number to raise")] double number1,
[Description("The power to raise the number to")] double number2
)
{
return Math.Pow(number1, number2);
}
[KernelFunction, Description("Take the log of a number")]
public static double Log(
[Description("The number to take the log of")] double number1,
[Description("The base of the log")] double number2
)
{
return Math.Log(number1, number2);
}
[KernelFunction, Description("Round a number to the target number of decimal places")]
public static double Round(
[Description("The number to round")] double number1,
[Description("The number of decimal places to round to")] double number2
)
{
return Math.Round(number1, (int)number2);
}
[KernelFunction, Description("Take the absolute value of a number")]
public static double Abs(
[Description("The number to take the absolute value of")] double number1
)
{
return Math.Abs(number1);
}
[KernelFunction, Description("Take the floor of a number")]
public static double Floor(
[Description("The number to take the floor of")] double number1
)
{
return Math.Floor(number1);
}
[KernelFunction, Description("Take the ceiling of a number")]
public static double Ceiling(
[Description("The number to take the ceiling of")] double number1
)
{
return Math.Ceiling(number1);
}
[KernelFunction, Description("Take the sine of a number")]
public static double Sin(
[Description("The number to take the sine of")] double number1
)
{
return Math.Sin(number1);
}
[KernelFunction, Description("Take the cosine of a number")]
public static double Cos(
[Description("The number to take the cosine of")] double number1
)
{
return Math.Cos(number1);
}
[KernelFunction, Description("Take the tangent of a number")]
public static double Tan(
[Description("The number to take the tangent of")] double number1
)
{
return Math.Tan(number1);
}
[KernelFunction, Description("Take the arcsine of a number")]
public static double Asin(
[Description("The number to take the arcsine of")] double number1
)
{
return Math.Asin(number1);
}
[KernelFunction, Description("Take the arccosine of a number")]
public static double Acos(
[Description("The number to take the arccosine of")] double number1
)
{
return Math.Acos(number1);
}
[KernelFunction, Description("Take the arctangent of a number")]
public static double Atan(
[Description("The number to take the arctangent of")] double number1
)
{
return Math.Atan(number1);
}
}
OpenAI Client
Next, we will create an instance of OpenAI client. You will need the endpoint of your Azure OpenAI service and the key which you can get from Azure Portal (you can also use Azure AD credentials instead of key if you want).
const string AZURE_OPEN_AI_ENDPOINT = "https://xyz.openai.azure.com/"; const string AZURE_OPEN_AI_KEY = "00000000000000000000000"; const string AZURE_OPEN_AI_MODEL_ID = "gpt-4-32k"; // create an instance of OpenAIClient. var openAIClient = new OpenAIClient(new Uri(AZURE_OPEN_AI_ENDPOINT), new Azure.AzureKeyCredential(AZURE_OPEN_AI_KEY));
Get Kernel
Kernel GetKernel()
{
var kernelBuilder = Kernel.CreateBuilder()
.AddAzureOpenAIChatCompletion(AZURE_OPEN_AI_MODEL_ID, openAIClient);
var kernel = kernelBuilder.Build();
kernel.Plugins.AddFromType<MathPlugin>();
return kernel;
}
What we are doing above is creating an instance of Semantic Kernel and then adding our plugin (with all the functions/tools) to that kernel.
OpenAI Prompt Execution Settings
Next, we will set the OpenAI prompt execution settings.
var promptExecutionSettings = new OpenAIPromptExecutionSettings()
{
ToolCallBehavior = ToolCallBehavior.EnableKernelFunctions,
Temperature = 0
};
There are two things I want to highlight in the code above:
- Temperature: Temperature parameter sets the creativity of the model. Since we want our model to be constrained and give predictive results, we would want this setting to be 0. Try removing this parameter and you will get some really funny results back!
- ToolCallBehavior: This parameter controls how tools will be called. By setting the value to
ToolCallBehavior.EnableKernelFunctions, we want Semantic Kernel to just pass the registered tools to OpenAI and return the matching tool. If we want Semantic Kernel to automatically invoke the matching tool returned by OpenAI, we would want to change this setting toToolCallBehavior.AutoInvokeKernelFunctions. There are other settings as well for this parameter that you can read here.
Get Tool
We are going to focus on invoking the function manually, so the code below gets the appropriate tool for the input prompt by using OpenAI Function Calling:
// select the tool best suited to execute our prompt.
async Task<OpenAIFunctionToolCall?> SelectTool(string prompt)
{
try
{
var chatCompletionService = new AzureOpenAIChatCompletionService(AZURE_OPEN_AI_MODEL_ID, openAIClient!);
var result = await chatCompletionService.GetChatMessageContentAsync(new ChatHistory(prompt),
promptExecutionSettings, kernel);
var functionCall = ((OpenAIChatMessageContent)result).GetOpenAIFunctionToolCalls().FirstOrDefault();
return functionCall;
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
Console.WriteLine(ex.StackTrace);
return null;
}
}
Here, Semantic Kernel takes our prompt and the functions loaded in the kernel and send them to Open AI. It then parses the response from Open AI and returns the first tool. Simple, isn’t it?
However, please keep in mind that Open AI may not find a matching tool. This could happen when a user asks an irrelevant question like “What is the capital of France?”.
Execute Function
Last step would be to execute the function and get the result!
var function = await SelectTool(prompt);
if (function != null)
{
// now we try to get the plugin function and the arguments.
kernel.Plugins.TryGetFunctionAndArguments(function, out KernelFunction? pluginFunction,
out KernelArguments? arguments);
Console.WriteLine($"Plugin function: {pluginFunction!.Name}");
if (arguments!.Any())
{
Console.WriteLine("Function arguments:");
foreach (var argument in arguments!)
{
Console.WriteLine($"Argument name: {argument.Key}; Argument value: {argument.Value}");
}
}
// execute the plugin function.
var result = await kernel.InvokeAsync(pluginFunction!, arguments);
Console.WriteLine($"{prompt}: {result.ToString()}");
}
else
{
Console.WriteLine("I'm sorry but I am not able to answer your question. I can only answer simple mathematical questions.");
}
Here, first we are getting the kernel function to invoke along with the arguments (kernel.Plugins.TryGetFunctionAndArguments(function, out KernelFunction? pluginFunction, out KernelArguments? arguments)) and then executing the function to get the result (var result = await kernel.InvokeAsync(pluginFunction!, arguments)).
That’s it!
Complete Code
Here’s the complete code:
using Azure.AI.OpenAI;
using Microsoft.SemanticKernel;
using Microsoft.SemanticKernel.Connectors.OpenAI;
using Microsoft.SemanticKernel.ChatCompletion;
using FunctionCallingWithSemanticKernel.Plugins;
// Azure OpenAI settings. You can get these settings from portal.
const string AZURE_OPEN_AI_ENDPOINT = "<your-azure-openai-endpoint like https://xyz.openai.azure.com/>";
const string AZURE_OPEN_AI_KEY = "<your-azure-openai-key like 44444444444444444444444444444444>";
const string AZURE_OPEN_AI_DEPLOYMENT_ID = "<your-azure-openai-deployment-id like gpt-4-32k>";
// create an instance of OpenAIClient.
var openAIClient = new OpenAIClient(new Uri(AZURE_OPEN_AI_ENDPOINT), new Azure.AzureKeyCredential(AZURE_OPEN_AI_KEY));
// get the kernel.
var kernel = GetKernel();
// set OpenAI prompt execution settings.
var promptExecutionSettings = new OpenAIPromptExecutionSettings()
{
ToolCallBehavior = ToolCallBehavior.AutoInvokeKernelFunctions,
Temperature = 0
};
Console.WriteLine("Hello, I am an AI assistant that can answer simple math questions.");
Console.WriteLine("Please ask me questions like \"What is 2 x 2\" or \"What is sqaure root of 3\" etc.");
Console.WriteLine("To quit, simply type quit.");
Console.WriteLine("");
Console.WriteLine("Now ask me a math question, I am waiting!");
do
{
var prompt = Console.ReadLine();
if (!string.IsNullOrWhiteSpace(prompt))
{
if (prompt.ToLowerInvariant() == "quit")
{
Console.WriteLine("Thank you! See you next time.");
break;
}
else
{
// get the tool/function best suited to execute the function.
var function = await SelectTool(prompt);
if (function != null)
{
// now we try to get the plugin function and the arguments.
kernel.Plugins.TryGetFunctionAndArguments(function, out KernelFunction? pluginFunction,
out KernelArguments? arguments);
// execute the plugin function.
var result = await kernel.InvokeAsync(pluginFunction!, arguments);
Console.WriteLine($"{prompt}: {result.ToString()}");
}
else
{
Console.WriteLine("I'm sorry but I am not able to answer your question. I can only answer simple math questions.");
}
}
}
} while (true);
// select the tool best suited to execute our prompt.
async Task<OpenAIFunctionToolCall?> SelectTool(string prompt)
{
try
{
var chatCompletionService = new AzureOpenAIChatCompletionService(AZURE_OPEN_AI_DEPLOYMENT_ID, openAIClient!);
var result = await chatCompletionService.GetChatMessageContentAsync(new ChatHistory(prompt),
new OpenAIPromptExecutionSettings()
{
ToolCallBehavior = ToolCallBehavior.EnableKernelFunctions,
Temperature = 0
}, kernel);
var functionCall = ((OpenAIChatMessageContent)result).GetOpenAIFunctionToolCalls().FirstOrDefault();
return functionCall;
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
Console.WriteLine(ex.StackTrace);
return null;
}
}
// create an instance of Kernel and load all plugins and functions in the Kernel.
Kernel GetKernel()
{
var kernelBuilder = Kernel.CreateBuilder()
.AddAzureOpenAIChatCompletion(AZURE_OPEN_AI_DEPLOYMENT_ID, openAIClient);
var kernel = kernelBuilder.Build();
kernel.Plugins.AddFromType<MathPlugin>();
return kernel;
}
You can download the complete solution from GitHub: https://github.com/gmantri/function-calling-with-semantic-kernel.
Conclusion
That’s it for this post. I hope you will find it useful. If you have any feedback or questions, please feel to provide them.
Happy coding and Happy New Year 2024!
