In my previous post about Azure Sidekick, I gave a general introduction about the tool and its capabilities. If you have not read that post so far, I would strongly encourage you to read that first. You can read that post here.
In this post, I will talk about the general flow of the application, prompt patterns used there and some other things.
So, let’s start!
Application Flow
From the outside, it looks like a very simple application: you ask it a question (about Azure or Storage accounts in your subscription) and then it answers it. However, a lot is happening behind the scenes and the application goes through multiple steps to answer your question.
1. Rephrase/Reword Step: When you ask a question, first thing the application does is that it tries to rephrase/reword the question by sending your question to LLM. LLM takes in the grounding rules, context and chat history (if available) into consideration and replies back with a rephrased question.
2. Intent Recognition Step: Next, the rephrased question is sent to the LLM for recognizing the intent of the question. It is a very important step because the routing happens based on the result of this and it also kind of serve as “input” guardrails.
3. Answering Step: This is the final step where based on the intent, appropriate plugins and tools are loaded and LLM provides and answer to the question.
However, depending on the intent there could be multiple steps involved in this step.
For example, if the question is about storage accounts, then the application will fetch the list of storage accounts from the selected subscription and then send that data to the LLM to generate an answer.
Similarly, if the question is about a specific storage account, then the application will first try to extract the storage account name from the question, fetch its properties and then send that data to the LLM to generate an answer.
I must confess that the code currently does not utilize the full potential of the LLMs especially its reasoning and planning capabilities. This is something I have started looking into recently and hopefully in next few releases, I will incorporate those.
Prompt Patterns
Now that you know how the application works, let’s talk about some of the prompt patterns I have used to build this application. I wrote a long post about prompt patterns sometime back which you can read here.
Here are the prompt patterns I have used so far:
Rephrase/Reword
What it does?
As the name suggests, it rephrases or rewords a user’s question so that it becomes easier for an LLM to understand and process. It makes use of grounding rules, context and chat history (if available) to rephrase a question.
Why it is needed?
During the course of development, I found many reasons as to why it is needed:
- Incomplete question: A user may have asked an incomplete question. For example, the user may just write “Lighthouse” (when they wanted to ask about Azure Lighthouse). Rephrase prompt will produce an output like “Please tell me about Azure Lighthouse and its features”.
- Lacking context: When we converse amongst ourselves, we can somehow understand the context of the question based on the previous questions however LLMs lack that capability. Chat history helps but relying on chat history alone is just not sufficient. Using this prompt pattern helps. For example, let’s say the user has first asked a question like “How many storage accounts are there in my subscription?” and then asks “Can you name them?”. Rephrase prompt will produce an output like “Please give me the names of the storage accounts that are there in my subscription?”.
Intent Recognition
What it does?
This prompt pattern is used to recognize the intent of the question i.e. to find out what the question is about. Is it about Azure, Azure Storage or something else, that’s what is determined using this prompt pattern.
Why it is needed?
There are many use cases for using this pattern. In context of this application, this pattern is used for the following things:
- Routing: A user may be asking a general question about Azure, or a question concerning their storage accounts, or a question related to a specific storage account. The user’s question may be ambiguous or may not even be related to Azure. Using this prompt pattern will help the application identify the intent of the question and route it accordingly.
- Input Guardrails: This pattern also help us enforce input guardrails. For example, let’s say you ask a general question like “What is the capital of India?”. Even though the LLM knows the answer to the question, because of this pattern, user will be informed that the purpose of this tool is to answer questions about Azure only. This becomes really important if you are building an AI application for your enterprise. You would want to constrain the application only to answer the questions related to your enterprise. If you need a proof of how bad it could be, please see this news about UK based parcel delivery company – DPD: https://www.bbc.com/news/technology-68025677.
- Relevant chat history: Another use case we found very useful in our application is that we filter the chat history based on the intent of the question and pass that to the prompt.
Named Entity Recognition/Extraction
What it does?
This pattern is used to recognize and extract the named entities in a question. For example, if the user asks a question like “Can I host a static website in xyz storage account?”, this pattern will recognize that the question is about “xyz” storage account and extract “xyz” it from there.
Why it is needed?
- Native code execution: In context of this application, we need this information to execute native code (get the properties of a storage account).
Grounding Rules
Next, I want to talk about setting the grounding rules. Grounding rules are very important in any AI application, as they set the overall constraints for your application. You can use these grounding rules to:
- Give your application a persona;
- Set the boundaries;
- Enforce legal restrictions;
- Overall tone of the responses etc.
For example, here are the grounding rules we use in the application:
- You are Azure Sidekick, an AI assistant specializing in Azure, tasked with providing accurate and knowledgeable responses to user inquiries about Azure. - Maintain honesty. If uncertain of an answer, respond with, "I apologize, but I currently lack sufficient information to accurately answer your question. - Uphold user privacy. Do not ask for, store, or share personal data without explicit permission. - Promote inclusivity and respect. Do not engage in or tolerate hate speech, discrimination, or bigotry of any form. Treat all users equally, irrespective of race, ethnicity, religion, gender, age, nationality, or disability. - Respect copyright laws and intellectual property rights. Do not share, reproduce, or distribute copyrighted material without the appropriate authorization. - Provide precise and concise responses. Maintain a respectful and professional tone in all interactions. - Wait for the user's question before providing information. Stay within your domain of expertise - Azure and related services. - Ensure responses are up-to-date and accessible. Avoid unnecessary jargon and technical language when possible.
Chat History
Now, let’s talk about chat history. Simply put, chat history is the history of the conversation between the user and the AI application. Usually, it contains the question asked by the user and the answer given by the application.
Chat history is an important part of any LLM application as it helps setting a proper context for the question being asked and thus it increases the chances of getting better answer to the question.
There are a few things to keep in mind when using chat history in the prompts:
- Use context-sensitive chat history: What I mean by that is use only that portion of chat history which is relevant to the question being asked. For example, if the user is asking a question about storage accounts, only include chat history related to storage in the prompt.
- Avoid unnecessary chat history: Even with context-sensitive chat history, you may end up with many items in the chat history relevant to that context. However, that does not mean you pass everything to your prompt. Just pick last 3, 5 or 10 context-sensitive chat history items to the prompt.
Following these will ensure that you are not unnecessarily bloating your prompts and thus save on input prompt tokens :), without compromising on the functionality.
Goal Reminder
This is an interesting thing I have found. I noticed that when I reminded LLM about their goal, I got better results. The best way to do so is include a summarized goal at the end of the prompt.
For example, this is how I am ending the rephrase prompt:
Considering the information provided to you, please rephrase or reword the current question to increase its clarity and specificity for a language model. Consider identifying the key elements or concepts within the question, ensuring the language is precise, and avoiding any ambiguity or overly complex language. Remember to incorporate the context provided by the previous questions and answers. Your goal is to create a revised question that maintains the original intent, but is more easily understood by an LLM when considering the conversation history.
Summary
That’s it for this post. It turned out to be somewhat lengthy but I couldn’t help it as I had so much to share :). I hope you have enjoyed this post.
In the next (and concluding post about this), I will talk about the lessons learned while building this. Stay tuned for that.
Until then, be well!
