Best way to learn a new technology is by building something (regardless of how big or small it is) with it. This was my primary intent behind building Azure Sidekick.
I had so much fun building this and learnt a lot of things along the way. Not only I learnt more about Microsoft Semantic Kernel and Prompt Engineering, I also picked up a thing or two in .Net as well (AddKeyedSingleton and IAsyncEnumerable<T> to name a few).
In my previous posts about this, I talked about the features and how I built it. You can read those posts here and here.
In this post I will talk about the lessons learned.
So let’s begin!
Lessons Learned
Here are some of the lessons I learned while developing this tool.
Writing prompts is hard
Writing great prompts is really-really hard. An LLM is a black box and we do not know how it is going to respond to our queries. Only solution to this problem is to write really good prompts. That requires one to have subject matter expertise as well as the ability to clearly communicate the ask, which unfortunately not many of us (including yours truly) possess.
Luckily, we can make use of LLMs to help us write great prompts. I wrote about how I used it building this tool in one of my recent posts that you can read here.
You really need to work with the LLM to get the right prompt. I spent countless hours just to get a single prompt working.
For example, when I was working on intent recognition prompt, just to get “Storage” intent returned correctly, I started by asking the question like “Given a question about Azure, how would you know that the question is about Azure Storage?”. LLM gave me some answer and then I asked follow up questions about its answer like “what about this” or “what about that” etc. until it gave me something I felt really comfortable with.
This is what it finally looks like:
<intent>
<name>Storage</name>
<description>
Return this intent if the question:
- Keywords: Has keywords such as 'Azure Storage', 'Blob Storage', 'Queue Storage', 'File Storage', 'Table Storage', 'Disk Storage', 'Data Lake Storage', 'Backup', or 'Azure Storage Explorer'.
- Context: Has context regarding Azure's storage capabilities, types of storage, data management, or other aspects specifically related to Azure Storage.
- Features: Mention of specific features such as security, scalability, redundancy, data replication, or pricing within the Azure Storage ecosystem.
- Operation or Commands: Asking for details about operations or commands in Azure Storage, like creating a storage account, uploading or downloading data, managing data, etc.
- Troubleshooting: Looking for troubleshooting tips, specifically seeking solutions for issues or errors encountered while using Azure Storage.
- Comparisons: Seeking comparison between Azure Storage and other cloud storage services like AWS S3, Google Cloud Storage, etc. and also comparison between various Azure Storage services.
- Rules: Looking for rules, restrictions and naming conventions applicable to storage accounts, blob containers & blobs, queues & messages, tables & entities and file shares, directories & files.
- Storage resources: Finding information about storage accounts, blob containers & blobs, queues & messages, tables & entities and file shares, directories & files in an Azure subscription.
- "How to" questions related to Azure Storage.
</description>
</intent>
Stop thinking like a developer
Us developers are very rational in our thinking. Somehow (through our experience or otherwise) it is hardwired in our brain that if we write code to add two numbers (and wrote it correctly), it will always give us the correct result. This is probably because we know how the programming languages work we kind of implicitly trust them.
Unfortunately, it is not the case when you are building an AI application. We do not anything about underlying LLM and how it would process our question and that creates a problem for us developers.
To solve this dilemma, stop thinking like a developer.
Do not make any assumptions like if I can write code that adds two numbers and give me proper result, LLM can also do the same.
Provide as much details as possible in the prompt even for the most trivial tasks. Do not leave anything behind.
When you are writing prompts, put on your “Business Analyst” hat on and write down each and everything. Think of prompts as writing requirements document for another developer who is going to write an application for you.
Streaming responses are always better
After I built the very first version, I made a small video about the application to share it with some folks. At that time, the application waited to receive complete response from the LLM before sending it to the user. As I was recording the video, I immediately realized that it is a very bad user experience (UX).
My first priority after that was to return the responses in a streaming way. It was much-much better UX as the user is immediately seeing the response from the LLM.
If you are building a chatbot kind of an application, please opt for streamed responses wherever possible. Find a hack if you have to, in order to achieve this. You will make your users very happy!
Don’t make it a chore
It’s more of an advise rather than my learning, but have loads of fun building what you are building (I know, I did). Don’t make it a chore!
Remember, you are building something because you do not have to but because you want to! There is a huge difference between “have to” and “want to”.
Because you are learning along the way, don’t be afraid of making mistakes. But keep on correcting your mistakes and keep on improving as you learn. For example, when I first implemented streaming response, I was not aware of “IAsyncEnumerable” and implemented it using events. When I came to know about it, I changed the code to make use of it. The code was much cleaner that way.
Summary
That’s it for this post and this series about Azure Sidekick. By no means I am done with this project (will be implementing planners shortly). As I make significant changes or learn new things, I will share them here.
In the next post, I am going to talk about a nifty little trick with Semantic Kernel I learned recently that could potentially lower your Azure OpenAI bill.
Keep learning and keep building!
