A-cx Llc

Custom LLM Using Retrieval Augmented Generation

We used retrieval augmented generation to implement a custom LLM endpoint for answering internal questions, improving documentation lookup speed, and creating company-specific documentation and marketing posts. Our client Undisclosed The challenge Our client wanted to encourage their employees to explore AI. Staff wanted to use ChatGPT to create marketing materials, answer frequently asked questions, and provide investor summaries. They were especially interested in creating more targeted documents, using ChatGPT to quickly change the tone, target audience, or language of various articles, ultimately allowing them to produce a wider variety of marketing content and respond to customer inquiries faster. How we helped There were three primary goals for our solution: Allow staff to use an LLM to create documents without requiring significant work gathering the information needed. Ensure that documents and answers generated by the LLM do not include incorrect data. Address any security concerns around data leakage. Retrieval Augmented Generation RAG is an approach to adding custom data to an LLM where instead of retraining the LLM on your custom data, you provide the data required to answer the query alongside the query itself. For example, if a user asks the LLM, “What does this widget do ” you might augment the query with information like “Widgets can be used for …” and then have the LLM answer the query using this provided information. Analyzing User Queries As a first step, we gathered a list of queries users commonly made. Our goal was to understand what types of data users typically requested and what sorts of questions they often asked. Determining Architecture For The Custom LLM We determined that we could use Microsoft Azure to host a simple architecture composed of an Azure Static Web App, an Azure Function, and an Azure Cognitive Search database. Our static web app would provide an endpoint for users to enter queries, providing a site similar to the ChatGPT interface users were already familiar with. Our Azure Function would handle queries sent from the web app, and use our LLM to classify the queries to determine what data and query augmentation was needed. Our Azure Function would then query an Azure Cognitive Search database containing the client’s documentation to find the relevant documentation needed to answer the user query. The Azure Function would then send the augmented query and documentation to the LLM. Finally, the Azure Function would return the query result to the web app, where it would be displayed. Indexing The Data Our first step was to gather and index the client’s data into the Azure Cognitive Search resource. We performed this step manually, scraping information from their intranet and then uploading it to the cognitive search resource. We enabled semantic ranking, and selected indexes based on what information was most likely to be looked up in queries. Creating the Custom LLM Endpoint Next, we created our Azure Function. We used Python to implement our function since it has libraries for querying both Azure Cognitive Search and OpenAI resources. It also had the nice advantage of allowing us to reuse code from our previous testing with our Machine Learning Studio notebooks. To store our code and deploy our function, we used Azure DevOps, a git repository that smoothly integrates with Azure resources. This greatly simplified the process of creating development and production functions for testing, as well as ensuring that we had a continuous development environment to keep things up to date. Adding A Web Interface Finally, we created a web interface using React, having our design team work directly with the client to ensure consistency with the rest of the client’s in-house tools. We once again used Azure DevOps to store and deploy the web app. Its direct integration with Azure means we could quickly deploy updates and manage the web application via code. Results Our custom LLM solution was able to safely and securely help users answer a variety of internal questions, making it easier for our client to create summaries of important documents and answer customer queries. We were also able to reduce the workload for their sales and marketing teams significantly. Languages Python Javascript Frameworks OpenAI Azure Document Search React Tools Visual Studio Code Azure ML Studio Cloud Azure