Recently we integrated Azure QnA Maker Service with SharePoint Search page for one of our DoD Customers so thought to share my experience and lesson learned with broader audiences. Generally, QnA Maker is used when designing a Chatbot, but, according to customer requirements, we integrated with an existing search page, so users do not need to go to multiple places for information. A single place to search all the contents, including static Questions and Answers (FAQs).

Introduction

Azure QnA Maker is one of the Azure Cognitive Services. It is a PaaS service, so no need to maintain any infrastructure. It is available within Azure Gov at DoD Impact Level (IL) 2 and can be configured to store the response data at higher DoD impact levels.

QnA Maker consists of two main parts, QnA Maker Portal site, which is a very user-friendly web interface to import questions and answers, and QnA Maker backend Azure service resources.

Use Case: If your organization has static questions and answers sets in word documents, PDFs, on web pages, etc., you can use the Azure QnA Maker portal to quickly import these questions and answers into QnA Maker to create knowledge bases. It can also automatically extract questions and answers sets from semi-structured contents, manuals, guidelines, etc. Once Questions and Answers are in QnA Maker, your application (Examples: Custom Bot or Custom Search Page) can call QnA Maker Rest API to answer the user’s question.

Asking questions through custom chatbot

High-Level Architecture

It is necessary to understand the QnA Maker architecture if you are working with Government clients and planning to upload answers in QnA Maker, then consider hardening the backend resources of Azure QnA Maker. Apply Network settings on App Service and Search Service. Check out this reference from Microsoft covering backend resources.

High-Level Architecture

Decision Making Between LUIS and QnA Maker Service

It is not hard to decide whether we should use LUIS or QnA Maker to design the Chatbot or any custom applications?

When to use the QnA Maker?
If your organization has lots of static questions and answers ( Example: Questions/Answers Sample), take advantage of out of the box features of QnA Maker. Upload your static questions and answers into QnA Maker Portal, and your application can call QnA API to search for questions asked by the user on the front end application and return the response.

When to use the LUIS?
If you would like to design the application which needs to extract the information from the user’s questions and further process their intents, then use the LUIS.

Example Application: I want to design Bot, where users can ask Bot to create various application development projects. Below are 2 example requests:

Hi, can you create .net core application in C:/Code folder?
Hi , can you create the node.js application in C:/Projects/Code folder?

In this case, LUIS API can extract the word “.net core,” “node.js,” “C:/Code”, and “C:/Projects/Code “ from the user requests, and our application can further process extracted data and can create the respective projects. See the below screenshot from LUIS output. Base on intent = CreateApplicationProject and AppType = node.js, the application gets enough data for the next steps.
LUIS Output Screen Shot

Lessons Learned

Below are the few lessons learned and essential points about the QnA Maker service.

  • You can easily use Postman to test various rest APIs of QnA.
  • Azure QnA Maker Permission can’t be set from Azure QnA Maker Portal. If you would like to share a knowledge base with multiple users, you must give permission on back-end resources using Azure role-based access control (Azure RBAC). You can create a group ( Example: QnAAdministrators) in Azure AD and grant access to the resource group (Resource group with all QnA Maker resources) by assigning a contributor role to a QnAAdministrator at this scope.
  • You can rename your knowledge base from QnA Portal anytime; it does not change the Knowledgebase Key or Endpoint Key.
  • Take advantage of the Metadata Tag feature, so you do not need to design the separate knowledge bases for all departments.
  • You can add a tag with each question.
  • QnA Maker works best if you import both questions and answers into the knowledgebase. If you would like to search only on questions, use the RankerType=QuestionOnly in the POST body of the GenerateAnswer request.
  • To turn on the Application Insight‘s telemetry (Logging), if you haven’t turned it on during QnA Maker Service creation time, you can turn it on afterward on back-end App Service Resource.
  • Application Insight telemetry is very useful for tracking all the unanswered questions and does not need to write custom code for it. Run the below query to view all unanswered questions.
requests
| where url endswith "generateAnswer"
| project timestamp, id, url
| parse kind = regex url with *"(?i)knowledgebases/"KbId"/generateAnswer"
| join kind= inner (
traces | extend id = operation_ParentId
) on id
| extend question = tostring(customDimensions['Question'])
| extend answer = tostring(customDimensions['Answer'])
| extend score = tostring(customDimensions['Score'])
| where  score  == "0" and message == "QnAMaker GenerateAnswer"
| project timestamp, KbId, question, answer, score
| order  by timestamp  desc
  • If you are using Train API Post call to send alternative phrases, it takes 30 to 40 mins to show up on QnA Portal. Once alternative phrases show up on Portal, you need to train and publish the knowledge base again. For Train API to work, make sure you turn on the Active Learning from QnA Portal Settings.
  • QnA Maker has a batch testing tool which is an exe file. Still, you cannot run it from your production environment desktops where an exe file upload might be blocked. In this case, we can easily write custom front end node.js code to read the various questions from the CSV file or Excel file, and for each question, call the GenerateAnswer API to get the answer and score and write it back to the output excel file.

Conclusion

A QnA maker’s beauty is the features of importing your questions and answers sets into QnA Maker from various files like Word, PDF, Text, Excel, etc. You can also bring problems and solutions from published web pages. Once your knowledgebases are in QnA Maker, we can quickly design questions/answers Bot applications without adding many custom codes. QnA Maker is smart enough to search your full question text or phrases from your questions and answers. It can also understand the spelling mistakes and returns the correct answer.

QnA Maker is not only about the QnA Maker Portal site. It does have back-end resources created in your Azure Subscription. If you plan to use QnA maker for larger knowledge bases, consider increasing the SKU of QnA Maker back-end resources and considering hardening the security of back-end resources.

If you plan to design the application that needs to extract the information from the user’s question and further process it, consider using the LUIS.