AI Analytics
LLM Bot Analytics is an analytical feature obtained from Meta Base to help users understand the performance of LLM (Large Language Model)-based bots. This analytics includes various key metrics that provide insights into bot usage, effectiveness, costs, and intent mapping.
Types of AI Analytics
Date Filter
Right at the top, you’ll find a section labeled Analytics along with a date filter (in this case, set from 2025/07/13 to 2025/08/12). This lets you narrow down your analytics view to a specific date range so you can track changes or spikes over time.
Cost Usage
Cost Usage displays the total cost of bot usage over a specific period. This cost is calculated based on the number of requests to the LLM model, the type of model used, and the interaction duration. By reviewing usage costs, businesses can manage AI budgets more efficiently and identify bot usage patterns that impact costs.
Information provided:
- Cost usage graph by date.
- Bot usage trends to optimize costs.
- Identification of cost spikes due to increased bot activity.
Current Usage
Current Usage shows the total costs incurred for bot usage up to the present time. With this monitoring, businesses can estimate future expenses and ensure efficient AI usage.
Knowledge Base Usage
Knowledge Base Usage indicates how often the knowledge base is used by the bot to answer questions. This data allows businesses to evaluate the effectiveness of the knowledge base and ensure its content remains relevant.
Information provided:
- Creation and update times of the knowledge base.
- Name of the knowledge base used.
- Model used in the knowledge base.
- Latency (response time) in answering questions.
- Estimated costs incurred for a specific knowledge base.
Bot Usage
Bot Usage provides an overview of how the bot is used in customer interactions. By understanding these usage patterns, businesses can optimize chatbots to be more responsive and efficient.
Information provided:
- Creation and update times of interactions.
- Name of the knowledge base used.
- AI model used in the chatbot.
- Average response time (latency) for each request.
- Costs incurred based on bot usage.
Total Room
Total Rooms shows the number of conversation sessions that have taken place in the chatbot. This metric helps businesses understand user engagement levels and measure how effectively the bot handles questions before being escalated to a human agent.
Information provided:
- Rooms: Total number of sessions (rooms) created by users.
- Rooms with handover: Number of sessions escalated to an agent.
- Rooms without handover: Number of sessions not escalated to an agent.
For example:
- 64 total rooms means there were 64 separate user conversations that is solved without handover
- 0 rooms with handover means none of those chats were escalated or transferred to a human agent
- 3 rooms without handover means chat sessions that were completely managed by the AI Chatbot, without needing to be escalated or transferred to a human agent.
Average Confidence
The confidence score is a key metric used to measure the system’s assurance in the relevance of a response based on the knowledge base. The confidence score ranges from 0 to 1, with the following interpretation:
- Low Score (Close to 0): Indicates that the system did not find a suitable answer in the knowledge base or that the given answer has a low confidence level. In this situation, the response may be inaccurate or less relevant.
- High Score (Close to 1): Indicates that an answer was found in the knowledge base and is highly relevant to the given question. The closer to 1, the more confident the system is that the answer is correct and appropriate.
This confidence score can be used to determine whether an answer needs review or to identify further optimization steps.
Intent Classification Match Log
Intent Classification Match Log displays intent matching for every question asked to the bot. With this information, businesses can improve accuracy in understanding user intent.
Information provided:
- Intent matching log based on user queries.
- Intent count information.
Intent Classification Tag
Intent Classification Tag displays the most frequently appearing intent tags in conversations between users and the bot. By understanding these intent patterns, businesses can refine bot responses to be more relevant.
Information provided:
- List of recognized intents in interactions.
- Intent tag count information.
Question Answers List with Confidence Score
Confidence Score is a metric that indicates the bot's confidence level in providing answers to user questions. The higher the confidence level, the greater the bot's assurance that the given answer is relevant and accurate.
Information provided:
- Prompt Name: The question asked by the customer.
- Confidence Score: The system’s confidence score in the provided answer.
- Answer: The response given by the bot.
- Time: The time the interaction occurred.
Key Data Points for Analytics
- Total Number of Interactions: The total number of questions handled by the LLM.
- Confidence Score: Breakdown of confidence scores for each response. The proportion of responses with a confidence score above a certain threshold indicates highly relevant answers.
Success Metrics
The success of this feature can be measured using several parameters, including:
- Improved Answer Accuracy: Reduction in the number of responses with low confidence scores over time through knowledge base optimization.
- Average Confidence Score Value: Indicates the bot’s average confidence level in providing answers. The higher the value, the more confident the bot is that its answers are correct.
Export as PDF
You can also export the analytic by clicking the Export as PDF button in the left up side of page. It will be saved in your local device.
