Testing / QA is a crucial part of software development. If testing/qa is not done properly then it leaves possibilities of bugs and errors in the system. For testing desktop, web or mobile based applications, there are various tools available. Also, we can develop our own testing tools using open source frameworks like Selenium.
Now as we are opening up a new frontier to reach out to customers in form of Chatbots, we are going to need testing tools for chatbots too. Normal desktop, web, mobile app testing tools might not work effectively for chatbots testing. We have to find some new ways. To bridge this gap, people at Botium has developed a framework – Botium which is claimed to be selenium for chatbots.
Botium claims to support multiple chatbots development platforms like Google Dialogflow, Microsoft LUIS, Microsoft Bot Framework, IBM Watson Assistant, Alexa Voice Service and many other platforms. As it looks promising we gave it a try and here is the tutorial of how to setup Botium and generate test cases for Dialogflow agent.
Installation & Configuration
1. Install Node.js first if your machine does not have. Botium cli would be installed as Node.js module.
2. Now run below command in terminal window:
npm install –g botium-cli
It will install botium cli in machine.
3. After “botim-cli” is installed successfully the command will be available to run. If you have done global installation then you can run “botium-cli” from any directory. If you have installed it in local directory then you will need to browse to that directory and run “npx botium-cli”
4. Now we need to configure connection between dialogflow agent and botium cli. The connection to the Dialogflow Agent is configured in a JSON-file. Create a file called “botium.json” with data in JSON as below:
{
<span class="hljs-attr"><span class="hljs-attr">"botium"</span></span>: {
u
m
<span class="hljs-attr"><span class="hljs-attr">"Capabilities"</span></span>: {
<span class="hljs-attr"><span class="hljs-attr">"PROJECTNAME"</span></span>: <span class="hljs-string"><span class="hljs-string">"Dialogflow Sample"</span></span>,
<span class="hljs-attr"><span class="hljs-attr">"CONTAINERMODE"</span></span>: <span class="hljs-string"><span class="hljs-string">"dialogflow"</span></span>,
<span class="hljs-attr"><span class="hljs-attr">"DIALOGFLOW_PROJECT_ID"</span></span>: <span class="hljs-string"><span class="hljs-string">"YOUR_PROJECT_ID"</span></span>,
<span class="hljs-attr"><span class="hljs-attr">"DIALOGFLOW_CLIENT_EMAIL"</span></span>: <span class="hljs-string"><span class="hljs-string">"YOUR_CLIENT_EMAIL"</span></span>,
<span class="hljs-attr"><span class="hljs-attr">"DIALOGFLOW_PRIVATE_KEY"</span></span>: <span class="hljs-string"><span class="hljs-string">"YOUR_AGENT_PRIVATE_KEY"</span></span>,
<span class="hljs-attr"><span class="hljs-attr">"DIALOGFLOW_USE_INTENT"</span></span>: <span class="hljs-literal"><span class="hljs-literal">false</span></span>
}
}
}
How to find required information for botium JSON
- We can find “google project id” in the settings of our agent in the dialogflow console.
- Service credentials to connect botium to your diaogflow agent. To find that, click on “Google Project ID” and it will redirect to Google cloud platform
- Now from side menu, go to IAM & Admin and then Service Accounts
- Create a new service account and make sure that you have assigned “Dialogflow API Admin” role for botium (Admin role is required to run a full dialogflow agent)
- Next step is to create private key for service account. Click on “Create Key” and select JSON as key type and download file. Open it in text editor, there is an JSON attribute is called “client_email” and “private _key” included, these are the values to copy to our botium.json file
Note: Private key should be included “Begin Private Key” and “End Private Key” texts with double quotes. Ex. “—–BEGIN PRIVATE KEY—–\………………..\n—–END PRIVATE KEY—–\n”
Complete and verify configuration
To complete the configuration, we set an environment variable pointing Botium the configuration file we just composed:
Linux/Bash:
export BOTIUM_CONFIG=path/to/your/botium.json
Windows cmd:
set BOTIUM_CONFIG path/to/your/botium.json
Note: If above command does not work for you, try to add environment variable manually by following these simple steps:
- Search for “Edit the environment variables for your account ”
- Under “User Variable” click on New and then fill variable name and give path of the botium.json file and save it
Now validate the configuration by running the following command in terminal. It will connect to your Dialogflow agent and will give a simple chat interface in the terminal.
Botium-cli emulator
Generate Test Cases
Now we would go for test case generation for dialogflow agents. In this phase, botium will connect to your dialogflow agent, analyze the conversation structure and extract the intent resolution, the training phrases and the conversation flow into test case automatically. We can create test case(called convo file in botium speak) for all intents, utterances(phrases) and conversation steps by using following commands.
export BOTIUM_CONVOS=path/to/some/directory/holding/test/cases
botium-cli import dialogflow-conversations
Output will be something like this in terminal:
Note: Botium does a static analysis of your conversation flow. The Dialogflow Agent won’t be activated during this analysis, no fulfillment action or anything else will happen. The generated convo files should be seen as starting point for manually refining your test cases.
Verify Test Output
To execute all generated test cases, run botium again in “run” mode using below command:
botium-cli run
Output will be something like this in terminal:
Test Report Generation using Mochawesome
Once test case execution is finished. All the test results display as PASSED and FAILED in CMD screen with green and red tick mark.
But imagine what if we can see test report in HTML template. Sounds good, right? Of course YES !!!
So, Mochawesome report generator is inbuilt functionality for botium-cli. So, We do not need to install it separately to your machine. We can generate test report in HTML and JSON by using simple command.
Use following command to generate test report in HTML or JSON formats:
botium-cli run mochawesome
Once test execution is finished, HTML and JSON formatted test reports will be saved in your botium directory. HTML report will look like below image:
Important Note: For Linux users, Terminal’s commands are different. So whenever we need to run command starts with “botium”, command should be starts with “npx”. For example we want to run the test in the linux machine, that command would be something like below command.
npx botium-cli run
That is our first tutorial for setting up Botium for chatbot testing. Do subscribe to our newsletter to get notification when we publish further tutorials in future.
Feel free to comment your doubts/questions. We would be glad to help you.
If you are looking for Chatbot Development or Natural Language Processing services then do contact us or send your requirement at letstalk@pragnakalp.com. We would be happy to offer our expert services.
