Botium Setup and Test Case Generation for Dialogflow Agent

Chatbots, Python Development, Machine Learning, Natural Language Processing (NLP)

Testing / QA is a crucial part of software development. If testing/qa is not done properly then it leaves possibilities of bugs and errors in the system. For testing desktop, web or mobile based applications, there are various tools available. Also, we can develop our own testing tools using open source frameworks like Selenium.

Now as we are opening up a new frontier to reach out to customers in form of Chatbots, we are going to need testing tools for chatbots too. Normal desktop, web, mobile app testing tools might not work effectively for chatbots testing. We have to find some new ways. To bridge this gap, people at Botium has developed a framework – Botium which is claimed to be selenium for chatbots.

Botium is claimed to support multiple chatbots development platform like Google Dialogflow, Microsoft LUIS, Microsoft Bot Framework, IBM Watson Assistant, Alexa Voice Service and many other platforms. As it looks promising we gave it a try and here is the tutorial of how to setup Botium and generate test cases for Dialogflow agent.

Installation & Configuration

1. Install Node.js first if your machine does not have. Botium cli would be installed as Node.js module.

2. Now run below command in terminal window:

npm install –g botium-cli

It will install botium cli in machine.

3. After “botim-cli” is installed successfully the command will be available to run. If you have done global installation then you can run “botium-cli” from any directory. If you have installed it in local directory then you will need to browse to that directory and run “npx botium-cli”

4. Now we need to configure connection between dialogflow agent and botium cli. The connection to the Dialogflow Agent is configured in a JSON-file. Create a file called “botium.json” with data in JSON as below:

{

  "botium": {

    "Capabilities": {

      "PROJECTNAME": "Dialogflow Sample",

      "CONTAINERMODE": "dialogflow",

      "DIALOGFLOW_PROJECT_ID": "YOUR_PROJECT_ID",

      "DIALOGFLOW_CLIENT_EMAIL": "YOUR_CLIENT_EMAIL",

      "DIALOGFLOW_PRIVATE_KEY": "YOUR_AGENT_PRIVATE_KEY",

      "DIALOGFLOW_USE_INTENT": false

    }

  }

}

How to find required information for botium JSON

  • We can find “google project id” in the settings of our agent in the dialogflow console.
  • Service credentials to connect botium to your diaogflow agent. To find that, click on “Google Project ID” and it will redirect to Google cloud platform
  • Now from side menu, go to IAM & Admin and then Service Accounts
  • Create a new service account and make sure that you have assigned “Dialogflow API Admin” role for botium (Admin role is required to run a full dialogflow agent)

  • Next step is to create private key for service account. Click on “Create Key” and select JSON as key type and download file. Open it in text editor, there is an JSON attribute is called “client_email” and “private _key” included, these are the values to copy to our botium.json file

Note: Private key should be included “Begin Private Key” and “End Private Key” texts with double quotes.  Ex. “—–BEGIN PRIVATE KEY—–\………………..\n—–END PRIVATE KEY—–\n”

Complete and verify configuration

To complete the configuration, we set an environment variable pointing Botium the configuration file we just composed:

Linux/Bash:

export BOTIUM_CONFIG=path/to/your/botium.json

Windows cmd:

set BOTIUM_CONFIG path/to/your/botium.json

Note: If above command does not work for you, try to add environment variable manually by following these simple steps:

  • Search for “Edit the environment variables for your account ”
  • Under “User Variable” click on New and then fill variable name and give path of the botium.json file and save it

Now validate the configuration by running following command in terminal. It will connect to your diaogflow agent and will give simple chat interface in the terminal.

Botium-cli emulator

Generate Test Cases

Now we would go for test case generation for dialogflow agents. In this phase, botium will connect to your dialogflow agent, analyze the conversation structure and extract the intent resolution, the training phrases and the conversation flow into test case automatically. We can create test case(called convo file in botium speak) for all intents, utterances(phrases) and conversation steps by using following commands.

export BOTIUM_CONVOS=path/to/some/directory/holding/test/cases
botium-cli import dialogflow-conversations

Output will be something like this in terminal:

Note: Botium does a static analysis of your conversation flow. The Dialogflow Agent won’t be activated during this analysis, no fulfillment action or anything else will happen. The generated convo files should be seen as starting point for manually refining your test cases.

Verify Test Output

To execute all generated test cases, run botium again in “run” mode using below command:

botium-cli run

Output will be something like this in terminal:

Test Report Generation using Mochawesome

Once test case execution is finished. All the test results display as PASSED and FAILED in CMD screen with green and red tick mark.

But imagine what if we can see test report in HTML template. Sounds good, right? Of course YES !!!

So, Mochawesome report generator is inbuilt functionality for botium-cli. So, We do not need to install it separately to your machine. We can generate test report in HTML and JSON by using simple command.

Use following command to generate test report in HTML or JSON formats:

botium-cli run mochawesome

Once test execution is finished, HTML and JSON formatted test reports will be saved in your botium directory. HTML report will look like below image:

Important Note: For Linux users, Terminal’s commands are different. So whenever we need to run command starts with “botium”, command should be starts with “npx”. For example we want to run the test in the linux machine, that command would be something  like below command.

npx botium-cli run

That is our first tutorial for setting up Botium for chatbot testing. Do subscribe to our newsletter to get notification when we publish further tutorials in future.


You can also contact us for Chatbots Development services. Get in touch with us at letstalk@pragnakalp.com to discuss about your chatbot development requirement.

Leave a Reply

Your email address will not be published. Required fields are marked *