1. Build a Conversational Chatbot with Rasa Stack and Python— Rasa NLU

Chatbots are gaining traction and as I mentioned in my previous article ‘Chatbots & Future of Conversational AI’ the day is not far when it’s gonna touch almost all aspects of life.

If you see biggies like IBM, Facebook, Google, Microsoft or Amazon all are working extensively on it and each building their own framework. I am sure you must be thinking why I chose Rasa over others. If I tell you frankly there is no such reason except I needed an open-source framework which can provide me sufficient control in building the bot. The bot which can understand the intent of the user, learn from it, respond intelligently and performs actions if required and all with an efficient learning mechanism. These requirements zeroed down me to use Rasa Stack to build my first ever Chatbot which I named as Wall-E (My love to movie Wall-E :))

Now lets first understand what is Rasa Stack and what all it provides —

RASA stack is an open-source AI tool and being an opensource framework, It is easy to customize. In fact, In many cases, Clients do not want to share their data and the majority of the tools available are cloud-based and provide software as a service. You can not run them internally in your environment. So you need to send your data to the third party. With RASA, There is no such issue. You can build, deploy or host Rasa internally in your server or environment with complete control on it.

Rasa comes up with 2 components —

Rasa NLU — a library for natural language understanding (NLU) which does the classification of intent and extract the entity from the user input and helps bot to understand what the user is saying.

Rasa Core — a chatbot framework with machine learning-based dialogue management which takes the structured input from the NLU and predicts the next best action using a probabilistic model like LSTM neural network.

NLU and Core are independent and one can use NLU without Core, and vice versa. Though Rasa recommends using both.

Rasa Flow

Before we further discuss, Let me explain a few keywords which would be used repeatedly here and in the future posts.

Intent — Intent is nothing but what the user is aiming for. For example — if the user says “Reserve a table at Cliff House tonight” the intent can be classified as to book the table.

Entity — Entity is to extract the useful information from the user input. From the example above “Reserve a table at Cliff House tonight” the entities extracted would be place and time. Place — Cliff House and Time — tonight.

Stories — Stories define the sample interaction between the user and chatbot in terms of intent and action taken by the bot. Like in the example above bot got the intent of booking the table and entities like place and time but still, there is an entity missing — no of people and that would make the next action from the bot.

Actions — Actions are basically the operations performed by the bot either asking for some more details to get all the entities or integrating with some APIs or querying the database to get/save some information.

Now as we know some basics, Let’s dive into some actual actions :). In this post, we will majorly focus on the Rasa Natural Language Understanding (NLU) which comprises of Intent Classification & Entity extraction and create a structured output which can be fed into Rasa Core.

If anyone wants to get into more details Rasa has provided some great documentation to understand the context better.

Rasa Natural Language Understanding (NLU)

As we already briefed on NLU above, we need to teach our bot to understand our messages first. For that, we have to train the NLU model with our inputs in a simple text format and extract structured data. We will achieve this by defining the intents and providing a few ways users might express them.

To make this work, we need to define some files. Lets first understand these files

NLU training file: It contains some training data in terms of user inputs along with the mapping of intents and entities present in each of them. The more varying examples you provide, better your bot’s NLU capabilities become.

Stories file: Though we will be discussing in details about it in next post but to brief this file contains sample interactions the user and bot will have. Rasa (Core) creates a probable model of interaction from each story.

Domain file: This file lists all the intents, entities, actions, templates and some more information. The templates which I just mentioned is nothing but the sample bot reply which can be used as actions. We will discuss this file in details when we discuss Rasa Core.

Objective

Before we create an NLU training file Lets first understand what our bot is going to do. We are going to build the Bot (I named it Wall-E) which will help the user to get the news from around the world and also in the specific categories like sports, business, entertainment, technologies and more.

Installation & Setup

Lets first do the environment setup and for that, we need to install python. I am using Conda (miniconda) to set up the virtual environment for Python and would highly recommend creating one as It would be good to have all the Rasa installation at one place.

First, install the Conda(miniconda) from here as per the OS. After installation check the Conda version

$ conda --version or conda -V

In case need to upgrade, run below command

$ conda update conda

Once Conda is installed we should create the virtual environment and proceed with further installation of Rasa NLU packages. For this post, I am using python 3.6

## Creating a virtual environment
conda create --name botenv python=3.6
## Activate the new environment to use it
LINUX, macOS: conda activate botenv
WINDOWS: activate bot
## Install latest Rasa stack
## Rasa NLU
pip install rasa_nlu
or Python3 install rasa_nlu
## For dependencies
## spaCy+sklearn (pipeline)
pip install rasa_nlu[spacy]
python3 -m spacy download en
python3 -m spacy download en_core_web_md
python3 -m spacy link en_core_web_md en

##Tensorflow (pipeline)
pip install rasa_nlu[tensorflow]

Once the package installation is done, Let's create the project structure. I have named the project as wall-e and so this is our base project directory. For training/data files, we create a data directory under wall-e and create the training file nlu.md in that.

$ mkdir wall-e
$ cd wall-e
$ mkdir data
$ cd data
$ touch nlu.md

nlu.md

The data provided will help in training the bot. More the data better the bot would get trained. ‘## intent:’ defines the name of the intent and [word](entity) defines the entity e.g — [world](category). We have defined multiple examples for Bot to understand it better.

Now we have to define the pipeline through which this data will flow and intent classification & entity extraction can be done for the bot. I have used ‘spacy_sklearn’ pipeline, created a file nlu_config.yml in the base project directory (i.e. wall-e) and added the pipeline in that.

nlu_config.yml

Rasa has provided some more information about pipeline here.

Now as our NLU data and pipeline are ready, it is time to train the bot. We can do so either by running the script in terminal or we can create a python file and run it.

If using the script in terminal (don’t forget to activate the virtual env where all the packages were installed. In our case it is botenv)

$ conda activate botenv
$ cd wall-e
$ python3 -m rasa_nlu.train -c nlu_config.yml --data data/nlu.md -o models --fixed_model_name nlu --project current --verbose

If using python file, create nlu_model.py in the base project directory (i.e. wall-e) and add below code to it.

nlu_model.py

$ cd wall-e
$ python3 nlu_model.py

This will train the NLU model and save it at ‘models/current/nlu’. The same path is passed to NLU Interpreter to parse some sample intents from the user to see if NLU is able to classify the intent and extract the entities correctly. The output looks something like this

If you see NLU is able to get the entity correctly though we didn’t provide an example of ‘What is going on in education’ still bot is able to guess the intent and extract the entity ‘category’ with value ‘education’. It is always good to train the model with a good amount of data so that it can learn from it.

Finally, we have successfully completed the training of Bot where it is able to understand the natural language but we still need to build the dialogues so that bot can respond to the messages. This we will cover in the next part Build a Conversational Chatbot with Rasa Stack and Python — Rasa Core

I would highly recommend to go through each section step by step and read Rasa Documentation to understand it better. In case you need the code here is the git repository.

Please share your thoughts, feedback, comments, and clarifications. It will help me to improvise.

If you liked this post, you can help me share it by recommending it below ❤

Follow me on Linkedin and Twitter