Building a Social Interactive Robot with Speech Recognition and a Custom Chatbot

Abstract

This paper describes a social interactive robot that combines speech recognition with a custom-built chatbot to 
support natural spoken interaction between a user and a machine. Speech recognition algorithms interpret spoken 
commands and queries, forming the input path of the system. Rather than using a generic conversational agent, a 
chatbot was built specifically for this robot, trained on an intent dataset defined for the tasks the robot is expected to 
handle, so that its responses stay within a scope it can actually serve. A hotword detector listens continuously for a 
trigger phrase and activates the recognition pipeline only when that phrase is heard, which keeps the robot from 
responding to ambient conversation and avoids running the full pipeline continuously on limited hardware. The robot 
is built on a Raspberry Pi running ROS, with an ESP32 for peripheral control, a microphone and 5 V speakers for 
audio, a Raspberry Pi camera, an HC-SR04 ultrasonic sensor for obstacle detection, a DHT11 for temperature and 
humidity, DC and SG90 servo motors driven through an L298N motor driver, and a touch-enabled TFT LCD. The 
chatbot is implemented in Python using NLTK for language processing and a TFLearn neural network trained on a 
JSON intent file, with Google Text-to-Speech generating the spoken reply. The mechanical design was produced in 
Fusion 360.