Sign Language on Arduino Nano

Sign Language PyTorch Model deployed to tinyML devices effortlessly

Photo by Sammi Schouten on Dribbble

Are you interested in making $20 device smart by running deep learning model with 97KB size to detect sign language gestures? This device can be incredibly useful for a mute person to be able to communicate critical phrases to people who don’t understand sign language.

I created the dataset by recording the ~200 data points per gesture. Fed the dataset to a PyTorch model and achieved 99.3% accuracy in cAInvas notebook. The trained model was compiled with deepSea compiler. The compiled library from deepSea was integrated in an application. The resulting binary was flashed to Arduino Nano board using Arduino IDE. Watch this process in action.

Technical Details

Running deep learning models in such tiny devices is a challenge for two reasons. First, tinyML devices such as Arduino Nano 33 are fairly resource constrained in terms of CPU cycles, RAM, storage space etc. Second, the runtimes for popular ML frameworks aren’t available on these tinyML devices. TF Lite Micro has an implementation but that lacks support for many operators such as RNNs, LSTMs etc.

AITS’ deepSea compiler takes in a deep learning model in Keras, TensorFlow protobuf or ONNX format. It creates a package which includes the C++ library, required header files and a readme that shows how to use the library. The deepSea compiler is optimized for tinyML devices with ARM, Xtensa and many other HW architectures.

The arduino application is available here.

Once you have followed the instructions provided below in Software Setup section and created the deepSea library from the cAInvas platform, please follow the instructions on github on how to put the library such that Arduino IDE finds the asl_imu.a library to link to your application.

Hardware used:

• Arduino Nano 33 BLE Sense with Headers — prefer one w/ Headers so you won’t have to do the soldering etc.
• USB-A to micro-USB cable

Software used:

• cAInvas — End-to-end tinyML development platform from AITS
• deepSea — AITS’ ML compiler to convert models to C++ std11 library
• Arduino IDE — IDE to program Arduino and other tinyML devices
• Arduino LSM9DS1 Library — Library to read sensors on Arduino Nano 33 BLE Sense

Software Setup:

• Download and install Arduino IDE
• Install supporting libraries for Arduino Nano 33 BLE Sense — link
• Install Arduino_LSM9DS1 library in Arduino IDE by traversing the menu as follows: Sketch → Include Library → Manage Libraries. Search for LSM9DS1 and install Arduino_LSM9DS1 v1.1.0 or later.

References:

1. Free signup on cAInvas
2. ASL Recognition with TinyML Devices using cAInvas
3. PyTorch sample notebook for ASL detection in PyTorch
4. Speed-Up Pre-Trained Deep Learning Models on Micro-Controllers
5. Deep Learning with deepC DNN Compiler — part 1
6. Deep Learning with deepC DNN Compiler — part 2
7. Arduino Nano 33 support on Arduino IDE
8. Person Detection on Arduino Nano