Sticky Notes
My company was hosting a little AI challenge back in 2020. The idea was to create a model that could take in an image of sticky notes on a wall, isolate each sticky note, and then perform OCR on the contents of it. The model could be used to digitize notes from a brainstorming session.
Collecting Data
My first step was to gather some data to train an object detection model. I found some sticky notes and started writing some random things on it with different markers and pencils. Next, I spun up CVAT and got my annotations.
The Models
Two models are needed to solve this problem. The first is an object detection model to isolate the sticky notes. The second is an optical character recognition model to read the handwriting. I went with ImageAI’s implementation of YOLOv3 and keras-ocr.
Results
The pipeline is 90% done. The object detection model works fine, but I could’ve done better with the OCR. Firstly, I didn’t use an OCR model that is specifically for handwriting. The one I used was trained on fonts/text. Lastly, I would’ve found a way to order the OCR outputs so the text output reads like the order on the sticky note. I made a Google Colab notebook so be sure to check it out if you want to give it a try!