In my last article I showed you how to do image classification in the browser.
Image classification can be a very useful tool, it can give us an idea of what’s in an image. However, sometimes we want more. It can be a little counterintuitive, but just because a machine learning model can tell what’s in an image, doesn’t mean it can tell us where it is in the image. We need a different architecture for that.
That’s where object detection comes into play.
Object detection opens up the capability of counting how many objects are in a scene, tracking motion and simply just locating an object’s position.
By the end of this tutorial we’ll have a fully functional real-time object detection web app that will track objects via our webcam.
Detecting Objects
To make object detection predictions, all we need to do is import the TensorFlow model, [coco-ssd](https://github.com/tensorflow/tfjs-models/tree/master/coco-ssd), which can be installed with a package manager like NPM or simply imported in a <script> tag. We can then load the model, and make a prediction.
Note: Loading the model can take several seconds. It is best to load the model once and save a reference to it.
The image we pass into the detection function is just a reference to the html <img> tag:
<img id="image" src="image_url">
After we get our prediction, we need a way to display it to the screen. Our prediction response will be a list of bounding boxes and their confidence scores. It should look something like this:
[{bbox: [x, y, width, height],class: "cat",score: 0.8380282521247864}]
To draw the bounding boxes we are going to take advantage of a <canvas> element:
<canvas id="canvas">
The canvas element allows us to use the strokeRect function, which maps perfectly with our prediction results:
Simple Detection Demo
Streaming from the Webcam
To run real-time detection on a webcam stream is almost as easy as changing from an <img> tag, to a <video> tag …with the simple exception of this giant blob of code to start up the webcam:
We can then just pass our video element to our model for detection. However, this time we are going to call requestAnimationFrame which will call our detection function over and over in an infinite loop as fast as it can, skipping frames when it can’t keep up.
Real-Time Detection Demo
TensorFlow.js — Real-Time Object Detection Demoz364noozrm.codesandbox.io
Note: You must use Google Chrome or a browser that supports webcam streaming (not Safari).
Link to the full code: https://codesandbox.io/s/z364noozrm.
Final Thoughts
I’ve been having a lot of fun using TensorFlow.js. They have made it extremely easy to play with machine learning with minimal headache. A similar project would have taken me days, but this only took hours. I think my next project will be to experiment with training some of my own models 🎉
Thanks for reading! If you have any questions, feel free to reach out at bourdakos1@gmail.com, connect with me on LinkedIn, or follow me on Medium and Twitter.
If you found this article helpful, it would mean a lot if you gave it some applause👏 and shared to help others find it! And feel free to leave a comment below.