In a recent publication, Apple explained how they used machine learning to directly recognize people in private photos on your iPhones and iPads without having access to your images to train their algorithms.
For those of you with Apple products, you can actually research by the person in the Photos app.
Indeed, using multiple machine learning-based algorithms that I will cover in this article, running privately on your device, you are able to accurately curate and organize your images and videos on iOS 15.
It will recognize the different people and allow you to research in your pictures where the person appears. If you have thousands of photos like I do, you will already have different clusters each representing different people.
For example, one such cluster could be all the photos where your friend John is in so that you can name it “John” and then search for images of John in your pictures to have them appear automatically.
Watch the video to learn more!
Watch the video
References:
► Read the Full article: https://www.louisbouchard.ai/how-apple-photos-recognizes-people/
►Apple, "Recognizing People in Photos Through Private On-Device Machine Learning", (2021), https://machinelearning.apple.com/research/recognizing-people-photos
►My Newsletter (A new AI application explained weekly to your emails!): https://www.louisbouchard.ai/newsletter/
►Apple, "Recognizing People in Photos Through Private On-Device Machine Learning", (2021), https://machinelearning.apple.com/research/recognizing-people-photos
►My Newsletter (A new AI application explained weekly to your emails!): https://www.louisbouchard.ai/newsletter/
Video Transcript
00:00
In a recent publication, Apple explained how they used machine learning to directly recognize
00:04
people in private photos on your iPhones and iPads without having access to your images
00:09
to train their algorithms.
00:11
I personally do not have an iPhone, so I cannot test it myself, but I am looking for an iPad
00:15
to draw explanations and write math equations and stream it during calls.
00:19
If some of you guys use tablets to do that, please let me know what you think is the best
00:24
to get!
00:25
For those of you with Apple products, you can actually research by the person in the
00:28
Photos app.
00:30
Indeed, using multiple machine learning-based algorithms that I will cover in this video,
00:34
running privately on your device, you are able to accurately curate and organize your
00:39
images and videos on iOS 15.
00:42
It will recognize the different people and allow you to research in your pictures where
00:45
the person appears.
00:47
If you have thousands of photos like I do, you will already have different clusters each
00:50
representing different people.
00:52
For example, one such cluster could be all the photos where your friend John is in so
00:57
that you can name it "John" and then search for images of John in your pictures to have
01:02
them appear automatically.
01:04
It can even recognize photos where the same people frequently appear, even if it doesn't
01:08
know the persons individually or hasn't been directly trained with it, and use it to share
01:13
memories like the "Together" feature shown here.
01:16
This is a super cool built-in application by Apple, and the best is that it even works
01:21
when the face is occluded or sideways, as we will see.
01:24
As I said, it seems to work really well.
01:27
It entirely runs on your device privately, and they are always improving the algorithms,
01:32
but it's even cooler to know how it works, so let dive into it!
01:36
This task of recognizing people in your own picture is extremely challenging because of
01:40
the variability your photos will have.
01:43
Different people, different angles, different scales, different lightings, occlusions because
01:47
your friend was catching a football, or even from other cameras.
01:52
If we would strictly base ourselves on the person's face, this would be pretty incomplete
01:57
as most of our pictures taken on the spot during an event aren't perfect images with
02:02
your friends smiling in front of the camera.
02:04
When you type in John, you'd like to see these events where John won the game by catching
02:09
this ball.
02:10
To attack this, they start by locating the faces and upper bodies of people visible in
02:15
the image using a first detection algorithm.
02:18
This algorithm was trained on many labeled human examples annotated with where the bodies
02:22
and the faces were.
02:24
Meaning that they trained a deep neural network with images sent as inputs, and the outputs
02:28
were only the cropped version of the image with either the bodies or faces of the people.
02:34
This is done by feeding many examples to the network, helping it showing where to focus
02:38
its attention with the correct identified sections.
02:42
This way, it can iteratively learn to find this body part by itself afterward if we show
02:47
it enough examples during training.
02:49
photo: figure 2 cursor + network (U-net, VAE?) with images and box over other image and incorrect
02:50
box until it's better?
02:51
By the way, if you find this interesting, don't forget to subscribe, like the video,
02:53
and share it with your friends or colleagues, it helps a lot!
02:57
Thank you!
02:58
Then, they match the bodies and faces of each individual to have even more data about the
03:03
person in case only one of the two appears in a future image.
03:06
You can see here that both the body and face are sent into a separate model that encodes
03:12
the information, creating embeddings.
03:14
These embeddings are simply the most valuable information about the face and body of the
03:18
person.
03:19
Here, we use another network to encode the information because we want our embeddings
03:23
to be similar for the same person and different for different individuals.
03:27
This is again done with another model that will look like this, inspired by mobilenet,
03:32
which I talked about in my convolutional neural network video.
03:36
It is a lightweight convolutional neural network that can run extremely efficiently, made for
03:41
mobile instead of GPUs.
03:43
If you are not familiar with CNNs, I strongly invite you to watch the video I made explaining
03:48
them simply.
03:49
Basically, it takes the cropped images and compresses the information in a smaller space
03:53
focusing on the most interesting details about the individual.
03:57
This is possible because such a model was trained on a lot of images to do exactly that.
04:02
Then, these embeddings are merged and saved in your phone's gallery unless they have poor
04:07
responses.
04:09
These poor responses may come from unclear faces or upper bodies and would be automatically
04:14
filtered out.
04:15
This is repeated with all your pictures to create clusters out of these embeddings.
04:20
These clusters will be the different people identified.
04:22
It will merge all similar embeddings in small groups where each group is a specific individual.
04:28
So this is the step where all the pictures where John was identified are put into a gallery.
04:33
And what's cool is that this automatically runs during nighttime when your phone charges
04:38
while you sleep and keeps on improving the more pictures you have.
04:42
So once these clusters are created, your new photos containing people are sent to the same
04:46
deep network to create a new embedding per person in the image.
04:50
This new embedding will either join a cluster if they find a match or create a new one based
04:55
on the difference between the embeddings you have in your phone and the new picture's embeddings.
05:00
Here, to find whether it is the same person or not, they focus primarily on the face.
05:04
If it's occluded or sideways, it uses the upper body coupled with what we have from
05:09
the face and takes the time of the photo into account to measure if the clothing could be
05:13
the same or different.
05:15
As you may suspect, the upper body isn't always helpful.
05:18
As they say, "We’ve carefully tuned the set of face and upper body distance thresholds
05:23
to get the most out of the upper body embedding without negatively impacting overall accuracy."
05:29
And this is how Photos regroup your friends within the application without you knowing
05:34
it!
05:36
Another concern was that they want to offer the same experience for all Apple users no
05:40
matter the photographic subject’s skin color, age, or gender.
05:44
It is great that they keep on improving the generalization and working to remove these
05:48
biases from their algorithm the best they can using the broadest datasets possible and
05:53
data augmentation to add variations to the training images.
05:57
If you have an iPhone or iPad, please let me know what you think of this feature in
06:01
the Photos app and how well it works!
06:04
Thank you for watching!