In a couple of minutes, you will know how the transformer architecture can be applied to computer vision with a new paper called the Swin Transformer.
As a bonus, make sure you stay until the end of the video for a giveaway sponsored by NVIDIA GTC!
References
►My Newsletter (subscribe here to have a chance to win!): http://eepurl.com/huGLT5
►Register to the GTC event: https://www.nvidia.com/en-us/gtc/?ncid=ref-crea-331503
►DLI courses: https://www.nvidia.com/en-us/training/
►Paper: Liu, Z., “Swin Transformer: Hierarchical Vision Transformer using Shifted Windows”, 2021, https://arxiv.org/abs/2103.14030v1
Video transcript
00:00
This video is about most probably the next generation of neural networks for all computer
00:05
vision applications: The transformer architecture.
00:09
You've certainly already heard about this architecture in the field of natural language
00:13
processing, or NLP, mainly with GPT3 that made a lot of noise in 2020.
00:19
Transformers can be used as a general-purpose backbone for many different applications and
00:23
not only NLP. photo: transformers, gpt3
00:25
In a couple of minutes, you will know how this transformer architecture can be applied
00:29
to computer vision with a new paper called the Swin Transformer by Ze Lio et al. from
00:35
Microsoft Research.
00:37
Before diving into the paper, I just wanted to tell you to stay until the end of the video,
00:41
where I will talk about my newsletter I just created and the next free NVIDIA GTC EVENT
00:47
happening in two weeks.
00:48
You should definitely stay or skip right to it as I will provide you with the timeline
00:53
as usual because I will be hosting a giveaway in collaboration with NVIDIA GTC!
00:58
This video may be less flashy than usual as it doesn't really show the actual results
01:03
of a precise application.
01:04
Instead, the researchers showed how to adapt the transformers architecture from text inputs
01:10
to images, surpassing computer vision state-of-the-art convolutional neural networks, which is much
01:15
more exciting than a small accuracy improvement, in my opinion!
01:19
And of course, they are providing the code for you to implement yourself!
01:23
The link is in the description.
01:25
But why are we trying to replace convolutional neural networks for computer vision applications?
01:30
This is because transformers can efficiently use a lot more memory and are much more powerful
01:36
when it comes to complex tasks.
01:38
This is, of course, according to the fact that you have the data to train it.
01:43
Transformers also use the attention mechanism introduced with the 2017 paper Attention is
01:48
all you need.
01:49
Attention allows the transformer architecture to compute in a parallelized manner.
01:54
It can simultaneously extract all the information we need from the input and its inter-relation,
02:00
compared to CNNs.
02:02
CNNs are much more localized, using small filters to compress the information towards
02:07
a general answer.
02:08
While this architecture is powerful for general classification tasks, it does not have the
02:13
spatial information necessary for many tasks like instance recognition.
02:18
This is because convolutions don't consider distanced-pixels relations.
02:23
In the case of NLP, a classical type of input is a sentence and an image in a computer vision
02:29
case.
02:30
To quickly introduce the concept of attention, let's take a simple NLP example sending a
02:34
sentence to translate it into a transformer network.
02:38
In this case, attention is basically measuring how each word in the input sentence is associated
02:44
with each word on the output translated sentence.
02:47
Similarly, there is also what we call self-attention that could be seen as a measurement of a specific
02:53
word's effect on all other words of the same sentence.
02:57
This same process can be applied to images calculating the attention of patches of the
03:01
images and their relations to each other, as we will discuss further in the video.
03:06
Now that we know transformers are very interesting, there is still a problem when it comes to
03:11
computer vision applications.
03:12
Indeed, just like the popular saying "a picture is worth a thousand words," pictures contain
03:18
much more information than sentences, so we have to adapt the basic transformer's architecture
03:23
to process images effectively.
03:26
This is what this paper is all about.
03:28
This is due to the fact that the computational complexity of its self-attention is quadratic
03:33
to image size.
03:35
Thus exploding the computation time and memory needs.
03:38
Instead, the researchers replaced this quadratic computational complexity with a linear computational
03:44
complexity to image size.
03:47
The process to achieve this is quite simple.
03:50
At first, like most computer vision tasks, an RGB image is sent to the network.
03:55
This image is split into patches, and each patch is treated as a token.
04:00
And these tokens' features are the RGB values of the pixels themselves.
04:04
To compare with NLP, you can see this as the overall image is the sentence, and each patch
04:10
is the words of that sentence.
04:13
Self-attention is applied on each patch, here referred to as windows.
04:17
Then, the windows are shifted, resulting in a new window configuration to apply self-attention
04:22
again.
04:23
This allows the creation of connections between windows while maintaining the computation
04:28
efficiency of this windowed architecture.
04:31
This is very interesting when compared with convolutional neural networks as it allows
04:35
long-range pixel relations to appear.
04:38
This was only for the first stage.
04:41
The second stage is very similar but concatenates the features of each group of two by two neighboring
04:47
patches, downsampling the resolution by a factor of two.
04:50
This procedure is repeated twice in Stages 3 and 4 producing the same feature map resolutions
04:57
like those of typical convolutional networks like resnets and VGG.
05:03
You may say that this is highly similar to a convolutional architecture and filters using
05:07
dot products.
05:08
Well, yes and no.
05:10
The power of convolutions is that the filters use fixed weights globally, enabling the translation-invariance
05:16
property of convolution, making it a powerful generalizer.
05:20
In self-attention, the weights are not fixed globally.
05:23
Instead, they rely on the local context itself.
05:26
Thus, self-attention takes into account each pixel, but also its relation to the other
05:32
pixels.
05:33
Also, their shifted window technique allows long-range pixel relations to appear.
05:38
Unfortunately, these long-range relations only appear with neighboring windows.
05:43
Thus, losing very long-range relations, showing that there is still a place for improvement
05:47
of the transformer architecture when it comes to computer vision,
05:51
As they state in the paper, "It is our belief that a unified architecture
05:55
across computer vision and natural language processing could benefit
05:59
both fields, since it would facilitate joint modeling of visual and textual signals and
06:04
the modeling knowledge from both domains can be more deeply shared"
06:08
And I completely agree.
06:10
I think using a similar architecture for both NLP and computer vision could significantly
06:15
accelerate the research process.
06:17
Of course, transformers are still highly data-dependent, and nobody can say whether or not it will
06:23
be the future of either NLP or computer vision.
06:26
Still, it is undoubtedly a significant step forward for both fields!
06:31
Now that you've stayed this far let's talk about an awesome upcoming event for our field:
06:36
GTC.
06:37
So what is GTC2021?
06:38
It is a weeklong event offering over 1,500 talks from AI leaders like Yoshua Bengio,
06:45
Yann Lecun, Geoffrey Hinton, and much more!
06:48
The conference will start on April 12 with a keynote from the CEO of NVIDIA, where he
06:53
will be hosting the three AI pioneers I just mentioned.
06:57
This will be amazing!
06:58
It is an official NVIDIA conference for AI innovators, technologists, and creatives.
07:04
The conferences are covering many exciting topics.
07:07
Such as automotive, healthcare, data science, energy, deep learning, education, and much
07:11
more.
07:12
You don't want to miss it out!
07:14
Oh, and did I forget to mention that the registration is completely free this year?
07:18
So sign-up right now and watch it with me.
07:21
The link is in the description!
07:23
What's even cooler is that NVIDIA provided me 5 Deep Learning Institute credits that
07:28
you can use for an online, self-paced course of your choice worth around 30$ each!
07:34
The deep learning institute offers hands-on training in AI for developers, data scientists,
07:40
students, and researchers to get practical experience powered by GPUs in the cloud!
07:45
I think it's an awesome platform to learn, and it is super cool that they are offering
07:49
credits to give away, don't miss out on this opportunity!
07:52
To participate in this giveaway, you need to mention your favorite moment from the GTC
07:57
keynote on April 12 at 8:30 am pacific time using the hashtag #GTCWithMe and tagging me
08:05
(@whats_ai) on LinkedIn or Twitter!
08:09
I will also be live-streaming the event on my channel to watch it together and discuss
08:13
it in the chat.
08:14
Stay tuned for that, and please let me know what you think of the conference afterward!
08:19
NVIDIA also provided me with two extra codes to give away to the ones subscribing to my
08:24
newsletter!
08:25
This newsletter is about sharing only ONE paper each week.
08:29
There will be a video, an article, the code, and the paper itself.
08:32
I will also add some of the projects I am working on, guides to learning machine learning,
08:37
and other exciting news!
08:39
It's the first link in the description, and I will draw the winners just after the GTC
08:43
event!
08:44
Finally, just a final word as I wanted to personally thanks the four recent Youtube
08:49
members!
08:50
Huge thanks to you ebykova, Tonia Spight-Sokoya, Hello Paperspace, and Martin Petrovski, for
08:58
your support and everyone watching the videos!
09:01
See you in the next one!