Deep learning models are extremely important. And they are going to have a pivotal role to play in the world to come. Which is why it is important for large corporations to ensure that they are not lagging behind in the race to train deep learning systems of their own. Along the same, Facebook has announced a major breakthrough in its deep learning capabilities.
In a paper published earlier, the company said that it had managed to redcue training time drastically. To put it in more concrete terms, Facebook said that it had managed to reduce the training time of a ResNet-50 deep learning model on ImageNet from 29 hours to just one.
The company managed this feat by distributing the training process in minibatches, and then assigning these minibatches to a lot of GPU.
This is essentially how training takes place however, Facebook really scaled thigns up. For instance, in the last benchmark, batches of 256 images were spread across eight GPUs. What Facebook did was, that it spread batch sizes of 8192 images across 256 GPUs. This way, the company was able tor educe the training time using a method only organizations with this many GPUs lying around could deploy.
However, there are a lot of questions still unanswered. Speaking on the topic, Pieter Noordhuis, a member of Facebook’s AML team said:
This work throws out more questions than it answers. There’s a tipping point beyond 8,000 images where error rates go up again and we don’t know why.
Visual recognition models can be used in a wide variety of purposes including, automatic payments from our account, ID checks and so on.
