What were the main advances in machine learning/artificial intelligence in 2016? originally appeared on Quora: the knowledge sharing network where compelling questions are answered by people with unique insights.
Answer by Xavier Amatriain, VP Engineering at Quora, former Netflix recommendations, researcher, professor, on Quora.
2016 may very well go down in history as the year of “the Machine Learning hype”. Everyone now seems to be doing machine learning, and if they are not, they are thinking of buying a startup to claim they do.
Now, to be fair, there are reasons for much of that “hype”. Can you believe that it has been only a year since Google announcedthey were open sourcing Tensor Flow? TF is already a very active project that is being used for anything ranging from drug discovery to generating music. Google has not been the only company open sourcing their ML software though, many followed lead. Microsoft open sourced CNTK, Baidu announced the release of PaddlePaddle, and Amazon just recently announced that they will back MXNet in their new AWS ML platform. Facebook, on the other hand, are basically supporting the development of not one, but two Deep Learning frameworks: Torch and Caffe. On the other hand, Google is also supporting the highly successful Keras, so things are at least even between Facebook and Google on that front.
Besides the “hype” and the outpour of support from companies to machine learning open source projects, 2016 has also seen a great deal of applications of machine learning that were almost unimaginable a few months back. I was particularly impressed by the quality of Wavenet’s audio generation. Having worked on similar problems in the past I can appreciate those results. I would also highlight some of the recent results in lip reading, a great application of video recognition that is likely to be very useful (and maybe scary) in the near future. I should also mention Google’s impressive advances in machine translation. It is amazing to see how much this area has improved in a year.
Recommended by Forbes
As a matter of fact, machine translation is not the only interesting advance we have seen in machine learning for language technologies this past year. I think it is very interesting to see some of the recent approaches to combine deep sequential networks with side-information in order to produce richer language models. In “A Neural Knowledge Language Model”, Bengio’s team combines knowledge graphs with RNNs, and in “Contextual LSTM models for Large scale NLP Tasks”, the Deepmind folks incorporate topics into the LSTM model. We have also seen a lot of interesting work in modeling attention and memory for language models. As an example, I would recommend “Ask Me Anything: Dynamic Memory Networks for NLP”, presented in this year’s ICML.
Also, I should at least mention a couple of things from NIPS 2016 in Barcelona. Unfortunately, I had to miss the conference the one time that was happening in my hometown. I did follow from the distance though. And from what I gathered, the two hottest topics were probably Generative Adversarial Networks(including the very popular tutorial by Ian Goodfellow) and thecombination of probabilistic models with Deep Learning.
Let me also mention some of the advances in my main area of expertise: Recommender Systems. Of course Deep Learning has also impacted this area. While I would still not recommend DL as the default approach to recommender systems, it is interesting to see how it is already being used in practice, and in large scale, by products like Youtube. That said, there has been interesting research in the area that is not related to Deep Learning. The best paper award in this year’s ACM Recsys went to “Local Item-Item Models For Top-N Recommendation”, an interesting extension to Sparse Linear Methods (i.e. SLIM) using an initial unsupervised clustering step. Also, “Field-aware Factorization Machines for CTR Prediction”, which describes the winning approach to the Criteo CTR Prediction Kaggle Challenge is a good reminder that Factorization Machines are still a good tool to have in your ML toolkit.
I could probably go on for several other paragraphs just listing impactful advances in machine learning in the last twelve months. Note that I haven’t even listed any of the breakthroughs related to image recognition or deep reinforcement learning, or obvious applications such as self-driving cars, chat bots, or game playing, which all saw huge advances in 2016. Not to mention all the controversy around how machine learning is having or could have negative effects on society and the rise of discussions around algorithmic bias and fairness.
