Uncertainty in Deep Learning

Yarin Gal:

So I finally submitted my PhD thesis (given below). In it I organised the already published results on how to obtain uncertainty in deep learning, and collected lots of bits and pieces of new research I had lying around (which I hadn’t had the time to publish yet). The questions I got about the work over the past year were a great help in guiding my writing, with the greatest influence on my writing, I reckon, being the work of Professor Sir David MacKay (and his thesis specifically). Weirdly enough, I would consider David’s writing style to be the equivalent of modern blogging, and would highly recommend reading his thesis. I attempted to follow David’s writing style in my own writing, explaining topics through examples and remarks, resulting in what almost looks like a 120 pages long blog post. So hopefully it can now be seen as a more complete body of work, accessible to as large an audience as possible, and also acting as an introduction to the field of what people refer to today as Bayesian Deep Learning. One of the interesting results which I will demonstrate below touches on uncertainty visualisation in Bayesian neural networks. It’s something that almost looks trivial, yet it has gone unnoticed for quite some time! But before that, I’ll review quickly some of the new bits and pieces in the thesis for people already familiar with the work. For others I would suggest starting with the introduction: The Importance of Knowing What We Don’t Know.