Cost Functions and the Backpropagation Derivation

There are dozens of ways to construct neural networks and it can be very difficult to decipher which are best used for which problems and why. However, while there are stark differences between different neural networks there are common elements to all. We will call these the vital elements. They are the DNA of the neural network. Each is different but each adheres to the same general principles. These vital elements are the cost function and the backpropagation calculation (the activation function is another vital element and we will discuss it somewhat here but I will get into more depth about the activation function in future posts). In this post I want to provide the general derivation of the backpropagation algorithm. The principles from this derivations are used to compute the gradients of all neural network. I will also provide a few of the more popular cost functions and there gradients at the conclusion of this post. We will not be going into great detail about the benefits of each but my hope is that by understanding the cost function and backpropagation in the general so we can better understand their applications in the specific.

Building a Neural Network from Scratch

Code to follow along with

Neural networks are one the most powerful learning techniques and with Python libraries such as TensorFlow, Theano, Torch, etc. their implementation is only getting easier. Machine learning and neural networks are buzz words being thrown around a lot in business and computer science circles. People seem to think these ideas will unlock the big breakthrough in AI we have been waiting 60 years for. They may be right but before we crown machine learning as the prized jewel of 21st century ingenuity let's first figure out how they work. This will hopefully give us some intuition about what is and is not possible with machine learning. To a majority of the public neural networks are a black box. We want to get over that and hopefully understand some of the practical uses of neural networks, as well as some of the potential drawbacks. I believe that the best way to learn about something is to build it yourself, test it, break it and test it again. This will really challenge our understanding of the algorithm. In this post my goal is to show you how I built a neural network from scratch.