Reproducing Leslie N. Smith’s papers using fastai

The following papers by Leslie N. Smith are covered

1. A disciplined approach to neural network hyper-parameters: Part 1 — learning rate, batch size, momentum, and weight decay. paper
2. Super-Convergence: Very Fast Training of Neural Networks Using Learning Rates. paper
3. Exploring loss function topology with cyclical learning rates. paper
4. Cyclical Learning Rates for Training Neural Networks. paper

Although the main aim is to reproduce the papers but a lot of research has been done since than and thus where needed I would change some things to match the state of the art practices. Most of these things are taught in the fastai courses, namely Practical Deep Learning for Coders, v3.

If you are not familiar with fastai, it is a deep learning library build on top of PyTorch and it contains the implementations of most of the state of the art practices, which keep changing over time. As a result of this you can get state of the art results in most of the tasks by using the defaults of the library.

I would explain all the concepts discussed in the paper and would provide a walkthrough with a CIFAR-100 example along the way. So there would be explanation of the topic and then the code for that. If you are to use these techniques for your own work, you can follow along the notebook from top-to-bottom. For the implementations of some concepts, I would use the fastai built in functions as fastai provides a callback system that really helps a lot when working on projects in real life. So if you do not know fastai, you can watch the course mentioned above or read the docs, as the docs contains ample examples. For Tensorflow users, as of now I am not aware if some library provides this much functionality as fastai but you can still follow on as the concepts discussed are general, only the implementation is different.

You can check the notebook here with all the details and the code.

I am thinking about shifting to jupyter notebooks for these tutorials, as it is easier to experiment in the notebooks.

So if you want to follow along and know when I am making a new notebook, you can check https://kushajveersingh.github.io/notebooks/. I will keep updating the list as I make new notebooks.

I will still make Medium posts but most of the paper implementations will be done in the Jupyter notebooks.