# CSC 578 HW 6: Convolutional Neural Networks

CSC 578 HW 6: Convolutional Neural Networks

1/5 CSC 578 HW 6: Convolutional Neural Networks
(Version 0.2, with much credit again to Prof. Tomuro)
Here is a dynamic multimedia video description of the assignment
Overview
The objective of this assignment is to enhance your understanding of
Convolutional Neural Networks (CNNs) while gaining experience with Kaggle
competitions at the same time. Kaggle is an online machine learning community. It
provides a large collection of public data sets and hosts competitions to solve data
science and machine learning challenges.
The assignment is to experiment with CNNs and write about your experience on
the hyperparameter tuning and the analysis of results. The CNN experimentation
part is made into a Kaggle inClass competition so that you can compete with
other students in the class. Your assignment submission will consist of your code
and your documentation of your journey of hyperparameter tuning along with the
The competition is to classify the CIFAR-10 image data: one of the most widely
used benchmark datasets for image classification. In this assignment, you will use
CNNs to classify the images. The dataset is already divided into training and test
sets, and you use the training set to train a CNN model (though you further split
the training set to training and validation subsets in the code). After training the
model, you generate predictions for the test set and submit the predictions to the
competition.
Note: the competition will close on Sunday, May 23rd, at 6:59 PM CDT (which is
11:59 pm UTC; 5 hours ahead of CDT).
Also note that this competition is intended for fun. Don't be concerned or take it
personal if you didn't place high in the leaderboard; ranking will not affect your
Code Development
Here's the pdf for it, using TF2.)
Note that there are three places in the code where you choose the
appropriate syntax/line(s) for the version of TensorFlow (TF version 1.X
or 2.Y) you are using. You can use either version of TF, but since not all
CSC 578 HW 6: Convolutional Neural Networks
2/5
TF2's syntax is backward compatible for some operations, you have to
use the right syntax for your version of TF.
For example, Google CoLab is now using version 2.4.1. If you prefer
TensorFlow, you have to install it on your end (to wit, you will get version
2). Or if you have access to an environment in which TensorFlow is
already installed, you should first check the version and choose the
appropriate lines in the code.
Then run the code to ensure it works in your system/environment.
B. Add code in the file to generate predictions for 'test_images', the held-out
input data.
The function to generate predictions is predict(), such as
'model.predict(...)'. You can look at the HW#4 code for syntax.
Save predictions in the code. So you know, they will be stored in a list of
lists in Keras, where each inner list is the prediction for a test image, i.e.,
a list of length 10, carrying the probability of each target class. For
example,
[[1.3942997e-06 1.3935754e-04 5.6055570e-03 6.6085613e-01 6.7078590e-04 2.8833924e-02 3.0388865e-01 5.1890177e-09 3.2569944e-06 8.7058385e-07] [9.5547803e-05 4.3761898e-02 2.9793470e-12 3.3167865e-12 1.0443895e-13 1.6440552e-17 4.0106508e-11 6.3863180e-22 9.5614260e-01 9.3374570e-09]
For a submission to Kaggle, you should create a csv (comma￾separated values) file. You may name the file as you wish, but
ensure that the extension is .csv. Also the content of the file must
be formatted exactly to the following specifications:
The file must have the header on the first row, then the
predictions of test images to follow.
You can simply traverse the saved prediction list from the
beginning and assign an integer sequence from 1 (NOT 0) to
10000 (the size of the test set).
Follow the ID number with the predicted values (10 of them)
for each class.
Repeat for each test image, one per line, as shown below.
C. Notes on hyperparameter tuning.
id,cat0,cat1,cat2,cat3,cat4,cat5,cat6,cat7,cat8,cat9 1,1.3942997e-06,0.00013935754,0.005605557,0.6608561,0.0006707859,0.0 2,9.55478e-05,0.043761898,2.979347e-12,3.3167865e-12,1.0443895e-13,1 ... 10000,4.3202136e-15,1.2549072e-13,1.8496052e-10,2.0123039e-11,5.0501
CSC 578 HW 6: Convolutional Neural Networks
3/5
The model in the start-up code has three convolution layers (and two
maxpooling layers) and one fully-connected layer in addition to the
input and output layers. Some hyperparameters are already plugged in
such as activation functions. You can play with any parameter, and as
many as you like. You may want to start with those mentioned in the
previous assignment HW#4. Some suggestions:
Change the number of filters.
Change the size of filters (e.g. 5*5, 7*7).
Change the size of stride – although you must choose the sizes that
'work'.
If you think the network is overfitting, try adding Dropout layers.
Other ideas include adding regularization and removing layers (to
make the model simpler).
Add BatchNormalization layers. Look at the Keras Documentation
As for automatic optimal parameter search such as GridSearch, you can
do so for the purpose of the competition, but I recommend you play
with one parameter at a time manually and observe the effect to
develop your understanding of the parameter. Automatic parameter
search is knowledge-free, therefore you don't learn much.
Also keep in mind the performance can be measured by other aspects
besides the competition evaluation metric, such as computational speed
and learning stability. Another thing to consider is the model complexity
vs. performance trade-off. Try to think from a higher point of view while
One more. You may want to look into an advanced technique called
Data Augmentation – which inject distorted images to make the model
less overfitting and more robust.
HOWEVER, you must NOT use a pre-trained network (such as VGG net
or MobileNet) in this assignment or competition.
If you want to train longer (i.e. more epochs), you can save the current
model and load it later to resume training. Look at the Keras tutorial
code 'Save and Load' for reference.
D. Notes on the Kaggle competition.
You are allowed to submit a maximum of 4 submissions in a day. The
number is intentionally set to small in order to force you to practice
using the validation set to monitor training, rather than having the
Kaggle results (for the held-out test set) influence your model
development.
Note that you will see the 'public' leaderboard during competition. Then
after the competition closes, the 'private' leaderboard becomes
available. The way it works is the test set is divided (by Kaggle) into
public and private subsets (when the competition is created), and the
CSC 578 HW 6: Convolutional Neural Networks
4/5
public leaderboard shows the ranking based on the performance for the
public test set. This is also to prevent overfitting to the competition
ranking (during competition). The final ranking will be the one for the
private test set, so you know.
You must make a successful submission to the competition. Failure to do
so will have a significant impact on the points that you can earn. Also,
this should go without saying, but you should NOT submit the
sample_submission.csv file which is used as the baseline in the
competition, though you may look at it to compare the formatting.
Your submission to the Kaggle competition will be evaluated with the
'LogLoss' function). LogLoss is defined as: $LogLoss = -\frac{1}{N} \sum_i^N \sum_j^M y_{ij} \cdot \ln(p_{ij})$ where $$N$$ is the number of
images in the test set, $$M$$ is the number of image class labels and \
(ln\) is the natural logarithm. $$y_{ij}$$ is 1 if observation $$i$$ belongs to
class $$j$$ and 0 otherwise, and $$p_{ij}$$ is the predicted probability that
observation $$i$$ belongs to class $$j$$. Since each image is labeled with
exactly one true class in this dataset, the metric is equivalent to
LogLikelihood. With some effort, you may be able to achieve somewhere
in the 0.6 range (or even lower) for this CIFAR-10 dataset.
E. Create a document that describes/reports your model development journey,
Submission
A. A code (Jupyter notebook) file and its html or pdf version. Be sure to add your
name, course/section number at the top of the code file.
Show at least the best model and its full trace of training with charts.
You can leave other models in the code too, but make sure you indicate
the best model clearly.
In general, organize the code file well, by inserting section headers and
points deduction.
B. Documentation.
In pdf.
Minimum 600 words / 2 full pages.
name at the top of the file.
Also at the top of the file, write, in bold, your Kaggle user name (as
displayed in the leaderboard) and ranking/score (public or private, or
both; at the time of writing).
Content:
Description of your best model (the one that produced the best
competition performance).
CSC 578 HW 6: Convolutional Neural Networks
5/5
Description of your journey of hyperparameter tuning. Starting
from the initial model, describe in in depth the hyperparameters
and values you tried and the results you got, and your prior
expectation and reaction to the results. Write insightful comments
as much as you can. A large portion of the grade will be placed on
this part.
Include in the journey description, pick one non-best model that
you caught your attention or want to put forward, and describe the
model and discuss its performance results.
Your final conclusions on the best model.
Your reaction and reflection on this assignment overall (e.g.