FIT5149 2021 S2 Assessment 2
Scientific Document Classification
Sep 2021
Marks 35% of all marks for the unit
Due Date 23:55 Friday 22 Oct 2021
Extension
An extension could be granted for circumstances.
Please refer to the university webpage on special
consideration. A special consideration application form
must be submitted. Please note that ALL special
consideration, including within the semester, is now to
be submitted centrally. All students MUST submit an
online special consideration form via Monash Connect.
Lateness
For all assessment items handed in after the official
due date, and without an agreed extension, a 10%
penalty applies to the student’s mark for each day
after the due date (including weekends, and public
holidays) for up to 5 days. Assessment items handed in
after 5 days will not be considered/marked.
Authorship
This assignment is a group assignment and the final
submission must be identifiable your group’s own work.
Breaches of this requirement will result in an
assignment not being accepted for assessment and
many result in disciplinary action.
Submission
Each group is required to submit two files, one PDF
file contains the report, and another is a ZIP file
containing the implementation and the other required
files. The two files must be submitted via Moodle. All
the group members are required to log in Moodle to
accept the terms and conditions in the Moodle
submission page. A draft submission won’t be marked.
Programming
language
Either R or Python
Note: Please read the description from the start to the end carefully before you start your work!
Given that it is a group assessment, each group should evenly distribute the work among
all the group members.
1
1 Introduction
Scientific document classification is a key step for managing research articles and papers in forums
like arxiv, Google Scholar and Microsoft Academic. In this assessment, you are given some abstracts
crawled from American Chemical Society, the task is to develop classification models which can
make predictions and return the corresponding scientific fields of the source documents. Different
from coarse grained classification tasks like sentiment analysis, this is a fine grained classification
task where there are 19 filed classes in total. There are many machine learning methods that can
be used in the classification task. They can be categorised into supervised method (like SVM) and
unsupervised method (like clustering). Figure 1 shows a typical framework used in the supervised
classification.1
Figure 1: A general framework for the supervised classification.
As shown in the figure, there are three major steps, including generating features, developing a
proper classifier, and applying the classifier to the unseen data. The feature extractor is shared by
both training and prediction, which tells us that data used in training and prediction should share
the same feature space.
The aim of this challenge is to develop a classifier that can assign a set of scientific abstracts
to their corresponding labels as correctly as possible.
2 Dataset
Data Source Data Type classes num. training examples num. testing examples
ACS Material abstracts 19 90,000 10,000
Table 1: Authorship Profiling data set.
We provide the following data sets (Table 1):
train data labels.csv contains training ids , abstracts and labels. It contains abstracts from
90,000 articles and acts as the training data.
test data.csv: only testing ids and abstracts are available. It contains the abstracts from
10,000 articles.
1The figure is download from https://www.nltk.org/book/ch06.html
2
Warning: Reverse engineering on the provided dataset is not allowed! Any information about
the test data cannot be used in training the classifiers.
3 Data Preparation & Feature Extration
Selecting relevant features and deciding how to encode them for a classification algorithm is crucial
for learning a good model. Free language text cannot be used directly as input to classification
algorithms. It must be pre-processed and transformed into a set of features represented in a
numerical form. In this section, we will discuss the basic text pre-processing steps and the common
features used in text classification.
The most common and basic pre-processing steps include
Case normalization: Text can contain upper- or lowercase letters. It is a good idea to just
allow either uppercase or lowercase.
Tokenization is the process of splitting a stream of text into individual words.
Stopwords are words that are extremely common and carry little lexical content. The list
of English stop words can be downloaded from the Internet. For example, a comprehensive
stop-word list can be found from Kevin Bouge’s website2.
Removing the most/least frequent word: Besides the stopwords, we usually remove words
appearing in more than 95% of the documents and less than 5% of the documents as well.
The percentages can be varied for corpus to corpus.
Those are only the common steps used in pre-processing text. Please note that the steps are of
your choice and there is no limitation on the pre-processing steps you can use in the
task.
Next, what kind of features one can extract from the free language text for document clas-
sification? There are some common features often considered in document classification, which
include
N -gram feature3: N -grams are basically a set of co-occurring words within a given window.
For example, for the sentence “The cow jumps over the moon”, if N = 2 (known as bigrams),
then the n-grams would be “the cow”, “cow jumps”, “jumps over”, “over the”, “the moon”.
If N = 3 (known as trigram), the n-grams would be “the cow jumps”, “cow jumps over”,
“jumps over the”, “over the moon”.
Unigram feature: a case of N-grams, if N = 1. Given the above sentence, the unigrams are
“The”, “cow”, “jumps”, “over”, “the”, “moon”.
POS tags4: part-of-speech annotation.
TF-IDF5 (Term Frequency-Inverse Document Frequency): It is a measure of how important
a word/n-gram is to a document in a collection.
You can choose to use either an individual feature or the combination of multiple features. The
features listed above are candidate features that you could consider in the task. However, you
can go beyond those features and try to find the set of features that can give you the best possible
classification accuracy.
There are many useful online tutorials on text preprocessing in either R or Python, for example,
Feature extraction in Scikit-learn6
Working with text data7
2https://sites.google.com/site/kevinbouge/stopwords-lists
3https://www.tidytextmining.com/ngrams.html
4martinschweinberger.de/docs/articles/PosTagR.pdf
5https://www.tidytextmining.com/tfidf.html
6https://scikit-learn.org/stable/modules/feature_extraction.html
7https://scikit-learn.org/stable/tutorial/text_analytics/working_with_text_data.html
3
R code: reading, pre-processing and counting text8
“Text Mining with R”9, a tutorial that discusses how the deal with text in R. It provides
compelling examples of real text mining problems
4 Classifier
The task is to develop a classifier that can give you the most accurate prediction in the scientific
document classification task. The algorithm that you can use are not limited to the algorithms
covered in the lectures/tutorials. The goal at this stage is to find the most accurate classifier. In
order to find the most accurate classifier, each group should empirically compare at least 3 different
types of classification methods in the context of scientific document classification, and then submit
the one perform the best in your comparison. Please note an algorithm with different input features
will only count as one type of classifier. For example, logistic regression will be count as one type
of classifier, no matter what features you use.
5 Evaluation
The evaluation method used in testing is the accuracy score, which is defined as the proportion of
correct predictions among all of the predictions.
Accuracy =
number of correct predictions
number of all predictions
You can use the existing python/R code to compute the Accuracy score, for example
Accuracy score in Python10
Accuracy score in R11
6 Submission
To finish this data analysis challenge, all the groups are required to submit the following files:
“pred labels.csv”, where the label prediction on the testing documents is stored.
– In your “pred labels.csv”, there must be two columns: the first one is the test id column,
and the second one is the label column. Remember the first row of your “pred labels.csv”
file should be “test id” and “label”.
– The “pred labels.csv” must be reproducible by the assessor with your submitted R/Python
code.
The R/Python implementation of your final classifier with A README file that tells
the assessor how to set up and run your code. The output of your implementation must
include the label prediction for all the testing documents. The use of Jupyter notebook or R
Markdown is not required. All the files that are required for running your implementation
must be compressed into a zip file, named as “groupName ass2 impl.zip”. Please note
that the unnecessary code must be excluded in your final submission. For example, if you
tried three different types of models, say multinomial regression, LDA and classification tree,
and your group decides to submit LDA as the final model, you should remove the code for
the other two models from the submission. The discussion of the comparison should
be included in your report. However, you should keep a copy of the implementation used
for comparison for the purpose of the interview.
8http://www.katrinerk.com/courses/words-in-a-haystack-an-introductory-statistics-course/
schedule-words-in-a-haystack/r-code-the-text-mining-package
9https://www.tidytextmining.com/index.html
10https://scikit-learn.org/stable/modules/generated/sklearn.metrics.accuracy_score.html
11https://www.rdocumentation.org/packages/rfUtilities/versions/2.1-4/topics/accuracy
4
A PDF report, where you should document in details the development of the submitted
classifier. The maximum number of pages allowed is 8. The report must be in the
PDF format, named a “groupdName ass2 report.pdf”. The report must include (but
not limited to)
– The discussion of how the data preprocessing/features selection has been done.
– The development of the submitted classifier: To choose an optimal classifier for a task,
we often carry out empirical comparisons of multiple candidate models with different
feature sets. In your report, you should include a comprehensive analysis of how the
comparisons are done. For example, the report can include (but not limited to)
A description of the classifier(s) considered in your comparison.
The detailed experimental settings, which could include, for example, the discussion
of how the cross-validation is set up, how the parameters for the model considered
(if applicable) are chosen, or the setting of semi-supervised learning (if applicable).
The semi-supervised learning process.
Classification accuracy with comprehensive discussion.
The justification of the final model submitted.
Warning: If a report exceeds the page limit, the assessment will only be based on the first
8 pages.
A signed group assignment cover sheet, which will also be included in your zip file.
Warning: typing name is not counted as a signature in the cover sheet.
7 How to submit the files?
The Moodle setup allows you to upload only two files
“groupdName ass2 report.pdf”: A pdf report file, which will be submitted to Turnitin.
“groupName ass2 impl.zip”’: a zip file includes
– the implementation of the final submitted model
– “predict label.csv”, where the label prediction on the testing documents is stored.
– the signed grouped assignment cover sheet
While submitting your assignment, you can ignore the Turnitin warning message generated for
the ZIP file.
Please note that
Only one group member need to upload the two files. But all the group members
have to login to their own Moodle page and click the submit button in order to
make the final submission. If anyone member does not click the submit button, the
uploaded files will remain as a draft submission. A draft submission won’t be marked!
The two files must be uploaded separately.