Essay: Evaluating Data Mining Models with WEKA Toolkit

For undergraduate modules, a score above 40% represent a pass performance at honours level.

• For postgraduate modules, a score of 50% or above represents a pass mark.
• Late submission of any item of coursework for each day or part thereof (or for hard copy submission only, working day or part thereof) for up to five days after the published deadline, coursework relating to modules at Levels 0, 4, 5, 6 submitted late (including deferred coursework, but with the exception of referred coursework), will have the numeric grade reduced by 10 grade points until or unless the numeric grade reaches or is 40. Where the numeric grade awarded for the assessment is less than 40, no lateness penalty will be applied.
• Late submission of referred coursework will automatically be awarded a grade of zero (0).
• Coursework (including deferred coursework) submitted later than five days (five working days in the case of hard copy submission) after the published deadline will be awarded a grade of zero (0).

This Assignment assesses the following module Learning Outcomes (from Definitive Module Document):

Successful students will typically:

2. be able to appreciate the strengths and limitations of various data mining models;
3. be able to critically evaluate, articulate and utilise a range of techniques for designing data mining systems;
4. be able to critically evaluate different algorithms and models of data mining.

Assignment Brief:

A dataset of text is provided in the assignment area on Canvas. Analyse this data using the WEKA toolkit and tools introduced within this module, comparing two different forms of preprocessing: Forexample, you may investigate the impact of using stemming, the effect of reducing the number of features, the impact of term frequency over a simple word count, etc.

Complete the following tasks:

1. Describe which question you will be investigating (e.g. “is stemming beneficial to improving performance?”, “is the reduction of features beneficial to improving performance?”, etc.)
2. Convert the text dataset into TWO different databases in ARFF format, based on your chosen question. Explain the conversion techniques and parameters that you have used, along with any other pre-processing you wish to do. (Do not include a screen shot of the attributes in WEKA – you need to describe them.)
3. For each database, produce a table and a graph of classification performance against training set size for the following three classifiers: decision-tree (J48), Naïve Bayes, Support Vector Machine. For the Support-Vector Machine you must determine the kernel,and its parameters.
4. Write a conclusion. You should at least compare the performance of the different learning algorithms on your databases, and answer the question you posed in part (1). Remember to explain the steps you have taken