ISY310 Data Mining And Business Intelligence

0 Download 85 Pages / 21,008 Words

22-09-2021

Question:

Decide on the data to be used for analysis. Criteria include relevance to the data mining goals, quality and technical constraints such as limits on data volume or data types.

Raise the data quality to the level required by the selected analysis techniques. This may involve selection of clean subsets of the data, the insertion of suitable defaults or more ambitious techniques such as the estimation of missing data by modeling.

Describe the decisions and actions that were taken to address the data quality problems reported during the Verify Data Quality Task. The report should also address what data quality issues are still outstanding.

Answer:

During the analysis, the set of data that are to be used has been collected from students of the university on the form of survey. In the survey the main roles played is by uncovering the modeling techniques and various analytics Data Mining. For the completion of this report, all the methods will be utilized as well as for the conventional questions to be answered, which are required to complete the analysis of this report. All these data sets are found in the survey process collection.

Defining the complex data while keeping in mind the goal of the task and hence gathering the data of the certain areas and data rows. The total collected data rows were almost 2000 that has been collected from different participants in the surveys. The data rows that have been removed had the following reasons:

A row in the age group has a value of -548. Since this, an invalid range of data thus the value is unacceptable and is discarded from the row.
The ‘Country of Birth’ column collected data for the value {0, 1, 2, 3}.
Data present in the column are in the area of major, minor and residence.
Blank data in rows cannot be utilized in the analysis.

Cleaning the data sets has been done on considering the above point and finally 1918 rows of data could be considered appropriate for the final analysis of the report.

Within the data sets all the data forms have been changed, if it was done within the texts the value could have hampered the process of creating the model building procession. The collected data has been used keeping in mind the clearance that has been done because of the unwanted data, which cannot be utilized in the analysis any further.

No new construction has been done in the column of data since all the data set has been gathered from the survey are thoroughly checked. Hence, new columns were not added in the data sets any further.

Excel sheet has been used to compile the above chart. Acceptance of the loan against the mode of study mapping has been done for the display in the chart. Acceptance of loan values, 0 as NO and 1 as YES. The mode of study determination can be further described as: 0 is F/T; 1 as P/T and 2 as external. As per the records of the chart given the highest values has been displayed of the student who are not willing to accept the loans and on a full time basis are willing to complete the course.

Excel sheet has been used to compile the above chart. From the chart we can observe that the students agreeing for the acceptance of loans depends on the mode of payment, since reliable the method easier to pay. Acceptance of loan values, 0 as NO and 1 as YES. The mode of payment 0 is HECS, 1 as by own medium and 2 for other. As per the record, the highest numbers of the students are of those who are willing to accept the loan technique and pay by HECS.

Excel sheet has been used to compile the above chart. From the chart, we can observe the loan acceptance of the students with respect to the current GPA of the student. Acceptance of loan values, 0 as NO and 1 as YES. Range of GPA ranges from 1 to 4, with each count of slabs. As per the records in the charts the highest number has been displayed, the records are of the students who are not willing to take loans and have GPA valuing 2.

There are two parts in which the splitting has been done in the data set analysis based on the characteristics of 66% of the whole set of data. In model building process data sets are used and rest of the data are used for the test procedure execution processes. Weka has been used for the divisions and thus manual divisions were not required. After this analysis the reports starts proceeding towards the building section of the models.

The analysis used three major algorithms:

Linear Regression: A linear approach that is used in the modeling to establish between an independent variable and a scalar variable.
Multi-Layer Perception: artificial to the nature this is known to be a feedback to the neutral network that consists of minimum 3 layers.
Random Tree: Selecting the major sets and has the possibility of turning into a tree are possibly be modeled.

For further analysis has the parameter in the column of “would accept loan”. It has been helpful to understand the perspective of the students about their willingness to take the loan or not from the university bank.

The rule-based analysis is the decision tree:

=== Run information ===

Scheme: weka.classifiers.trees.RandomTree -K 0 -M 1.0 -V 0.001 -S 1

Relation: data_clean

Instances: 1918

Attributes: 16

Gender

Hours worked

Age

Months in current job

credit rating

Number of loans applied for

Number of Current loans

Country of Birth

Area of residence

Current GPA

Major

Minor

Mode of study

Mode of payment

Number of Dependents

Would accept loan

Test mode: split 66.0% train, remainder test

=== Classifier model (full training set) ===

RandomTree

==========

Hours worked < 95.5

| credit rating < 19

| | Mode of study < 0.5

| | | Hours worked < 41.5

| | | | Major < 0.5

| | | | | | Age < 65.5

| | | | | | | | | | Age < 21 : 0 (2/0)

| | | | | | | | | | Age >= 21 : 1 (1/0)

| | | | | | | | Age < 22.5

| | | | | | | | Age >= 22.5 : 0 (3/0)

| | | | | | Age >= 65.5

| | | | Major >= 0.5

| | | Hours worked >= 41.5

| | | | Hours worked < 63.5 : 0 (59/0)

| | | | Hours worked >= 63.5

| | | | | | | Age < 48

| | | | | | | | | | | | Age < 29.5 : 0 (1/0)

| | | | | | | | | | | | Age >= 29.5 : 1 (1/0)

| | | | | | | | | Age < 39 : 0 (47/0)

| | | | | | | | | Age >= 39

| | | | | | | Age >= 48 : 0 (21/0)

| | Mode of study >= 0.5

| | | Current GPA < 1.5 : 1 (1/0)

| | | Current GPA >= 1.5 : 0 (2/0)

| credit rating >= 19

| | Age < 20.5

| | | Country of Birth < 0.5 : 0 (6/0)

| | | Country of Birth >= 0.5

| | | | Number of loans applied for < 0.5

| | | | Number of loans applied for >= 0.5 : 0 (3/0)

| | Age >= 20.5 : 0 (101/0)

Hours worked >= 95.5

| Number of Dependents < 0.5

| | Age < 37.5

| | | Age < 22.5

| | | | Months in current job < 2.5

| | | | | | | Age < 21.5

| | | | | | | Age >= 21.5 : 0 (7/0)

| | | | | | | | | Age < 20.5

| | | | | | | | | | | Age < 11 : 0 (1/0)

| | | | | | | | | | | Age >= 11 : 1 (1/0)

| | | | | | | | | Age >= 20.5 : 1 (11/0)

| | | | | | | Age < 21 : 0 (22/0)

| | | | | | | Age >= 21

| | | | Months in current job >= 2.5

| | | | | | | Age < 11.5

| | | | | | | Age >= 11.5

| | | | | | | | | | | | | | | | Age < 20.5

| | | | | | | | | | | | | | | | Age >= 20.5 : 0 (1/0)

| | | | | | | | | | | | | | | | Age < 20.5

| | | | | | | | | | | | | | | | Age >= 20.5 : 0 (6/0)

| | | | | | | | | | | | | | | | | Age < 21.5 : 0 (22/0)

| | | | | | | | | | | | | | | | | Age >= 21.5

| | | | | | | | | | | Number of loans applied for >= 0.5 : 0 (41/0)

| | | | | | | | | | | Number of loans applied for < 0.5 : 1 (1/0)

| | | | | | | | | | Age < 20.5

| | | | | | | | | | | | | Number of loans applied for < 0.5 : 1 (1/0)

| | | | | | | | | | | | | Number of loans applied for >= 0.5 : 0 (1/0)

| | | | | | | | | | Age >= 20.5

| | | | | | | | | | Number of loans applied for < 0.5 : 0 (1/0)

| | | | | | | | | | | | Age < 21 : 0 (1/0)

| | | | | | | | | | | | Age >= 21 : 1 (1/0)

| | | | | | | | | | | | | | | | | Number of loans applied for >= 2.5 : 0 (3/0)

| | | Age >= 22.5

| | | | Country of Birth < 0.5

| | | | | | | Age < 24.5

| | | | | | | | | | Age < 23.5

| | | | | | | | | | Age >= 23.5 : 0 (7/0)

| | | | | | | | | | Age < 23.5

| | | | | | | | | | Age >= 23.5

| | | | | | | Age >= 24.5

| | | | | | | | Age < 36.5

| | | | | | | | Age >= 36.5

| | | | | | | | | Age < 23.5

| | | | | | | | | Age >= 23.5 : 0 (7/0)

| | | | | | | | | | Age < 33 : 1 (1/0)

| | | | | | | | | | Age >= 33 : 0 (1/0)

| | | | | | | | Age < 33

| | | | | | | | | | | Age < 23.5 : 0 (1/0)

| | | | | | | | | | | Age >= 23.5

| | | | | | | | Age >= 33

| | | | | | | | | | Age < 35

| | | | | | | | | | Age >= 35 : 0 (3/0)

| | | | | | | | | Number of loans applied for >= 2 : 0 (1/0)

| | | | | | | | Age < 24.5

| | | | | | | | | Age < 23.5 : 0 (14/0)

| | | | | | | | | Age >= 23.5

| | | | | | | | Age >= 24.5 : 0 (21/0)

| | | | | | | Number of Current loans >= 0.5 : 0 (24/0)

| | | | Country of Birth >= 0.5

| | | | | | | | | Age < 33 : 0 (43/0)

| | | | | | | | | Age >= 33

| | Age >= 37.5

| | | Age < 61

| | | | Months in current job < 2.5

| | | | | | | Age < 57.5

| | | | | | | Age >= 57.5 : 1 (4/0)

| | | | Months in current job >= 2.5

| | | | | | | | | | Age < 59.5 : 0 (7/0)

| | | | | | | | | | Age >= 59.5 : 1 (1/0)

| | | | | | | | Number of loans applied for >= 0.5 : 0 (4/0)

| | | | | | | | | | | | | Number of loans applied for < 1.5 : 0 (1/0)

| | | | | | | | | | | | | Number of loans applied for >= 1.5 : 1 (1/0)

| | | | | | | | | | | | Age < 39 : 0 (1/0)

| | | | | | | | | | | | Age >= 39 : 1 (2/0)

| | | | | | | | Age < 57.5

| | | | | | | | Age >= 57.5

| | | | | | | | Age < 56.5

| | | | | | | | Age >= 56.5 : 0 (3/0)

| | | | | | | | | | Age < 38.5

| | | | | | | | | | | Number of loans applied for >= 0.5 : 1 (1/0)

| | | | | | | | | | Age >= 38.5 : 0 (3/0)

| | | | | | | | | | Age < 58.5 : 1 (1/0)

| | | | | | | | | | Age >= 58.5 : 0 (1/0)

| | | | | | | | | | | | | Age < 56

| | | | | | | | | | | | | Age >= 56 : 0 (9/0)

| | | | | | | | | | | | Age < 59 : 0 (9/0)

| | | | | | | | | | | | Age >= 59 : 1 (1/0)

| | | Age >= 61

| | | | Major < 1.5

| | | | | | Age < 69.5

| | | | | | | | | | | | | | Age < 64 : 0 (2/0)

| | | | | | | | | | | | | | Age >= 64

| | | | | | | | | | | | Age < 62.5 : 1 (1/0)

| | | | | | | | | | | | Age >= 62.5

| | | | | | | | Age < 66.5

| | | | | | | | | Age < 62.5 : 1 (1/0)

| | | | | | | | | Age >= 62.5

| | | | | | | | Age >= 66.5 : 1 (3/0)

| | | | | | Age >= 69.5 : 0 (12/0)

| | | | Major >= 1.5 : 1 (3/0)

| Number of Dependents >= 0.5

| | credit rating < 29 : 0 (7/0)

| | credit rating >= 29

| | | Hours worked < 446.5

| | | | Minor < 1.5

| | | | | | | | | | Number of loans applied for < 2.5 : 1 (1/0)

| | | | | | | | | | Number of loans applied for >= 2.5 : 0 (1/0)

| | | | | | | | Age < 23.5 : 0 (5/0)

| | | | | | | | Age >= 23.5

| | | | | | | | Age < 22.5 : 0 (2/0)

| | | | | | | | Age >= 22.5

| | | | | | | | | | Number of loans applied for < 0.5 : 0 (4/0)

| | | | Minor >= 1.5

| | | | | Number of loans applied for < 0.5 : 1 (3/0)

| | | | | | Age < 22.5

| | | | | | Age >= 22.5 : 0 (6/0)

| | | Hours worked >= 446.5 : 1 (1/0)

Correlation coefficient 0.1812

Mean absolute error 0.1771

Root mean squared error 0.4204

Relative absolute error 92.9332 %

Root relative squared error 138.4155 %

Total Number of Instances 652

The multilayer perception (neural network topology) is a form of opaque models:

=== Run information ===

Scheme: weka.classifiers.functions.MultilayerPerceptron -L 0.3 -M 0.2 -N 500 -V 0 -S 0 -E 20 -H a

Relation: data_clean

Instances: 1918

Attributes: 16

Gender

Hours worked

Age

Months in current job

credit rating

Number of loans applied for

Number of Current loans

Country of Birth

Area of residence

Current GPA

Major

Minor

Mode of study

Mode of payment

Number of Dependents

Would accept loan

Test mode: split 66.0% train, remainder test

=== Classifier model (full training set) ===

Linear Node 0

Inputs Weights

Threshold -0.7934874747063402

Node 1 1.5863171423229618

Node 2 1.933436142639909

Node 3 0.15393051522400075

Node 4 1.1407746615953174

Node 5 2.2277342441589583

Node 6 0.32932988228714843

Node 7 1.8980166584005944

Node 8 3.375913912274132

Sigmoid Node 1

Inputs Weights

Threshold -17.633691718807214

Attrib Gender -0.6063023892897429

Attrib Hours worked 7.769587442011242

Attrib Age -1.9705291676168797

Attrib Months in current job -8.027325718113659

Attrib credit rating 2.8555606542937024

Attrib Number of loans applied for 4.615692836961333

Attrib Number of Current loans -6.8789009342525205

Attrib Country of Birth -5.10611624719799

Attrib Area of residence -8.01950853056682

Attrib Current GPA 13.842612102565939

Attrib Major -4.446132689206738

Attrib Minor -0.28767537744344157

Attrib Mode of study 6.944442535352257

Attrib Mode of payment 1.0089412379034852

Attrib Number of Dependents 9.768092701218661

Sigmoid Node 2

Inputs Weights

Threshold -16.662521676828288

Attrib Gender -1.8569101323430144

Attrib Hours worked 12.553016217247958

Attrib Age -7.047755961381406

Attrib Months in current job -1.1814674694863674

Attrib credit rating -27.004113126631356

Attrib Number of loans applied for 2.108416633761098

Attrib Number of Current loans 0.4394847112117454

Attrib Country of Birth -10.09760708591061

Attrib Area of residence -4.319783310655949

Attrib Current GPA -1.1393223312855167

Attrib Major -1.9256158427839145

Attrib Minor 10.842215576061717

Attrib Mode of study 10.538531092664792

Attrib Mode of payment -2.6448169101240184

Attrib Number of Dependents 14.427558862848413

Sigmoid Node 3

Inputs Weights

Threshold -15.180133064782485

Attrib Gender -11.85854190242291

Attrib Hours worked -5.642654054575123

Attrib Age 6.734479452913219

Attrib Months in current job 6.644944391003227

Attrib credit rating -1.5772815051610865

Attrib Number of loans applied for -4.451480623271361

Attrib Number of Current loans -3.4140282991401145

Attrib Country of Birth 0.6212144172375925

Attrib Area of residence 4.975731597001943

Attrib Current GPA 0.8696636116516945

Attrib Major 13.60039960988846

Attrib Minor 8.463929640984745

Attrib Mode of study 0.8796336138382383

Attrib Mode of payment -2.4352022646989653

Attrib Number of Dependents -3.485405604800929

Sigmoid Node 4

Inputs Weights

Threshold -20.702433133742577

Attrib Gender 8.339707995607522

Attrib Hours worked 6.206277387030351

Attrib Age 15.453258268362376

Attrib Months in current job -3.805045342827591

Attrib credit rating 11.19564486368074

Attrib Number of loans applied for -8.306517782295417

Attrib Number of Current loans -9.648072497592967

Attrib Country of Birth -8.932812796364326

Attrib Area of residence -1.1495312804303115

Attrib Current GPA 2.4227049249038237

Attrib Major 3.5155383736363297

Attrib Minor 9.042089557017377

Attrib Mode of study -2.129547650242301

Attrib Mode of payment -1.5910651838042669

Attrib Number of Dependents 11.321894013448372

Sigmoid Node 5

Inputs Weights

Threshold -17.244449953145104

Attrib Gender -4.783529172262689

Attrib Hours worked 5.038608731398995

Attrib Age 11.574330545482498

Attrib Months in current job 7.876586149178689

Attrib credit rating -2.0759988797110047

Attrib Number of loans applied for 11.534641041850826

Attrib Number of Current loans 3.2997533417585627

Attrib Country of Birth -3.5132256401850714

Attrib Area of residence -4.542441660925225

Attrib Current GPA -2.018915695070438

Attrib Major -1.0841011712890576

Attrib Minor -6.761690424410381

Attrib Mode of study -4.654800320178031

Attrib Mode of payment -0.9976538120691812

Attrib Number of Dependents 7.612288733619478

Sigmoid Node 6

Inputs Weights

Threshold -6.137098232720996

Attrib Gender -5.716361951315588

Attrib Hours worked 0.6446702039710765

Attrib Age 13.730044805406774

Attrib Months in current job -10.985504521926952

Attrib credit rating -0.8736645257369247

Attrib Number of loans applied for -1.4799999919776694

Attrib Number of Current loans -0.15622368042300766

Attrib Country of Birth 3.8405253978569744

Attrib Area of residence 5.501185218542164

Attrib Current GPA 3.4433435667248062

Attrib Major 7.791172520388

Attrib Minor -5.4161975993087195

Attrib Mode of study -1.0072208481631482

Attrib Mode of payment 1.9816853875908345

Attrib Number of Dependents -0.5650051980405012

Sigmoid Node 7

Inputs Weights

Threshold -15.797389333164505

Attrib Gender -3.7311214548937186

Attrib Hours worked -5.729013058124828

Attrib Age -6.670133334291946

Attrib Months in current job -12.333635957861258

Attrib credit rating -10.650408713981454

Attrib Number of loans applied for 11.382376535313366

Attrib Number of Current loans -3.03469746185878

Attrib Country of Birth -12.894772818107546

Attrib Area of residence -5.1331761080267135

Attrib Current GPA 1.820379876394602

Attrib Major 14.72059133688235

Attrib Minor -1.149652204631702

Attrib Mode of study 8.948749469595402

Attrib Mode of payment 8.014480302654984

Attrib Number of Dependents -6.6864334005121755

Sigmoid Node 8

Inputs Weights

Threshold -14.949834298329886

Attrib Gender -3.5991344246252295

Attrib Hours worked -3.7810856106446775

Attrib Age 7.161906999607669

Attrib Months in current job 2.0342436041298284

Attrib credit rating 3.7353519159896997

Attrib Number of loans applied for -8.651771591411611

Attrib Number of Current loans -1.2697881898939134

Attrib Country of Birth 3.2686778154025995

Attrib Area of residence -4.001677096418436

Attrib Current GPA 9.083856263353365

Attrib Major 6.672251464020582

Attrib Minor 2.8087590082515224

Attrib Mode of study 0.5598027035044083

Attrib Mode of payment 6.694634673793227

Attrib Number of Dependents -1.7862681319840856

Correlation coefficient 0.2197

Mean absolute error 0.1839

Root mean squared error 0.4119

Relative absolute error 96.4854 %

Root relative squared error 135.6216 %

Total Number of Instances 652

Linear regression also falls under the form of opaque model:

Scheme: weka.classifiers.functions.LinearRegression -S 0 -R 1.0E-8 -num-decimal-places 4

Relation: data_clean

Instances: 1918

Attributes: 16

Gender

Hours worked

Age

Months in current job

credit rating

Number of loans applied for

Number of Current loans

Country of Birth

Area of residence

Current GPA

Major

Minor

Mode of study

Mode of payment

Number of Dependents

Would accept loan

-0.0782 * Gender +

0.0001 * Hours worked +

0.0026 * Age +

-0.0078 * Months in current job +

-0.0022 * credit rating +

-0.0254 * Area of residence +

0.1029 * Major +

0.0665 * Minor +

0.08 * Number of Dependents +

0.1331

Correlation coefficient 0.2262

Mean absolute error 0.1859

Root mean squared error 0.2981

Relative absolute error 97.5223 %

Root relative squared error 98.1293 %

Total Number of Instances 652

The analyzed data depends on the column of “would accept loan”. The survey participants were selected based on the analysis of the university that resembles some form of loans. Based on the analysis we can describe weather the student is able to accept the loan or not, again on full time basis or per time basis. Single kind of data is not sufficient to make the decision in the case of analysis and thus the report to contain the collected data. Hence, the analysis has been done based on collected data sets. The equation that stands for this report: “RESULTS = MODELS + FINDINGS”.

The report contains three forms of model building process:

Linear Regression
Random tree
Multi-Layer Perception

In the required analytical data set can be used as model of Linear Regression for its simplicity. Multi-Layer Perceptions time consuming helps in finding the data. This also requires many training completion for the analysis of the Random Tree in data sets. This is visualized on large scale, which is more difficult to understand keeping in mind the best outcome that we can get from the analysis.

Bibliography

D’Oca, S. and Hong, T., 2015. Occupancy schedules learning process through a data mining framework. Energy and Buildings, 88, pp.395-408.

Dua, S. and Du, X., 2016. Data mining and machine learning in cybersecurity. CRC press.

Fan, C., Xiao, F. and Wang, S., 2014. Development of prediction models for next-day building energy consumption and peak power demand using data mining techniques. Applied Energy, 127, pp.1-10.

Fournier-Viger, P., Gomariz, A., Gueniche, T., Soltani, A., Wu, C.W. and Tseng, V.S., 2014. SPMF: a Java open-source pattern mining library. The Journal of Machine Learning Research, 15(1), pp.3389-3393.

García, S., Luengo, J. and Herrera, F., 2015. Data preprocessing in data mining (pp. 59-139). New York: Springer.

Ilardi, E.A., Vitaku, E. and Njardarson, J.T., 2013. Data-mining for sulfur and fluorine: An evaluation of pharmaceuticals to reveal opportunities for drug design and discovery: Miniperspective. Journal of medicinal chemistry, 57(7), pp.2832-2842.

Rutkowski, L., Pietruczuk, L., Duda, P. and Jaworski, M., 2013. Decision trees for mining data streams based on the McDiarmid's bound. IEEE Transactions on Knowledge and Data Engineering, 25(6), pp.1272-1279.

Xiao, F. and Fan, C., 2014. Data mining in building automation system for improving building operational performance. Energy and buildings, 75, pp.109-118.