0

I'm not sure what the difference between these 2 is: 

classifier = NaiveBayesClassifier.train(train_d)

d1 = (nltk.classify.accuracy(classifier,train_d[:500]))*100
d2 = (nltk.classify.accuracy(classifier,train_d[:600]))*100
d3 = (nltk.classify.accuracy(classifier,train_d[:700]))*100
d4 = (nltk.classify.accuracy(classifier,train_d[:800]))*100
d5 = (nltk.classify.accuracy(classifier,train_d[:900]))*100
d6 = (nltk.classify.accuracy(classifier,train_d[:1000]))*100
d7 = (nltk.classify.accuracy(classifier,train_d[:1100]))*100
d8 = (nltk.classify.accuracy(classifier,train_d[:1200]))*100
d9 = (nltk.classify.accuracy(classifier,train_d[:1300]))*100
d10 = (nltk.classify.accuracy(classifier,train_d[:1400]))*100
dvd_results = [d1,d2,d3,d4,d5,d6,d7,d8,d9,d10]

df1 = pd.DataFrame(list(zip(sample_sizes,dvd_results)),columns=["Sample Size","Accuracy"])
display(df1)

Which gives me the results: 

 Sample Size    Accuracy
0   500     99.400000
1   600     99.500000
2   700     99.285714
3   800     99.000000
4   900     99.111111
5   1000    99.100000
6   1100    99.181818
7   1200    99.250000
8   1300    99.153846
9   1400    99.071429

In comparison to what I would have thought would be the same: 

classifier_d1 = NaiveBayesClassifier.train(train_d[:500])
classifier_d2 = NaiveBayesClassifier.train(train_d[:600])
classifier_d3 = NaiveBayesClassifier.train(train_d[:700])
classifier_d4 = NaiveBayesClassifier.train(train_d[:800])
classifier_d5 = NaiveBayesClassifier.train(train_d[:900])
classifier_d6 = NaiveBayesClassifier.train(train_d[:1000])
classifier_d7 = NaiveBayesClassifier.train(train_d[:1100])
classifier_d8 = NaiveBayesClassifier.train(train_d[:1200])
classifier_d9 = NaiveBayesClassifier.train(train_d[:1300])
classifier_d10 = NaiveBayesClassifier.train(train_d[:1400])
d1 = (nltk.classify.accuracy(classifier_d1,train_d))*100
d2 = (nltk.classify.accuracy(classifier_d2,train_d))*100
d3 = (nltk.classify.accuracy(classifier_d3,train_d))*100
d4 = (nltk.classify.accuracy(classifier_d4,train_d))*100
d5 = (nltk.classify.accuracy(classifier_d5,train_d))*100
d6 = (nltk.classify.accuracy(classifier_d6,train_d))*100
d7 = (nltk.classify.accuracy(classifier_d7,train_d))*100
d8 = (nltk.classify.accuracy(classifier_d8,train_d))*100
d9 = (nltk.classify.accuracy(classifier_d9,train_d))*100
d10 = (nltk.classify.accuracy(classifier_d10,train_d))*100
dvd_results = [d1,d2,d3,d4,d5,d6,d7,d8,d9,d10]

Which gives me the results: 

Sample Size Accuracy
0   500     50.000000
1   600     50.000000
2   700     50.000000
3   800     60.142857
4   900     88.000000
5   1000    93.500000
6   1100    93.785714
7   1200    96.428571
8   1300    97.428571
9   1400    99.071429

I honestly can't see what the difference is between the 2 chunks of code, as they're both already trained by classifier, and it is simply getting the accuracy where it seems to be messing up. Also if someone could fill me in on a reason for why my accuracy would only be 50% for sample sizes 700 and below! Partly because of this I'm going to assume that the 1st chunk is the correct way to do it whilst the 2nd chunk I've just messed up with the classifier. Alas I do not know why!

Improve this question
1
  • Please don't edit your question into an entirely different one; it helps others if you leave it up so others searching can find it. You're welcome to ask another, though!
    – Draconis ♦
    Dec 6, 2018 at 2:27

1 Answer 1

Reset to default
3

In the first case, you're training with the entire data set, then testing on different chunks of it.

In the second case, you're training with chunks of the data set, then testing on the whole thing.

You probably don't want to be doing either. You want to train on as much data as possible, but you never want to test on the same data you trained with (if you do, you'll get incorrectly inflated results).

Try breaking the data into a training set and a test set (a common split is somewhere around 80% training, 20% testing). Train on the entire training set, then test on the entire testing set.

Improve this answer
6
  • So if I wished to see how changing the training data amount impacted my results? I will need to train my classifier over and over again? And if that is the case should I keep my testing data ratio of 80% to 20% throughout? - I will edit post to show what I mean
    – bemzoo
    Dec 6, 2018 at 1:43
  • 1
    @bemzoo If you want to see how changing the training data affects it, you'd want to train several classifiers on different amounts of data, like in your second example. Just make sure not to test on anything you trained with!
    – Draconis ♦
    Dec 6, 2018 at 1:48
  • I've updated it so now I have many different classifiers, each with different amounts of training data. My total training data is 1400 so I've gone up to there, and the amount of test data I have is 600. So I'm now testing the same amount of data against differing training data amounts. Though is that rational? Should I be keeping the ratios intact with each classifier?
    – bemzoo
    Dec 6, 2018 at 1:56
  • @bemzoo Test with as much as you have; normally the 80-20 split is for when you have a limited amount.
    – Draconis ♦
    Dec 6, 2018 at 2:01
  • Thank you Draconis, one last thing do you know why some of my training data sizes seem to have a baseline accuracy of 50% pre sample size of 1000?
    – bemzoo
    Dec 6, 2018 at 2:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.