I'm not sure what the difference between these 2 is:

classifier = NaiveBayesClassifier.train(train_d)

d1 = (nltk.classify.accuracy(classifier,train_d[:500]))*100
d2 = (nltk.classify.accuracy(classifier,train_d[:600]))*100
d3 = (nltk.classify.accuracy(classifier,train_d[:700]))*100
d4 = (nltk.classify.accuracy(classifier,train_d[:800]))*100
d5 = (nltk.classify.accuracy(classifier,train_d[:900]))*100
d6 = (nltk.classify.accuracy(classifier,train_d[:1000]))*100
d7 = (nltk.classify.accuracy(classifier,train_d[:1100]))*100
d8 = (nltk.classify.accuracy(classifier,train_d[:1200]))*100
d9 = (nltk.classify.accuracy(classifier,train_d[:1300]))*100
d10 = (nltk.classify.accuracy(classifier,train_d[:1400]))*100
dvd_results = [d1,d2,d3,d4,d5,d6,d7,d8,d9,d10]

df1 = pd.DataFrame(list(zip(sample_sizes,dvd_results)),columns=["Sample Size","Accuracy"])
display(df1)

Which gives me the results:

 Sample Size    Accuracy
0   500     99.400000
1   600     99.500000
2   700     99.285714
3   800     99.000000
4   900     99.111111
5   1000    99.100000
6   1100    99.181818
7   1200    99.250000
8   1300    99.153846
9   1400    99.071429

In comparison to what I would have thought would be the same:

classifier_d1 = NaiveBayesClassifier.train(train_d[:500])
classifier_d2 = NaiveBayesClassifier.train(train_d[:600])
classifier_d3 = NaiveBayesClassifier.train(train_d[:700])
classifier_d4 = NaiveBayesClassifier.train(train_d[:800])
classifier_d5 = NaiveBayesClassifier.train(train_d[:900])
classifier_d6 = NaiveBayesClassifier.train(train_d[:1000])
classifier_d7 = NaiveBayesClassifier.train(train_d[:1100])
classifier_d8 = NaiveBayesClassifier.train(train_d[:1200])
classifier_d9 = NaiveBayesClassifier.train(train_d[:1300])
classifier_d10 = NaiveBayesClassifier.train(train_d[:1400])
d1 = (nltk.classify.accuracy(classifier_d1,train_d))*100
d2 = (nltk.classify.accuracy(classifier_d2,train_d))*100
d3 = (nltk.classify.accuracy(classifier_d3,train_d))*100
d4 = (nltk.classify.accuracy(classifier_d4,train_d))*100
d5 = (nltk.classify.accuracy(classifier_d5,train_d))*100
d6 = (nltk.classify.accuracy(classifier_d6,train_d))*100
d7 = (nltk.classify.accuracy(classifier_d7,train_d))*100
d8 = (nltk.classify.accuracy(classifier_d8,train_d))*100
d9 = (nltk.classify.accuracy(classifier_d9,train_d))*100
d10 = (nltk.classify.accuracy(classifier_d10,train_d))*100
dvd_results = [d1,d2,d3,d4,d5,d6,d7,d8,d9,d10]

Which gives me the results:

Sample Size Accuracy
0   500     50.000000
1   600     50.000000
2   700     50.000000
3   800     60.142857
4   900     88.000000
5   1000    93.500000
6   1100    93.785714
7   1200    96.428571
8   1300    97.428571
9   1400    99.071429

I honestly can't see what the difference is between the 2 chunks of code, as they're both already trained by classifier, and it is simply getting the accuracy where it seems to be messing up. Also if someone could fill me in on a reason for why my accuracy would only be 50% for sample sizes 700 and below! Partly because of this I'm going to assume that the 1st chunk is the correct way to do it whilst the 2nd chunk I've just messed up with the classifier. Alas I do not know why!

  • Please don't edit your question into an entirely different one; it helps others if you leave it up so others searching can find it. You're welcome to ask another, though! – Draconis Dec 6 at 2:27
up vote 3 down vote accepted

In the first case, you're training with the entire data set, then testing on different chunks of it.

In the second case, you're training with chunks of the data set, then testing on the whole thing.

You probably don't want to be doing either. You want to train on as much data as possible, but you never want to test on the same data you trained with (if you do, you'll get incorrectly inflated results).

Try breaking the data into a training set and a test set (a common split is somewhere around 80% training, 20% testing). Train on the entire training set, then test on the entire testing set.

  • So if I wished to see how changing the training data amount impacted my results? I will need to train my classifier over and over again? And if that is the case should I keep my testing data ratio of 80% to 20% throughout? - I will edit post to show what I mean – bemzoo Dec 6 at 1:43
  • 1
    @bemzoo If you want to see how changing the training data affects it, you'd want to train several classifiers on different amounts of data, like in your second example. Just make sure not to test on anything you trained with! – Draconis Dec 6 at 1:48
  • I've updated it so now I have many different classifiers, each with different amounts of training data. My total training data is 1400 so I've gone up to there, and the amount of test data I have is 600. So I'm now testing the same amount of data against differing training data amounts. Though is that rational? Should I be keeping the ratios intact with each classifier? – bemzoo Dec 6 at 1:56
  • @bemzoo Test with as much as you have; normally the 80-20 split is for when you have a limited amount. – Draconis Dec 6 at 2:01
  • Thank you Draconis, one last thing do you know why some of my training data sizes seem to have a baseline accuracy of 50% pre sample size of 1000? – bemzoo Dec 6 at 2:02

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.