I have about 40,000 documents( of which 6,000 are test set) I ran cross validation, selected a model that works well for me and recorded a 98% recall with a document classification task. Now, to finally deploy my model, I have a question! :

Do I train my model on all my data(all 40,000 of them)? If so, how do I know I'm not over-fitting to it? Stop it the very first time it reaches 100% accuracy?

Thanks! :)

  • What's your system designed to do? Usually you train only on a small part of the data to avoid over-fitting. – Draconis Jul 19 '17 at 3:24
  • My system is designed to essentially classify documents into that are inherently either class A or class B type. We absolutely cannot afford to miss class A when it's deployed, and hence I maximized recall for class A. In Real life, class A has a distribution of about 4% of the total docs( 96% belong to class B). So when training, I trained it on 50% - 50% data and did CV and achieved a good 98% recall on A on the test set. Now, for the final model(which will be deployed), do I simply retrain the same model architecture on all of my data? or I always need to set aside some data for validation? – Daniel Dsouza Jul 20 '17 at 4:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.