I have used the Google LM 1B corpus in the past for testing written language, e.g. other periodicals.

However, I'm now using text from an IM chat, which is conversational. I would like to measure predictability, but wasn't sure how accurate probabilities given by the LM 1B would be, given that it was mostly trained on newswire text.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.