4

I try to find corpus with technical language to train my model, but I cannot find it.
Do you know if there are any tagged PoS corpus in English related with technical language?

4
  • 1
    A good model trained on general data should work on technical language too. Did you try spaCy and it didn't work? From experience, the topic matters less than the type of text (title / description / email / comment ...). – Adam Bittlingmayer Mar 5 '19 at 7:26
  • I tried OpenNLP and results showed that model trained on general data can't recognize for example hexadecimal number 0x12ab34 as a number, so I have tried to extend general corpus by technical data and compare results, but the main problem is availibility of technical dataset. – Benji Mar 6 '19 at 18:37
  • My guess is that they are all just trained to see it as a named entity. So then you should do a separate test on whether it is valid hex. – Adam Bittlingmayer Mar 6 '19 at 21:26
  • It may be a good idea in general to perform some text normalization before tagging if your texts are domain-specific. For instance, converting all the hex numbers to some special string like <HEX>. In this case, you would be able to use a general text collection or models trained on them as @Adam suggested. Normalization will be necessary anyway, even if you find a technical corpus and train an off-the-shelf PoS tagger on it. – Max Ionov Mar 20 '19 at 11:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.