2

I've built myself a little corpus that I'm using for language-learning.

However, I'm reaching the limits of standard command-line tool-chain: grep and friends.

For example, I might look for sentences where a word is used as an adverb, maybe together with a particular type of verb. If you want to do this 'correctly' you end up wanting to parse to sentence and categorise words in the sentences by their grammatical function. But queries like "this word is not first" are often good enough to rapidly find relevant use cases.

**Are there any good tools for this sort of querying?* Even just a a general tool to cleanly parse and separate sentences would be useful.

I quite like command-line tools because they increase the chance that the tool is composable and that it is easy to both install and set-up. This is particularly important in niche fields where tools can be immature and installation is not made easy. However, if there is some magic framework that is guaranteed to work and solve all my problems then that is good as well.

Improve this question
3
  • 1
    Apache Lucene lucene.apache.org/core is a widely used text storage/indexing/search framework that might be useful for you. Apache UIMA uima.apache.org is another popular framework more focused on representing and operating with natural language content. Both might do what you want.
    – peschü
    Jul 27, 2017 at 5:11
  • I've played with lucene before. My understanding is that the matching is done at the document level whereas for corpora you tend to want sentence level querying. Uima looks interesting.
    – Att Righ
    Jul 27, 2017 at 10:49
  • Might be interesting to consider things like python nltk, and running commands from ipython3 or similar.
    – melissa_boiko
    Jul 27, 2017 at 11:16

2 Answers 2

Reset to default
2

The Open Corpus Workbench originally developed at IMS Stuttgart has such a tool, called cqp (Corpus Query Processor). There are other implementation of similar tools, to mention one, there is Poliqarp.

The tool can do really amazing things when your corpus is annotated (e.g., for part-of-speech), you can issuer mixed queries for word forms and part-of-speech.

Improve this answer
3
  • Looking into poliqarp it seems to require documents in XCES as XCES. An xml metadata format.
    – Att Righ
    Jul 30, 2017 at 14:52
  • 1
    @AttRigh: When metadata complexity is an issue for you, the original cqp does't require much markup, the vrt-Format it uses as input is very simple and easy to create.
    – Sir Cornflakes
    Jul 31, 2017 at 12:01
  • treetagger (free but non open-source license) seems like one approach to annotate corupusesm
    – Att Righ
    Aug 5, 2017 at 12:56
0

Answering my own question here.

My first terrible approach was to use:

Advantages

  • Easy to do (no unknown steps and plain text formats)
  • Lots of people already know how to use grep and command line tools

Disadvantages

  • Knowing and searching for words by their grammatical function could be useful.
  • Lack of indexing means searching becomes very slow for huge corpuses, other tools will get around this by indexing.
Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.