2

I've built myself a little corpus that I'm using for language-learning.

However, I'm reaching the limits of standard command-line tool-chain: grep and friends.

For example, I might look for sentences where a word is used as an adverb, maybe together with a particular type of verb. If you want to do this 'correctly' you end up wanting to parse to sentence and categorise words in the sentences by their grammatical function. But queries like "this word is not first" are often good enough to rapidly find relevant use cases.

**Are there any good tools for this sort of querying?* Even just a a general tool to cleanly parse and separate sentences would be useful.

I quite like command-line tools because they increase the chance that the tool is composable and that it is easy to both install and set-up. This is particularly important in niche fields where tools can be immature and installation is not made easy. However, if there is some magic framework that is guaranteed to work and solve all my problems then that is good as well.

  • 1
    Apache Lucene lucene.apache.org/core is a widely used text storage/indexing/search framework that might be useful for you. Apache UIMA uima.apache.org is another popular framework more focused on representing and operating with natural language content. Both might do what you want. – peschü Jul 27 '17 at 5:11
  • I've played with lucene before. My understanding is that the matching is done at the document level whereas for corpora you tend to want sentence level querying. Uima looks interesting. – Att Righ Jul 27 '17 at 10:49
  • Might be interesting to consider things like python nltk, and running commands from ipython3 or similar. – melissa_boiko Jul 27 '17 at 11:16
2

The Open Corpus Workbench originally developed at IMS Stuttgart has such a tool, called cqp (Corpus Query Processor). There are other implementation of similar tools, to mention one, there is Poliqarp.

The tool can do really amazing things when your corpus is annotated (e.g., for part-of-speech), you can issuer mixed queries for word forms and part-of-speech.

| improve this answer | |
  • Looking into poliqarp it seems to require documents in XCES as XCES. An xml metadata format. – Att Righ Jul 30 '17 at 14:52
  • 1
    @AttRigh: When metadata complexity is an issue for you, the original cqp does't require much markup, the vrt-Format it uses as input is very simple and easy to create. – jk - Reinstate Monica Jul 31 '17 at 12:01
  • treetagger (free but non open-source license) seems like one approach to annotate corupusesm – Att Righ Aug 5 '17 at 12:56
0

Answering my own question here.

My first terrible approach was to use:

Advantages

  • Easy to do (no unknown steps and plain text formats)
  • Lots of people already know how to use grep and command line tools

Disadvantages

  • Knowing and searching for words by their grammatical function could be useful.
  • Lack of indexing means searching becomes very slow for huge corpuses, other tools will get around this by indexing.
| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.