6

I'm making a Corpus.
I have searched for good practices on how to build it.
I've downloaded some Corpus to check their internal structure and the extension file, but I didn't find many.
So, I am lost and confused, because I don't know how to chose the good structure, for instance: Should I chose a grid, a CSV, or a JSON structure, or something else?

Here, some articles about "How to make it":

Corpus building and investigation for the Humanities

Steps for Creating a Specialized Corpus and Developing an Annotated Frequency-BasedVocabulary List

Thanks a lot for your advice.

Improve this question
4
  • 1
    A corpus doesn't need to be any format other than text. Or audio I guess. Anything else is post processing, and in principle shouldn't be too complicated to convert between them.
    – curiousdannii ♦
    Commented Sep 29, 2019 at 5:22
  • I understand that, but that is the problem, because programmatically I can do a process which read line by line (if it is a grid to which we refer) or load a JSON completely in a JSON Object (but I was thinking a memory consumption) and that is because I think in a good way to structure the data.
    – XOREZ
    Commented Sep 30, 2019 at 3:48
  • I'm voting to close this question as off-topic because those are computer-related constraints and formats, or at best, complexity-related questions in data structures in computer science. I don't see a real connection with linguistics here, especially in the light of comments below.
    – LjL
    Commented Oct 3, 2019 at 14:59
  • 1
    It really depends on the purpose, the scale etc. In translation very often each column (conceptually) is a separate file. Not necessarily recommending that, but top researchers implicitly believe it's the best choice, because they do it.
    – Adam Bittlingmayer
    Commented Nov 12, 2019 at 16:25

1 Answer 1

Reset to default
3

Well, there are several considerations:

1 What is your research question?

You always make a corpus with a purpose. So you design a corpus that is useful with respect to your research question. When you want to analyse the language differences according to social class, make sure that you have balanced amounts of material assigned to speakers or writers of different social classes, when you want to analyse diachronic development of language, be sure to have some diachronic depth of the material.

2 Think of FAIR principles from the beginning

The FAIR principles revolve around the re-usability of the corpus. It needs a good description with metadata to be findable, and it needs a suitable licence to be re-usable. It may be necessary to choose texts according to the licence because altering licences of texts is often difficult, time-consuming, and not guaranteed to work at all.

3 Think of the format

The most important thing is that the format is sustainable over a long time. Plain text obviously works, the vertical file format (vrt) of the Corpus Query processor (cqp) is a very simple XML format, TEI is a sophisticated XML format that can do at lot of detail. csv is a viable alternative to that, and there are some more formats outside, you may chose whatever fits to your tools chain. Refrain from the file formats of commercial word processors (they may change or become obsolete without prior announce).

4 Think of a repository

It is probably a good idea to plan for depositing the completed corpus with a specialised repository. There are some criteria that you may want to apply in the selection of the repository, like

  • Has it undergone some certification like Core Trust Seal
  • Does it belong to larger community, like CLARIN
  • Is it based on a research institution or is it commercial
  • When it is commercial, what is their buisyness model
Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.