4

Do you know any cases of corpus linguistics tools or corpora, using JSON to "mark" or delimited linguistics units in morphology, syntax,semantics? It seems all marking systems use some kind of XML or derivative to mark text. I would like to know if there is an initiative using JSON instead.

4

I have seen a lot of corpora and corpus tools, but none using JSON. The only alternative to an XML based format is plain text coming in two flavours

  • vertical format: One token per line with annotations separated by TABs
  • inline format: plain text with annotations appended to each token like this: plain/ADJ text/NOUN with/PREP ... (the seperating character may vary, and there can be more than one layer of annotation)
1
  • There is also the text-fabric format, which is similar to the vertical format but uses separate files for each column, which makes it easier to combine different data sets.
    – Keelan
    Jul 2 '18 at 16:27
4

There are several promising new developments where JSON is being used to provide improved modern data formats for computational linguistics, NLP, and corpus research.

There is an extensive project under way to create a JSON framework for linguistic fieldwork resources, including lexicon, texts, etc:
Hieber, Daniel W. (2018). Data Format for Digital Linguistics.
Check it out at https://format.digitallinguistics.io/
Or see the project on GitHub: https://github.com/digitallinguistics

Another useful source for JSON for syntax and semantics representations is:
Forbes, Angus G., Lee, Kristine, Hahn-Powell, Gus, Valenzuela-Escárcega, Marco A. & Surdeanu, Mihai (2018). Text Annotation Graphs: Annotating complex natural language phenomena. In Goggi, S. & Mazo, H. (Eds.), Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018). Miyazaki, Japan: European Language Resources Association (ELRA). https://www.aclweb.org/anthology/L18-1169

Information about this JSON approach is available from Angus Forbes' Creative Coding Lab on GitHub:
https://github.com/CreativeCodingLab/TextAnnotationGraphs

Ines Montani of Spacy makes an interesting proposal for a new approach to JSON formatting for linguistic data/NLP:
https://github.com/explosion/spaCy/issues/2928

Basic documentation for Spacy's (existing) JSON NLP format is here:
Spacy (2018). JSON input format for training.
https://spacy.io/api/annotation#json-input

Stanford NLP describes a JSON formatter: https://nlp.stanford.edu/nlp/javadoc/javanlp/edu/stanford/nlp/pipeline/JSONOutputter.html

Finally, at the Rezonator project (https://github.com/johnwdubois/rezonator) we are working to create a JSON format for the 2nd edition of the Santa Barbara Corpus of Spoken American English (SBCSAE). This is work in progress; stay tuned.

2
  • Very interesting information. Thanks!! Jul 4 '19 at 16:39
  • I would love to hear your thoughts on any of these initiatives, based on how they meet (or not) your needs. Jul 6 '19 at 5:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.