3

Words contain letters (A to Z and a to z), but also apostrophes, such as in the case:

don't

and even:

'em

and even hyphens in the case of:

ice-cream

or:

long wind-
ed sentence

With regard to English, is there any other symbols that can compose a word? More precisely, I want to know how I should identify words when processing a corpus.

Additionally, I would be interested in learning about other European languages and their use of such symbols within words which fall outside those defined by the alphabet.

Improve this question
  • 1
    My experience with corpora and word counting is that all sorts of different definitions are used. A simple one is to count the spaces in a text (-1 and you have the number of words, provided all double spaces have been removed). I found that to be the best workable method. You could also use a lemmatizer and count the lemmas identified by it. Also, it's not uncommon to consider contractions like don't as two words. I haven't heard of any accepted standards, and in practice you just have to watch out that you compare like with like. – robert Sep 23 '13 at 22:30
  • 1
    The "definition of word" problem also leads to "Is 'Hong' a word?' As in "Hong Kong". Is "tit-for-tat" one word or three? What about "vis-a-vis"? These are basically things you need to decide for your needs and based on the capabilities of your tools. – hippietrail Sep 24 '13 at 1:04
  • In my code I always have a set of extra characters to treat as parts of words always plus for each language I optionally define a set of extra characters. And even this is not enough since both apostrophes and single quote marks are the same character and are not straightforward to disambiguate, etc. – hippietrail Sep 24 '13 at 1:11
  • Check out the Wikipedia article on "text segmentation". – hippietrail Sep 24 '13 at 1:18
  • Couldn't you just use regular expressions with predefined word rules? I'm sure there is plenty of code around for this. – user2498 Sep 24 '13 at 7:35
4

In English, you're fine with apostrophes. Compounds with hyphens, such as ice-cream, are mostly treated as separated tokens in corpora. As for other languages in Europe, the "long" L in Catalan is denoted by a "bullet", e.g. pel·lícula.

Improve this answer
  • In Catalan, sometimes a specific character ĿL/ŀl (occupying two spaces total) is used instead of an l and interpunct L·L/l·l. It is common to see a period used also (though this is prescriptively incorrect) L.L/l.l. – brazofuerte Dec 18 '17 at 18:04
  • @tjfuke Yes, I think the ligature is even the preferred way. – Atamiri Dec 19 '17 at 18:09
3

I think, the key point answering this question would be by understanding the distinction between an alphabet and a script.

Some boring definitions

Alphabet is defined as a set of letters. Examples are Latin, Cyrillic, etc.

Script, or, more formally, Writing system, is a language-specific set of rules used to encode a text in a particular language.

For each language, script includes (a part of) a certain Alphabet, certain type of Numerals, Punctuation marks, and other symbols.

Now, let's get to the question

With regard to English, is there any other symbols that can compose a word?

English writing system includes:

  • Latin letters, letter-case and capital-case, obviously;
  • Arabic numerals, obviously;
  • Diacritic marks used for loanwords;
  • Punctuation symbols - make sure your software properly processes similarly looking symbols, e.g. apostrophes used to mark possession, and to mark contractions;
  • For Natural Language Processing (NLP) software, you have to be aware that most characters with diacritic marks have their own Unicode points, so, for instance, parsing a sequence of A and ◌̂ (a zero-width combining circumflex accent modifier, U+0302, not to be confused with a standalone character ^) should work exactly as parsing a single code point of  (Latin A with circumflex)
  • If you are parsing Unicode text, you have to be also aware about ligatures, here's the list, and there may be stylistic ligatures as well;

Additionally, I would be interested in learning about other European languages and there use of such symbols within words which fall outside those defined by the alphabet.

Again, don't look into an alphabet. Look into writing system of a corresponding language.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.