I am currently looking for text data in Arabic. However, for this I need to know whether there exist some significant differences among dialects in its written form.

Surely, dialects very likely differ in how they pronounce words but does this apply to text as well? If so: What differences are we talking about?

Are there any sources where I can get Arabic textual data split by dialects?


Yes, major differences, but do you actually have or want text data in Arabic dialects?

Almost any formal Arabic text corpus you can find will be in Modern Standard Arabic, not in the actual local dialect.

There are in practice different variants of standard Arabic, just as there are different variants of standard English.

(In software and localisation, these are generally referred to as locales.)

The differences can be lexical, but also nuances like numbers and punctuation, and moreover do not map so neatly to locale as one might imagine. Just like for English locales.

For example, in Arabic the use of the Indian numerals and the Arabic numerals (the ones you know from English) varies by country and by context.

You can get text split by locale simply by taking Common Crawl pages in Arabic and filtering by country TLD. Again, just like for English locales, this will be a correlation, not perfect, and have other noise.

If you actually have or want data in spoken Arabic dialects - which thanks to SMS and the internet now exist in writing - note that they are essentially different languages.

| improve this answer | |
  • The internet has vast amounts of written Arabic in the form of comments, e.g. youtube.com/watch?v=ZBJQDehTTmU. Can you discern with reasonable certainty what form of Arabic is written here? – user6726 Nov 17 '19 at 21:11
  • Yes, as I noted. Are you asking me if I personally can manually identify them, or if one can do so programmatically and reliably? Language identification works well enough on long texts, but not on short strings, and keep in mind that in diglossia, especially diglossia involving related languages where one of the languages does not have a defined standard, the lines are very blurry. – Adam Bittlingmayer Nov 18 '19 at 6:51
  • In any case fastText language identification and CLD2 (the one in Chrome and CommonCrawl) do not support most of those languages, just ar, so one would probably need to train one, which leads back to the problem of collecting data. (See fasttext.cc/blog/2017/10/02/… and github.com/CLD2Owners/cld2#supported-languages, note that fastText ie Tatoeba does support arz - Egyptian Arabic - but I doubt it works well, knowing what I know.) – Adam Bittlingmayer Nov 18 '19 at 6:54

"Arabic" can refer either to Modern Standard Arabic, which as the name implies is standardized and relatively uniform in the Arabic speaking world, and there is "dialect", which is highly variable. Which version of Arabic you will encounter depends on context, for example MSA is used in formal lectures and books; local dialects are what people speak conversationally. In MSA there may be regional indicators such as the names of calendar months (March = آذار vs. مارس) or orthographic differences ( مسؤول vs. مسئول) but MSA is basically uniform.

However, people will write in their dialect as well. I consulted with an Arabic speaking friend who confirms my impression that local dialect is generally used as the written form in informal contexts, such as Youtube, Twitter, or Facebook (at least if they are making an informal comment). Dialects are much easier to distinguish, though from texts you can't distinguish sub-dialects which differ in having [i] versus [a] in a certain context (since nobody will mark the vowels). In fact, LDC offers MSA, Baharna, Egyptian, Gulf, Mesopotamian, North Mesopotamian, North Levantine and South Levantine. Their collection includes online written content in specific dialects (this is Egyptian).

So you can either get or avoid dialect data, depending on what you want (and pick your source accordingly). Pick more formal sources if you want uniformity.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.