Linguistics Stack Exchange is a question and answer site for professional linguists and others with an interest in linguistic research and theory. It only takes a minute to sign up.
Sign up to join this community
I am working in a project where i need to use an ngram model. So, i want to know if an Arabic ngram corpus exist. I have tried to find a corpus but all my researches failed. I know that for languages like English there exist such a corpus (Google ngram corpus for instance but i want an arabic corpus).
Thanks in advance
I would love to find something that could mimic what the Google Ngram does, too. Unfortunately, I have yet to find one. In the meantime, here are a couple of things you can do / resources that can get somewhat close. I am going to do this using just a single word. I know that the beauty of the Google Ngram is that it allows you to search for a single word or up to four words at a time:
You can do that, to some degree, with a Google search, but it will be a lot more difficult and if your graph looks cool at the end, it's because you had much more of a hand in it. To demonstrate this, I'll use a couple of words that made me stumble on your question in the first place. I wanted to find out the difference between
My efforts were, for the most part, quite fruitless, so I wondered about their relative frequency to see if I should even bother going down that rabbit hole. I was about to let it go and move on to other things when I saw your post and I thought to myself, Well, I really haven't ever contributed much to the Linguistics Stack Exchange, so why not give it a go.
So, this is what I did. I simply did a Google search, filtering for time frame and for language. For time frame, I used five-year increments going back to 1996. (For all the obvious reasons, a search going back to the 1800s wouldn't make a whole lot of sense.) For language, naturally I used "Arabic." I collected the numbers from each search. (If you don't know how to do this or where to look, let me know and I can explain it further.) I plugged a spread sheet with those numbers and then built a chart from it. It isn't the most sophisticated chart in the world, but it does the trick:
So, that is one thing you can do. The other thing you can do is pay a visit to Google Trends. Using this to study frequencies of language usage is a bit limited in that this is simply a tool you can use to tell you the frequency of Google search terms used, but it does allow you to search worldwide, break it down by category (everything from Arts and Entertainment to Travel), by type of search (e.g., Web, Image, News, Google Shopping, and YouTube), and it gives you a couple of neat graphics. Using the same example I used above, this is what it looks like:
If you want to see where the other search term(s) were most used, scroll down the page:
I only used two search terms, but you are not limited to that and you can search for phrases as well.
Like I said, this is a bit limited, but one kind of cool thing it can do for you is really pin down your search. For example, if you click on any of the countries that surface, it will show you, by color gradation, where the search term was most often used. To illustrate this better, I ran another search using an Arabic word for "star." This is how it breaks out in Yemen:
Like I said, you can use it to really pin down where a search term is being used. To show you this, I went back to the previous example and selected Turkey. I thought to myself, Where in Turkey is someone using Arabic script to search for the Arabic word for "prepare?" This is where:
If my other answer wasn't so long, I would have just edited it, but I think this thread will be easier to read if I just add another answer, so here it is.
The day after posting my first answer in this thread, I discovered that my usual tool for searching English and Spanish collocations was not functioning. This made me go searching for another, and, in the process, I stumbled upon a collocation tool that includes a corpora for Arabic. It's called Sketch Engine and it is one of the best things I've found in a long time. But collocation is just one of its many features; it also has an N-gram! But it isn't like the Google Ngram in that you don't get to choose what it searches and then have a neat graph come out at the end. It simply lists the most common collocations in order of frequency. You get to choose 2, 3, 4, 5, 6, or any combination thereof for the N-gram length (i.e. length of the phrase).
Unfortunately, I'm currently only sampling a free account (which lasts for 30 days) and though this provides me with two Arabic corpora:
Arabic Web 2012 (arTenTen12, Stanford tagger)
KSUCCA (Classical Arabic)
I don't know if a paid subscription might give me access to more.
Regardless, if you go to this site and use this feature, this is how it will look if you run an n-gram length of 5 for the "Arabic Web 2012 (arTenTen12, Stanford tagger)" corpora (which I use for all of the experimentation included in this post):
I translated the results using Chrome machine translation and added that on the right.
There are so many features on this site that the 30-day free trial (which does not require credit card information up front), while seemingly generous, is just enough time to give you a taste of what it offers (but if you enjoy tinkering with such things, you might be hooked on this within the hour).
Since this site does not offer exactly what the Google Ngram offers, I thought I'd explore it just a bit to show you some of the things you could do with it.
Word Sketch Difference
Beyond the collocations I showed you earlier, it also has something called "Word Sketch Difference." With this, you can compare collocations. When I first used this, I just kind of winged it and didn't read too much about it first. Later, when I read the instructions, I read something that made me think that I should have used a lemma instead, so I experimented with both the root and the conjugated form of the two words used in the other answer I posted for this thread. This is what it looks like:
To enlarge the image, bring up the page in Chrome, right click on the image, and open it in a new tab. This applies to all images in this post.
As you can see, the collocations are slightly different and if you are familiar enough with Arabic to have at least had a small taste of roots and verb forms, then you already know that small changes to the root form can result in another meaning altogether. The big takeaway from this experiment, though, is this: You don't have to use a lemma. Having said that, Sketch Engine does mention that:
Rich word sketches with lots of collocates in all grammatical relations are important.
Another neat thing about Sketch Engine's Word Sketch Difference is that it gives the option of creating a cool little graph. This is what it looked like for my conjugated forms:
I added the English words to it later in a separate application not associated with Sketch Engine.
What does it mean? How do you interpret what you see in this graph? That's best left to the Sketch Engine website, but here's a small excerpt from its "About" page on "Word Sketch Difference":
My personal opinion is that this works a bit better with languages other than Arabic. When you use this feature with Arabic, the interface (namely, the labels for the columns of results) aren't similar to what I saw in the tutorials and other supporting materials from Sketch Engine. Then again, in the tutorials and supporting materials I looked at, another language was used — English. I found them interesting, but you might be a bit perplexed when you go to run Arabic words through this same tool.
Concordance
I just visited the LinkedIn profile of the OP, so I'm pretty sure he doesn't need me to lay down a definition for "concordance," but for those of you who may be new to linguistics, one of the simplest, best definitions I've seen of it comes from Collins and it is this:
A concordance is a list of the words in a text or group of texts, with information about where in the text each word occurs and how often it occurs. The sentences each word occurs in are often given.
So, to use Sketch Engine's "Concordance" feature, you first need to set it up. I'm not going to break it down step by step, but eventually you'll want to get to something that looks like this:
Then, in the upper right hand corner of your screen, you'll want to click on the "Frequency" button:
Then you'll want to select your parameters for collocations. As you can see, Sketch Engine gives you plenty to choose from:
This is what you'll see next. (Be patient ... you may have to wait a while as it processes your search parameters.)
But don't just be content with that when you can also take a look at part of speech (POS) frequencies:
This is what a query for part of speech frequency can show you:
For a page that lists the POS tags for this corpora, go here.
After having written all of this, I realize that although Sketch Engine may be many things, it is still not a Google Ngram. But who knows ... maybe you or someone like you will figure out how to make something similar. If it's any consolation, I know you and I are not the only ones looking for such a thing. Just so you know, I belong to/follow a couple of Arabic groups on Quora and sent out this question just a few seconds ago. If anyone should answer it, I'll update this post.
In the meantime, I highly encourage anyone with an interest in this type of thing to check out Sketch Engine. It has a beautifully designed interface with a lot of neat features and tutorials to assist you with your searches. Check it out:
After writing all of the above, I did one final check of something and came across an academic paper that mentioned this thing called the "Historical Arabic Concordancing and Searching System" (HACSS). I haven't read the entire paper just yet, but I did quickly skim over it and it looks as if this is something developed by some Ph.D. students attending the University of Jordan. (Unfortunately, this is the one thing I wasn't able to find a Wikipedia page for, even after visiting the Arabic Wikipedia site.)
I also visited the website for the University of Jordan. It does have a linguistics program up to the doctorate level and, in fact, one of its courses in its linguistics doctoral program is "Historical Linguistics," but I could find no mention of HACSS. Nevertheless, I think you might get something from reading the paper.
If you'd like to get in touch with someone at the University of Jordan's Language Center, you'll find some contact information here.
But you may have better luck just trying to contact the author of the academic paper. Three students are on the byline of it, but I sense that the first one listed — Osmaima Ismail — is the lead on it.
I also came across a Prezi presentation that, initially, I thought might be a waste of my time, but it is actually one of the most comprehensive Prezi presentations I've seen. It devotes an entire section to HACSS, beginning on slide 75. In fact, it's by the same person (Osmaima Ismail) who wrote the academic paper I mentioned earlier. I have no proof that this thing called HACSS even exists or not, but apparently this is the type of graph it can generate:
Anyway, if you want to try contacting Osmaima Ismail about this, you can reach out to her via LinkedIn. Not only does she have a profile on it, but, under projects, she mentions "Historical Arabic Corpus, Natural Language Processing (thesis)."
This is all I've got for now, but best wishes to you and your quest for a tool that is long overdue in the world of computational linguistics.
phoneticswhich you do not explain further in your question. Corpora and ngrams usually deal with orthography and Arabic orthography usually doesn't indicate vowels. If you want ngrams of Arabic phonemes or transcriptions or with fully marked vowels you should say so. Otherwise you probably should either explain or remove the phonetics tag.