0

Is there any way to have a good heuristic study on the frequency of special characters and uppercase ?

Would studying a freely available corpus (around 15m words) generates a good estimation?

Edit: I would like to study the frequency of occurrence in English of special characters ( e.g. !@#$%^&*()_+?~-/[]{} ) compared to letters

where I would like to construct a frequency table for all characters ( e.g. saying the frequency of occurance of @ is 0.05% of all texts )

  • What are special characters? What language and subject matter do you want their frequencies for? – sumelic May 10 '15 at 20:53
  • for example: !@#$%^&*()?":/;-~ and the language is english. Frequency of occurrence compared to other characters ( say compared to letters an such ) where I can say that @ has a frequency of 0.05% and so on – Tt22 May 11 '15 at 3:55
  • What is the application (what do you need this for)? – sumelic May 11 '15 at 3:58
  • We would like to expand on the frequency studies of letters to include uppercase/special characters/numbers – Tt22 May 11 '15 at 4:01
  • So you just want to know for general written English? It will be different depending on the subject matter; for example, the frequency will probably be higher in math papers or programming guides and lower in novels. – sumelic May 11 '15 at 4:03
2

Regarding corpus size: If you need meaningful results for the less frequent characters, you'll need a larger corpus, such as the ones at http://corpus.byu.edu.

Their web interface accepts queries consisting of punctuation. A sample query is .|,|:|;|!|?|'|"|@|#|$|%|^|&|*|(|)|-|~, which for the Contemporary American English, Global Web-based English and Wikipedia corpora produces these frequencies:

COCA               GloWbE             Wikipedia
 1 , 23849941       1 , 83847760       1 , 109960441
 2 . 21304861       2 . 80894692       2 '  79264498
 3 "  7008881       3 " 17572733       3 .  73633968
 4 )  1694276       4 )  8668155       4 #  28560000
 5 (  1688162       5 (  8337855       5 (  25894803
 6 ?  1620720       6 ?  7853684       6 )  25851855
 7 :  1360298       7 :  6323921       7 *  18572182
 8 '   990783       8 '  6179597       8 :  10649543
 9 ;   782692       9 !  4029929       9 ;   5709259
10 #   545703      10 ;  2511484      10 -   4912002
11 -   392700      11 -  2228750      11 !   3248850
12 !   368622      12 *   137176      12 ?    378872
13 $   133473      13 @    66127      13 @     20287
14 &   109713      14 %    27920      14 %     16899
15 *    81934      15 $    23881      15 $     10455
16 @    74050      16 #    22256
17 %     1761

(The Wikipedia corpus has " encoded as '' and uses # as a special character.)

However, if you need to study digits, capitals, or the symbols [ ] / | < >, you need to download the corpora and do your own analysis.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.