Why is the vocalized dental fricative [ð] so rare across languages? Is it just a coincidence or is there a pattern behind this?
Actually, [ð] is not so rare across languages. Apart from English, it exists in Icelandic, Swedish and Norwegian dialects, Danish, Kven, Saami, Welsh, Spanish, Catalan, Basque, various Italian languages, Albanian, Greek, Arabic, Tiberian Hebrew, Berber, Hawrami, Mari, Bashkir, Turkmen, Somali, Dahalo, Moro, Didinga, Kikuyu, Kamba, Swahili, Tamil, Burmese, Thao, Iai, Palauan, Fijian, Aleut, widely in Athabascan, Shoshone, Lakota, Osage. It is not as common as [t] or [s], but certainly not rarish like [ʕ, ƛ'] or rare like [ʘ].
The main thing that [ð] has working against it is that it is acoustically harder to detect. Because of the lingual constriction, it has reduced airflow which means less acoustic energy, and the flow is not turbulent (which makes a sound noisier, thus easier to detect). The energy is spread out over the frequency spectrum (compared to [ʒ]) and such peak as it has is relatively higher (compared to [γ]). The latter fact is relevant because sounds in the "frequency center" are most audible and those at the high and low ends are less audible to human ears, given the same sound pressure level (and as I said, it also has a lower net amplitude).
It is possible that one can accurately say that a phonemic contrast between d and ð is rare (often, as in Spanish, d and ð are in an allophonic relation), though there is no systematic database of phonemic contrasts that we can consult to see what its frequency is.
There's a lot of rare phones. Its well-known that many phones are more common than others. I should note though that there are none that are universal. No one really knows the reason for this, just that it exists. There's speculation that it may be because some phones are easier to pronounce than others, but this is an argument you mostly see in the auxlang community. Still, it may explain why some phones are virtually universal (the sounds m p t and k occur in over 90% of languages each) while others are rare or in a few cases unattested (there is no documented language that has a lateral alveolar trill).
Linguistics in general is more of an analytic study. We study how languages work, but there's little effort being put into why languages work the way they do. Its also unknown why most languages on this planet are subject/agent first. Or why grammar changes over time (its pretty well understand how sound change works, but little is known about how grammar changes). Similarly, we don't know why there's such a discrepancy in how common specific phones are. For all we know it may just be happenstance. There are trends though. For instance, front vowels are normally un-rounded while back vowels are rounded. Rounded front vowels and un-rounded back vowels are rare. As a matter of fact, the 'u' sound (a high back rounded vowel) is the most common phone on earth, while its un-rounded counterpart is the rarest. Non-sibilant fricatives (like ð) are also quite rare. Humans also seem to lean away from consonant clusters (languages seem to prefer simpler syllable structures overall, even if the language allows some rather elaborate consonant clusters). Also, all languages have at least one voiceless plosive. There is no other place of articulation that is universal.
I guess it could be said there really is no answer to this. We can find patterns behind things like this, but we just don't know what causes these patterns.
I won't say that a voiced dental fricative is "so rare". With a frequency of 4.88% in the UPSID sample (see http://menzerath.phonetik.uni-frankfurt.de/upsid_find.html for a nice interface to the UPSID data) it is neither particularly frequent nor particularly rare.
There are of course patterns in the sound systems of the languages of the world, many of them under heated debate. There is some agreement on the vowel systems that the vowels tend to take maximal distance in the vowel space (so a three vowel system comes out as /a/, /i/, /u/ and not, say /e/, /ə/, /o/). For consonants, things are much more difficult (e.g, there is no such thing as an easy-to-picture consonant space).
BTW, in the UPSID sample, the vowels /i/ and /a/ are more frequent than /u/.