I think it is a matter of presence/absence of a phonetic system within a given language as a milestone for comparison-contrast procedure, either historically or diachronically. Or maybe there is no relation between a phonetical inventory and a number of allophones at all.
The point is, if number of allophones might be regarded within the scope of same inventory (consonants/vowels only), or does it cover the whole phonetical system of a language.
E.g. Sanskrit has three primal vowels (a/i/u), which produce long vowels and diphtongs, yet its consonants ṭ,ṭh, t and th (= aspirated t) are regarded as separate consonants, not as allophones. In fact, its vowel phonetic inventory might not be taken as really 'minimal' together with (semi)vowels ṛ,ṝ,ḷ or ḹ.
The languages like English or Chinese has no inner comparison/contrast systems, but the number of consonant and vowel allophones in English is bigger as compared to Chinese, perhaps due to the polycentrism of English. Chinese almost has no vowel allophones, and position of phonemes zh, j, x and q is dubious. Well, to a Westerner's ear, at least.
One of the languages with a relly minimal vowel inventory is Abkhaz, which has only two distinctive vowels, but a number of Abkhazian consonants equals 58. They are, however, regarded not as allophones of, say, t/tʼ/tʷ, d/dʷ, or h/χʲ/χ/χʷ/χˤ/χˤʷ/ħ/ħʷ sounds, but as separate consonants.
In Arabic, (which has not just a/æ/e and i/y allophones, but also a special [o:] phoneme for its Egyptian variety), there is a big set of consonants. There are tˤ/t and x~χ/ħ~ʜ/h phonemes, which are also recognised as different consonants.
Let us take as an example the Estonian language which has all the possible dyphtongs and three-grade vowel length (short, long and overlong). All this abundence of phonetics, and overlong vowels as well, happens in initial syllables only, while the rest of Estonian word has either short or long back vowels and long or short consonants. Not too much of vowel allophones, but a system of front/back vocals and long/short vowels and consonants instead.
Finnish has the same system (long/short vowels and consonants and front/back vowels together with vowel harmony). Not too much of allophones, either (with the exception of nasalised vowels and n-allophones).
Modern Irish has broad/slender comparison paradigms for consonants together with a number of consonant allophones. A number of vowel allophones has survived in Ulster dialect up to day. Additionally, there has been lenis/fortis comparison-contrast paradigm for sonorants.
Finally, Russian has palatalised/unpalatalised dychotomy for its comparison-contrast paradigm and a large set of vowel allophones.
