I don't think it's 3 or 4. Any rule that exists is not particularly "strict" (see the various kinds of counterexamples listed below). The diphthong /aʊ/ does sound similar to /æl/ or /ɑl/, but it's hard for me to imagine how this could have influenced the distribution of the sound /aʊ/.
1 seems the closest, although a different, more diachronic-rules-oriented spin that you could put on it is "there happen not to have been any processes that historically acted to create (tautosyllabic) /aʊm/, /aʊp/, /aʊb/, /aʊf/, /aʊv/ sequences in modern English". The reason for this is actually an exception to a historical sound change that otherwise changed /uː/ to /aʊ/: depending on how you analyze it, the existence of this exception could be seen as something that supports 2 (there is something about the English sound system that "doesn't like" these sequences—or at least, that "didn't like" these sequences at some point in history).
The gap can be described in terms of /aʊ/ + a labial consonant in the same syllable
There is a descriptive rule that sequences of the diphthong /aʊ/ + a labial (or labiodental) consonant don't exist in English vocabulary. Certain di- or polysyllabic words are exceptions to this rule:
compound words such as cowpox /ˈkaʊˌpɑks/, where /aʊ/ and /p/ belong to separate syllables (other examples include cowpoke, cowpea, cowpie; cowbell, cowboy, cowbird; cowfish)
endowment, where /aʊ.m/ occurs across syllable boundaries because of the presence of the unstressed suffix -ment
If we want to make a generalization that acknowledges the existence of the above words, we could say more specifically that /aʊ/ is not followed by a labial consonant in the same syllable in any English word. I don't know of any exceptions to this form of the generalization in "normal" English vocabulary.
Would a "foot"-based rule be preferable? It's not clear to me, but there seem to be some counterexamples
As far as I know, in all of the above examples, /aʊ/ is separated from the following labial consonant not only by a syllable boundary, but also by a "foot" boundary. (The exact definition of a "foot" is a bit complicated, but it can be thought of as a unit in a language's "metrical" system: in English, this is related to stress. Compound words like cowpox have a kind of stress on the second element, which is shown by the presence of an unreduced vowel in the second syllable; also, the voiceless plosive /p/ is likely to be aspirated in this context.)
However, I'm not sure whether it would make sense to argue for the slightly stronger generalization that /aʊ/ is not followed by a labial consonant in the same foot (assuming we allow for HL feet; if not, I think this generalization would just be a restatement of the previous syllable-based one). There might be theoretical reasons to prefer this formulation of the generalization, but I know of several apparent counterexamples (all fairly marginal or obscure, though, unlike the first set of examples):
the uncommon blend word "cowfeteria", which can be analyzed as starting with the trochaic HL foot /ˌkaʊfɪ/ or /ˌkaʊfə/.
the mispronunciation of the surname "Cowper" as /ˈkaʊpər/, which Greg Lee mentioned in the comments
The jocular name "Cowvin" (presumably pronounced as /ˈkaʊvɪn/ or /ˈkaʊvən/) which is attested in various web documents: e.g. Urban Dictionary ("The official name of the GarageSaleCow.com cow mascot: Cowvin the Cow enjoys going to garage sales."), various sites mentioning the "Cowvin Cookie" (which apparently was named after a real calf that was given the name "Cowvin"), "Six Degrees of Cowvin Bacon"
If we follow the principle of maximizing syllable onsets, then these would not be counterexamples to the syllable-based generalization about the gap.
I don't know if it's significant or just a coincidence that many of the exceptions start with the word "cow" (or in the case of "Cowper", just the letters C O W).
English speakers don't show any particularly strong tendency to avoid pronouncing /aʊ/ + a labial consonant in new words that they encounter
Whether or not there is a rule in the sense of "active constraint" in the phonology of modern English against sequences like /aʊm/ and /aʊp/ is kind of disputable. Clearly, it's not that difficult for most English speakers to perceive and pronounce sequences like /aʊm/, /aʊp/, /aʊf/ in new vocabulary words that they encounter: various German words with these sequences are used and commonly pronounced with /aʊ/ by English speakers, such as Lebensraum, Hauptmodul, and Aufbau in "aufbau principle". I think the surname Baum (of German origin) is typically pronounced /bɔːm/ or /bɑːm/, but this can fairly easily be explained as a spelling-pronunciation, rather than some kind of synchronic phonological adaptation of German /aʊm/ to English /ɔːm/.
I think English speakers would intuitively feel that words containing sequences like /aʊm/ or /aʊp/ don't sound very "native English", but such a feeling might arise from the absence of native words that are pronounced this way, even if there is no current "rule" per se against these sequences.
An explanation of the gap in terms of historical sound changes
From a certain standpoint, this gap can be explained by looking at the historical sound changes that created the modern English /aʊ/ diphthong. Most instances of /aʊ/ in English originated from Middle English /uː/ via the "Great Vowel Shift", and for some reason, this change of /uː/ to /aʊ/ was inhibited before labial consonants. So, for example, Old English "dūn" corresponds to Modern English down /daʊn/, Old English "rūm" corresponds to Modern English room /ruːm/.
I would guess that this exception to the sound change can probably be explained at least in part on articulatory grounds. I'm not sure exactly how to formulate an explanation like this, though.
We don't see a similar effect after labial consonants—modern English has /aʊ/ after onset /m p b f v/ in words like mouth, pouch, about, found, vow. The /aʊ/ in mouth is known to come from Old English [uː] (in mūþ). So an articulatory explanation would have to account for why coda labial consonants had this effect on a preceding /uː/, but onset labial consonants did not similarly affect a following /uː/.
It might be relevant that the rounded portion of the diphthong /aʊ/ is at the end, but I'm not sure exactly how to formulate an explanation that makes use of this fact.
"On the Markedness of Diphthongs", by Haruo Kubozono, references a discussion of the historical development of this sound in Ekwall 1965/75: 53 ("Historische neuenglische Laut- und Formenlehre.", English translation by A. Ward 1975. A History of Modern English Sounds and
Morphology.)
/aʊ/ also doesn't occur before coda velar consonants
I have seen it noted that /aʊ/ also does not occur before a tautosyllabic velar consonant in modern English (we don't see words with /aʊk/, /aʊg/ or /aʊŋ/). The historical explanation for the absence of /aʊ/ + velar sequences seems to be a bit different than the explanation for the absence of /aʊ/ + labial sequences. In the case of /aʊg/, it fits with the general absence of /g/ after "long vowels" or diphthongs in native English words; the historical explanation for this is that native English g was vocalized after a vowel except for when it was a geminate or "long" consonant (which shortened the preceding vowel). Likewise, the absence of /aʊŋ/ can be explained in terms of the origins of /ŋ/ from the heavy cluster /ng/.
The absence of /aʊk/ in modern English seems to be the strongest candidate for another gap (aside from the /aʊ/ + labial gap) that could be accounted for by a rule that specifically involves /aʊ/.
The history of the /aʊk/ gap
Unfortunately, there does not seem to be as much data about the historical evolution of the relevant back vowel sounds before /k/ as there is about the historical evolution of Middle English /uː/ before labial consonants. The absence of /aʊk/ looks almost (but not quite) accidental: I have only been able to find a small number of words where we might expect to see /aʊk/ in present-day English. All of them have shortened vowels before the /k/ instead:
the verb brook (< OE brūcan). The Oxford English Dictionary says that "The phonetic history is unusual" and describes the modern English form as a descendant of "a Middle English brōken, found already, as a by-form, in Layamon" (this form is also supposed to be the source of "Scots bruik"). In this case, the short /ʊ/ in present-day English would be derived from earlier long [uː] from Great-Vowel-Shift raising of Middle English long [oː].
the verb (and noun) suck (< OE sūcan). Spellings with ou/ow existed in the Middle English period. The OED doesn't give any detailed explanation of the development of this word's pronunciation.
the noun duck. The Oxford English Dictionary says that the length of
the vowel in Old English is uncertain, and that already in Middle English
we find the three "types" dukke/duk, dōke/dook, and douke/dowke.
the obsolete verb tuck and related noun tucker from Old English tūci(ġ)an. Some Middle English examples of touker are mentioned in Peter Kitson's review of Gillis Kristensson's Survey of Middle English Dialects... (p. 145).
There is also a word “louk” < OE lūcan (now obsolete or dialectal) which as far as I can tell has been pronounced with /aʊk/. It seems to me that there aren't really enough examples to establish a clear generalization for the development of OE ūk / Middle English /uːk/, but there does seem to be some support for the idea that /aʊk/ was somehow avoided by the application of various sound changes to words that would otherwise be expected to have /aʊk/—or possibly in some cases by the loss of words where this sound sequence would have developed (the OED has an entry for an obsolete/dialectal word for body bouk from Old English būc.
The remaining consonants that /aʊ/ does combine with are all coronal
The absence of /aʊ/ + velar consonant may be relevant for certain theories about the phonology of modern English because it means that the coda consonants that /aʊ/ does combine with can be described as those that belong to a particular phonological class: they are all "coronal consonants".
Namely, /t, d, n, θ, ð, s, z, tʃ, dʒ/ (as in shout, loud, gown, mouth (n.), mouth (v.), house, rouse, couch, gouge) as well as /r l/ (as in hour, owl) for speakers without pre-rhotic or pre-lateral breaking of /aʊ/. The "coronal"-based rule for the distribution of /aʊ/ is given in Beginning Linguistics, by Laurie Bauer, 2012 (p. 105).
In some theories, coronal consonants may be expected to have some kind of special status that allows them to occur more freely than other kinds of consonants at the end of words (or syllables). For comparison, English also has one (not too infrequent) word that ends in /aɪnt/—pint—but no words that end in /aɪmp/ or /aɪŋk/.
/kʌup/does exist in some English dialects with influence from Scots/Ulster Scots (see also Scots gowp, howp, lowp, nowp, roup etc).