It is correct that all sounds which can be part of a language can in principle be produced by any human. It is a bit of an overstatement that infants practice all of the possible human language sounds, but it is true that babbling infants make sounds that they do not hear in the language of their environment. There are odd combinations of phonetic properties which might not ever be attested in an infant's repertoire, for example an implosive labiopalatalized breathy voiced retroflex stop. In fact it is likely that sounds which require complex timing of gestures are not part of the set of generally-babbled sounds. But I am not presently inclined to attribute anything to infant motor skills and difficulty of articulating.
The kind of explanation which seems to curry the most favor in the field is an appeal to those factors that disfavor the creation of rare sounds, and which favor the creation of common sounds. Here is an example. t, s are very common sounds, and the interdental ejective affricate t͡θ' is rare. It occurs in Halkomelem, and as far as I can tell (I have limited exposure to the language) it derives from proto-Salishan *t͡s', which is *t͡s' itself is a relatively rare phoneme. The sound t͡θ' also appears in Tahltan (coincidentally, also spoken in British Columbia but far enough away that contact is not a reasonable explanation).
The fricative θ is at a perceptual disadvantage compared to s, that is, it is harder to distinguish θ from other similar sounds than it is to distinguish s from other similar sounds. The reason for this is the acoustic structure of θ, which has a very flat low amplitude spectrum, and it is difficult to distinguish it from simple noise. The fricative s on the other hand has a distinctive spectrum that concentrates energy at a perceptually-optimal frequency. It is easier to hear s that it is to hear θ.
Whatever event caused the shift in Salishan fricatives in the development of the infrequent sound t͡s into t͡θ' was perceptually dysfunctional, taking a sound that was reasonably easy to perceive and making it harder to perceive. Such a change is, on general evolutionary ground, less likely to be successful. Now we can reframe the question as, what would make it likely that a language would have t͡θ' as a sound? Modern Halkomelem has it because pre-Modern Halkomelem has it, and so far, no sound change has wiped it out of existence (or perhaps it has in dialects). How did pre-Modern Halkomelem get this sound? Through an unlikely change applied to an uncommon sound. The historical changes that would give rise to that sound are simply unlikely.
Many natural sound changes will result in the stop t. It often arises from d, tʰ, dʰ, ɗ, t' by eliminating the laryngeal flourishes that distinguish those segments. It also arises from eliminating place of articulation distinctions among lingual consonants, where ʈ, t̪, t reduce to plain t. Voicing and glottalization of consonants involve extra articulatory gestures that incur an aerodynamic penalty (voicing of stops is difficult, compared to not voicing them; glottalization messes with the normal flow of air through the vocal tract). If there is to be some elimination of distinctions in d, tʰ, dʰ, ɗ, t', ʈ, t̪, t, these factors favor t as the outcome, rather than ᶑ.
Generally, people say that it is "hard" to pronounce certain sounds, which is cognitively true (they don't know how to do it), but not physiologically true. There are aerodynamic or acoustic (perceptual) explanations for the asymmetry between various phonetic properties, and that seems to be what "hard to do" comes down to.
