Stops and fricatives (the obstruents) are the segments that are most-conducive to a voicing contrast. The sonorants typically do not have a voicing contrast, and are usually voiced (save for the laryngeal glides). The main physical difference between the obstruents and the sonorants is that obstruents have more-significant impedance of transglottal airflow – complete blockage in the case of stops, substantial in the case of fricatives.
In order to produce voicing during the production of obstruents, you have to do something special in order to maintain the airflow necessary for voicing. It is possible, but more challenging, to produce voiced obstruents, by lowering the larynx, raising the velum (or allowing nasal leakage), and often voiced obstruents are not fully voiced. The sonorants pose no such problem, but they pose a different problem that it is hard to prevent voicing during their production – the vocal folds have to be significantly abducted, to prevent natural vocal fold vibration.
There are sub-generalizations within the class of obstruents for example that [g] is at a disadvantage compared to [b] or [d], and [ɢ] is very rare compared to [g]. This is explained by the fact that [g,ɢ] have a short supraglottal cavity behind the consonantal constriction, where air pressure rapidly builds up and the transglottal pressure drop becomes zero (no voicing).
There is probably also a historical explanation for some asymmetries. For instance, intervocalic spirantization often results from shortening of a stop's constriction, so that the stop becomes "fast" and the closure is incomplete – it sounds like a voiced fricative. The peripheral consonants (labials and dorsals) often are reanalyzed as fricatives, but the coronals tend to be reanalyzed as flaps (indeed, IPA doesn't have a dorsal flap symbol, and only recently added the extremely rare labial flap symbol). This asymmetrically increases the supply of fricatives at peripheral places of articulation. The physical reason for this central / peripheral difference is probably related to the massiveness of the tongue body or lips as articulators, vs. the agility of the tongue tip.