A "wide-spread assumption" means there is no unique formal argument. Rather, following the observed rarity (potentially none), a rationalization has to be imagined.
First of all, it might be excedingly unlikely that a language with the need and opportunity to loan a suffix wouldn't also take the opportunity to loan whole words. Further, for the acceptance of the loan beyond bilinguals, a naturalized loan word would help to establish the phonotactics, and more importantly, the meaning. Coralary, speakers of the target language are unlikely to bracket and isolate the suffix correctly from sparse input. In successful loanwords, the suffix becomes reanalyzed and reinterpreted, respectively, often enough, in which case it's clear that neither the stem nor the suffix were really loaned independently. This implies that a suffix alone doesn't obviate it's meaning. If it would, it would probably become perceived of as a stem.
In the same sense, if the suffix is originally inflecting as it tends to be in synthetic languages, the whole inflection paradigm is even less likely to be borrowed due to its complexity (and the difference to endings is difficult to maintain in that sense). An uninflecting suffix would loose a lot of its functionality, which is offensive to the native speakers who are most likely to employ the words as lenders to begin with.
Vice versa, in analytic languages, isolated words would be more likely than suffixes (stems, as I said).
These are really just my own presumptions.