EDIT: Having found out that my answer has been voted down by three people, I have decided to revisit the topic and explain in a bit more detail my view regarding some cases of man-formations being suffixes rather than compound members. In order to do so, I have summarized my views at the very end of my answer and provided two corroborating quotations from references I have recently discovered on the Internet.
I don't think this is an easy question to answer, because whether a particular morph/component represents an affix rather than a compound member may differ from word to word, depending on a number of factors. In addition, you may need to specify what exactly you mean by English, because there seem to be differences from variety to variety.
As you know, there is a continuum spanning from phrases to compounds to derivations to inflections on the one hand, just as there is one spanning from free forms to enclitics to bound forms (and that is already a simplification) on the other, so it may be difficult to define rules differentiating between them within a particular language, and, perhaps, sometimes we may simply have to accept the fact that a particular word represents a transitional formation. Terms such as affixoid have sometimes been applied to formatives occupying the position somewhere between affixes and compound members by linguists.
Now, it's true that phonological reduction may be an indicator of the component's being an affix, but that's just one aspect of the formal side of the equation. You may also want to look at the semantics, and ask yourself questions such as "Is the meaning of the resultant form predictable from the meaning of its components?" and "If so, in what ways and to what extent?" Cross-linguistically, however, semantic criteria have shown to be not quite reliable either.
As far as the man-words in English are concerned, you may need to look into things such as "when the word was coined" and "whether the differences in pronunciation correlate with that", but that may only reveal the diachronic origin and may not be relevant from synchronic perspective. Speaking of which, I have come across an interesting Language Log post here followed by a discussion you may also find worth reading. And here's another page worth looking at (referred to from the Language Log post, in fact, and presumably by linguist Byron W. Bender).
You can also compare the -man component with, say, the suffix -er. Adding the latter to a verbal base XV derives the "doer of X". But what exactly does the former derive when we add it to a nominal base XN? How exactly does the -man derive the new meanings? How regular and predictable is the outcome? And if we split -man to -man1 and -man2, i.e. /mən/ or /mn̩/ and /mæn/, respectively, does the division correlate with anything in the language (e.g. geographical or dialectal distribution, semantic transparency, period of coinage)? How restrictive are the rules combining -man with X as to its form and part of speech? What about words such as snowman [ˈsnəʊ.ˌmæn] and frogman [ˈfrɒɡ.mən / ˈfrɒɡ.mn̩]? And so forth.
Some of these questions may prove to be more relevant than others, of course, and there may be a number of different angles depending on ones linguistic-theoretical background. I could restate my current opinion here, namely that -man may be seen as (becoming) a suffix in some words while a compound member in others, correlating at least in part with the relative recentness of their coinage and the two pronunciation variants reflecting this. The situation is somewhat reminiscent of the history of like, which has (1) remained an independent word (though transformed into a preposition, hence like a man), (2) became a transparently cognate suffix(oid) (manlike) and (3) evolved into a non-transparently cognate suffix (manly). Something limilar might happen to man if its semantics gets reduced, which is difficult to predict.
I hope you will find at least some of my thoughts useful. To read more on the issue of demarcating the borders between compounds and derivations, you may want to have a look at Angela Ralli's Compounding versus Derivation (pre-publication draft available as a PDF here), for example, and I'd suggest that you also read Understanding Morphology by Haspelmath & Sims (2010, 2nd edition).
Other references worth reading include Sign Language: An International Handbook by Pfau, Steinbach & Woll (2012) and Morphology Now by Aronoff (1992). I offer two quotations below, one from the former (p. 103), the other from the latter (p. 71):
How can one identify affixes in a language? What distinguishes them from compound members? First, and affix recurs in the language, co-occurring with many different base words, while compound members are confined to few bases. The suffix -ness, for example, is listed as occurring in 3058 English words (Aronoff/Anshen 1998, 245), while green (as in greenhouse, greengrocer, greenmail) occurs in about 30. In addition, affixes are more distant from their free word origin. While members of compounds usually also occur as free words in the language, affixes in many cases do not. Therefore, a morpheme that recurs in many lexical items in a language and in addition does not appear as a free form is an affix and not a compound member. Finally, allomorphy is much more typical of affixes than of compound members. This is to be expected, since affixes are more fused with their bases than compound members with each other. However, the difference between an affix and a compound member is a matter of degree, not a categorical difference, and can be hard to determine in particular cases.
And, more importantly:
According to [Grammaticization Theory], affixes go back to earlier lexical items that are reduced semantically and phonologically and become bound to the stem. For many English affixes the diachronic origin in a full lexical item has long been known to historical linguistis - for example, -hood (cf. Gothic haidus 'kind, manner'), -dom (cf. Old English (OE) dōm 'judg[e]ment, doom', Old High German tuom 'position, condition'), -less (cf. OE lēas 'devoid (of), free (from)'), -ly (OE -līc, second member of exocentric compounds based on Germanic *līk- 'appearance, form, body'; thus friend-ly is originally 'having the appearance of a friend'). For some English affixes, the origin in free lexical items is to some degree even synchronically transparent - for example, -ful (cf. full), -man (cf. man).
- First of all, we need to restrict ourselves to cases in which -man shows reduced pronunciation, namely /mn̩/;
- Since this may differ variety from variety, we need to stick to a single variety at a time; after all, a variety (such as a dialect) can and should be considered a system of its own, with its own rules etc.;
- We need to check whether the reduction is phonologically or otherwise conditioned, i.e. whether we can predict it from things such as word-length, its components etc. Quite clearly, we cannot - if we are to believe the above-mentioned sources. Examples like <Batman> /ˈbæt.ˌmæn/ vs. <batman> /ˈbæt.mn̩/ vs. (hypothetical) *<bat man> /ˌbæt.ˈmæn/ appear to prove this. See also no. 8.
- Orthography is a source of bias we need to take into account, hence the spelling <man> shared by both /mæn/ and /mn̩/ has to be considered irrelevant.
- Now, unlike /mæn/, /mn̩/ doesn't ever appear as an independent word. In my view, the latter form is much closer to being a suffix, whereas the former is indubitably closer to being a compound member instead.
- Even though it is obvious that it was a specific stress pattern that has given rise to the reduced pronunciation, and that the formations once were true compounds, strictly synchronically, they need not to be, just like in woman, which is not only not a compound synchronically, it should even be considered monomorphematic in my view. Again, bear in mind that spelling has to be ignored here.
- The relevance of the fact that literate native speakers consider a word as a compound may also be questioned, mainly due to the above-mentioned spelling bias, but also due to synchronic co-existence of affixes and words sharing similar origin and parallel miniparadigms (the connection of /-mn̩/ to /mæn/ may well be secondarily reinforced by the homophonous /wʊ.mn̩/ ~ /-wʊ.mn̩/).
We could provisionally assume that the words mentioned in no. 3 represent the continuum (a) derivative ~ (b) compound ~ (c) phrase and generalize the pattern based on stress:
- a. primary-none
- b. primary-secondary
- c. secondary-primary.