Your assertion that any human speech sound must be represented by some phoneme jumps the logical gun. The ubiquitous fact that cannot be denied is that humans make speech sounds. The claim that such things are "represented" (which is a mental claim) is built on that fact: what does it mean to say that sound is "represented"? The essence of that claim is that humans have various sound concepts, mental abstractions that represent classes of physical objects, assigned to one symbol. Those symbols include [k], [p], [g], [kʰ] and so on. The name (or, one name) for those symbols is "phone". Another is "segment".
Phones have all sorts of imaginable relationships. [p,t] have a certain relationship of phonetic similarity, [b,p] have a different relationship of phonetic similarity, similarly [p,pʰ], and so on. One of those relationship is the "allophonic" relationship, which distinguishes relations between phones in a certain way – [p, pʰ] are allophones, [p,b] are not allophones.
One cogent reason to not make a claim about a relationship being phonemic vs. allophonic is that you do not know whether that relation holds between the two sounds in question. And a very cogent reason to not make that claim is that you don't know what distinguishes allophones from phonemes. That uncertainty arises because there are multiple definitions of phoneme vs. allophone, and multiple criteria for analyzing sounds one way or the other. So if you are not strongly committed to the taxonomic phonemic definition of phoneme, then you may not wish to say, for example, that [t] and [ɾ] are "allophones" in English (some people think they are, but that is not possible given the taxonomic theory). Still you will want to talk about the sounds, and "phone" is the term we use to talk about sounds without specifically committing to the derivational claim embodied in saying "phoneme" versus "allophone". Insofar as the criteria for calling something a "phoneme" vs. "allophone" are at this point as solid as the criteria for calling sounds "marked" versus "unmarked", avoiding dubious distinctions is a reasonable choice.