I think what the question boils down to is whether phonological primitives have intrinsic phonetic content. I can first give a "consensus" answer, and then point to some issues which seem to me to be unresolved.
Consensus view: This view is most clearly set out in Chomsky & Miller (1963) and Chomsky (1964) [the latter being a substantially revised version of Chomsky's plenary address at the 1962 ICL]. There are three basic ways of defining phonological features: (i) as distinctive features, which are used to organize contrast-bearing differences, and (ii) as indexes for grouping classes of sounds which uniformly undergo or trigger a phonological alternation, and (iii) indexes for grouping sounds by their positional characteristics. The features of (i) are thought to have a physical interpretation which is language-independent. Furthermore, it is assumed that there is a small number of such features, and their logical combinations form a universal phonetic alphabet.
Such a phonetic alphabet (e.g., the IPA) can be used for representing the contrasting segments in any language. Under the consensus view, however, the same set of features can be used for (i), (ii), and (iii). This means that it is assumed that all phonological regularities can be characterized with reference to labels which have a lanaguage-independent physical interpretation. Since a phoneme represents a group of sounds related by phonological regularities,
a phoneme can be represented as a symbol taken from the universal phonetic alphabet, its allophones differing from each other by feature-changing operations. Under this view, it makes perfect sense to use IPA symbols to represent phonemes. Such a view is assumed implicitly in most work in theoretical phonology today.
Some criticisms: There are two basic problems that have been identified with the consensus view. The first is that languages show consistent but subtle differences between similar-sounding sounds. This fact renders the enterprise of developing a universal phonetic alphabet less attractive. This kind of argument is made in several works by Ladefoged (e.g., 1980). The second issue is that there are so-called "crazy rules" in phonology which are difficult to account for in terms of phonetically specifiable features (therefore calling into question the identity between features of type (i) and of types (ii)--(iii)). This issue is touched upon in Fudge (1967) and subsequent work by different scholars. If these types of criticisms of the consensus view are accepted, then it does not make sense for reasons other than convenience to use IPA symbols to represent phonemes.
Noam Chomsky. Current issues in linguistic theory.
Mouton, The Hague, 1964
Noam Chomsky and George Miller. Introduction to
the formal analysis of natural languages. In
Robert Duncan Luce, Robert R Bush, and Eugene
Galanter, editors, Handbook of mathematical
psychology, volume 1, pages 271–321. John Wiley
and Sons, Inc., New York, 1963.
Erik C Fudge. The nature of phonological primes.
Journal of Linguistics, 3(1):1–36, 1967.
Peter Ladefoged. What are linguistic sounds made
of? Language, 56(3):485–502, 1980.