The answer to the question in the title is "it depends". The concept you are referring to is called a "distinctive feature" (or just "feature") in phonology. I can't provide a literature overview, since it is a complicated and extensively discussed topic, but here is an article by Arnold Zwicky that discusses the concept: "Phonemes and Features", published in Innovations in Linguistics Education 2.2 (1982).
As an example of how linguists can use phonemic contrasts to provide justifications for feature decompositions, I would offer this portion of the article:
The English phoneme /p/ would then be seen as an assemblage of the properties VOICELESS, LABIAL, and STOP, therefore as distinguished from /t/ and /k/ by being labial rather than alveolar or velar, from /f/ by being a stop rather than a continuant, and from other English phonemes by differences in two or more of these properties.
(page 64)
If we just consider the mathematical angle, these kinds of factorizations are obviously not necessarily unique (indeed, a language's phoneme inventory itself is not uniquely determined by the data, since we are not given phoneme divisions but must decide where they fall). However, most linguists rely on phonetic considerations, as well as the system of contrasts in a language, when deciding what features a phoneme has. I am not aware of any rigorously formalized method for doing this; thus, it seems to me that the process has a definite subjective element (my viewpoint perhaps leans more towards the "phonemes and features are theoretical constructs" than Zwicky's article, which suggests we can identify a "psychological reality of phonemes and features", p. 55).
Aside from the phonetic approach to features, a popular approach to identifying features is to look at which phonemes are treated similarly by rules of the language; e.g. in English, the fact that a vowel is inserted before the plural suffix /z/ after the phonemes /s, z, ʃ, ʒ, t͡ʃ, d͡ʒ/ could be used as a justification for considering these phonemes to be distinguished from all others by some common feature (which we might, based on their shared phonetic qualities, name "sibilant").