I'm a high school student who will be going to college to study linguistics next fall. I'm already knowledgeable about some areas, but I'm currently trying to expand my knowledge in phonology. I have learned about both phoneme and distinctive feature theories. I think I understand their principal ideas, but I'm still unclear as to the justification for distinctive features.
I don't know a lot about semantics, but I'm aware of two basic theories of lexical meaning. One sees meaning as determined by the presence or absence of certain semantic characteristics associated with a word, but this one was rejected a long time ago for a prototype-based theory which sees meaning as a general categorization of this related by their varying degrees of similarity to a prototypical example. I'm sure semantics is a lot more complicated than that, but I don't see why phonology can't be viewed in a similar way. Distinctive feature theories are like the first semantic theory because they think speakers distinguish segments by the presence or absence of relevant features. It seems like there could be a phoneme theory that works like prototype theory, i.e., speakers distinguish segments by their degree of similarity to a certain auditory or articulatory standard for each phoneme.
I feel like the justification for phonemes is very obvious: speakers can tell when a segment is one phoneme or another. But distinctive features seem quite a bit more arbitrary. They assume that a particular characteristic is the specific thing allowing people to make the distinction between two segments, and they often seem to assume that many other distinctions in the language are made due to the exact same characteristic.
For example, the SPE theory [+/- anterior] seems particularly absurd: Doesn't [f] differ from [s] in the exact same way it differs from [ʃ], and how is having a constriction at or in front of the alveolar ridge relevant those distinctions?
Phonemes also seem less arbitrary since there is less room for uncertainty about what the phonemes of a particular language are. Different people will come to generally similar conclusions as to what the phonemes of a particular language are, but the features they include or the number they use could probably differ quite a bit. I mean, you could create completely random features like [+/- my friend sounds silly when she makes this sound]. Without the theory being 100% based in psychological evidence, I don't see any reason to unequivocally claim that one distinctive feature model is more valid than another (except maybe for reasons for computational efficiency, which doesn't have to exist a priori). That makes the justification for their existence at all seem dubious.
What is the justification for distinctive feature theories? I'm especially interested in psychological or cognitive justifications since those would surely be the most important, right? Content I've seen so far just says things like "there are some psychological justifications" without elaborating at all and switches to talking about other things like natural classes and allophonic rules.