I don't think there is a general age. Some people retain the ability into old age; some people retain it into their 30's, maybe it declines for some around 20. Since p and b, f and v are contrastive phonemes of German, very few German speakers lose the ability to distinguish those phonemes. There might be a question about German speakers detecting it in word-final position, though.
English speakers who don't speak Logoori don't generally have the ability to distinguish i ɪ; u ʊ in that language, even though English has that contrast. The reason is that those vowels in Logoori are much closer together than they are in English.
There is a fundamental methodological problem with answering the question, that there is no good way to test if a person can "distinguish" two sounds. You would have to present listeners with two recordings and ask if they are different or the same. Assuming these are sounds of a language that they don't know, they don't have any idea about phonemic boundaries in that language, so you're effectively asking them to set aside their own language, and listen for absolute acoustic differences. You could perform the test by giving similar words, different tokens of the same word, or playing the exact same token twice – the question would be "are these the same, or different". I don't know of any good studies looking at sampled speech that way, though I've done this with pure tones with marginally different physical properties.
If someone does the experiment, they will have to include children in the subject pool, because we don't know how children (say age 8) are perform on this test. We know that ultimately they could learn any language, but this kind of experiment is not the same as learning a language.