As described (empirical similarity from many listeners), I think you want confusability, e.g. a ‘confusion matrix'. Here’s an example, from Munson et al. (2002) JASA: tc.umn.edu/~munso005/ConfusionMatrices.pdf

As described (empirical similarity from many listeners), I think you want confusability, e.g. a ‘confusion matrix'. Here’s an example, from Munson et al. (2003): https://doi.org/10.1121/1.1536630

Full reference:
Munson, B., Donaldson, G. S., Allen, S. L., Collison, E. A., & Nelson, D. A. (2003). Patterns of phoneme perception errors by listeners with cochlear implants as a function of overall speech perception ability. The Journal of the Acoustical Society of America, 113(2), 925-935.