There are two approaches to this problem: phonological comparison with phonetic assumptions, and computing acoustic similarity. I will only gloss over the latter approach. For the former, you need a standard decomposition of segments into phonetic properties, such as the SPE feature system. Then the difference between "cat" and "cad" is 1 because there is a single feature difference between the words; "cat" and "cab" are different by 2 (2 features), and so on. This is more fine-grained than your all-or-nothing phone comparison. It probably does not match "perceptual similarity" ideally, because "cat" would be equidistant from "cap" and "cad" whereas "cat" is probably closer to "cad" than it is to "cap". This can be remedied by assigning particular feature differences, so that place feature differences would increase distance. The only problem is that we don't actually have particularly good evidence about relative perceptual similarity. This has been a major problem for similarity-based theories of phonological change.
The acoustic similarity approach would have you compute least RMS difference between two waveforms (suitable trimmed), but you need somewhat normalize the data for overall performance differences (amplitude, noise, speech rate) while not obliterating real differences in amplitude between [a] and [u], duration of long and short segments (plus number of segments), and the greater noisiness of fricatives compared to sonorants.