The problem is, the IPA isn't quite as precise as it looks. Its goal is to capture every distinction made in human languages, and it does a reasonably good job of that—many languages have a contrast between some sort of [p] and some sort of [b], for example, so the IPA has separate letters for them—but that's not always the same as capturing the exact sounds made.
For example, both Spanish and Mandarin have a distinction between [p] and [b], but in isolation, a Mandarin [b] is very much like a Spanish [p]! You can imagine the degree of "voicedness" being a spectrum, and both Spanish and Mandarin have drawn a line somewhere on that spectrum, dividing their [b] from their [p]. But the Mandarin dividing line is so far toward the [p] side, and the Spanish dividing line so far toward the [b] side, that there's no easy way to decide what the "true" [p] should be.
Perhaps you object that the Mandarin stops should be analyzed as having an aspiration contrast instead of voicing, so calling them [pʰ] and [p] instead of [p] and [b]. That works. But then what to do about the English stops? English also has a dividing line on the "voicedness" spectrum, but the English one is in between the Spanish and Mandarin ones!
Fundamentally, the IPA just doesn't provide enough detail to synthesize speech without reference to the phonology of some particular language. Details like where exactly the dividing lines between [pʰ] and [p] and [b] are drawn simply cannot be answered in the abstract—only in specific instances. If you want more precision, you need a recording or a spectrogram.
