Many native EndlishEnglish speakers have yourthe same reaction as you, because they use the fortis/lenis (strong/weak) articulatory difference to distinguish [pa]/[ba], not the unvoiced/voiced difference. This leaves them free to devoice the [b], which in fact they often do, since it is less effort. So, it is likely that your pronunciation of "bah" has a devoiced "b", which makes it [pa], really.
The reason /b/ tends to be phonetic [p] is that there is no easy way to maintain the voicing of [b], because the air that must continue to escape through the glottis has no way to get out of your mouth. One thing you can do, which I refer to as the Bing Crosby maneuver, is to loosen the muscles of your cheeks to let them puff out, so the oral cavity has room to contain the extra air for the brief time the [b] needs it. Otherwise, the increasing air pressure in the mouth will choke off the voicing in the glottis by slowing the flow of air there. Dropping the jaw or the larynx has a similar effect.