It depends on what language you are talking about, what language you speak, and whether you are talking about phonological analysis, or physical implementation.
Terminologically speaking, we usually talk about classes of sounds with categorical phonological terms, such as "voiced", "(un)aspirated", "ejective", "implosive" and so on. However, the way that sounds of a particular category are not physically produced in just one way, there is a range of known pronunciations. Another way to talk about language sounds is in terms of measurable physical properties, such as "vocal fold vibration" or "voice onset time" – ultimately you are looking for a number. In the case of voicing and aspiration, there has been a quest (for over a half century) to understand the relationship between phonological categories like "voiced" and "aspirate" and their physical phonetic realization.
Assuming that you are a native speaker of English, and with reference to your feeling about the p in "pot" and the b in "bought", there is an experiment that could be performed on you to test these feelings. The underlying issue is that you almost certainly can hear that the p in "spot" is distinct as a physical sound from the b in "bought", but in terms of finding the closest auditory analog, unaspirated [p] (created by editing initial /sp/ clusters) is not identical to initial /b/, but it is closer to initial /b/ than it is to initial /p/ (which is realized as [pʰ]). In other words, if you ask a question that probes auditory perception, you get one result, but if you as a question that probes the connection between perception and phonological analysis, you get a different result. In English (US English, I don't know about UK speakers).
There is a debate for English whether one should treat /p/ as /pʰ/ and /b/ as /p/, as opposed to the conventional /p,b/ analysis. One way that people have attempted to resolve the discussion is by appeal to other languages – /b/ of English is physically distinct from /b/ of French. This is somewhat appropriate since people usually think that the classificatory labels refer to physical properties. However, it is also the case that /b/ in English is not physically realized identically in all contexts including dialects: in general, language sounds don't have just one physical realization. English isn't the only language like this – North Saami stops are phonetically rather similar to English stops (but not in Finland dialects).
Advocates of the /p,pʰ/ hypothesis don't deny that their /p/ is sometimes voiced (produced with vocal fold vibration), they just attribute that to a separate rule of phonetic implementation. This is the converse of the position held by advocates of the /b,p/ hypothesis that sometimes /b/ is realized as [p] but remember that in those contexts, /p/ is realized as [pʰ] hence they are still distinct. In pre-pausal position, there is physical devoicing hence "height" and "hide" are phonetically different and there isn't aspiration. If you inspect spectrograms, you will see that /d/ in "hide" is partially-devoiced, at the right edge of the stop, whereas there is no voicing in "height" (instead, there is early glottal closure).
This is the sense in which voicing and aspiration are "the same" – it depends on the kind of analysis you are performing. I think the confusion largely arises from assuming that phonological analysis is the same as phonetic analysis.
