First, there is a lot of variation in English, so don't expect the facts to be the same for all speakers. Second, it's unclear what you mean by "really". There is phonological analysis, and there is acoustic analysis. The standard mostly-phonological analysis is that "pin" has aspirated [pʰ] and "spin" has unaspirated [p], but they reflect a single phoneme /p/, which contrasts with /b/. It is often said that that /b/ is voiced, but for most speakers it is not always voiced in the way it is in French, so there is an alternative analysis of the phonemic contrast /p,b/ as being one of aspiration (the phoneme "p" is treated as aspirated and "b" is voiceless and unaspirated).
If we say that "p" is /pʰ/ and "b" is /p/, the description of the allophonic difference between the variants of "p" is harder to explain. One solution is to invoke an additional property, tense / lax or fortis / lenis, so /p/ is fortis and /b/ is lenis, and then you are free to say that fortis stops can be aspirated in syllable-initial (or foot-initial) position. Without some additional feature, and if you are not allowed to say that /b/ is voiced (given the presumption that there is no vocal fold vibration during its production), then you need something to distinguish the labial stops of "pit, bit, spit, happy, cabby". Actually, in words like "cabby", "b" is produced with vocal fold vibration. The confusion situation is because people do not agree on the criteria for saying what some language sound "really" is – is it a claim about phonological pattern, or a claim about phonetics.
It may be useful to gather some good recordings of yourself saying "spit", "pit", "bit" and so on, and perform the editing test, where you (use Praat to) remove the fricative from "spit". Then the question is, can you tell that "spit" with [s] removed is distinct from initial "b", or do "(s)pit" and "bit" become indistinguishable. It depends on the task that you set for yourself. If you set it up to ask "does this sound most like pit or bit", you will never think that edited (s)pit sounds like "pit", therefore you will always say "sounds like bit". But if you do an a-b comparison (do these two sound the same), you might distinguish (s)pit from bit. That is, phonetically they do sound different. (You have to be very dedicated to the question, in order to be willing to put in the work necessary to do this experiment). Alternatively, you could just make many recordings of "spit" and "bit" and measure voice onset time to see if there is any difference in VOT between /p/ and /b/ in this context.
