When I whisper, none of my consonants is voiced. But I can tell the difference between voiced and voiceless consonants. How is that possible?
Whispering involves a low-amplitude non-periodic noise source at the glottis. This is harder to hear than modal phonation, but is enough to allow phoneme discrimination. In English, voicing is encoded in a number of ways, sometimes with vocal fold vibration, but also with duration, aspirations, and constriction size. Probably the most salient feature is aspiration, and this actually gives some evidence in support of the claim that voiceless fricatives are aspirated in English, since "sue" and "zoo" are still distinguishable (as long as the room is quiet). Also, there are a number of allophonic differences that are still applied, such as vowel raising before voiceless (rapid has a higher, shorter vowel; rabid has a longer, lower vowel, and this is maintained under whispering). It would be interesting to see how whispering plays out in in tone languages, and in languages with a contrast between voiced stops vs. voiceless stops with negligible voice lag.
Human speech is built in a very, very robust way to ensure mutual comprehension and thus it is quite rare that only one feature expresses some value. Typically there are other features that coincide with it in some way and if the original feature is absent, their presence indicates that it is intended to be there.
With voiced/voiceless consonants, it is typically tense/lax category, where voiceless tend to be more tense, in English actually to the point of aspiration (in Germanic languages, this is so important that actually phonologists typically do not even use the voiced/voiceless classification but actually go for the tense/lax or aspirated/non-aspirated).
Another important thing is that we typically underestimate the extra-linguistic factors in comprehension - when two people talk, there is actually a surprising amount of lip reading (to the point that in films or video games, people are heavily bothered by bad lip sync), which further helps to identify the sounds and again the tense/lax can have strong visual clues.