Since syllable-final voiceless consonants are also not aspirated ([ɹæt], not *[ɹætʰ]), we generally focus on saying when you get aspiration, and don't say that voiceless stops are intrinsically aspirated. So the rule for assigning aspiration to otherwise unaspirated voiceless stops is that they are aspirated syllable-initially. In skill, /k/ is not syllable initial, so there is no aspiration. It's therefor not about /s/ per se, it's about syllable position and the only thing in English that comes before a stop in a syllable onset is /s/ (or /ʃ/).
There is a problem with the syllable-initial analysis, that there is no aspiration in [ˈhæpi], that is between vowels where the first vowel is stressed and the second is unstressed (cf. also [ˈlɛɪˌtʰɛks]). Either you have to treat "happy" as [ˈhæp.i], or you re-state the rule to refer to the stress foot (a two- or three-syllable unit with stress in the first syllable). People have also tried to relate non-aspiration to physical properties of /s/, e.g. saying that /s/ is aspirated and this is a dissimilation process, but given words like mistrial (two stress feet) blaming the problem on /s/ does not get you far enough.