The other answers here are excellent and give good evidence. But here's another way of looking at it, from a phonological viewpoint.
Phonologically, the distinction is really between "syllabic" (acts as a nucleus) and "nonsyllabic" (doesn't act as a nucleus).
The division into "consonant" and "vowel" is pretty arbitrary, and you can't always draw a nice line between the two. In Chinese, fricatives can be syllabic; in English, nasals and laterals can be. Many languages have pairs of phonemes that are exactly the same except one is syllabic and the other isn't (such as
/j/). In PIE,
*j are reconstructed as allophones: is that phoneme a consonant or a vowel? In the context of this answer, when I say "vowel" I mean "something on the IPA vowel diagram", and when I say "consonant" I mean "something on the IPA consonant diagram"—but either of those can be either syllabic or non-syllabic.
Sometimes, it's pretty clear that you have a vowel + glide sequence, with two separate phonemes. For example, in Japanese,
/j/ acts just like any other non-syllabic consonant: you can have
/aja/, and even
/a.ika/, but not
*/ajka/. So in Japanese, it makes sense to analyze
[aj] as an underlying
/a/ (syllabic) plus a
Sometimes, though, it's pretty clear that you only have a single phoneme. In English, the sequence
[aj] acts just like any other syllabic phoneme. It can come before all sorts of codas, as in geist and likes, where other syllabic+non-syllabic sequences can't. And the
[j] never comes "detached", even to satisfy the Maximal Onset Principle: in my dialect, for example, your is
[jɹ̩], but liar is
[laj.ɹ̩]. So it makes sense to say that
/aj/ is a single phoneme in English, because that analysis is more elegant and explains all the data nicely.
And that's what a diphthong is: a series of multiple different phones, that acts as a single phoneme. The same can be said for affricates: phonetically,
[tʃ] is a stop followed by a fricative. But if you call it an affricate, you're saying that those two sounds act as a single phonemic consonant. (In Swahili, for example,
/tʃ/ patterns with the stops: it acts like a
/k/ or a
/t/, not like a sequence of consonant-plus-consonant.)
P.S. Like with anything in phonology, there's no indisputable "proof" of this. Phonemes aren't something we can see or measure directly. Instead, I'm saying that this theory, with diphthongs in it, explains the English evidence in a nice way. Maybe someday someone will come up with a different theory with no diphthongs involved! But for now, this theory is the most solid one I know of, and that's sort of the general consensus among linguists as a whole.