A lot of words in European and other languages come from Latin and Greek, if English (It's an example) words come from Latin and Greek, where Latin and Greek words come from?
This graph gives you a decent picture of the evolution of Indo-European languages:
The first line under Proto-Indo-European would benefit from a "Proto-" prefix, though (Proto-Indo-Iranian, Proto-Hellenic, etc.)
Derivation and borrowing are two different phenomena. Kernel English words derive from Proto-Indo-European, via Proto-Germanic, West-Germanic, Anglo-Frisian, Old English and Middle English. Words that come from Latin and Greek (or French, Old Norse, Celtic, Arabic or Nahuatl) are borrowings. You can spot the difference because the normal transformations that apply to derived words do not apply to borrowings. For instance, initial Latin /p/ corresponds to English initial /f/:
Pater - Father
Pisces - Fish
But English paternal, with an initial /p/ - which means it was borrowed from Latin, at a time when the transformation p->f was no longer taking place.
Latin and Greek, just like English, are Indo-European languages: they're descended from an ancestor language called Proto-Indo-European (PIE), which we know relatively little about because it was spoken before the invention of writing. Many Latin and Greek words can be traced back to PIE, though others cannot (sometimes because they're borrowings from other languages, sometimes because their etymology is unknown).
Of course, your question can be taken a step further back -- where did Proto-Indo-European come from? The answer to that is that, although many people have speculated, no one really knows, because there isn't enough evidence to work with.
(Btw, many or most common English words do not come from Latin and Greek, but directly from Proto-Indo-European.)