Actually, there is at least one more possibility:  the similarities might result from intensive influence of one on the other.
Now, to the answer:
Genetic (or genealogical, as I prefer to call it) affinity is quite difficult to prove beyond doubt. Perhaps the best way is to find a large number of regular sound correspondences in the most basic parts of the vocabulary, such as affixes, pronouns, low numerals, etc. No one can really tell what large exactly means in this context (see point 3), but if you can do that, the two languages will be considered related. Most Indo-European languages belong here.
An older version is only possible if at least one of the two languages is already dead. So, for at least one of them we only have a limited number of texts. If definite statements are to be made on this basis, these texts must either be rather long, or very obviously related, i.e. the great majority of affixes and basic words used in them must demonstrably and regularly correspond to the respective forms in the other language. For example, Latin has a large base of texts and they can be shown to regularly correspond to Italian.
Intensive influence is what is suspected when large from point 1 is not large enough for everyone, i.e. there remains a considerable number of affixes, etc, which do not correspond between the two languages. The Altaic "family" (Turkic + Mongolic + Manchu-Tungus + Korean + Japanese) is a good example of that. Linguists have argued for around a century now whether they are a family in the genealogical sense, or just have influenced one another very deeply in prehistoric times.
So, the main problem is the quantitative proportion. You might think that it would be enough to count the number of correspondences between the cases we are already certain about, and see if the appropriate numbers are higher or lower in the unclear cases. Unfortunately, no. Linguistics does not live in a void and in practice linguists always take historic, geographical, archeological, etc, evidence into account. Sometimes a relatively low number of correspondences is enough to pronounce affinity because the other pieces of the puzzle are very clear, but at other times the other pieces are few and so harder evidence is required on the linguistic side.
To sum up, linguistics alone can determine how two languages are related but not in every case is the conclusion certain. Sometimes, it is merely more likely than the alternative possibilities.