Assuming the test subjects are not trying to fool the test (by intentionally picking the wrong answer or feigning knowing when they don't), then it's entirely reasonable that you can derive a reasonably confident list of distinctions the test subject is able to hear.
From that, you may be able to match that with trained (or manually curated) models on languages' expected matrices.
On the positive side, a test subject may be able to distinguish between lake and rake, but what would this tell us? Very little. But while it wouldn't allow us to confidently eliminate languages that lack that sort of a distinction, it might provide a strong clue.
On the negative side, a test subject may not be able to distinguish between lake and rake. This is slightly more informative as we might be able to eliminate many languages with distinctive contrasts of that sort.
Overall, I am strongly confident that you would be able to create a test that can be used to differentiate speakers of two drastically different languages. Given good statistical modeling, I am also confident that you'd be able to make a fairly strong n-way distinction.