There is a computational metric which gives you the "distance" (in phonological features) between an arbitrary test string and all other words of English (it requires a phonetic dictionary of English). The original idea was that [bnzk, bnɪk, blɪk] are not English words, and [brɪk] is; but in the non-word set, [bnzk] is the least word-like, [bnɪk] is next most-word like; [blɪk] is a possible word, although it isn't an actual word. As far as I know there is no empirical underpinning whatsoever to the metric, that is, it hasn't been tested to see if there is any psychological reality to the metric, and the original form of the test is not actually computeable (d'oh!).
The usual alternative is a simple yes/no intuition test, i.e. "is [flæmp] a possible word of English?". This seems to be the test that you are looking for. The test is widely employed by phonologists, generally using themselves as test subjects. The method underlies decades of research on syllable-construction rules, where the goal is to capture the generalization that [flæmp] is a possible word even though it is not a word, but [flimp] (with a tense / long vowel) is not even a possible word. There are numerous border skirmishes involving marginal, Yinglish and anomalous words (sphere, Zbigniew, shpritz, pueblo), but there is a clear set of impossible words, and a somewhat clear set of possible non-existent words. The patterns were discussed in the literature on autosegmental theories of syllabification. If you go with standard phonologists' judgments of possible vs. impossible words, that gives you [bnɪk,bnzk] as not-possible and [blɪk, brɪk] as possible.
Here is where it's not clear what you want. Do you want to exclude just [bnɪk, bnzk]; or [bnɪk, bnzk, Zbigniew]; or [bnɪk, bnzk, Zbigniew, pueblo]...? The problem is that "possible" as opposed to "actual" is about a system of rules. We cannot directly verify what the rules are, so we substitute speaker feelings about words to fine-tune what the rules must be. So we have to separate the speakers (and their intuitions) used to construct the rules from the test subjects used to test this possible / impossible distinction (assuming that those subjects can accept "possible but non-existent" as different from "impossible and non-existent").
Since we are in the realm of the little-explored, the domain is ripe for pilot experiments. For example, if you give Hindi speakers a Sanskrit text, "how Hindi" are the words judged to be (controlling for actual knowledge of Sanskrit) – maybe just giving uninflected roots? Same with Bengali, Sindhi, Dogri or Sinhalese – in Devanagari. And likewise Tamil or Telugu; or Pashto; or Khmer; or Dinka. The segmental inventory of Dinka is so different that you would face serious spelling problems (vowel phonation, tone, diphthongs). I think I could distinguish a Tamil Devanagari corpus from a Hindi Devanagari corpus (having only a superficial linguist's knowledge of either) based on the "appearance" of Tamil, provided you could come up with a consistent scheme for converting தமிழ் அரிச்சுவடி to देवनागरी (dealing with the extra vowel and consonant letters). I don't think that [māṭu] violates any strong phonotactics of Hindi (cf. जाति [jāti]), which I suppose is a tadbhava word so not "really" Hindi). My conjecture is that after eliminating the Tamil e/ē (etc.) contrast, admissible Tamil strings are technically a proper subset of admissible Hindi strings, but if you did a statistical study of words, the majority of Tamil words have a different phonotactic pattern (more syllables, severe coda restrictions) from that of Hindi (fewer syllables, fewer polysyllabic V-final words), which could influence judgments of Hindi-ness.