I recently came across oeuvre, which in the two out of two times I've heard the word spoken (in an English context), sounded like it does in Merriam-Webster's online audio pronunciation, that is, with a vowel that does not occur in any other English word, loanword or otherwise. (As a side note, various other dictionaries seem to list some very peculiar pronunciations, together with hilariously pronounced audio samples that do not match the transcriptions.) On the other hand, a quick check here and here suggests that the word has really been around in English long enough to have gotten more Anglicized. Are there any other English words with /œ/, or other non-English phonemes? What would have to happen to qualify /œ/ as an English phoneme? Is there a name for the use of phonemes that are not part of a language?
A good question, but I'm afraid there's no good answer to it.
The phenomenon has been noticed by linguists some time ago, and we have two terms: loan words and foreign words. For the latter also the German term is sometimes used, Fremdwort (pl. Fremdwörter) -- incidentally, itself a good example of its own meaning. There is at least one more like that in English, namely menu. I don't know why it hasn't been adapted better. Maybe because it's a 'vocabulary word', maybe because there aren't many more like it, and maybe both.
But the second possibility is key here: the feeling of nativeness of a phoneme/word/anything is a function of its frequency. English native speakers don't perceive [œ] as native because it's only there in very few (possily just one) words, and not a very frequent one to that. As it is, it's a foreign word. If there were many more like it, they would become loan words.
Polish originally did not have /f/. When the Latin name Stephanus was borrowed, it was adapted to Szczepan [ščepan]. But later many other words with /f/ have been borrowed and Poles began to perceive it as a native sound. The connection between Stephanus and Szczepan became obscure, and the name was borrowed again, yielding Stefan [stefan] this time. (Both exist in modern Polish and are seen as two separate names.)
Turkish originally did not have either /f/ or /ž/. Since the 10th century, however, it has borrowed a large number of words with /f/ but only relatively few with /ž/. As a result, the former is now seen as a native sound much more than the latter.
So you can see that the distinction is a gradient and comes down eventually to the proportion. Unfortunately, no scale or mathematical formula seems to exist which would allow to specify the feeling.
The process is not limited to phonemes. I wrote "phoneme/word/anything" above. It's because this is how we perceive familiarity in general. In the 19th c., English purists attempted to convince the society to abandon French, Latin &c. loan words and thus 'purify' the language. The slogan they came up with was Use exclusively Anglo-Saxon monosyllables. In case you wondered, use is a borrowing from French.