I was reading a paper called Loanword adaptation as first-language phonological perception for a term paper I'm currently writing. I don't know enough about the literature to know if this is a particularly novel idea, but I think that their OT-based perception analysis (as seen in section 3 of that paper) is relevant here. I'll describe the theory first and then I'll talk about how I think it applies to your problem.
For people who don't know, Optimality Theory is basically an alternative to rule-based systems where instead of having specific inputs undergo specific transformations to become specific outputs, there is simply a bunch of preferences where some have a higher priority than others (and thus are less likely to be violated).
Usually when people talk about OT they're talking about the production process (from UR to SR). However, that paper mentions an OT-based approach to perception. So, for example, consider a language that does not contain /ɛ/.There must be some highly ranked constraint *[ɛ]/ɛ/ (meaning the phonetic sound [ɛ] must not be interpretted as the phonological unit /ɛ/). This prevents speakers of that langauge from perceiving /ɛ/ when they hear [ɛ]. This would be desirable because, due to the lack of /ɛ/ in that language, any SR containing /ɛ/ would be unexpected.
Let's compare this to a language that does contain /ɛ/ as well as /e/ (like French). It is important to note that in OT all constraints exist across all languages only the rankings change. Therefore, that same *[ɛ]/ɛ/ must exist in this language too except that now it is much much much lower priority. In this langauge, since /e/ and /ɛ/ are different, we know there needs to be another constraint *[ɛ]/e/ (do not interpret [ɛ] as /e/) that has higher priority than *[ɛ]/ɛ/. This means that when speakers hear [ɛ] they are more likely to perceive it as /ɛ/ because perceiving it as /e/ is a greater violation (*[ɛ]/e/ >> *[ɛ]/ɛ/).
The same paper proposes that the ranking of these constraints is somewhat dynamic and based on a "probability-matching listener". This means that, theoretically speaking, if a French speaker is systematically forced to accept [ɛ] as /e/ (for example, by speaking for an extended period of time with an L2 French speaker from a language without /e/, only /ɛ/) then the *[ɛ]/e/ constraint slowly loses priority and the French speaker slowly becomes more ready to identify [ɛ] as /e/ (and of course, the process will slowly undo itself when the French speaker returns to speakering to other native speakers). Taken to its logical extreme, this could explain why certian languages merge certian sounds while others do not (i.e. if a language does not distinguish between two sounds, violations of the relevant constraints become so lowly ranked that faithfully perceiving phonetic input becomes more costly than approximating it).
So here is how I think this applies to your problem: in English the constraints *[ɛ]/e/ and *[e]/ɛ/ are both very lowly ranked, allowing English speakers to freely interpret one as the other. Conversely, in French, where misinterpreting one as the other causes a loss of meaning, the constraints are much higher ranked and thus it is much less likely that they will confuse to two. By this approach, it isn't that speakers "lose comprehension" when faced with similar phonetic sounds, it's that the natural "error correcting" in their brain interprets the sounds as whatever sound makes the most sense in a given context.
As mentioned earlier, I'm not too familar with the literature, but I've found another author who's written a few papers around a similar idea of a productive perception as well as both sets of authors cite various other authors who came before them.
