I don't think it's a matter of direction—going to a language that makes extra distinctions being harder than going to one that leaves them out. Rather, it's a matter of how often the distinction you're gaining/losing is… I'm not sure how to put it, maybe "unexpectedly grammatically salient"?
Going from English to a language without gendered pronouns can definitely be a problem. Consider these English examples:
- John was telling me about Mary. He said… ("He" is clearly John.)
- John was telling me about Mary. She said… ("She" is clearly Mary.)
- John was telling me about Bill. He said… ("He" is clearly John.)
- John was telling me about Bill. ??? said… (There is no pronoun that can refer to Bill.)
As a native English speaker, I don't consciously think about the fact that (2) only works when John and Mary are different genders. So when I speak a language without gendered pronouns,1 without thinking about it, I may start trying to form a sentence like (2) and then have to correct myself when I realize I'm in the same situation a (4). It comes out something like:
- John was telling me about Mary. He/she said—I mean he/she Mary, not he/she John—he/she said…
I could imagine (although I don't have any research, or even anecdotal evidence) that this is an even bigger problem for native speakers of languages with more distinctions,2 or that make even more grammatical use of them.3
So, why aren't you tripped up by losing a 2.sing vs. 2.pl or 1.pl.excl vs. 1.pl.incl distinction?
Well, how many sentences or discourses have "just you" in one role and "you and John" in another role, or "me and you" in one role and "me and Mary" in another? It does happen, but nowhere near as often as having a male in one role and a female in another.
Going in the other direction, I'm rarely thrown off by having to add such a plural or clusivity distinction, either. It feels like (yeah, I know, real scientific… but I'll bet there's actual psycholinguistics on this) I apparently have that information readily at hand and can plug in without thinking, even though I generally don't need it in English.
But adding a gender distinction is harder. Why? Because there are plenty of cases where you clearly don't have that information readily at hand. You're talking about a generic person—or, worse, about a specific person whose gender is unspecified or unknown. As an English native, I'm used to dealing with that all the time, so I know going into the sentence that I'm going to have to deal with he/she. As a second-language speaker, you're not, so it hits you by surprise halfway through producing the sentence, and you're stuck with "I saw you talking to someone last night. What did… um… he/she?… say".4,5
1. In practice, many languages aren't as pronoun-happy as English; it's more idiomatic to repeat names than in English, or to use descriptions or titles in place of pronouns, etc., in which case I can just say "John was telling me about Mary. The woman said…" But that's not true for all languages.
2. For example, with a native language with 13 noun classes instead of 2 genders, or first-second-proximate-obviate person instead of first-second-third, or with whole extra axes like honorific, presumably you'd run into that repair situation more often than I do, and in more second languages, too.
3. Consider the way Bantu languages use noun-class agreement on the verb to distinguish subject and object, allowing relatively free word order in all sentences except those where the subject and object have the same class. Presumably, native speakers don't think about that any more than I think of when I do or don't have an extra pronoun distinction to use. And trying to repair a sentence where you intended to topicalize the object, but instead turned it into the subject… you probably have to scrub the whole sentence after you're more than halfway through and just start over.
4. English being pronoun-happy is probably relevant here, too.
5. And look at finer grammaticalized distinctions here. Proximate vs. obviate is rarely a problem to come up with on the fly, but when it is, it probably trips you up badly. Noun classes are frequently a problem, especially in a language that doesn't have a simple generic fallback class. "Whatever you have in your back, let me see… oops, I don't care whether it's a long thing or a round thing or a flat thing, but I have to pick a class anyway, so… um… let me start the sentence over and structure it completely differently so I can omit that word, OK?"