More theory than history for you, but one take on it:
Language evolution is an eternal tug-of-war between ease of articulation and information density. We want to say things quickly and learn how to say them easily, but we also want to be able to communicate with nuance.
English is a fairly extreme case. For regular verbs, we have only two forms across all person-gender-number combinations: I run, you run, we run, they run but he runs. However, we're hugely inefficient; we could halve our numbers right now and go with Swedish's one form for all combinations!
However, there's a tradeoff. English has six subject pronouns: I, you, he, she, we, they. Swedish has seven: jag, du, han, hon, vi, ni, de. You might argue that this is not a direct tradeoff, since the extra pronoun differentiates "you" (sg.) from "you" (pl.) rather than 3rd-person singular, but such tradeoffs generally aren't one-for-one. The point is that there's some effort needed to encode and decode meaning, and some kind of equilibrium must be maintained.
Likewise, on the opposite end of things, you have Spanish. There, you have many forms: quiero, quieres, quiere, queremos, queréis, quieren. Six forms, but guess what? No pronouns needed. Which is easier: always having to tack on "I, you, we, they" or always having to tack on "o, es, emos, eren"? In fact, are these even two different strategies from a theoretical point of view?1
Noun and pronoun classes and endings: OK, they're hard for non-natives. But they serve a purpose. They tell you the role words play in the sentence. As a Russian speaker, you'll appreciate that the meaning of Влад любит Ану and Ану любит Влад are the same. This is not the case in English, where we must obey strict rules about sentence order and avoid saying "Ana loves Vlad" if we mean that Vlad loves Ana!2
In this case, there's actually no tradeoff between encoding and decoding difficulty; rather, there are two different strategies that yield more or less equivalent results. In English, the encoder is forced to use a sentence template of a certain structure in order to yield the correct decoding, but needn't worry about word forms. In Russian, the encoder is forced to worry about word forms, but needn't worry about a sentence template.
"Aren't some strategies much worse than others?" people often ask. "I mean, I would far rather have to say things in a logical order than memorize two dozen personal pronouns in German." First, the perceived difficulty is relative to your starting point; the fewer cases and the more word order concerns you have in your native language, the harder you'll find German.3 Second, there's a pretty high limit on complexity. Children learn such things perfectly — sure, it might take longer to learn this or that feature of some language, but there's always a plateau that the vast majority reach after only a couple years.4 At that point they don't perceive it as complex (unless through analytical education...).
Sometimes the systems change. The encoder gets tired of always having to decide between "ins Bett" and "in dem Bett",5 or maybe she speaks too quickly to distinguish what she's saying anyway, or the environment is too noisy. Maybe she's speaking to a child and makes just enough mistakes that the kid fails to realize only one is correct and produces the two in free variation for the rest of his life. Luckily, the context makes it clear often enough that, lo and behold, in a few generations it's acceptable to some significant subset of the population, and a case system begins to collapse.
Meanwhile, somewhere in the southern US, someone is overheard saying too loudly, "I'm inviting you to a barbecue!" and a bystander remarks: "Um, do you mean 'you' as in 'you alone' or as in 'all of you'?" The inviter hesitatingly replies, "Er, you all. You all are invited." Soon there's an inside joke about "You all" and lo and behold, in a few generations there's a new distinction in plurality.
Then, in both cases, the desire for analogy across paradigms can suppress or reinforce the change. Like airflow on a small fire, sometimes it extinguishes what's too exceptional to keep standing out, and sometimes it feeds what's just big enough to absorb the attention.
1 Depends on the theory. Also, not all the affixes are regular, but this added difficulty is made up for in other ways.
2 I'm no native speaker of Russian; if this example doesn't work, please correct me.
3 By the by, a couple more nice things in German: adjectives are invariable after nouns, and there are no adverbs. Wait, what?! Yup... I mean, there is that function, but no form attached to it — you just use adjectives. That's an advantage over even English where someone might chide you for saying "I'm doing good" instead of "I'm doing well."
4 The same goes for phonology. Just yesterday I heard a little boy say to his mom, pointing to a closed-off trailway, "We can' go fwu daew." In a year or two he'll be able to perfectly pronounce the pesky English sequence "through there".
5 Not consciously, of course. I'm thinking of a psycholingustics lit review I did in undergrad on the hypothesis that much of the executive function related to language is selection: we are constantly selecting between alternatives, inhibiting some and okaying others that surface as speech. Anyway, not that anyone should put stock in random undergrad paper, but an interesting idea and there were some interesting findings I cobbled together to argue for it.
