It shouldn't be controversial to say that the grammar of certain languages is more complex than what communication calls for. For instance, some languages have gendered nouns, and it is often unclear what is the communicative purpose.
Is there any consensus on why grammar becomes "more complex than needed"? If not, what are the leading theories?
Speech (or signing, though there's been less research done on this) is an inherently imperfect communication channel. Information gets lost or corrupted very easily, and there's just no way to avoid it—communicating in a crowded party, or shouting across a long distance, or holding a conversation while music plays in the background, is often necessary.
As a result, human language has a whole lot of redundancy built into every aspect. Yu cn prbbly undrstnd Englsh wth all wrd-intrnl vwls ersd, even though that destroys a whole lot of information. Or you can hold a whole conversation in whispers (removing voicing information), or have entire words drowned out by noise, and still understand everything. This isn't a bug, it's a feature.
For your specific example (noun gender agreement), you can think of it as redundantly specifying what category a noun falls into, by putting that marking on adjectives and verbs as well as nouns. Or you can think of it as redundantly specifying which words go together, by marking them explicitly instead of just relying on word order. Either way, it doesn't (usually) add information you wouldn't get otherwise—it just adds redundancy in case one source of that information fails.
Complexity makes a grammar hard to learn, but not hard to use. Speakers of a language, having already learned the language, are not really interested in reducing its complexity. It is also not a conscious choice to adhere to the language's rules, so they don't feel "restricted" by them.
What an language speaker wants from their language is the ability to get their point across accurately and fast. So a grammar has to reduce ambiguousness while still allowing to form short sentences. That is what they need for "communication purposes".
If we measure complexity as the number of rules a grammar has, we see, that:
We need rules to make sentences less ambiguous: Forcing the word order to be SPO or marking cases on nouns gives an unambiguous way to parse a sentence.
Adding more rules to a grammar can also make it less verbose.
For example, to be being irregular in English allows for the shorter I'm, he's etc.
But making a language less ambiguous also entails forcing speakers to give some extra information:
For example, you can drop the subject in Japanese and it has to be understood from context, sometimes leading to misunderstandings. German and English force you to specify the subject to counter this possible ambiguousness. This, on the other hand, makes the language slightly more verbose.
Most grammar rules I can think of reduce either verbosity or ambiguousness.
Regarding gendered nouns: German forces its speakers to mark the gender of almost every person as soon they are mentioned by using gendered nouns. So German speakers, in return, think it is very strange to hear from "a teacher" and not know if they're male or female.
This may not sound so useful, but gendering nouns comes at a very low cost in German: it is mainly marked on articles and adjectives that are already inflected for case, and the inflections are very short and easy to pronounce.
The biggest benefit of gendered nouns is that it reduces the ambiguousness of demonstrative, relative and personal pronouns, because only nouns with the right gender could be meant.
In the sentence
Das Pferd ging in den Stall und aß eine Karotte. Sie war orange.
meaning
The horse went into the stable and ate a carrot. It was orange.
the it can only be the carrot, not the horse. Using the other pronoun "es" we could have made it the horse.
To summarize: Complexity is not really a problem of a language's speakers, and adding more complexity can reduce ambiguousness and verbosity, which bothers speakers the most.
The question appears to assume that the language only exists as a method of communicating factual descriptions and making demands between humans. Once you consider the full range of things that language is actually used for, you should be able to see that mere communicative efficiency can be subordinated to other purposes, such as obfuscation and artistic expression. Language is as complicated as is needed, and the linguistic needs of humans goes far beyond low-level informational exchanges. It is as complicated as is necessary for all of the purposes of humans.
I see everyone answering regarding the neutral reasons why grammar gets complex and stays this way, but no one wrote about the more "political" aspect, so allow me:
Grammar (and by extent, Orthograph), was often normalized when only rich/powerful people knew how to write. Back then, while matching the oral language was still the main goal, written complexities also helped to prevent the "commoners" to access a higher level of education. This doesn't apply to the grammar you can hear orally of course, but there are languages (for instance French) where most grammar doesn't exist orally.
Besides, even when the oral language evolved and sometimes got rid of old grammar rules, they often stay at the written form, again for social reasons.
