In many Indo-European languages, like Latin, the masculine is less "marked" than the feminine, meaning that it's the more basic or fundamental form: the one you use by default unless there's a reason to do otherwise. While sexism might play a role in this (certainly the ancient Romans weren't particularly feminist), there's also a more mundane historical reason. The feminine gender seems to have been a later development in the history of Proto-Indo-European, which made it more marked than the masculine or the neuter—in other words, the three genders were originally "animate", "inanimate", and "this special new marking for specifically-feminine things". If something wasn't specifically feminine, it didn't get the special new marking. This seems to have led to the convention that was inherited by Latin, that groups of people and generic individuals used the masculine gender.
Of course, this was thousands of years ago. The generic masculine in modern English is a recent development, as you noted: English used the non-gendered "they" for groups of people and hypothetical/non-specific individuals until prescriptive efforts arose to make it more like Latin. (You can find lots of traces of these prescriptive efforts in modern English: "don't split infinitives" and "don't strand prepositions" are similar rules imposed to make English more like Latin, which are still taught in schools but most people don't really follow.)
Other languages may have the convention for other reasons. In Proto-Afro-Asiatic, there seems to have been a two-way masculine/feminine gender distinction—but when using an adjective generically, this changed to an animate/inanimate (or sometimes concrete/abstract) split. So while "good man" would be masculine and "good woman" feminine, "some good person" would also be masculine, and "some good thing" or "quality of goodness" would be feminine. This is the system that appears in Ancient Egyptian and Akkadian, and likely also in some modern Semitic languages (though I don't know any of them in enough detail to say for sure). This leads to a "generic masculine" convention, but for fairly arbitrary reasons: repurposing their existing morphology to distinguish between "good person" and "quality of goodness".
Tl;dr this happened for different reasons in different languages; sexism may well have had something to do with it, but there are other (often-arbitrary) historical forces in play.
