The origin of grammatical gender is not necessarily well understood, but presumably it originated like any other inflectional feature and then became associated with gender when it was noticed that some prominent things of one natural gender fell into one paradigm and things of another into another, upon which those paradigms might have been generalised to other things of the same natural gender and, eventually, things that couldn't reasonably be said to have a natural gender. As noted, grammatical gender doesn't need to have anything to do with natural gender—in plenty of languages the distinction is between animate and inanimate things, for example, or more complex than that—it just happens that in Indo-European languages there's an overlap between grammatical gender and natural gender, so the two became associated in our languages. (I initially wanted to blame the Greeks or the Romans, but the word their grammarians used, γένος/genus, just means "kind" and has no connotation of natural gender.)
The other part of your question, why various languages have different gender systems despite being in the same family, and specifically in the case of Indo-European languages, is more concrete and easier to answer: because sound changes and other processes of language change lead to loss of contrast between the genders.
Late Proto-Indo-European had three genders: masculine, feminine, and neuter. Roughly speaking, in Germanic languages, the free accent of PIE was replaced by an accent on (basically) the first syllable of words, and one that was so strong that it led to the progressive weakening of vowels in unstressed syllables (an ongoing process in e.g. English), and eventually the loss of some of them, especially at the end of words. This meant the loss of all or most of the case system in all Germanic languages, as well as most traces of the gender system: it mainly survives through articles. Proto-Germanic had a demonstrative pronoun *sa that became the definite article in the West Germanic languages. In Old English it was still inflected for case and gender, but as it was unstressed and English continued the process of vowel erosion more vigorously than some of its siblings, by Middle English it had just become þe in all contexts (with a þ by analogy with some of the other demonstrative pronouns), and English lost its genders. German, on the other hand, kept declining its definite article into modern times (der, die, das), and retained three genders as a result. (Why did it keep inflecting the article? Probably for the same reason it retained a bit of its case system.) Dutch is somewhere in the middle: its descendent of *sa eroded into undeclined de, but it innovated a new article for neuters along the way (het, cognate with English it), so today it's often said to have two genders: common (formerly masculine/feminine) and neuter (but N.B. that almost all dialects retain a three-way contrast in the indefinite article, e.g. ne, een, e, and thus three genders; Hollandic, and therefore Standard Dutch, does not).
In another branch of the family tree, Latin had three genders inherited from PIE, but even though Latin nouns were (relatively) highly inflected, the neuter gender was mostly not very distinct from the masculine anymore. Already in the Classical period we find graffiti in Italy confusing one for the other, and with the export of Latin across Europe and the simplification of language that close language contact often entails, the contrast was eventually deemed too trivial to maintain and lost almost entirely in Vulgar Latin. Today, the Romance languages (effectively) have two genders; because the merged gender stands in contrast with the feminine, we call it masculine instead of e.g. common. In Italian and Spanish you can often still tell the gender of nouns by their ending (not always, but about as often as in Latin), but in French the endings underwent an erosion nearly as severe as in Germanic, and the articles are again the only clue.
The history of the Indo-European languages is broadly one of loss of inflectional complexity, though the details vary from language to language. As a result, if an IE language's gender system differs from its parent it will be in a reduction in genders. That's not an iron law: genders can be innovated as well. Early Proto-Indo-European itself probably had two genders (animate and neuter), and what was originally a collective/abstract suffix *-h₂ was reinterpreted as a feminine ending at some point (becoming, for example, the -a of Latin first-declension nouns). That seems to be rarer, though.