What counts as a letter is a matter of convention. In some languages, letters with diacritics are not considered separate letters, whilst in others, they are. Some languages even consider some separate letters and others not (e.g. in Spanish ñ is considered a separate letter, but á é í ó ú ü are not).
In some languages, some digraphs are even considered separate letters (e.g. in Welsh, ch dd ff ng ll ph rh & th are considered separate letters, and Hungarian goes even further, considering the trigraph dzs as a separate letter).
Romance languages tend not to consider vowels with diacritics separate letters, whilst Germanic languages usually do. As English derives its alphabet from the Old French alphabet, it makes sense that it would follow Romance norms here (also note that vowel diacritics are pretty uniformly optional in English, although for proper nouns they are usually preferred). Hungarian, and also Slavic Latin alphabets are largely based on the German alphabet (with heavy alteration for the differing phonology), and so follow the Germanic norm.
It's also worth noting that before the modern era, the alphabet was much less fixed than it is today, and many glyphs we would now consider punctuation were often included e.g. Tironian et ⁊ and the ampersand & (which derives its name from the phrase "and, per se, and" which is how it was usually named when it was placed at the end of an abecedary).
So to sum up, the reason for this difference is historical accident.