To some extent, this is just a question of terminology. In some languages, it is conventional to speak of "genders"; in others, "noun classes"; in some languages, the plural is considered to be one of the genders/noun classes, while in others, it isn't. In general, though, my impression is that linguists tend not to consider plural to be a distinct grammatical gender.
The number system of a language often interacts with the gender system, but that doesn't mean that they can only be analyzed as a single system. To make an analogy to another part of linguistics, the "tense" system of a language often interacts with the "aspect" system, but that doesn't mean that it is useless to have separate words for "tense" and "aspect". That said, some people do find it useful to be able to not distinguish between tense and aspect, and just speak of "TAM" (Tense-Aspect-Mood).
A good overview of the issues involved is "Gender" by Anna Kibort & Greville G. Corbett (Grammatical Features. 7 January 2008.)
They point out that it is often useful to distinguish between controller genders and target genders. Controllers are nouns, and targets are the words that show agreement with their associated nouns. The most typical, uncontroversial types of targets for gender agreement are other words in the noun phrase (like adjectives, determiners, or demonstratives) or the head of the associated clause (e.g. subject-verb agreement); anaphoric pronoun choice is also sometimes considered to be a type of agreement, but this is a bit more controversial.
If a language has n types of target gender marking, there may be more or fewer than n types of noun paradigms of controller gender.
Kibort and Corbett give Romanian as an example:
A good example of a language whose gender system has been the source of continuing disagreement is Romanian (for references to the extensive literature on this topic see Corbett 1991:150). The argument, which has gone on for decades, is whether we have two genders or three. In terms of agreement classes, the situation is clear: there are three classes that should be set up as follows (where the agreement endings are typical allomorphs for each target gender):
a. class I nouns taking -∅ in the singular and -i in the plural (e.g. bǎrbat 'man')
b. class II nouns taking -∅ in the singular and -e in the plural (e.g. scaun 'chair')
c. class III nouns taking -a in the singular and -e in the plural (e.g. fatǎ 'girl')
However, simply to say that Romanian has three genders suggests that it is like German, Latin or Tamil, even though in each of these languages, intuitively, the situation is rather different. It can be seen that Romanian has three controller genders (i.e. the genders into which nouns are divided), and it has two target genders (i.e. the genders which are marked on adjectives, verbs, and so on, depending on the language) in both singular and plural.
For Bantu languages, plural and singular nouns are often considered to belong to different noun classes. Sometimes one plural noun class corresponds to more than one singular noun class.
Kibort and Corbett say
'For two nouns to be in the same agreement class, they must take the same agreements under all conditions - that is, if we hold constant other features such as case and number. (...) If two nouns differ in their agreements when factors such as case and number are held constant, then they belong to two different agreement classes and normally they will belong to two different genders' (Corbett 2006:750). Note that the earlier Bantuist tradition treated nouns as being in different noun classes when singular and plural, and it is often stated that Bantu languages have twenty noun classes. Counted in the way outlined above, the number is typically between seven and ten.
To address the matter of German brought up in the question, it could be said to have three target genders in the singular, and one target gender in the plural. But the nouns fall into three controller-gender paradigms, not four.
