The Beti-Fang subgroup of languages of Cameroon (and I suspect other related Bantu languages) have this property. Example languages are Ewondo, Fang, Ntumu, Bulu. Like most Bantu languages, there is a class system involved in agreement and singular / plural formation. Adjectives, which are pre-nominal, have a lexical class, but demonstratives and numerals, which are post-nominal do not have any lexical class. The agreement properties (NP-internal as well as subject agreement on verbs) of the NP are determined by the first class-bearing word, which would be the noun if there are no adjectives, but the adjective (first adjective) if there are. Alas, I can't put my hands on examples at the moment, but I will insert examples when I get them.
[Edit]
Some examples, from Ntumu. Numerals refer to the class number, and nouns have a prefix reflecting their lexical class. Demonstratives are post-nominal; [ém-boro ny-í] "this-1 person-1", [é-boro b-á] "those-2 people-2", [ʌ́-nló !w-í] "that-3 head-3", [mí-nló !m-í] "those-4 heads-4". Possessive pronouns are post-nominal; [ém-boro wɔ-m] "my-1 person-1", [é-boro bâ-m] "my-2 people-2", [á!-bɔ́ dâ-m] "my-5 leg-5", [má!-bɔ́ dâ-m] "my-6 legs-6". Numerals are post-nominal; [bi-tɔ bí!-bɛ́ŋ] "two-8 chairs-8", [mʌ-kɛ́ mʌ́!-bɛ́ŋ] "two-6 leaves-8". "All" is post-nominal; [boro bʌ́!-sʌ̂] "all-2 people-2", [mi-nkwɛ́n mí!-sʌ̂] "all-4 crops-4", [ʌ-jwi á!-sʌ̂] "all-5 (the) banana-5" (i.e. the whole banana). Wh-modifiers are postnominal; [m-boro m-bé] "which-1 person-1", [boro bʌ-vé] "which-2 people-2".
Here are some adj+N examples. In these, there is a class-agreeing prefix on the noun which I gloss as "AGR" (it is also used in N of N constructions), and there is a fair amount of phonological complexity to its realization but basically the AGR prefix plus the noun's lexical prefix can't be more than one syllable. Given citation nouns [fâm] "male-1", [bʌ-fâm] "males-2", [e-lé] "tree-7", [bi-lé] "trees-8", we have [e-tun ʌ́+fâm] "short-7 AGR-7+male-1", [bi-tun bí+fâm] "short-8 AGR-8+male-2" [e-tun e+lé] "short-7 tree" [mʌ-yab m+élé] "tall-6 AGR-6+trees-7", [bi-tun bí+lé] "short-8 AGR-8+trees-8". I think this is enough to show the general pattern that the agreement pattern is controlled by the NP-initial adjective, and it can cause deletion of the lexical class prefix of the noun.
The noun ([ʌn-sîn] "enemy-1", [bʌ-sîn] "enemies-2") governs post-nominal agreement as expected in [ʌn-sín ʌ́!m-bé] "which-1 enemy-1", [bʌ-sín bʌ́!-vé] "which-2 enemies-2", and also subject agreement on the verb in [ʌn-sin a-ku] "enemy-1 fell-1" (sorry, no tones in this section), [bʌ-sin ba-ku] "enemies-2 fell-2". But when there is an adjective, the class of the adjective controls the agreement pattern: [a-yab ʌ+n-sin da-ku] "tall-5 AGR-5+enemy-1 fell-5" ("a tall enemy fell"), [mʌ-yab mʌ+sin mʌ-ve ma-ku] "tall-6 AGR-6+enemies-2 which-6 fell-6" ("which tall enemies fell").
True, more needs to be said about the details, but the point is that in this set of languages, adjectives do have lexical class, and can actually control agreement.