Although adverb agreement in gender/noun class is far from ubiquitous, there seem to be (apparent) examples of this kind of agreement in a fair number of languages. I am most familiar with examples of gender-agreeing adverbs from Indo-European, since that is a large and well-studied family containing many languages with gender systems. But there do seem to be examples from other language families as well.
Some Indo-European languages have words that despite being analyzed as adverbs show gender agreement in certain contexts. All of the examples that I have seen so far seem to be limited either to particular words or particular contexts: I don't know of any Indo-European language that always shows gender agreement for all adverbs. Also, all of the examples of gender-agreeing adverbs that I've seen so far seem to share their forms with non-adverbs. By this I mean the forms used for gender agreement are not exclusively used as adverbs, but are also used in other contexts as e.g. quantifiers or adjectives.
Gender agreement on adverbs can be found in some Romance languages for specific types of adverbs. In some Romance languages (e.g. French and Spanish), many deadjectival adverbs are formed with a suffix -ment-: as far as I know, -ment- adverbs never show agreement. But some other kinds of adverbs may show agreement.
(Interestingly, Hummel (2017) suggests that prescriptive traditions may have increased the frequency of invariant -ment- adverbs in the Romance languages that have them (p. 27). In traditional grammatical descriptions of Latin, adverbs are treated as a category that is distinct from adjectives and characterized by a lack of inflection. Even though Latin didn't use the -ment- ending to form adverbs, -ment- adverbs have a similar morphological structure to Latin -ē and -ter adverbs and can be described using the same grammatical concepts. This may have helped them to become established in the standardized forms of some Romance languages.)
It seems that there is a long history in Romance languages of using adjective forms (which generally means forms with the same ending as nouns of a certain class, since Romance adjective terminations tend to be a subset of noun terminations) adverbially—often without agreement, but sometimes with agreement with another word in the phrase/clause. Even in Latin, one significant method of forming adverbs was with the singular neuter accusative of an adjective; this construction seems to have contributed to the form of certain adverbs in some Romance languages, whereas Latin's adverb-specific suffixes like -ē and -ter left no visible traces.
Romance constructions with gender-agreeing adverbs seem to have fairly recently been the topic of some study and attention from linguists interested in determining what exactly is going on there. I'm not on top of the literature (I haven't even finished reading all the works cited in this post), so this answer is just a small sample of the information that is out there.
Adverbs modifying adjectives
One situation where gender agreement may show up in Romance languages is with adverbs modifying adjectives: there are some adverbs that show gender agreement with the adjectives that they modify or with the nouns that the adjectives modify.
The example I am most familiar with is French [tu] (masculine form) 'all': the feminine form [tut] is found in phrases like "une galerie toute petite" and "des galeries toutes petites", where it shares its gender with the feminine adjective petite 'small' and feminine noun galerie 'gallery' (examples taken from "The Principle of Phonology-Free Syntax: four apparent counterexamples in French", by Miller, Pullum and Zwicky, 1996).
I referred to the pronounced forms rather than the spelling of the French adverb because the standard spelling of this word is rather complicated and perhaps artificial: Miller, Pullum and Zwicky argue that the number agreement that is present in the standard spelling of "des galeries toutes petites" does not actually exist as part of the grammar of non-written French. Such number agreement is not found when the adverb is in a position where liaison (morphologically conditioned pronunciation before a vowel-initial word of a word-final consonant that is not pronounced in most contexts) occurs; instead, the word is spelled tout and pronounced [tut] in this context.
Apparently, in Spanish mucho can show gender agreement (taking the form mucha) as an adverbial modifier of an adjective that is followed by a feminine singular noun, but mucho does not show agreement in either gender or number before a plural adjective and noun ("Gender agreement on adverbs in Spanish", Antonio Fábregas & Isabel Pérez). Fábregas and Pérez give the example mucha mejor intención 'much-fem better intention-fem' (p. 27), and report that this agreement pattern only occurs with abstract nouns, not with animate nouns or count nouns.
This next part is not directly relevant to your question (because it doesn't involve gender agreement), but the French and Spanish examples remind me of the use in English of many as an adverb in the expressions "many more" and "many fewer" ("much fewer" is also used), despite the usual use of much as an adverb. French tout/toute, Spanish mucho/mucha, and English much/many are all quantifiers that have common pronominal and adnominal uses in addition to their uses as adverbs.
Adverbs modifying verbs
"Adverb Agreement in Urdu and Sindhi", by Miriam Butt, Sebastian Sulger,
Mutee U Rahman, and Tafseer Ahmed, discusses another construction where a manner adverb based on an adjective may show gender agreement with a noun while modifying a verb. In addition to the two languages mentioned in the title of the paper, Butt et al. cite Punjabi and Southern Italian dialects as languages that have gender-agreeing adverbs of this type (as well as non-agreeing adverbs).
Butt et al. briefly mention the existence of adverbs that show agreement in gender/noun class in Daghestanian languages.
In Daghestanian, adverbs agree with either the agent (e.g., in Archi) or the patient (e.g., in Avar) (Evans 2000, Kibrik 1979).
In the Archi example in (5) the adverb dītaru ‘early’ agrees in (feminine) class ii
and singular number with buwa ‘mother’, the agent of the overall predicate.
(It cannot agree with dez ‘me’ since dative NPs are generally not available for
(5) buwa dez dītaru x̄ₒalli
mother:ii:sg:nom 1:ii:sg:dat early:ii:sg bread:iii:sg:nom
‘Mother was baking me the bread early.’ (Kibrik 1979, p. 70)
- In the Avar example in (6) the adverb xar ‘here’ agrees in (non-human) class iii and plural number with ‘icalgi ‘apples’, the patient of the overall predicate
(6) ṛex xar dedebe ‘icalgi
3:ii:sg:erg here:iii:pl father:iii:pl:dat apple:iii:pl:nom
‘She was buying father the apples here.’ (Kibrik 1979, p. 76)
Butt, Miriam; Sebastian Sulger,
Mutee U Rahman, and Tafseer Ahmed. 2016. "Adverb Agreement in Urdu and Sindhi"
Fábregas, Antonio and Isabel Pérez. 2008. "Gender agreement on adverbs in Spanish"
Hummel, Martin. 2017. "Adjectives with adverbial functions
in Romance", in Adjective Adverb Interfaces in Romance, edited by Martin Hummel and Salvador Valera, Linguistik Aktuell/Linguistics Today (LA) Volume 242.
Miller, Philip H.; Geoffrey K. Pullum, and Arnold M. Zwicky. 1996. "The Principle of Phonology-Free Syntax: four apparent counterexamples in French"