There are languages that have a dual/plural number distinction as well as an inclusive/exclusive distinction in the first person, so there definitely could be separate inclusive first-person pronouns corresponding to "[me] and [you (sing)]" and "[me] and [you (plural)]". "Inclusive/Exclusive Distinction in Independent Pronouns", by Michael Cysouw (Chapter 39 in the World Atlas of Language Structures) says
There are many languages [with a 1st-person singular/inclusive/exclusive distinction] that also mark dual
number in their pronouns. The basic and most common way to mark
duality is exemplified in (4) by the pronouns from Lavukaleve
(Solomons East Papuan; Solomon Islands). A special dual pronoun exists
both for the inclusive and for the exclusive, both marked by a suffix
(4) Lavukaleve (Terrill 2003: 170)
el ‘exclusive, exactly two’
e ‘exclusive, more than two’
mel ‘inclusive, exactly two’
me ‘inclusive, more than two’
Another strategy is to mark duality only in the inclusive (Plank 1996:
130-131). In such paradigms, the dual inclusive aligns structurally
with the singular pronouns, yet strictly speaking it is of course not
singular in reference. The term minimal inclusive is used to refer to
such a dual inclusive. Paradigms with a dual only in the inclusive are
known as minimal-augmented structures (Thomas 1955). This lexical
structure is exemplified in (5) by the pronouns from Southern Sierra
Miwok (Penutian; California).
(5) Southern Sierra Miwok (Broadbent 1964: 93)
mah·i ‘exclusive, two or more’
ʔoti·me ‘inclusive, exactly two’
ʔotic·i ‘inclusive, more than two’
The opposite distribution of duality – dual in the exclusive but not
in the inclusive – exists among the world’s languages, but it is
extremely rare (Cysouw 2003: 221-222). In the present sample it is
attested in the pronouns from Yagua (Peba-Yaguan; Peru; Payne and
Payne 1990: 369-370). Just as exotic is the division attested in
Gooniyandi (Bunaban; Australia). Here the inclusive dual is expressed
by the same pronoun as the exclusive ngidi, but is different from the
inclusive plural yaadi (McGregor 1990: 167-173). There are more cases
like this among the world’s language, but not many (Cysouw 2003: 93).
I haven't heard of the other thing you mention ("different forms of the exclusive we depending whether the excluded you is singular or plural") and it sounds a bit unlikely to me--I have read that no language has a clusivity distinction between things like "you (all of whom I am addressing right now)" and "you (the ones I'm addressing right now and others that aren't here)" and that distinction sounds a bit similar to me.
My impression is that despite the name, the "exclusive" element of the "exclusive we" isn't necessarily the most salient part of its use. Its meaning can be explained to a speaker of a language that doesn't have the distinction as "we, excluding the second person", but I have the impression that a speaker of a language with the distinction would be just as likely or more likely to think of it in terms of "first person + third person(s)" rather than "first person plural excluding (some specific) 2nd person(s)".
For comparison, the third person pronouns also could be described as "excluding" the second person, but I haven't heard of languages that use different third-person pronouns depending on whether the addressed (and technically excluded) party is singular or plural.
I don't actually know whether any languages have this, though (pronouns that inflect based on the number of the listener, rather than or in addition to the number of the referent).