Going through the derivation might help. Since your question is framed in terms of the generative framework, i'll give my answer also from a generative PoV - specifically, from the PoV of the Minimalist Program, where movement is feature-driven. I presuppose i certain amount of familiarity with minimalism in my answer, so apologies in advance.
We start with the following 'Numeration'/'Lexical Array'. I've only included featural specifications (which are given in square brackets) where they will be relevant to the derivation. Note that i'm also going to ignore subject-auxiliary inversion in the derivation. I also use bare phrase structure labelling, so no distinction between maximal projections and intermediate levels.
{Whom[wh], will, you, invite, C[*u*Wh]}
First we merge 'whom' and 'invite', so that 'invite' satisfies its internal theta-role, and 'whom' is assigned accusative case. Note that 'whom' possesses an interpretable Wh feature.
(1) [V invite whom[wh] ]
Next we merge the subject, 'you', which satisfies the external theta-role of 'invite':
(2) [V you [V invite whom[wh] ]
Next we merge 'will':
(3) [I will [V you [V invite whom[wh]]]]
Movement of the subject 'you' to SpecIP is triggered (by, e.g. an EPP feature on I):
(4) [I you [I will [V you [V invite whom[wh]]]]]
Interrogative C is merged:
(5) [C C[*u*Wh] [I you [I will [V you [V invite whom[wh]]]]]]
Now we've reached the crucial part of the derivation. Interogative C possesses an uninterpretable wh-feature, which must be deleted before spell-out, or the derivation will crash. The [*u*Wh] feature 'probes' downwards for a matching 'goal' - since 'whom' posseses an interpretable [wh] it counts as a matching goal. Once the goal is found, movement of 'whom' to SpecCP is triggered, and [uWh] is deleted in a spec-head configuration under AGREE with 'whom'.
(6) [C {whom[wh]} [C C [I you [I will [V you [V invite {whom[wh]}]]]]]
The answer to your question is that 'you' doesn't count as an intervener, since it doesn't possess a [wh] feature. Interrogative C probes specifically for [wh] - anything not possessing [wh] isn't visible to the probing operation. Intervention is relativised - only elements of the relevant sort count as interveners.
We can see that other wh-words do count as interveners by considering the well known superiority effects:
(a) Who t bought what?
(b) What did who buy t?
Since the subject is closer to interrogative C than the object, it counts as an intervener to wh-movement of the object, as attested by the ungrammaticality of (b).