First of all, it should be noted that in nearly all generative theories--even in ones which generate subjects inside the VP--the subject practically never stays there for long. Subjects generally move upwards into a position where they can receive Case; by most X' accounts, they move up into SpecTP (or SpecIP, etc.).
This is what BillJ's response is referring to; when they say, "[subjects] are different from other types of complement in an obvious way: they are positioned outside the VP", they are not referring to the place where subjects are generated, they're referring to the place where subjects generally end up after all the various kinds of movement have occurred. Objects and other verbal complements can also wind up outside the VP via movement due to various processes (topicalization, wh-fronting, etc.), and the promotion of an object to a location outside the VP is obligatory in object-first languages. But subjects are unique in that they seem inherently 'fated' to wind up somewhere far-flung from the VP. [Caveat: In clauses with a TP. Assume that I'm talking about clauses with a TP in all of this--"tensed" clauses, "finite" clauses, however you want to call them.]
We know that the subject of a verb has to be generated in a position similar to the verb, because, for instance the verb selects for the thematic roles of all its various complements. Yet, the hierarchical positions of the various verbal complements are assymmetrical: a verb can seemingly select for and assign the Case of a direct object, but not of a subject. So, in the past, we generated the subject in SpecVP. This allowed us to maintain the closeness of the subject and the verb, but allowed the verb to be closer to the object.
In some languages, we can distinguish a class of verbs called "light" verbs which seem to behave differently from other verbs. We'll call those other verbs "heavy" verbs--they're not called that in the literature, this is just for illustrative purposes. They always possesses a weak semantic meaning; prototypical examples include things like GO, TAKE, GET, DO, MAKE, and so on. Their properties vary from language to language, but here are just some examples:
- In Inuit languages, light verbs can incorporate a direct object, but heavy verbs cannot, even when the semantics are nearly the same. In Eastern Québec Inuit, "he bought a table" (heavy verb) is niuviq-tuq (bought-3ms) saa-mik (table-obj.). But "he got a table" (light verb) is saa-taaq-tuq (table-get-3ms).
- In English, a small number verbs of semantically weak verbs (have, make, take, get, give, etc.) form a disproportionate amount of lexicalized predicates equating to light verb + indefinite article + noun (make a mistake, have a smoke, take a break, give a fuck, do an inspection, etc.). These often interchange with heavy verbs: take a nap vs. nap; make a claim vs. claim; do a revision vs. revise; give a performance vs. perform; have a smoke vs. smoke; do a dance vs. dance; and so on.
- In Yiddish, two verbs of great generality--ton 'do' and gebn 'give'--can be combined with a nearly any heavy verb root in order to create a sense of immediacy or intensity: Er tut/git a bak beygelekh (he does/gives a bake bagels) 'He is baking bagels [as we speak!]'.
- In many Australian languages, only a small, closed handful of verbs of great semantic generality are capable of inflecting. All verbal expressions consist of such a (finite) verb plus a non-finite verb. "To dive" would be to go dive, "to hunt an animal" would be to do hunt an animal, and so on. The constructed language Kelen claims to be verbless, but in reality, it is simply doing what these Australian languages do--it restricts its verb set to a small number of semantically broad light verbs.
Which verbs can/cannot be considered "light" depends on the language, but they can obviously appear in addition to heavy verbs. And they can often appear together with heavy verbs and with other 'auxiliary verbs' like aspectual verbs and tensed verbs (I shouldn't have taken a bet on that horse; Yiddish: Ikh hob gegebn a hilf dem man 'I helped the man straight a way [I have given a help the man]'.)
So where do we put them in the tree? The fact that they can be lexicalized in ex. English and some Australian languages tells us that they have to be base-generated in a position nearby the verb and all of its complements. But we can't put them in SpecVP--that's where we put subjects! And we can't put them in CompV--that's where we put objects!
So, we created a new layer above the VP. We call it the vP, with a lowercase v, and that's where the light verbs go. But now what's the specifier of vP?
Well, this is when we run into the problem of ditransitive verbs and indirect objects. If we put the indirect object and direct object both as complements of V0, this gets us into some trouble. For one, it gives us a tertiary branching structure, which X' and later Bare Phrase Structure absolutely cannot abide. For another, this predicts that the c-command effects of direct and indirect objects should be exactly equal--V0 should c-command the DO and IO, and the DO/IO should c-command the V0 and each other. But we can tell that this is not the case--various evidences from binding effects/anaphora show that the IO c-commands the DO and the V0, but the DO does not c-command the IO. This suggests that the IO is higher than the DO. (In langauges like English; secundative languages like West Greenlandic show the opposite pattern.)
This works out for us because a new appartment above the VP just opened up in SpecvP. We can tell that the IO is more closely linked the verb than the subject is; IOs c-command the VP, but not the subject, so they cannot be higher than the subject, so the IO cannot go in SpecvP. This means that we have actually found ourselves a new home for the subject, up in SpecvP. (Also, subjects would have had to land temporarily in SpecvP anyways in the course of movement upwards to SpecTP even if they were generated in SpecVP, just like they land temporarily in SpecNegP and so forth.)
This has a number of consequences, and some of them turn out to be solutions to other outstanding problems. An interesting problem comes up with theta roles. You've no doubt noticed that some of the things called 'light verb constructions' are sometimes two verbs, but they're often a verb and a noun. Where does the subject get its theta-role, and does the "object" have a theta-role? The "object" of He took a drink is synonymous with He drank; so the subject is receiving an agent or experiencer theta-role, but the "object" is arguably not receiving one at all, as if it were fulfilling the role of the V0. In the aforementioned Australian languages, specific kinds of light verbs are associated strongly with specific subject theta-roles. And in English, there is a correlation--not strict, but present--between choice of light verb and the subject's theta-role. If the vP hypothesis is true, it is likely that we can actually define the v0 node as the element responsible for assigning theta-role to the subject. This changes how we represent verbs at an underlying level: all verbs are now associated with a V-node, which gives theta-roles to the IO and DO, and a v-node, possibly null, which gives theta-roles to the subject.
This is, it turns out, actually a very very handy thing. It gives a good reason for why subjects seem to be so much less strongly associated with the semantics and syntax of the verb than objects/direct objects do, and it definitively divides "internal arguments" (objects and adpositional complements etc.) and "external arguments" (subjects). The basic idea of "internal" and "external" arguments already had support before the vP hypothesis, but the vP hypothesis gave it an even firmer standing.
The account I've presented here is too simplistic, and it does not follow the original argumentation for the vP hypothesis; but it's the line of reasoning I've found most successful in trying to understand the vP hypothesis myself. We've of course developed even more complicated kinds of internal structure in the "VP shell" with things like AgrIOP and AgrOP, but the overall picture is consistent. There's further evidences for the vP hypothesis in some phenomena related to the differences between unaccusative and uneragative intransitives, where unaccusative subjects seem 'closer' to verbs in certain kinds of constructions (because unaccusative subjects are generated in the VP and unergative subjects are generated in SpecvP), and I believe also in some aspects of syntactic phases. But this should suffice to lay the very general outline of why we think (most) subjects are very probably generated in their own shell outside of the VP, and that most lexical verbs are actually syntactically pairs of a (usually null) light verb a (usually overt) heavy one.
It's been some years since I was last in a syntax research environment, to my dismay, so I do apologize if I have bungled the finer points.
