Linguists frequently describe utterances as "ungrammatical", and in the context of linguistics, this is a fairly meaningful, objective and well-defined statement (although what exactly it means depends on the linguist). I think most linguists would disagree with the statement that "Language 'is' grammar" if we define "language" as all utterances of any kind that anybody makes, for whatever reasons.
Linguists have various opinions about grammar, but typically, "descriptive" grammarians think that there are rules (in the sense of "scientific laws", not in the sense "things that you'll be punished for breaking") underlying language. They don't think of grammar as just being an after-the-fact summary of the different utterances that people have made, but as some kind of system, underlying speech, that can be studied by linguists.
The point of descriptivism is that descriptive linguists expect any grammatical rules that are proposed to be useful for explaining linguistic data. Other criteria may also be applied (e.g. restrictions on the kind of rules that are allowed to be in the grammar) as long as they are motivated by a theory of how language works, rather than by a theory of how language "should be".
Many theories of how language work distinguish a grammatical component from some other components that contribute to the phenomenon of language; e.g. the limitations of human memory.
Aside from there being limitations on language that aren't based in grammar, it's also the case that people can draw on resources other than the naturally acquired grammatical system of their native language to form utterances. For example, an utterance like "Dost thou thinketh that I beeth a fool?" isn't a grammatical modern English sentence, but the ungrammatical parts ("thinketh" etc.) may be intentionally used to produce a certain effect. This isn't a matter of idiolects, either: as far as I know, nobody has an idiolect where "Dost thou thinketh that I beeth a fool?" is actually grammatical.
Obviously words like "wrong" and "incorrect" look like value judgements, which is why I've mainly used the word "ungrammatical" in the preceding parts of this post. But in the context of descriptive linguistics, "grammatically incorrect" can be used as a synonym of "ungrammatical". Saying that an utterance is "grammatically incorrect" doesn't mean that it's "wrong" from a moral or rational standpoint. It just means that it's not consistent with—or not a valid product of—whatever grammatical system you are talking about.
Linguists and non-linguists don't mean the same thing by "grammatically incorrect"
When non-linguists say that someone's grammar is "wrong", they usually mean something different, and more complicated to define. (Since clothing is frequently used an an analogy to the social aspects of language, compare the concept of "wrong" fashion.) The meaning of "incorrect grammar" as used by a non-linguist isn't entirely meaningless, but it's typically a value-based judgement where part or all of the meaning being expressed is "I don't like this usage" or "I think this usage is socially inferior". If that's who you're talking about when you say "how could people say someone's grammar was "wrong" (or grammatically incorrect)?", then you're right to say that these people are making an essentially arbitrary and subjective judgement in regarding the rules of non-prestige dialects as "wrong".