I know most of the phrases in English are headed phrases, like noun is the head of NP. But what is non-headed phrase?
A non-headed phrase is a phrase without a head, of course. There is one clear candidate for such a status in modern syntactic analyses. One can, namely, view coordinate structures as phrases that lack a head, e.g.
(1) [Frank and Bob] have arrived.
(2) [The old woman and the young man] are a couple.
The proper nouns Frank and Bob are conjoined in (1), and together they form the subject noun phrase. But should the one be viewed as had over the other? Probably not, although the analysis of such coordinate structures has been debated a lot over the decades. I, for one, assume that such coordinate structures lack a single head, which means they are in fact headless in the relevant sense. Similar comments apply to the conjoined noun phrases the old woman and the young man in (2).
Note, though, that almost all other constructions and structures in natural language are viewed as having heads, and it is even a matter of debate whether headless structures exist at all, as the comments above demonstrate. I think they do, as the discussion of the coordinate structures just above demonstrates.
The issue of headed vs. headless phrases is characterized in terms of endo- and exocentric structures. Wikipedia has a helpful article on this distinction. The next diagrams are from that article:
As can been seen from these diagrams, an exocentric (headless) structure is one in which the category status of the whole is entirely distinct from that of the parts.
Until about the 1970s, prepositional phrases were viewed as excocentric. Leonard Bloomfield is known for putting forward this stance in his book Language (1933). That changed in the 1970s, though, when the position was widely adopted that the preposition is the head of the prepositional phrase.
More importantly, the traditional analysis of the sentence in phrase structure grammar viewed it to be an exocentric structure, e.g.
Since the whole, a sentence S, is distinct in its category status from both its parts, an NP and a VP, it is in fact exocentric. Most modern phrase structure analyses no longer view sentences as exocentric constructions, though. They instead view the verb as projecting its category status to the whole in terms of tense (T) or inflection (I).
Before concluding, I would like to address the structures mentioned in Greg Lee's answer. That answer suggests that gerund phrases and free relative clauses are headless phrases or clauses. I disagree with that stance. Certainly given a dependency grammar (DG) approach to syntax, the gerund is the head of the gerund phrase and the wh-word is the head of the free relative clause, e.g.
Gerunds like discussing in the first tree are a mixed category. They distribute like nouns with respect to their heads but more like verbs with respect to their dependents. The what of the free relative clause in the second tree can be construed as the head of the relative clause by virtue of the fact that it fulfills a grammatical function in the matrix clause (the subject function) as well as a grammatical function in the relative clause (the object function). This fact is more evident in languages that have case, e.g. German:
Sentence (3a) is good because the nominative pronoun Wer functions as the subject in both clauses. In contrast, sentences (3b) and (3c) are both bad because the interrogative pronouns Wer and Wen cannot be both nominative and accusative at the same time. There is a basic case conflict that arises based upon the syntactic functions that the relative pronoun must fulfill in the two clauses.
Noun phrases without a noun head are the most obvious examples of phrases without a head. Gerunds, for example, are clearly NPs, since they occur as subject, direct object, and object of a preposition, yet internally they are VPs -- no noun. Also common are headless relative clause constructions ("What I saw was strange.").
Reading over the comments above, it is remarkable how many people are in the dark about this.