When you post a question next time, please mention what theory it is based on. Do not assume that the terms (and implicit assumptions) are necessarily shared and/or easily recognized by everyone. Your OP is a good case in point.
It looks like your question is based on Cognitive Grammar.
In Cognitive Grammar,
a complement is understood as "a conceptually autonomous component structure that elaborates the profile determinant, which is conceptually dependent",
whereas
a modifier is "a conceptually dependent component structure that is elaborated by the profile determinant, which is conceptually autonomous" (Evans and Green 2006: 587).
Note the key word "conceptual," because this is very different from other formal syntax theories.
For instance, from the point of view of generative syntax, in both of your examples, we can see complements.
their incredible story of the trip in space, [of the trip in space] is complement;
the noisy yellow airplanes that scared the children in the yard, [that scared the children in the yard] is a complement clause.
Also, I don't think the term "modifier" is very common in generative syntax and is used as an umbrella term for anything attached to the head, the more common opposition being complement-specifier-adjunct (all of those are different types of modifiers).
In some other syntactic theories, the difference between a complement (sometimes also called an argument) and a modifier is that complements are obligatory (cf. *their incredible story of) whereas modifiers are optional (their incredible story).
Putting theoretical differences aside, I am afraid you didn't quite understand your professor or textbook very well. For instance, you ask us, "Why is it that certain nouns takes modifiers and others complements?"
The thing is that all English nouns can take (select) complements and what you call "modifiers":
their story that scared me a lot (complement clause; your "post-modifier")
their story of the trip in space (complement; your "complement")
their incredible story (specifier; your "modifier")