This originates in linguistics with Trubetzkoy, who spoke of distinctive "marks" (in the sense of "indication"). -s is a "mark" of plurality, also d has a "mark" of voicing. You may be interested in the recent volume Beyond Markedness in Formal Phonology. Subsequent developments of the theory equated "unmarked" with "common, natural", and thereby expanded the range of things that could be called "marked" or "unmarked". The notion of "natural" is widely used in the literature, but there is not a clear understanding of what "natural" actually means.
I should emphasize that "marked" is comparative, not absolute: A is marked compared to B, B is marked compared to C – that does not mean that C is completely unmarked. Some theories may stipulate that you have to set up your primitives and distribution so that there is always an item that is "completely unmarked", but that's an added theoretical stipulation.
In practice, markedness seems to mean "most frequent", although there are other "If P then Q" implications. Probably all languages have syllables with an onset (there are some reported cases of languages where all words begin with a vowel), and any language that allows onsetless syllables also allows syllables with onsets – the possibility of onsetless syllables presupposes the possibility of onsetful syllables, but not the opposite. Likewise with codas – all languages allow open syllables, not all languages allow closed syllables. Such implicational relations are often called on to determine what is marked vs. unmarked.
It is "most natural" in the implicational and frequency senses for a syllable to have an onset, and to be open. Therefore, the least-marked syllable would be CV. Syllables of the type V, and also CVC, are more marked compared to CV, but no ordering can be established (none has been proposed) between V and CVC. If you look only at single C and V possibilities in the syllable, the last syllable type, VC, is the most marked, in being marked compared to both CVC and V. VCCC would be even more marked compared to VC (but, it's not a core syllable in Clements & Keyser terminology). You cannot compare the degree of markedness of CCV and VCC syllables, at least not in C&K theory.
The problem with asking why something is marked is similar to questions of why phlogiston has the properties it has. But we can make that be a question about how this is expressed in a particular theory of markedness, if we assume there is such a thing. In Clements & Keyser theory, this derives from taking CV to be the universal primitive syllable template, and deriving other syllable types from transformations of that – adding a coda, or deleting the onset. If you do both things, you have added two degrees of markedness. In OT, CV is the one un-punish syllable type; V (and VC) violate Onset (the requirement to have an onset), and CVC (and VC) violate No-Coda. VC syllables get two marks against them. But you can also sensibly ask, what are the functional phonetic considerations that make it good for syllables to have onsets and to not have codas.