This is a fundamental question in morphology that has consequences going far beyond the simple distinction between case endings and postpositions (which, by itself, is effectively quite thorny in many languages). I would say that it pertains to the problem of defining such notions as grammatical category, paradigm and fusional/flective typology in general. Indeed, in order to postulate case endings for a language we should first assure that it has nominal paradigms, which, in turn, implies the existence of inflexional grammatical categories for the noun.
But how would we discriminate between flective languages (i.e. languages with paradigms) and, let's say, agglutinative languages (i.e. those without paradigms)? Some criteria can be suggested here, but the list can be easily emended or enlarged. For the sake of simplicity I am exemplifying every statement with facts from nominal inflexion, even if this approach is valid for the paradigmatic morphology in general.
- Grammaticality implies obligatoriness: cases are obligatory on all nouns in all instances. Such grammatical categories as case or number or gender can't be freely avoided by the speakers. In agglutinative languages this is not always the case: sometimes no grammatical marker is also an admissible option (this happens with Japanese postpositions, especially in the spoken language).
- Grammatical categories imply the possibility of making a grammatical error, since they are selected syntactically: if you say many mouse it would be clearly an error that any speaker of English will be able to emend into the correct form many mice, which fits better the syntactic context. The postpositions, on the other hand, are not selected strictly syntactically since they can vary according to the communicative intentions of the speaker. Therefore, there is no such thing as "wrong postposition" (if not stylistically).
- Grammatical categories imply morphological variability and inflectional classes. Indeed, in a language that don't vary its case endings, you basically can't tell the difference between cases and postpositions, since both will appear as grammatical morphemes without any variation. On the other hand, in prototypical flective languages, cases are instantiated by sets of allomorphs, which are either contextual (i.e. depend on the phonological form of the stem) or, importantly, non contextual (i.e. are lexically bound, forming inflectional classes, such as the five nominal declensions in Latin).
- Grammatical categories are better when they are semantically uniform, i.e. refer to a set of uniform meanings. Thus, cases can apparently convey many different meanings, but such meanings would not usually overlap with other semantic content, like gender or number or, say, tense (i.e. in languages with cases, no "feminine", "future" or "plural" case would be, normally, attested). On the other hand, postpositions can bear many different functions, from syntactic relations to honorific connotations.
- Cases usually belong to cumulative morphemes which also include gender, number and possibly some other grammatical categories; the postpositions, instead, are simple morphemes by definition.
- Some minor features can be added:
- regularity of markedness (e.g. case do not usually allow markedness shifts: if the nominative singular case is unmarked in some nouns, then the dative plural or the instrumental dual cannot be unmarked in other nouns; however, this rule is less strict; contrariwise, let us think to the Japanese verbal transitivity markers: some verbs have unmarked intransitive form, while some others have unmarked transitive form, which strongly suggest that transitivity is a derivational category, rather than an inflectional one;
- you usually cannot have multiple case markers at the same time; on the other hand, you can have serialisation of postpositions (think of such Japanese NP structures as N ni wa, N de no, N no ga, and similar);
- case categorial values usually don't exceed a dozen; systems with up to 40 cases are obviously agglutinative.
The labels "flective" and "agglutinative" may be as antiquate and outdated as you want; however, defining a prototypical flective language (vs. a prototypical agglutinative language), on the ground of the above mentioned criteria, provides a useful analytical tool. Thus, a prototypical flective language will have: rigid paradigms defined by the cumulative marking of two or more obligatory, error-provoking, syntactically selected grammatical values, with non contextual allomorphy, semantic uniformity, no shifts of markedness, no serialisation, no optionality, and with a limited number of grammatical values.
