Modest ends in /st/, unlike vacant or secret
If we assume that secrecy, vacancy and modesty all contain one derivational suffix (the same for each), along with one base morpheme each (the adjectives secret, vacant and modest respectively), and that the form of each of these three nouns is given by some morphological rule, then the most likely explanation is that modesty ends in /ti/ rather than /si/ because of the presence of the consonant cluster /st/ at the end of the base modest.
This is probably the answer you are supposed to give.
Challenging some assumptions
The assumptions mentioned in the first paragraph may be a bit problematic. English has more than one morpheme used for deriving nouns from adjectives: this is shown by the existence of multiple nouns that correspond to the same adjective such as falsehood/falsity/falseness. And even when we're dealing with nouns that are formed from adjectives by suffixing the same derivational morpheme, there is no guarantee that there will be a reasonable rule that accurately predicts the phonological form of all of the nouns based on the phonological form of the adjective. For example, for many speakers the nouns clarity and sincerity do not rhyme, even though the corresponding adjectives clear and sincere do rhyme: there is no way of explaining this by phonological rule (unless you resort to something ad hoc).
The historical basis of these forms (which is almost certainly not the same as any synchronic rule that could be used to explain these forms)
From a diachronic perspective, /t/ is present in modesty but not in secrecy and vacancy because of a postclassical Latin sound change that turned t into the affricate [ts] before [j], except for when the t was in a st cluster. As far as I know, this was (at this point) a purely phonologically conditioned process. The effects of this process are also seen in various English words ending in -tion, -tious, -tial, -tian or -ce (e.g. relation, nutritious, partial); note that we see different pronunciations for words ending in -stion, -stial, -stian (e.g. question, bestial, Christian).
It's clear that English does not have any purely phonologically conditioned process like this that changes the t from secret to /s/ in secrecy. There are many examples of /ti/ in English, including after schwa or /n/ (e.g. ability, puberty, maggoty, certainty).
But your question brings up the topic of "morphophonological" rules. As far as I know, the exact meaning of "morphophonology" is a bit unclear, but we can say that it involves both morphology and phonology. In particular, "morphological" rules are often described as only applying in "derived environments", which would allow us to ignore words like certainty where the /ti/ sequence occurs within a single morpheme (the suffix -ty). Maggoty has /ti/ in a derived environment, but there are other parts of morphological theory that I think could be used to explain away this fact.
I won't try to pursue this in more depth though because even though it seems theoretically possible that English has a synchronically active morphological rule that turns t into /s/ in secrecy based on some aspect of the phonology of the suffix -y, it seems unlikely to me. I just don't think there's that much evidence for the existence of such a rule:
Obviously, in its surface form “secrecy” does not contain /j/ after the /s/. We also see /s/ that is word-final (on the surface, at least) in words like silence. So the conditions that applied for this sound change in Latin do not apply on the surface in English. We could try to formulate some very abstract analysis that derives both secrecy and silence from "underlying" forms containing /j/, but it's not straightforward to come up with appropriate underlying forms, and it might not really be "worth it" in terms of the amount of explanatory power such an analysis provides. See user6726's answer to my question What are current perspectives on analyzing word-final /i/ in English words like "potency" as synchronically derived from /j/?
It seems to me that it would be simpler to just say that in the morphological system of Modern English, the phonological form of the (assumed) suffix -y is unrelated to the fact that it selects for the /s/-final allomorph of "secret" (and "vacant", etc.).
The nominalizing suffix found in secrecy is not very productive in modern English, and the fairly recently coined example of normalcy (along with the standard spelling of bankruptcy) suggests that some people think of -cy as a suffix. This raises the possibility of an alternative morphophonological analysis where the /s/ is part of the suffix, and the t is deleted rather than palatalized. That's not the diachronic origin of the form, but as a synchronic explanation, it seems fairly plausible (consider that many -cy words are -ncy words, and the contrast between /nts/ and /ns/ is neutralized for many English speakers). See Greg Lee's answer to this English SE question: What are the rules to pronounce the suffix “-tion” in English, “/-tʃən/” or “ʃən”? If this explanation is correct, than there is no need for a rule that turns t to /s/ before -y.
I'm not totally sure how to explain modesty according to this model; one option might be to say that it has been reanalyzed as containing the suffix -ty, which is historically the source of the -ty in the similar-looking word honesty (from Latin honestas/Old French honesté). Or we could just say that it has -y, since there are still words like recovery or villainy that show -y as a nominalizing suffix.
One problem for the t-deletion analysis as well as the t-fricativization analysis is the existence of the word baronetcy, with /ts/.