If the same word is pronounced two distinct ways with one sound different, say "tomato" and "to-mah-to", is the sound that differs a different phoneme?
If the same word can always be pronounced two distinct ways with sound X or sound Y, "tomato" and "to-mah-to", that is known as free variation. If two distinct words are distinguished by a single difference in sound, then the pair of words is a minimal pair, and the two sounds are distinct phonemes. But "tomato" and "to-mah-to" are not distinct words, they are the same word, pronounced different ways. (Actually that's a terrible example because it's a song lyric, not a linguistic example, but there are analogous examples like the first syllable of "economic" which can, for many people, be either [ɛ] or [ɪj]).
The nuclei [ɛj] and [ɑ] are not in free variation in English, see for example "date" and "dot" or "sate", "sot" and "sought" (the latter two because the vowel of "tomahto" is either the vowel of "sot" or that of "sought", depending on how you neutralize the difference, if you do). Uncontroversially, [ɛj] and [ɑ] are distinct phonemes as established by minimals pairs. In fact "tomato" ~ "tomahto" is non-systematic and doesn't reflect a rule: they come from different linguistic systems. Likewise, "pasta, Mazda" are pronounced with [æ] north of the border and with [ɑ] south of the border, so these vowels are also not in free variation. There is no potato ~ potahto variation outside of humor. Such two-pronunciation cases generally reflect competing strategies for dealing with spelling when there are multiple possibilities for pronunciation.