When you try to remove word-initial glottal stop [ʔ] before vowels, it turns to voiced glottal fricative [ɦ].
First of all, [Ø] (no consonant) is not what you think it is. It exists as a phonemic element (/Ø/), but phonetic initial vowel [ØV], a bare utterance-initial vowel, doesn't actually exist. I'll use  for phonetic means and // for phonemic in the rest of this post.
When we make a sound for speaking, it requires some air flowing through our glottises without exception. When trying to remove the initial [ʔ], due to the airflow, it becomes [ɦ] (voiced glottal fricative) instead of "[Ø]" which we desire. The characteristic of [ɦV] is that the consonant completely merges with the vowel, unlike ALL other consonants. The ways to pronounce [ɦV] and [ØV] are generally the same: form the vowel, then let the air flow to vibrate the cord while all other parts of our vocal tracts are at their default (relaxed) positions. The only difference is between vowels, where [ɦ] aspires more but [Ø] stays the same. For example, Japanese word 情報 [dʑoːɦoː] (information) has extra aspiration at [ɦ] (/h/) than 女王 [dʑo'oː] (queen).
It won't be a problem when [ɦ] and [Ø] are between vowels because the air is already flowing. However, it will be a dilemma to distinguish initial [ɦ] and [Ø] because the air is just about to flow, thus no comparison of "extra air" right after it breaks silence, or we can say it is always extra air comparing to no air. If you think you can get initial [ØV] by pushing less air, you are wrong: it will be just a quieter [ɦV], and a consonant cannot be determined by its volume.
Another fact to see is that the ancient alphabet sets, like Proto-Canaanite and Phoenician, have glottal stop (the origin of alphabet "A") as their very first alphabet. The past people already recognized glottal stop as "default" of a syllable, that's why many people didn't realize the existence of initial [ʔ] in their languages until modern days.
So, you won't find any difficulties to distinguish initial /ʔ/ and /Ø/ even if your native language doesn't have phonemic differences between [ʔ] and [Ø]. The problem is our vocal system disallows removing both [ʔ] and [ɦ] before an initial vowel. Luckily, Hawaiian language has /Ø/, /ʔ/, and /h/ but not /ɦ/, thus no problem to recognize initial [ɦ] as /Ø/ and intermediate [ɦ] as /h/.
There is no way to avoid [ʔ] becoming [ɦ] in your attempt, as the "hardware", our vocal system, just doesn't support pronouncing without air passing through the glottis.