Is there any rule in Old English / Modern English a/o (ham/home, ban/bone, stan/stone), a/oa (fam/foam, hlaf/loaf, gat/goat) transformation?
This difference is arbitrary: all of goot, gote, goote, goate, goat are attested by the end of Middle English in the 16th century, and the standardisation that happened afterward does not reflect any diachronic phonological difference.
The oCV orthography (as in gote, stone) is the older one, as it reflects the consequence of early Middle English /o/ undergoing open syllable lengthening to /ɔː/.
The "competing" value for the orthographic o, the closed /oː/, generally came from long vowels in early Midde English, so had settled into oo as its orthography.
BOAT - BOTE - BOOT
MEAT - METE - MEET
These equivalent orthographies have had different outcomes in Modern English, a merger having happened between /ɛː/ and /eː/ (the MEAT-MEET merger) in the Great Vowel Shift.