Is there any rule in Old English / Modern English a/o (ham/home, ban/bone, stan/stone), a/oa (fam/foam, hlaf/loaf, gat/goat) transformation?
This difference is arbitrary: all of goot, gote, goote, goate, goat are attested by the end of Middle English in the 16th century, and the standardisation that happened afterward does not reflect any diachronic phonological difference.
The oCV orthography (as in gote, stone) is the older one, as it reflects the consequence of early Middle English /o/ undergoing open syllable lengthening to /ɔː/.
The "competing" value for the orthographic o, the closed /oː/, generally came from long vowels in early Midde English, so had settled into oo as its orthography.
BOAT - BOTE - BOOT
MEAT - METE - MEET
These equivalent orthographies have had different outcomes in Modern English, a merger having happened between /ɛː/ and /eː/ (the MEAT-MEET merger) in the Great Vowel Shift.
I found a picture on Wikipedia which lead me to believe that it was ðe old English long 'a' which evolved into modern English 'oa' as in 'hláf' to 'loaf' and 'gát' to 'goat' or sometimes 'o_e' as in 'hám' to 'home' or 'bán' to 'bone'. Ðis change does not take place if ðe 'a' is short, so if ðou have a textbook or oðer reference which marks long vowels wiþ perhaps a macron, ðat would be of great help. However, try as I might, I have failed to find any rule governing where it becomes 'o_e' and where it becomes 'oa', ðis, quite alike many oðer irregularities in English orþography probably occured due to variations in ðe preferences of fifteenþ century scribes, and ðerefore is quite unpredictable, sorry. One oðer problem is ðat not all words wiþ 'o_e' originated as having a long 'a', meaning ðere is even more uncertainty in knowing ðe origins of such words, so I am unable to help ðee ðere as well. However, if ðou are willing to do your own research, ðis is ðe Wikipedia page on which I found ðe picture, and it will also lead ðee to oðer pages regarding boþ ðe orþographical and sound changes ðat have occurred in English history, so I hope ðat comes to be of help.