I think that the main factors are going to be language external. Japanese people do not know English very well for reasons having to do with how English is used in Japan, and reasons having to do with the general language ecology in Japan.
Scandinavians tend to have very good English, and there are language-internal reasons (their native language has a large number of vowels, there are many Germanic cognates, etc.) as well as external reasons (English has become established as a lingua Franca in Northern Europe, there is intensive English education and ample opportunities to practice, living in a multilingual society, they are comfortable communicating in a language that they don't master, etc.). So it could be debatable whether the language-internal or the language-external reasons are more important in determining the English skills of Scandinavians.
But consider English in Nigeria. Roughly 400 languages are spoken in Nigeria, most of which are as different from English as Japanese is from English, but educated Nigerians tend to have a high success rate in learning English as a second language. In this case, there is nothing about their native languages that helps to explain their success in learning English. The status English has in government and business in that country, as well as the multilingual nature of the society, have to play a large role.
My hunch then is that grammar is not the main barrier for Japanese people to learn English, because grammar is not a barrier for English learning in countries where the sociolinguistic situation is favorable for English learning; I think that all of the main barriers have to do with the sociolinguistic situation in Japan: that the society is basically monolingual, and that languages other than Japanese do not have an important status in any major public sphere.