Foreigners who say English is easy also often claim in the same breath that it "has no grammar." What they mean by that is that each word has relatively few forms that need to be learned. Irregular plurals and irregular verb forms are fairly limited in number. If you compare this to French, where people need to learn numerous conjugated forms of verbs, you'll see the difference. Since the memory of learning conjugation tables is emblematic for many people of the negative experience they had in school learning some language they may never have made any progress with, they tend to associate that kind of learning with what they consider difficult languages.
But there is of course much more than that to grammar, and that kind of morphology is usually not the hardest part of learning a language competently. When it comes to deciding which verb form to use, I've observed that second-language speakers probably make more mistakes in English than they do in French (sorry for the unscientific answer). Phrasal verbs are an additional difficulty, and I believe learners need to be very advanced before they can really get a good feel for the meanings of the particles. (They might know "butter", but would they "butter" their boss or "butter him up"?)
All in all, I would say I've seen more second-language speakers with near-perfect French than near-perfect English. This is perhaps less true as regards immigrants; I mainly have foreign-language speakers in mind.
These days, there is one important way in which English may well be easier than practically any other language, for most nationalities anyway. That's that you've already learned hundreds of English words before you so much as set foot in a classroom. This is because in many countries advertising uses English words, there may be many borrowings from English in the language, and it tends to be present in people's visual environment for other reasons too (for example, if they go to the default webpage for a multinational company).
I would hypothesize that Americans have a head start in Spanish for the same reason, making it easier to learn for them at the initial stage than Portuguese, for example. I'm not sure how much of this is the learning actually being easier as opposed to a kind of emotional response to seeing familiar words come up. I don't think there are many Americans who don't know that el, la, los and las mean "the" in Spanish, at least passively. How many know the corresponding words in Portuguese? If you are learning Portuguese, it won't take long to learn them, but it may feel good if you don't have to.