I think the major idea behind the linguistics perception of all languages (except the pidgin ones I guess) is that all languages are good to what they are made for: to make people using it communicate clearly and effectively. That said all languages include all words and meanings necessary to achieve the communication. So, when people compare languages, words, vocabulary size etc they often project to a different language idea of their own language.
Example of the latter are when we are looking for a specific word to a different language that we cannot find. Or that the equivalent word is broader in sense than the one we are looking for. This often reminds me about the story of Eskimos having a huge amount of word to refer to the snow (I don't know how many words there are and I guess it's probably true if we consider they live an environment full of snow) while other languages does not have this diversity.
For me apart from your opinion (with which I agree) that the
relationship to one's native language
is the main factor which determines the difficulty to learn a language. Other factors that can affect someone trying to learn a language is:
- The use of different alphabet or an alphabet different than an already known well enough. To make it clearer: I am Greek native speaker, we use Greek alphabet which resembles the English one but it's obviously not the same. So, since my English are good enough I can apprehend German for example relatively easier being familiar with English alphabet than if I weren't familiar with.
- If in the language learning you also include pronunciation, then the closeness to your native or a well comprehensive language is also important. By this I mean only phonetic closeness. Example, Greek and Spanish are two distinct languages with different alphabet but really close phonetically: the only real differences is that Spanish do not use the 'z' sound and have a rough 'rr'. This fact make me learn easier this language in the sense that I don't need to invest time learning the pronunciation of various words and also I can understand quite easily the words being spoken.
- Another aspect that I can think of is the vocabulary shared between the language to learn and the already comprehensible by the learner languages. Example: English has made a really good job at inserting a huge amount of Latin-originated words through French mostly. This makes really easier for a person knowing English to find the correct French word. Thus he must just adopt to the French case system (non existing in English).
- You don't clarify what the previously mentioned relationship means so if it doesn't include the inflectional system of language to be learned. For example English native speaker find it harder to understand adjective cases than a person with an already comprehensive knowledge of language having cases.
I am not refering to other factor that are assumed to be equal for all subject learning a language:
Living in a country which the language is spoken or having friends speaking the language etc.
One good link for language diversity (and you can make your own conclusions over complexity what ever sense you give the word yourself is this)