Are those terms totally interchangeable in all contexts (finite = conjugated) (non-finite = conjugated) or are there slight meaning differences?
As these terms originate in Indo-European grammar, I will focus on Indo-European languages; these names may not apply as well to non-Indo-European languages.
The conjugation of a verb can mean two things:
1.) All the finite forms of a verb.
2.) All the inflected forms of a verb, including non-finite forms, like the infinitive and participles. I believe this usage is more common.
A finite form of a verb is one that has an ending; conventionally, the suffixes used to form infinitives (what's in a name?) and participles are not considered endings, although one might question the rationale behind this.
A verb ending is normally that part of a verb which expresses the features person and number (and usually mood). That is, a non-ending normally cannot express person or number or mood; but an ending can express more than person and number and mood (such as tense). You could say endings that express more than person, number, and mood are a contraction of some suffix with an ending, like the Latin imperfect (e.g. stabam, "I stood"), which is either analysed as present stem + imperfect suffix -ba- + ending (sta-ba-m), or as present stem + imperfect ending (sta-bam). We usually say the tense is included in the ending in such cases, of which there are many, but in other situations it is more practical to treat the suffix or contraction as a separate morpheme.
Strictly speaking, an ending / finite form is a kind of inflected form (finite inflexion), and an inflected form is a kind of suffixed form. Often, but not always, we use suffix to mean "non-inflexional suffix", because we already call inflected forms "inflected"— although inflected forms are also said to have inflexional suffixes in some contexts. "Inflected forms" normally does include all finite forms / endings.
Tense is a difficult concept of which different people use different definitions. An ending can be said to express tense, although there are endings that do not express tense. The forms of the Greek optative, for example, are always finite, but they never express a tense, while the endings of the imperfect do express tense (even though they are always finite too). Luoi = "may he unbind" (optative), which can be used in present, future, or past sentences alike, i.e. there is no real tense in it.
Aspect can be expressed in suffixes, in (suppletive) stems, in additional words, but also in endings, if you consider something like -bam above to be a single morpheme.
In English, most endings have contracted or disappeared in ways that make different persons, moods, and numbers invisible. There is still -s in the 3rd person present singular indicative for most verbs, and the most important and frequent verb of all, to be, still has many distinctive endings; but most other Indo-European languages have many more visible endings. The "invisible endings" in English finite verbs are often called zero endings, an kind of zero inflexion.
Because of this, many linguists have resorted to using the present tense v. the (synthetic) past tense as a more practical way of distinguishing between finite and non-finite forms. English doesn't really have endings that can be conveniently separated from tense, so a problem as with the Greek optative endings above (which have no tense) does not occur in English. So a way to test whether a verb form is a finite form or not is by testing whether it can be put in the (synthetic) past tense and present tense without changing the structure of the sentence. He goes to school => he went to school: that is possible, so it must be a finite form.
In Latin grammar the term "finite verbform" means the use of this verbform is limited to the use with a subject or a pronoun of the first or second or third person. Infinite verbforms as infinitive, participles or gerunds don't have this limited use. They are not connected with a subject or personal pronouns.