What specific label to assign to participants in events/states of affairs is a highly controversial issue because it is practically impossible to 'reduce' (without residue!) the innumerable semantic functions that predicates universally assign to their arguments to a small list of 'proto-roles' or 'archi-roles'. Many linguists more or less agree that some labels are adequate to characterize most of the more important and frequent ones, e.g., 'Agent' (or a much more general one: 'Actor'), 'Instrument' (or 'Means', or both), 'Possessor', 'Experiencer' (or 'Patient', or both), 'Affected', 'Theme' (or both), 'Resultant'/'Effected', 'Recipient', 'Beneficiary/Damnified','Goal', 'Source', 'Path', ...'Time', 'Place'/'Locative', etc., are all popular, but do not expect a consensus on the number of roles each theory distinguishes, on the specific names different linguists prefer to label them, on their definition, or, consequently, on their actual range of applicability. Fillmore tried hard and failed, in my view, and most of the others have not even tried hard. In some theories of the syntax-semantics interface, as a matter of fact, roles are specifically determined by the individual predicates that assign them; for example, 'give' will assign the roles of 'giver', 'given' and 'givee', rather than Agent, Theme and Recipient, whereas 'pay' will assign 'payer', 'paid' and 'payee', etc. Of course, that 'solves' the classification problem, but at a high price: loss of generalization (which is bad, because the original reason why thematic/semantic roles/Cases were 'invented' was to reduce semantic patterns to a small number of types and exploit such a reduction to establish recurrent syntax-semantics correspondences, projection regularities, recurrent alternations, etc.).
However, what you seem to be more puzzled about is a slightly different issue: whether it is only 'participant arguments' (syntactically: subjects and complements) that discharge thematic roles or 'circumstants' - syntactically, 'adjuncts' (of time, place, manner, purpose, etc.) also do, as would seem logical on purely semantic grounds. Thus, with respect to your examples (James got a ball yesterday, The cat chased the dog for a long time) you ask whether yesterday and for a long time are instances of a TIME role or just 'adjuncts'.
Well, of course, syntactically they are adjuncts (i.e., not subjects or complements), but, semantically, they unquestionably express 'time' and 'duration' circumstances, and, in principle, you could perfectly well claim that they also 'discharge' roles, in this case TIME and DURATION, perhaps.
Beyond this, matters have become highly theory-bound and answering your question in a way acceptable to the profession as a whole is probably impossible. The earliest 'modern' theories of 'roles' (mid 1960's), roughly those that coined the terms 'thematic roles' (Gruber's) or 'Cases' (Fillmore's), aimed rather at accounting for syntactic alternations than at defining LF and the syntax-semantics interface, and, with such an aim in view, only 'participants' (= arguments of the main predicate) were supposed to discharge 'thematic roles'/Cases. However, as linguists, from the late 1960's onwards, shifted their focus of interest from syntax sensu stricto to semantics/LF and the syntax-semantics interface, things changed substantially.
On the one hand, the mere distinction between complements and adjuncts had already proved rather difficult to justify in a substantial number of cases, and by the late 1980's major syntactic theories like Chomsky's (especially following Larson's work on 'shells') were generating subjects, certain complements (not all) and certain adjuncts (not all) in exactly the same position, i.e., in 'specifier of X', which cancelled the clear structural distinctions initially made in early X-bar theory (1970's, early 1980's) between subjects, complements, and adjuncts.
On the other hand, Davidson's influential early attempt (1967) to analyse the LF of clauses and the truth-conditional contribution of their syntactic constituents yielded flat sequences of conjoined predicate logic formulae in which 'participants' and 'circumstants' were treated in exactly the same way, i.e., as second arguments of theory-internally defined predicates - usually labelled according to their semantic 'role' (Agent, Instrument, Theme, Goal, yes, but also Time, Place, Manner, etc.) - whose first argument was the 'event' as a whole.
Thus, in your first sentence, for example, Davidson would have said that the verb got denotes a 'Getting' event (>'e') that is claimed to have existed [at some point in the past] via a prefixed existential quantifier over the variable 'e', the subject James would be rendered by the Pred-Logic clause 'Agent of (e, James)', the object a ball would appear at LF as 'Theme of (e, a ball)', and the 'adverbial' noun yesterday would be rendered by a similar Pred Logic clause, i.e., 'Time of (e, yesterday)'. Such a 'flat' analysis has its limitations, but roughly suffices to capture the original sentence's truth conditions and many people since then (1967) have been happy enough to have at least that as an LF to work with. Assuming that approach is adequate, since all constituents of the sentence incrementally contribute truth conditions to it, the null hypothesis is to try to represent them all in the same way, and, to that extent, syntactic adjuncts are as capable of discharging 'semantic roles' as subjects and complements are. The deep reason is that ultimately all are 'arguments' of one or another subsentential predicate; whether they are arguments of the main predicate (the verb) or of some 'subsidiary' one (say, a preposition introducing an adjunct of time, place, purpose, etc. that 'augments' the arity of the main predicate) is irrelevant from the Davidsonian viewpoint: Time, Place, etc. are also 'arguments' of predicates internal to the clause, and arguments are functionally defined and must have roles. [Actually, the idea of reducing a complex sentence to a conjunction of atomic predications is much older than Davidson's paper; a practically equivalent analysis had been offered by Wundt in his Die Sprache in 1900, and I think Wund may have taken inspiration from earlier work by Von der Gabelentz and other 19th c. German grammarians, although I can no longer off the cuff offer precise page references to work I read decades ago].
In sum, depending on the character and aims of your (or your teacher's) favourite theory of 'roles', yesterday and for a long time will indeed have the 'roles' of Time and Duration, or not, but even if that theory says that only participant arguments of the main verb can receive 'roles', it will still have to distinguish between 'adjuncts', and adding to the general syntactic term 'adjunct' notional labels like 'of Time' or 'of Duration' is still probably the commonest and easiest way to do so.