Is there a term for the set of conversational actions/tools that instead of exchanging direct information, seek to instead affect a certain emotional state? Examples of such actions would include compliments, taunts, threats, (to express) doubt, etc. Is there any branch of linguists that studies this aspect of verbal communication?
In J. L. Austin's speech act theory, complimenting, persuading, frightening, and so on are termed perlocutionary acts: the effect of the speech on other people. Note that unintentional effects also fall into this category: if I say something innocuous that causes you to feel afraid, I've frightened you (a perlocutionary act), whether or not that was my intent.
This is contrasted with locutionary acts (the actual words you say and their literal meanings) and illocutionary acts (the implications of those words).
Austin's theory, and various alternatives to it, generally fall under the heading of pragmatics.