I am transcribing an oral text into IPA in order to compare it with the "correct" pronunciation of words (e.g. according to the dictionary). I am using slashes for both versions, but it feels incorrect. Example: (actual written text); /hɛː san was/ (text as pronounced by the speaker); /hə(r) sʌn wəz/ (text according to dictionary). Should I use square brackets for the transcription of the actual pronunciation? Thank you!
Short answer: use slashes.
Slashes are used for phonemic transcriptions or for very approximate phonetic transcriptions, and brackets are used for phonetic transcriptions. A phonemic transcription has only enough information about the pronunciation to identify which words are being said, for someone who knows this language and dialect. A phonetic transcription has additional details about the pronunciation, which would be enough information for a knowledgeable person to make himself understood, even if he is not familiar with the language being spoken.
Phonetic transcriptions are language independent, at least in principle, though practically speaking, they will often omit some important information (for instance information about pitch and stress).
Phonemic transcriptions are language dependent, and they omit information which a person who knows the language would not need in order to understand what is transcribed.
In your example, I don't see a lot of phonetic detail -- the vowel of "son" is not written with a tilde indicating nasalization, for instance, even though the vowel probably is nasalized. So I'm guessing that you are only trying to capture phonetic information that you think is relevant to what is being said. That's why I recommend slashes.
Since your stated goal is to compare someone's actual pronunciation with some standard pronunciation, you should use square brackets, to indicate that you are talking about actually pronounced sounds. It does not matter whether the level of detail is narrow or broad; square brackets are for phones, slash brackets are for phonemes.
If you use slash brackets, you are asserting that what you write is a phonemic representation of speech, which should represent all (and only) the sounds that you believe to be contrastive sound categories in the talker's or listener's mental representation of the language (phonemes are abstract constructs, and are never spoken or heard).
To use your example, if you write
/hɛː san was/ then you are asserting that the vowels in the second and third words are identical in the speaker's mental representation. That may or may not be the case; e.g., they could be different phonemes that have a partially overlapping distribution for this speaker, such that in this particular sentence they came out sounding very similar to you. Or, they could be totally non-overlapping distributions, but each of their sounds happens to overlap with different parts of your distribution for the phoneme you represent as
/a/, so you classify them as the same sound, even though the speaker thinks they are producing two different sounds.
The Original Poster is going to have a massive problem here, brackets or no brackets. When we use slanty brackets, we use language-specific symbols to represent different phonemes. For example, the convention for transcribing the STRUT vowel in RP English is to use the upturned 'v': /ʌ/. This symbol does not relate in any meaningful way to the position of the notional sound on the vowel quadrilateral used on the international IPA chart.
If the Poster feels that a particular word produced uses a different sound from the one we would expect from the dictionary transcription, then, if they use slanty brackets, they are going to have to choose a different English-specific IPA symbol to represent that sound. It is highly unlikely that the sound produced by the speaker will neatly fit into the same slot as a different English phoneme. So for example, if they are saying /bed/ but with a very open /e/, the poster might decide to use an /æ/ instead. But of course, this sound is very unlikely to actually be more like a typical /æ/ than an /e/. So using an /æ/ here is not really appropriate. Apart from anything else the speaker clearly will be using their own allophone of /e/, not /æ/.
So that would seem to suggest that we need to use square brackets so that we can use an appropriate symbol from the international IPA set to describe the sound. But then the problem is that we can no longer use the remaining set of symbols chosen specifically for the transciption of English. The reason for this is that the strut vowel will no longer be able to be transcribed using /ʌ/, because the proper symbol for this when using the international set of symbols is going to be [ɐ], not [ʌ]. Similarly nearly all the other vowel symbols are going to have to change or have some diacritic added. This means that it will be very difficult to compare the output with what is given by the dictionary. Apart from this we can never be certain what target sound the dictionary transcription is intending anyway - because their transcription was from the set of English specific symbols relating to phonemes, not necessarily a clearly identified sound on the IPA vowel quadrilateral!
Although this second method might be better, it will take a lot more time and thought on the part of the Original Poster, and will necessarily be flawed to some degree.