SPE-era phonologists used the term "matrix" for features, primarily because that is a term used by Chomsky and Halle, but users were generally unaware of the mathematical properties of matrices, as the term is used in mathematics (that elements are strictly ordered). Essentially, all that people were committed to was the graphic presentation.
There were two ways of presenting a phonological string: as an ordered set of column vectors (each called a "matrix"), or as one nxm matrix of n segments. The latter representation was not particularly common, and the brackets were generally taken to specifically represent "segment". In rule formalism, you never encounter an nxm matrix, instead you encounter an ordered sequence of columnar "matrices". But as discussed here, ordering of features within a segment is redundant (and the number of value-attribute pairs in any segment is not predictable by convention, so features must be "labeled"), so since there is no reason to additionally impose order on features within a segment, "matrix" is not really an appropriate term describing a segment.
It would be appropriate to say that a phonological string is a row vector of sets of features: elements in each set (segment) are not ordered, but each set (segment) is ordered w.r.t. other segments. Which is exactly the situation that you described.
