Is it orthography in general, maybe punctuation, or something else?
Capitalization is part of "orthography in general". Which is usually not considered part of linguistics at all,1 but can at least be studied in similar ways.2
I'm not aware of any formal division of orthography into different areas, but I think in general you'd treat capitalization as a top-level area of study, alongside punctuation, spelling, letter forms, etc., not as part of one of those areas.3
Of course there will always be some overlap between the different areas, and some questions where it's not clear where they fall. But let's look at some examples:
- Are "ares" (the plural of "are") and "Ares" (the god of war) homographs? That seems to be a spelling question.
- Which words are capitalized wherever they appear? In English, it's proper names and "I"; in German, it's both common and proper nouns and "Sie".4 That might be a spelling question.5
- How are titles capitalized? English capitalizes everything but a closed class of "small words";6 Spanish treats titles like sentences. That's clearly at a different level than spelling (roughly, phrases, rather than words).
- How are digraphs (that are considered a single letter) capitalized? At first this seems like just a question of letter forms—Dutch has a capital form "IJ" to go with "ij"; Croatian has "Nj" to go with "nj". But when you look at Czech, which capitalizes "ch" as "CH" in some contexts and "Ch" in others, it's clear that there's something else going on here—again, at a higher level than individual letters.
I think that shows that capitalization really is separate enough from other areas of orthography that it's worth treating as its own thing.
1. There are both philosophical and methodological reasons that linguistics studies only, or at least primarily, spoken language, which are too deep to get into here. At any rate, orthography seems to be on-topic for this site, even if it isn't on-topic for many academic linguistics departments.
2. As with (spoken) language, it can be studied descriptively—working out the actual rules people use in practice, rather than trying to work out some set of rules from first principles and then prescriptively teaching people to follow those rules. In fact, lexicographers often do an emic descriptive study of the prescriptive rules of both language and orthography; I'm pretty sure Geoffrey Pullum has a good blog post on that somewhere, but I don't remember where.
3. For what it's worth, Wikipedia says: "Orthography is largely concerned with matters of spelling, and in particular the relationship between phonemes and graphemes in a language. Other elements that may be considered part of orthography include hyphenation, capitalization, word breaks, emphasis, and punctuation." So, they treat capitalization as one of the top-level areas. But it seems weird that they don't separate out the question of which (variant of which) script a language uses from the question of how phonemes are mapped to graphemes in that script. Wikipedia has articles on the Roman alphabet, the set of CJK characters, etc. independently of articles on individual languages' orthography, and it seems like it would be mad not to do that.
4. But that's an oversimplification, and in fact the actual rules literate Germans use are not identical to the rules published in the Reform der deutschen Rechtschreibung von 1996.
5. You can just say "I" is spelled with a capital "I", and "me" is spelled with lowercase "m" and "e". But what about "the lord God" vs. "the god of the hearth", or "President Trump" vs. "a past president"? Unless those are actually separate words, there's more going on than just how each word is spelled.
6. … except that there are some cases where people capitalize all of the words. For example, album titles almost always do this: Beat This! The Best Of The English Beat, Classic Rock Best Of The Year 2018, etc. So the rule is more complicated than what you learned in school, and might be worth studying to tease out what it actually is. Which is exactly the same kind of thing linguists do to discover the actual syntactic, etc. rules that speakers use, even if it's not linguistics.
Linguistics itself isn't really concerned with orthography at all, because linguistics aims at describing how natural languages work as they are (descriptivism), while orthography is usually understood as imposing rules on how written language should work (prescriptivism) - of course, as abarnert points out, the real-life application of orthography could also be studied descriptively, but de facto most discussions about orthography revolve around the question how things should be spelled and not on how they are spelled, and that's just what linguistics normally wants to do. Besides that, linguistics primarily focusses on the study of spoken rather than written language; sounds rather than letters are seen as the primary units of language and subject of interest. The comments on this post and this post discuss some reasons why.
Within the field of orthography, capitalization is certainly not part of punctuation, but would probably constitute its own category, alongside
- general rules for the phonemes-to-graphemes (sounds-to-letters) mapping
- exceptions/specific spelling of individual words
- punctuation (where to set commas, quotation marks, apostrophes, etc.)
- separate vs. hyphenated vs. compound spelling
- heuristics for the representation of foreign words and proper names
- and maybe others.
Which precise categories would be seen as basic depends on the language - punctuation is likely a relevant question in pretty much all written languages, whereas e.g. separate vs. compund spelling or capitalizaton is less of an issue in English than it is in German or so.