Can a grammar formalism be something that's neither a dependency grammar nor a constituency grammar?
In a manner of speaking, yes. The analysis of phrases into immediate constituents is not really a matter of theory, but more a matter of fact, which any grammatical theory must provide for. So, in this sense, every grammar formalism must tell you about constituency (even dependency grammar).
If, however, by constituency grammar you mean a grammatical theory with constituents at its most primitive level, which is probably what you do mean, then context free phrase structure grammar (psg) is a constituency grammar, because production rules, aka phrase structure rules, are primitives of the theory, and we can equate the right hand sides of productions with constituents.
Surveying an appropriate dependency diagram for a particular phrase, you will not see all the constituents of the phrase given explicitly, though you can probably deduce them, so dependency grammar is probably not to be counted as a constituency grammar.
Now, finally, to your question. Arc Pair Grammar (with Relational Grammar) gives graphs to describe the structure of phrases, and, as with dependency grammar, looking at an arc pair diagram, you don't see the constituents given explicitly.
My own theory of grammar, 2psg (How ... vertical dimension ...?) is another example of a theory in which constituents are not primitives. In 2psg, phrases are built up from "forms", which incorporate both constituency information (the pronunciation) and valency information (the category of a constituent). Discontinuous constituents have a straightforward and natural account in 2psg (though they do not in psg).