A piece could absolutely be written in B dorian, but the octave is irrelevant. Octave will never be indicated in the key of a song. If pitch level is important to the song, it has to be indicated in the notation of the music and the instrumentation.
Any triad with the intervals Root, flat third, perfect 5th is a minor triad, and will allow for a consonant resolution. Thus, B dorian is a minor key because the triad built (using the root third and fifth) in that mode gives you a minor triad. Because B dorian shares a key signature with A major (A ionian) you would write the key signature for A major in the piece. You would communicate to the musicians that the piece is in B dorian through the music - the fact that the music resolves to B dorian. Context is the only way that you know what musical home the piece is in. This is the case for any piece of music, and any of the 7 modes.
The key signatures are named after their most common usage which is still the major scale/ionian mode.
For example, the famous jazz song “So What” from the Miles Davis album “Kind of Blue” is in D dorian. The songs musical focus is D dorian, the resolutions are all to D dorian. It begins and ends on a D minor chord. The song (on paper) is written with the key signature of C major - no sharps, no flats, because those are the notes in D dorian.
I think the problem may be one of terminology.
Key signatures could (and probably should) be named for the notes that make them up, not for a single note and system that the key signature creates.
Meaning - The key of A major has 3 sharps, creating a 7 note pattern that repeats through all octaves infinitely. It could be called The Key of 3 Sharps. Or the key of ABC#DEF#G#, or it could be called the 3-o-clock Key (because of where it sits on the circle of fifths). A less harmonic-centric name for the Key Signature might help in understanding the key of 3 sharps to be simultaneusly the key of:
A major (ionian)
B dorian
C# Phrygian
D Lydian
E Mixolydian
F# minor (aeolian)
G# locrian
Because the most referenced, popular and historically significant pieces written with this system favor the major or minor scale (ionian or aeolean) the nomenclature for key signatures remains simplified as: the key of 3 sharps is the key of A major.
You’re absolutely correct that music theory is a mess. It is a system created for the aid in studying great compositions of the past. It comes up with terminology and rules and descriptions to describe things that already happened, and in the meantime musicians are bending and breaking all the rules and pushing music into new sonic places.
Learning music theory is frustrating in this way.
It starts out with - here are the rules and names for things, and you can’t break them or its wrong.
Then you go to your next year of theory and the rules change: OK, now you can break all those first rules, but here are the new rules and names for things, and these are the ones you can’t break.
Then you get to 20th century music and the rules get ridiculous. OK, so this composer made up his own rules, and to compose like this, here are the rules to follow. Also, all the rules can be broken now, so long as you know they’re there.
And then, finally… you end up in theory Nirvana:
If it sounds good, it’s right.
Not sure how this lands or if it helps.
I sure do love talking about this stuff though.
(I still have to read your post with full attention, but didn’t want this detail to linger)
So, it is often about knowing which scales to use to improvise on certain chords!
