If you're interested in efficient x86 code, see the links in the x86 tag wiki. There's a lot of good stuff, esp. Agner Fog's guides.
You have key in AL, but your cmp instructions all use a memory operand. There's a special opcode for cmp al, imm8, so cmp al, 75h is only a 2 byte instruction. Using an absolute displacement to address key makes a much longer instruction. Also, cmp mem,imm can't macro-fuse with a conditional jump. And every insn needs the load port.
The rest of your code looks suspiciously like it uses memory operands too much, and is indented strangely. (UP looks like it's part of the COLOR block, but actually there's an unconditional jump at the end of COLOR, so it doesn't fall into UP.)
Of course, a long series of cmp/je is nowhere near optimal, since all the je targets are the same. You don't need to figure out which key actually matched.
One strategy you can use for a check like that is to see if al is in the right range, then use it as an index into a bitmap.
Compilers use this strategy (Godbolt compiler explorer) for a switch or multi-condition if like this. This is why we use compilers instead of manually writing asm most of the time: they know lots of clever tricks and can apply them where applicable. We get 1<<c for the switch, but the if actually compiles to a bt with GCC. (GCC9 has a regression where the switch compiles to a jump table, though.)
See my answer on another ASCII question for an explanation of the unsigned-compare trick (ja .non_alphabetic) and an example of an efficient loop.
MOV [key], AL ; store for later use
or al, 20h ; lowercase (assuming an alphabetic character)
sub al, 'a' ; turn the ascii encoding into an index into the alphabet
cmp al, 'z'
ja .non_alphabetic
mov ecx, (1<<('a'-'a')) | (1<<('e'-a')) | (1<<('i'-a')) | (1<<('o'-a')) | (1<<('u'-a')) ; might be good to pull this constant out and use an EQU to define it
; movzx eax, al ; unneeded except for possible partial-register issues
bt ecx, eax ; test for the letter being set in the bitmap
jc UP ; jump iff al was a vowel
.non_alphabetic:
CMP dx,VK_ESCAPE ; this test could be first.
JE OVER
(bt masks its input, only using the low bits as the "shift count" so you don't really need movzx. But if you do need to avoid partial-register stalls on some uarches or something, use movzx edx, al instead of movzx eax, al because mov-elimination only works with different registers.)
This is significantly fewer instructions, and far fewer branches, so it uses up fewer branch-predictor entries.
Don't keep the constant in memory for bt: bt mem,reg is slow because of crazy-CISC semantics where it can access a different address is the bit index is higher than the operand-size. It only masks the bit-index when bt is used with a register first operand.
An alternative to bt is to do if(mask & 1 << (key - 'a')):
movzx ecx, al ; avoid partial-reg stall or false dep on ecx that you could get with mov ecx,eax or mov cl,ca respectively
mov eax, 1
shl eax, cl ; eax has a single set bit, at the index
test eax, 1<<('a'-'a') | 1<<('e'-a') | 1<<('i'-a') | 1<<('o'-a') | 1<<('u'-a')
jnz .vowel
This is more uops, even though test/jnz can macro-fuse, because variable-count shifts are 3 uops on Intel Sandbridge-family CPUs. (Again, crazy-CISC semantics slow things down).
Or right-shift the mask instead of creating 1<<c. You can even arrange to skip a test al,1 by having your mask right-shifted by 1 bit already, so the bit you want to branch on is shifted into CF by the shr.
But on Nehalem and earlier, reading the flag-result of a variable count shift stalls the front-end until the shift retires from the back-end, and on SnB-family it's still 3 uops for a variable-count shift.
Since comments are discussing SSE:
; broadcast the key to all positions of an xmm vector, and do a packed-compare against a constant
; assuming AL is already zero-extended into EAX
imul eax, eax, 0x01010101 ; broadcast AL to EAX
movd xmm0, eax
pshufd xmm0, xmm0, 0 ; broadcast the low 32b element to all four 32b elements
pcmpeqb xmm0, [vowels] ; byte elements where key matches the mask are set to -1, others to 0
pmovmskb eax, xmm0
test eax,eax
jnz .vowel
section .rodata:
align 16
vowels: db 'a','A', 'e','E'
db 'i','I', 'o','O'
db 'u','U', 'a','a'
times 4 db 'a' ; filler out to 16 bytes avoiding false-positives
A byte broadcast (SSSE3 pshufb or AVX2 vpbroadcastb) instead of a dword broadcast (pshufd) would avoid the imul. Or use or eax,0x20 before broadcasting so we don't need upper and lower case versions of every vowel, just lowercase. Then we could just broadcast with movd + punpcklbw + pshufd or something like that.
This requires loading a constant from memory, rather than a 32bit bitmap that can efficiently be an immediate in the instruction stream, so this is probably not as good even though it only has one branch. (Remember that the bitmap version need to branch on non-alphabetic, and then on being a vowel).
