x86: limit RegRex64 use

The special property really only applies to the "extended" byte regs
having legacy word/dword counterparts.

While touching involved code also drop redundant byte checks from a
conditional in establish_rex(): The other remaining RegRex64 uses only
exist on registers which can't be used as register operands anyway.
Hence RegRex64 as an attribute of a (valid) register operand implies
that it's a byte reg.