x86/APX: respect {vex}/{vex3}

Even when an EVEX encoding is available, use of such a prefix ought to
be respected (resulting in an error) rather than ignored. As requested
during review already, introduce a new encoding enumerator to record use
of eGPR-s, and update state transitions accordingly.

The optimize_encoding() change also addresses an internal assembler
error that was previously raised when respective memory operands used
eGPR-s for addressing.

While this results in a change of diagnostic issued for VEX-encoded
insns, the new one is at least no worse than the prior one.