[gdb/tdep] Fix vmovdqu decoding
PR tdep/31952 reports that displaced stepping over an instruction pointer
relative insn "vmovdqu 0x20(%rip),%ymm1" gives the wrong results.
This is caused by misclassification of the insn in amd64_get_insn_details,
which results in details.modrm_offset == -1, while the instruction in fact
does have a modrm byte.
The instruction is encoded as follows:
...
400557: c5 fe 6f 0d 20 00 00 00 vmovdqu 0x20(%rip),%ymm1
...
where:
- "0xc5 0xfe" is the vex2 prefix,
- "0x6f" is the opcode,
- "0x0d" is the modrm byte, and
- "0x20 0x00 0x00 0x00" is a 32-bit displacement.
The problem is related to details.opcode_len, which is 1.
While it is true that the length of the opcode in the insn (0x6f) is 1 byte,
the vex2 prefix implies that we're encoding an 2-byte opcode beginnning
with 0x0f [1].
Consequently, we should be using the twobyte_has_modrm map rather than the
onebyte_has_modrm map.
Fix this in amd64_get_insn_details, and add a selftest to check this.
Tested on x86_64-linux.
Bug: https://sourceware.org/bugzilla/show_bug.cgi?id=31952
[1] https://en.wikipedia.org/wiki/VEX_prefix
1 file changed
