amd64-tdep: need_modrm = 1 for VEX/EVEX instructions, except vzeroall/vzeroupper

VEX and EVEX-encoded instructions generally require a ModR/M byte, with the
notable exception of vzeroall and vzeroupper (opcode 0x77), which do not
use ModR/M.

This change sets need_modrm = 1 for VEX instructions, and adds an exception
for instructions where *insn == 0x77, following Intel’s SDM.

EVEX has no exceptions and thus always sets need_modrm to 1.

Additionally, the legacy twobyte_has_modrm table cannot be used for VEX and
EVEX instructions, as these encodings have different requirements and
exceptions. The logic is now explicit for VEX/EVEX handling.

Add vpblendw to selftest amd64_insn_decode.

The Intel SDM says the following:

  1. Intel® 64 and IA-32 Architectures Software Developer’s Manual

  Section 2.2.1.2 — Instruction Prefixes

  "The VEX prefix is a multi-byte prefix that replaces several legacy prefixes
  and opcode bytes. The VEX prefix is not an opcode; it is a prefix that
  modifies the instruction that follows."

  Section 2.2.1.3 — Opcode Bytes

  "The opcode byte(s) follow any instruction prefixes (including VEX). The
  opcode specifies the operation to be performed."

  Section 2.2.2 — Instruction Format

  "If a VEX prefix is present, it is processed as a single prefix, and the
  opcode bytes follow immediately after the VEX prefix."

  Source: Intel® SDM Vol. 2A, Section 2.2.1.2 and 2.2.2 (See Vol. 2A,
  PDF pages 2-4, 2-5, and 2-7)

  2. ModRM Byte Requirement

  Intel® SDM Vol. 2A, Table 2-2 — VEX Prefix Encoding

  "Most VEX-encoded instructions require a ModRM byte, except for a few
  instructions such as VZEROALL and VZEROUPPER."

  Source: Intel® SDM Vol. 2A, Table 2-2 (See Vol. 2A, PDF page 2-13)

Approved-By: Tom de Vries <tdevries@suse.de>
1 file changed