x86: Rework how VEX prefixes are decoded.

Remove redundant information from the ExtMachInst, hash the vex
information to ensure the decode cache works properly, print the vex info
when printing an ExtMachInst, consider the vex info when comparing two
ExtMachInsts, fold the info from the vex prefixes into existing settings,
remove redundant decode code, handle vex prefixes one byte at a time and
don't bother building up the entire prefix, and let instructions that care
about vex use it in their implementation, instead of developing an entire
parallel decode tree.

This also eliminates the error prone vex immediate decode table which was
incomplete and would result in an out of bounds access for incorrectly
encoded instructions or when the CPU was mispeculating, as it was (as far
as I can tell) redundant with the tables that already existed for two and
three byte opcodes. There were differences, but I think those may have
been mistakes based on the documentation I found.

Also, in 32 bit mode, the VEX prefixes might actually be LDS or LES
instructions which are still legal in that mode. A valid VEX prefix would
look like an LDS/LES with an otherwise invalid modrm encoding, so use that
as a signal to abort processing the VEX and turn the instruction into an
LES/LDS as appropriate.

Change-Id: Icb367eaaa35590692df1c98862f315da4c139f5c
Reviewed-on: https://gem5-review.googlesource.com/3501
Reviewed-by: Joe Gross <joe.gross@amd.com>
Reviewed-by: Jason Lowe-Power <jason@lowepower.com>
Maintainer: Anthony Gutierrez <anthony.gutierrez@amd.com>

1 files changed, 8 insertions, 6 deletions

diff --git a/src/arch/x86/decoder.hh b/src/arch/x86/decoder.hh
index 2e5e83764..93e5955c8 100644
--- a/src/arch/x86/decoder.hh
+++ b/src/arch/x86/decoder.hh
@@ -178,9 +178,10 @@ class Decoder
         ResetState,
         FromCacheState,
         PrefixState,
-        TwoByteVexState,
-        ThreeByteVexFirstState,
-        ThreeByteVexSecondState,
+        Vex2Of2State,
+        Vex2Of3State,
+        Vex3Of3State,
+        VexOpcodeState,
         OneByteOpcodeState,
         TwoByteOpcodeState,
         ThreeByte0F38OpcodeState,
@@ -199,9 +200,10 @@ class Decoder
     State doResetState();
     State doFromCacheState();
     State doPrefixState(uint8_t);
-    State doTwoByteVexState(uint8_t);
-    State doThreeByteVexFirstState(uint8_t);
-    State doThreeByteVexSecondState(uint8_t);
+    State doVex2Of2State(uint8_t);
+    State doVex2Of3State(uint8_t);
+    State doVex3Of3State(uint8_t);
+    State doVexOpcodeState(uint8_t);
     State doOneByteOpcodeState(uint8_t);
     State doTwoByteOpcodeState(uint8_t);
     State doThreeByte0F38OpcodeState(uint8_t);