News for hackers. Stuff that matters.
Re:Weird things I learned while writing an x86 emulator by dmitrygr on Wednesday July 10, 2024 @01:20PM (#40929146)
I've written fast emulators for a dozen non-toy architectures and a few JIT translators for a few as well. x86 still gives me PTSD. I have never seen a messier architecture. There is history, and a reason for it, but still ... damn
Re:Weird things I learned while writing an x86 emulator by trealira on Wednesday July 10, 2024 @01:35PM (#40929301)
Studying the x86 architecture is kind of like studying languages with lots of irregularities and vestigial bits, and with competing grammatical paradigms, e.g. French. Other architectures, like RISC-V and ARMv8, are much more consistent.
Re:Weird things I learned while writing an x86 emulator by x0x0 on Wednesday July 10, 2024 @02:49PM (#40930101)
I think English may be a better example; we just stapled chunks of vulgar latin to an inconsistently simplified proto-germanic and then borrowed words from every language we met along the way. Add in 44 sounds serialized to the page with 26 letters and tada!
Re:Weird things I learned while writing an x86 emulator by WalterBright on Wednesday July 10, 2024 @06:37PM (#40932174)
The Norman conquest of England means English is a language with barbarian syntax and French nouns. It's a happy mess of cultural appropriation!

Squeezing the lot into 26 characters was simply genius - enabling printing with movable type, Morse code, Baudot code, ASCII, etc.

Of course, then icons came along and ruined everything.

Re:Weird things I learned while writing an x86 emulator by aengelke on Wednesday July 10, 2024 @04:31PM (#40931185)
> Other architectures, like [...] ARMv8, are much more consistent.

From an instruction/operation perspective, AArch64 is more clean. However, from an instruction operand and encoding perspective, AArch64 is a lot less consistent than x86. Consider the different operand types: on x86, there are a dozen register types, immediate (8/16/32/64 bits), and memory operands (always the same layout). On AArch64, there's: GP regs, incremented GP reg (MOPS extension), extended GP reg (e.g., SXTB), shifted GP reg, stack pointer, FP reg, vector register, vector register element, vector register table, vector register table element, a dozen types of memory operands, conditions, and a dozen types of immediate encodings (including the fascinating and very useful, but also very non-trivial encoding of logical immediates [1]).

AArch64 also has some register constraints: some vector operations can only encode register 0-15 or 0-7; not to mention SVE with it's "movprfx" prefix instruction that is only valid in front of a few selected instructions.

[1]: https://github.com/aengelke/disarm/blob/master/encode.c#L19-...

Re:Weird things I learned while writing an x86 emulator by trealira on Wednesday July 10, 2024 @05:42PM (#40931810)
Admittedly, I never wrote an assembler, but the encoding of x86-64 seems pretty convoluted [0] [1], with much of the information smeared across the some bits in the Mod/RM and SIB bytes and then extended by prefix bytes. It's more complicated than you would assume from having only written assembly in text and then having the assembler encode the instructions.

One of the things that sticks out to me is that on x86-64, operations on 32-bit registers implicitly zero out the upper 32-bits of the corresponding 64-bit register (e.g. "MOV EAX, EBX" also zeroes out the upper 32-bits of RAX), except for the opcode 0x90, which, logically, would be the accumulator encoding of "XCHG EAX, EAX" but is special-cased to do nothing because it's the traditional NOP instruction for x86. So you have to use the other encoding, 87 C0.

The fact that AArch64 has instructions that are almost always 4 bytes, and the fact that they clearly thought out their instruction set more (e.g. instead of having CMP, they just use SUBS (subtract and store condition codes) and store the result in the zero register, which is always zero), is what made me say that it seemed more regular. I hadn't studied it in as close detail as x86-64, though. But, you've written an an ARM disassembler and I haven't; you seem to know more about it than me, and so I believe what you're saying.

> AArch64 also has some register constraints

x86-64 also has some, in that many instructions can't encode the upper 8 bits of a 16-bit register (AH, BH, DH, CH) if a REX prefix is used.

[0]: https://wiki.osdev.org/X86-64_Instruction_Encoding

[1]: http://www.c-jump.com/CIS77/CPU/x86/lecture.html

Re:Weird things I learned while writing an x86 emulator by jcranmer on Wednesday July 10, 2024 @07:56PM (#40932629)
One of the projects I work on every now and then is the "World's Worst X86 Decoder", where I'm trying to essentially automatically uncover x86 assembly semantics without ever having to build a list of x86 instructions, and this has forced me to look at the x86 ISA in a different way. The basic format I build for an opcode is this:

  enum Group1Prefix { Lock = 0xf0, Repnz = 0xf2, Repz = 0xf3 }
  enum Group2Prefix { Cs = 0x2e, Ds = 0x3e, Es = 0x26, Fs = 0x64, Gs = 0x65, Ss = 0x36 }
  enum Group3Prefix { OpSize = 0x66 }
  enum Group4Prefix { AddrSize = 0x67 }
  enum ModernPrefix {
    Rex { w: bool },
    Vex { w: bool, l: bool },
  }
  struct Opcode {
    pub group1: Option<Group1Prefix>,
    pub group2: Option<Group2Prefix>,
    pub group3: Option<Group3Prefix>,
    pub group4: Option<Group4Prefix>,
    pub modern_prefix: Option<ModernPrefix>,
    pub opcode_map: u8,
    pub opcode: u8,
  }
And all opcodes can optionally have an immediate of 1, 2, 3, 4, or 8 bytes and optionally have a ModR/M byte (which is a separate datastructure because I don't enter that information myself, I simply run a sandsifter-like program to execute every single opcode and work out the answer).

This isn't quite accurate to how an assembler would see it, as Intel will sometimes pack instructions with 0 or 1 operands into a ModR/M byte, so that, e.g., 0F.01 eax, [mem] is actually the SGDT instruction and 0F.01 ecx, [mem] is SIDT (and 0F.01 eax, ecx is actually VMCALL).

As long as you're internally making a distinction between the different operand forms of instructions (e.g., 8-bit ADD versus 16-bit versus 32-bit versus 64-bit, and register/register versus register/immediate versus register/memory), it's actually not all that difficult to deal with the instruction encoding, or even mapping IR to those instructions, at least until EVEX prefixes enter the picture.

Re:Weird things I learned while writing an x86 emulator by trealira on Thursday July 11, 2024 @08:21AM (#40935875)
That's pretty cool; thanks for showing me. I guess x86 instruction encoding isn't as bad as I thought, then.
Re:Weird things I learned while writing an x86 emulator by WalterBright on Wednesday July 10, 2024 @06:31PM (#40932142)
> you've written an an ARM disassembler

Here's my AArch64 disassembler work in progress:

https://github.com/dlang/dmd/blob/master/compiler/src/dmd/ba...

I add to it in tandem with writing the code generator. It helps flush out bugs in both by doing this. I.e. generate the instruction, the disassemble it and compare with what I thought it should be.

It's quite a bit more complicated than the corresponding x86 disassembler:

https://github.com/dlang/dmd/blob/master/compiler/src/dmd/ba...

Re:Weird things I learned while writing an x86 emulator by bonzini on Wednesday July 10, 2024 @07:00PM (#40932311)
x86 has a lot of special cases and it's becoming harder and harder to find non-code material with all instructions in a nice tabular format. But overall the encoding isn't that complicated. What's complex in the architecture is the amount of legacy stuff that still lives in privileged code.

> x86-64 also has some, in that many instructions can't encode the upper 8 bits of a 16-bit register (AH, BH, DH, CH) if a REX prefix is used.

You can't use both the high registers and the extended low registers in the same instruction, indeed. But in practice nobody is using the high registers anymore so it's a relatively rare limitation.

Re:Weird things I learned while writing an x86 emulator by WalterBright on Wednesday July 10, 2024 @06:25PM (#40932104)
As I'm currently implementing an AArch64 code generator for the D language dmd compiler, the inconsistency of its instructions is equaled and worsened by the clumsiness of the documentation for it :-/ But I'm slowly figuring it out.

(For example, some instructions with very different encodings have the same mnemonic. Arrgghh.)

Re:Weird things I learned while writing an x86 emulator by brendank310 on Thursday July 11, 2024 @09:31AM (#40936610)
It's probably no longer maintained, but a former colleague of mine did some work on this for C++: https://github.com/ainfosec/shoulder. Obviously if the docs are lying it doesn't help much, but there was another effort he had https://github.com/ainfosec/scapula that tried to automate detecting behavior differences between the docs and the hardware implementation.
Re:Weird things I learned while writing an x86 emulator by corsix on Thursday July 11, 2024 @03:40AM (#40934474)
For an implementation of logical immediate encoding without the loop, see https://github.com/LuaJIT/LuaJIT/blob/04dca7911ea255f37be799...
Re:Weird things I learned while writing an x86 emulator by jcranmer on Wednesday July 10, 2024 @01:29PM (#40929237)
> I have never seen a messier architecture.

Itanium. Pretty much every time I open up the manual, I find a new thing that makes me go "what the hell were you guys thinking!?" without even trying to.

Re:Weird things I learned while writing an x86 emulator by snazz on Wednesday July 10, 2024 @03:57PM (#40930859)
What sorts of projects are you working on that use Itanium?
Re:Weird things I learned while writing an x86 emulator by jcranmer on Wednesday July 10, 2024 @04:59PM (#40931416)
None, really. I just happened to get a copy of the manual and start idly reading it when my computer got stuck in a very long update-reboot cycle and I couldn't do anything other than read a physical book.
Re:Weird things I learned while writing an x86 emulator by Arech on Wednesday July 10, 2024 @01:27PM (#40929219)
Haha, man, I feel you :DD You probably should have started with it from the very beginning :D