You can not select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
298 lines
15 KiB
298 lines
15 KiB
Design and Implementation Notes
|
|
===============================
|
|
The overall approach is:
|
|
|
|
* For each bytecode create an assembler routine
|
|
* Create a dispatch table with pointers to the assembler routines; the dispatch table is stored in the Lua global_State structure
|
|
but this could change in future
|
|
* Each assembler routine will (after completing its action) fetch the next bytecode instruction and jump to the next
|
|
assembler routine using the dispatch table (equivalent to computed goto)
|
|
* The assembler routines are **not** C functions - they are part of one whole program. Hence they make assumptions about
|
|
register usage which will be documented below. The VM as a whole will have a fixed set of register allocations so that most
|
|
important information is held in registers.
|
|
|
|
Implementation Considerations
|
|
-----------------------------
|
|
* The dispatch table is stored in global_State - it is not clear yet whether it is worth making a local stack copy of it when the
|
|
VM starts executing.
|
|
|
|
Why dynasm
|
|
----------
|
|
The implementation requires following key abilities that dynasm has:
|
|
|
|
* Obtain the offsets of the bytecode labels so that these can be gathered into a dispatch table.
|
|
* Use C code to calculate various structure offsets etc.
|
|
* Macros to create aliases for registers, and generate common sequences.
|
|
|
|
I am not sure whether this combination of features is available in other approaches such as using inline assembler in C code. I have briefly looked at:
|
|
|
|
* Inline assembly in gcc / clang - too ugly syntactically to work with
|
|
* Inline assembly in D - almost okay - no macros however so simple stuff like creating aliases for registers requires ugly mixin templates. The generated code appears to lack enough controls (such as disabling the frame pointer register, and ensuring correct unwind data on Win64). Also do not know how to get the offsets from generated code.
|
|
|
|
Using an assembler like yasm has the problem of computing offsets of C structures.
|
|
|
|
Issues with dynasm
|
|
------------------
|
|
On Windows 64-bit the generated code requires UNWIND information however the mechanism for this was in LuaJIT specific files (buildvm_peobj) and not fully reusable. I have modified this to de-couple from LuaJIT. This took some effort because LuaJIT's code
|
|
has numerous magic numbers with no explanation of what the code is doing. Not very helpful for anyone trying to work out what
|
|
the code is doing unless you already know what needs doing.
|
|
|
|
I wish dynasm could generate annotated assembly source file that is human readable. This would allow debuggers to display the
|
|
source code and would make debugging the assembly instructions much easier.
|
|
|
|
Register Allocations
|
|
--------------------
|
|
The VM will use a fixed set of registers mostly with some register usage varying across routines. The following table shows the
|
|
planned usage.
|
|
|
|
Nomenclature
|
|
|
|
* cs - callee saved, if we call a C function then after it returns we can rely on these registers
|
|
* v - volatile, these registers may be overridden by a called function so do not rely on them after function call
|
|
* `(n)` - used to pass arg n to function
|
|
|
|
+--------------------+------------------+------------------------------+------------------------------------------+
|
|
| Windows X64 reg | Linux X64 reg | Assignment | Notes |
|
|
+====================+==================+==============================+==========================================+
|
|
| rbx (cs) | rbx (cs) | PC | Pointer to next bytecode |
|
|
+--------------------+------------------+------------------------------+------------------------------------------+
|
|
| rbp (cs) | rbp (cs) | L | Pointer to lua_State |
|
|
+--------------------+------------------+------------------------------+------------------------------------------+
|
|
| rdi (cs) | rdi (v) (1) | | |
|
|
+--------------------+------------------+------------------------------+------------------------------------------+
|
|
| rsi (cs) | rsi (v) (2) | | |
|
|
+--------------------+------------------+------------------------------+------------------------------------------+
|
|
| rsp (cs) | rsp | | Stack pointer |
|
|
+--------------------+------------------+------------------------------+------------------------------------------+
|
|
| r12 (cs) | r12 (cs) | CI | CallInfo (Lua frame) |
|
|
+--------------------+------------------+------------------------------+------------------------------------------+
|
|
| r13 (cs) | r13 (cs) | LCL | Current function's LClosure |
|
|
+--------------------+------------------+------------------------------+------------------------------------------+
|
|
| r14 (cs) | r14 (cs) | DISPATCH | Ptr to Dispatch table |
|
|
+--------------------+------------------+------------------------------+------------------------------------------+
|
|
| r15 (cs) | r15 (cs) | KBASE | Ptr to constants table in Proto |
|
|
+--------------------+------------------+------------------------------+------------------------------------------+
|
|
| rax (v) | rax (v) | RCa = rax, RC = eax | Scratch - also eax used for |
|
|
| | | | the B,C portion of bytecode |
|
|
+--------------------+------------------+------------------------------+------------------------------------------+
|
|
| rcx (v) (1) | rcx (v) (4) | RAa = rcx, RA = ecx | Scratch - also ecx used for |
|
|
| | | | the value of A in bytecode |
|
|
+--------------------+------------------+------------------------------+------------------------------------------+
|
|
| rdx (v) (2) | rdx (v) (3) | RBa = rdx, OP = edx | Scratch - also edx used for |
|
|
| | | | the OpCode |
|
|
+--------------------+------------------+------------------------------+------------------------------------------+
|
|
| r8 (v) (3) | r8 (v) (5) | BASE | Pointer to Lua stack base |
|
|
+--------------------+------------------+------------------------------+------------------------------------------+
|
|
| r9 (v) (4) | r9 (v) (6) | TMP3 | Scratch |
|
|
+--------------------+------------------+------------------------------+------------------------------------------+
|
|
| r10 (v) | r10 (v | TMP1 | Scratch |
|
|
+--------------------+------------------+------------------------------+------------------------------------------+
|
|
| r11 (v) | r11 (v) | TMP2 | Scratch |
|
|
+--------------------+------------------+------------------------------+------------------------------------------+
|
|
|
|
Stack space
|
|
-----------
|
|
On Win64 every function gets a 32-byte shadow space for the 4 register arguments, which we can use. But we also need
|
|
to provide a shadow space for function calls inside the VM. Basically these 4 stack positions cancel out if we make use
|
|
of the slots provided by the caller.
|
|
|
|
VMBuilder
|
|
---------
|
|
The `VMBuilder <https://github.com/dibyendumajumdar/ravi/tree/master/vmbuilder/src>`_ is a tool that generates the code for the VM. It is a sub-project and the only component that needs to use dynasm.
|
|
|
|
To build the tool, you have to use CMake. Following steps are for Windows::
|
|
|
|
cd vmbuilder\src
|
|
mkdir build
|
|
cd build
|
|
cmake -G "Visual Studio 15 Win64" ..
|
|
|
|
Above will generate the Visual Studio project that you can open in VS2017. Do a BUILD, followed by INSTALL. The resulting buildvm.exe
|
|
will be installed under ``vmbuilber/bin``.
|
|
|
|
On Linux, the process is as follows::
|
|
|
|
cd vmbuilder/src
|
|
mkdir build
|
|
cd build
|
|
cmake ..
|
|
make
|
|
make install
|
|
|
|
Running VMBuilder on Windows
|
|
----------------------------
|
|
VMBuilder tool is run as follows on Windows::
|
|
|
|
cd vmbuilder\bin
|
|
buildvm -m peobj -o vm.obj
|
|
buildvm -m bcdef > ravi_bcdef.h
|
|
|
|
If you have LLVM installed you can obtain the assembly source as follows::
|
|
|
|
llvm-objdump -unwind-info -d -t vm.obj > vm.asm
|
|
|
|
Running VMBuilder on UNIX systems
|
|
---------------------------------
|
|
On Linux, the steps for unning ``buildvm`` are::
|
|
|
|
cd vmbuilder/bin
|
|
./buildvm -m elfasm -o vm.s
|
|
./buildvm -m bcdef > ravi_bcdef.h
|
|
|
|
Note that you should not have to run VMBuilder manually this way as the Ravi CMake build will do these steps anyway.
|
|
|
|
Files generated by VMBuilder
|
|
----------------------------
|
|
On Windows VMBuilder generates an object file. On Linux and Mac OSX it generates assembly source file.
|
|
The generated file is then linked with the main Ravi library.
|
|
|
|
Additionally a header file is generated named 'ravi_bcdef.h' - containing an array of offsets that can be used to obtain the
|
|
address of each assembly routine by adding the offset to the 'ravi_vm_asm_begin' symbol.
|
|
|
|
Current Issues
|
|
--------------
|
|
* Some additional work may be necessary to link the ASM routines when shared library builds are on - at least on Windows where
|
|
I noticed that the ASM functions were not properly being invoked. Have switched to static builds for now.
|
|
|
|
Exported Symbols
|
|
----------------
|
|
The main public symbols that are accessed externally are:
|
|
|
|
* ravi_vm_asm_begin - this is the start of the generated code, and all the assembly routines are at offets relative to the address of this symbol.
|
|
* ravi_luaV_interp - this is the entry VM point, equivalent to luaV_execute().
|
|
|
|
Setup of dispatch table
|
|
-----------------------
|
|
Currently this occurs in `lstate.c <https://github.com/dibyendumajumdar/ravi/blob/master/src/lstate.c>`_ in function ``dispatch_init()`` which is shown below::
|
|
|
|
#ifdef RAVI_USE_ASMVM
|
|
/* Initialize dispatch table used by the ASM VM */
|
|
static void dispatch_init(global_State *G) {
|
|
ASMFunction *disp = G->dispatch;
|
|
for (uint32_t i = 0; i < NUM_OPCODES; i++) {
|
|
/*
|
|
Following computes an offset for the assembly routine for the given OpCode.
|
|
The offset is relative to the global symbol ravi_vm_asm_begin that is
|
|
generated as part of the VMBuilder code generation. All the bytecode
|
|
routines are at some offset to this global symbol.
|
|
*/
|
|
/* NOTE: enabling ltests.h modifies the global_State and breaks the assumptions about
|
|
the location of the dispatch table */
|
|
disp[i] = makeasmfunc(ravi_bytecode_offsets[i]);
|
|
}
|
|
}
|
|
#endif
|
|
|
|
OpCode structure
|
|
----------------
|
|
The OpCode is stored in LuaJIT inspired format when the ASM VM is enabled. This is showns below::
|
|
|
|
+---+---+---+----+
|
|
| B | C | A | Op |
|
|
+---+---+---+----+
|
|
| Bx | A | Op |
|
|
+-------+---+----+
|
|
| Ax | Op |
|
|
+-----------+----+
|
|
|
|
Note that the bytecode decoding assumes above and will break if Lua style encoding is used.
|
|
|
|
Progress
|
|
--------
|
|
The only op codes implemented so far are:
|
|
|
|
* OP_RETURN (some scenarios to be tested, e.g. upvalues)
|
|
* OP_LOADK
|
|
* OP_MOVE
|
|
* OP_RAVI_FORPREP_IP and OP_RAVI_FORPREP_I1
|
|
* OP_RAVI_FORLOOP_I1 and OP_RAVI_FORLOOP_I1
|
|
|
|
Here is a `link to the generated assembly code on Windows X64 <https://github.com/dibyendumajumdar/ravi/blob/master/vmbuilder/asm/vm-win64.asm>`_.
|
|
|
|
The equivalent `Linux assembly code is <https://github.com/dibyendumajumdar/ravi/blob/master/vmbuilder/asm/vm-linux64.s>`_.
|
|
|
|
It seems hard to test and validate the assembly code. I have to step through the code instruction by instruction.
|
|
|
|
On Windows I debug in Visual Studio 2017.
|
|
|
|
On Linux I use Eclipse CDT to debug the code. I use the instruction step mode.
|
|
|
|
On Mac OSX I use Xcode.
|
|
|
|
I think it might be useful to create a test harness that mocks the Lua structures
|
|
and functions so that each bytecode instruction can be tested in isolation. This will need some work however, so not yet sure.
|
|
The other alternative is to check by running Lua scripts which is how we test Ravi normally.
|
|
|
|
Windows X64 Specifics
|
|
---------------------
|
|
On Windows the VMBuilder tool generates object code rather than an assembly source file. Win64 also requires some special data
|
|
for stack unwinding in case of exceptions. I believe even longjmps trigger this functionality.
|
|
|
|
The way we handle this now is by generating following in the object file::
|
|
|
|
Unwind info:
|
|
|
|
Function Table:
|
|
Start Address: ravi_vm_asm_begin
|
|
End Address: ravi_vm_asm_begin + 0x06a2
|
|
Unwind Info Address: .xdata
|
|
Version: 1
|
|
Flags: 0
|
|
Size of prolog: 0
|
|
Number of Codes: 9
|
|
No frame pointer used
|
|
Unwind Codes:
|
|
0x00: UOP_AllocSmall 40
|
|
0x00: UOP_PushNonVol R15
|
|
0x00: UOP_PushNonVol R14
|
|
0x00: UOP_PushNonVol R13
|
|
0x00: UOP_PushNonVol R12
|
|
0x00: UOP_PushNonVol RBX
|
|
0x00: UOP_PushNonVol RSI
|
|
0x00: UOP_PushNonVol RDI
|
|
0x00: UOP_PushNonVol RBP
|
|
|
|
Basically above tells Windows what the function epilogue (stack) looks like so that Windows can correctly restore the registers when
|
|
unwinding the stack. Note that the unwind information applies to the entire generated code and not a specific function. In particular
|
|
the assumption is that there any entry point in the code needs to have a prologue that is the exact inverse of the
|
|
epilogue described above.
|
|
|
|
Here is the function prologue::
|
|
|
|
ravi_luaV_interp:
|
|
651: 55 pushq %rbp
|
|
652: 57 pushq %rdi
|
|
653: 56 pushq %rsi
|
|
654: 53 pushq %rbx
|
|
655: 41 54 pushq %r12
|
|
657: 41 55 pushq %r13
|
|
659: 41 56 pushq %r14
|
|
65b: 41 57 pushq %r15
|
|
65d: 48 83 ec 28 subq $40, %rsp
|
|
|
|
And the epilogue::
|
|
|
|
ravi_vm_return:
|
|
691: 48 83 c4 28 addq $40, %rsp
|
|
695: 41 5f popq %r15
|
|
697: 41 5e popq %r14
|
|
699: 41 5d popq %r13
|
|
69b: 41 5c popq %r12
|
|
69d: 5b popq %rbx
|
|
69e: 5e popq %rsi
|
|
69f: 5f popq %rdi
|
|
6a0: 5d popq %rbp
|
|
6a1: c3 retq
|
|
|
|
As you can see the unwind information basically tells Windows what the epilogue is supposed to be, and where to find the saved
|
|
values of the registers.
|
|
|
|
Building Ravi With New ASM VM
|
|
-----------------------------
|
|
This is only for the brave who want to hack with the code.
|
|
|
|
To enable the new VM first build and install VMBuilder as described above.
|
|
Then build Ravi using the cmake flags ``-DSTATIC_BUILD=ON`` and ``-DASM_VM=ON`` enabled. Don't enable JIT.
|
|
|
|
Right now the ASM VM is exercised via the ``test_asmvm`` sub project. The ASM VM is only invoked in special cases, i.e. a function has small number of instructions and only contains supported instructions, and additionally as OP_CALL is not yet implemented, you can only call the new VM via the Lua C api (see `test_asmvm() in test_asmvm.c <https://github.com/dibyendumajumdar/ravi/blob/master/tests/test_asmvm.c>`_).
|