* Built-in C pre-processor, parser and JIT compiler
* A `distribution with batteries <https://github.com/dibyendumajumdar/ravi-distro>`_.
* A `distribution with batteries <https://github.com/dibyendumajumdar/Suravi>`_.
Documentation
=============
@ -47,17 +47,17 @@ Documentation
Lua Goodies
===========
* `An Introduction to Lua <http://the-ravi-programming-language.readthedocs.io/en/latest/lua-introduction.html>`_ attempts to provide a quick overview of Lua for folks coming from other languages.
* `Lua 5.3 Bytecode Reference <http://the-ravi-programming-language.readthedocs.io/en/latest/lua_bytecode_reference.html>`_ is my attempt to bring up to date the `Lua 5.1 Bytecode Reference <http://luaforge.net/docman/83/98/ANoFrillsIntroToLua51VMInstructions.pdf>`_.
* `Lua 5.3 Bytecode Reference <http://the-ravi-programming-language.readthedocs.io/en/latest/lua_bytecode_reference.html>`_ is my attempt to bring up to date the `Lua 5.1 Bytecode Reference <http://luaforge.net/docman/83/98/ANoFrillsIntroToLua51VMInstructions.pdf>`_.
Compatibility with Lua
======================
Ravi should be able to run all Lua 5.3 programs in interpreted mode, but following should be noted:
Ravi should be able to run all Lua 5.3 programs in interpreted mode, but following should be noted:
* Ravi supports optional typing and enhanced types such as arrays (described above). Programs using these features cannot be run by standard Lua. However all types in Ravi can be passed to Lua functions; operations on Ravi arrays within Lua code will be subject to restrictions as described in the section above on arrays.
* Ravi supports optional typing and enhanced types such as arrays (described above). Programs using these features cannot be run by standard Lua. However all types in Ravi can be passed to Lua functions; operations on Ravi arrays within Lua code will be subject to restrictions as described in the section above on arrays.
* Values crossing from Lua to Ravi will be subjected to typechecks should these values be assigned to typed variables.
* Upvalues cannot subvert the static typing of local variables (issue #26) when types are annotated.
* Certain Lua limits are reduced due to changed byte code structure. These are described below.
* Ravi uses an extended bytecode which means it is not compatible with Lua 5.3 bytecode.
* Ravi uses an extended bytecode which means it is not compatible with Lua 5.3 bytecode.
+-----------------+-------------+-------------+
| Limit name | Lua value | Ravi value |
@ -80,19 +80,21 @@ When JIT compilation is enabled there are following additional constraints:
History
=======
* 2015
* 2015
- Implemented JIT compilation using LLVM
- Implemented libgccjit based alternative JIT (now discontinued)
* 2016
- Implemented debugger for Ravi and Lua 5.3 for `Visual Studio Code <https://github.com/dibyendumajumdar/ravi/tree/master/vscode-debugger>`_
* 2017
- Embedded C compiler using dmrC project (C JIT compiler)
* 2016
- Implemented debugger for Ravi and Lua 5.3 for `Visual Studio Code <https://github.com/dibyendumajumdar/ravi/tree/master/vscode-debugger>`_
* 2017
- Embedded C compiler using dmrC project (C JIT compiler)
- Additional type-annotations
* 2018
* 2018
- Implemented Eclipse OMR JIT backend
- Created `Ravi with batteries <https://github.com/dibyendumajumdar/ravi-distro>`_.
- Created `Ravi with batteries <https://github.com/dibyendumajumdar/Suravi>`_.
@ -15,7 +15,7 @@ For building Ravi with JIT options please read on.
Build Dependencies
==================
* `CMake <https://cmake.org/>`_ is required for more advanced builds
* `CMake <https://cmake.org/>`_ is required
* On Windows you will need Visual Studio 2017 Community edition
* LLVM versions >= 3.5
@ -23,7 +23,8 @@ LLVM JIT Backend
================
Following versions of LLVM work with Ravi.
* LLVM 3.7, 3.8, 3.9, 4.0, 5.0, 6.0
* LLVM 3.7, 3.8, 3.9, 4.0, 5.0, 6.0, 8.0.1
* LLVM 7.0 was skipped because of unstable ORC api changes
* LLVM 3.5 and 3.6 should also work but have not been recently tested
Unless otherwise noted the instructions below should work for LLVM 3.9 and later.
@ -31,56 +32,54 @@ Unless otherwise noted the instructions below should work for LLVM 3.9 and later
Since LLVM 5.0 Ravi has begun to use the new ORC JIT apis. These apis are more memory efficient
compared to the MCJIT apis because they release the Module IR as early as possible, whereas with
MCJIT the Module IR hangs around as long as the compiled code is held. Because of this significant
improvement, I recommend using LLVM 5.0 and above.
improvement, I recommend using the latest version of LLVM.
Building LLVM on Windows
------------------------
I built LLVM from source. I used the following sequence from the VS2017 command window::
cd \github\llvm
mkdir build
cd build
cmake -DCMAKE_INSTALL_PREFIX=c:\LLVM -DLLVM_TARGETS_TO_BUILD="X86" -G "Visual Studio 15 2017 Win64" ..
cmake -DCMAKE_INSTALL_PREFIX=c:\Software\llvm801 -DLLVM_TARGETS_TO_BUILD="X86" -G "Visual Studio 15 2017 Win64" ..
I then opened the generated solution in VS2017 and performed a INSTALL build from there. Above will build the 64-bit version of LLVM libraries. To build a 32-bit version omit the ``Win64`` parameter.
..note:: Note that if you perform a Release build of LLVM then you will also need to do a Release build of Ravi otherwise you will get link errors.
..note:: Note that if you perform a **Release** build of LLVM then you will also need to do a **Release** build of Ravi otherwise you will get link errors. Ditto for **Debug** builds.
Building LLVM on Ubuntu
-----------------------
On Ubuntu I found that the official LLVM distributions don't work with CMake. The CMake config files appear to be broken.
So I ended up downloading and building LLVM from source and that worked. The approach is similar to that described for MAC OS X below.
Building LLVM on Ubuntu or Redhat
---------------------------------
The approach is similar to that described for MAC OS X below.
Building LLVM on MAC OS X
-------------------------
I am using Max OSX El Capitan. Pre-requisites are XCode 7.x and CMake.
I am using Max OSX Mojave. Pre-requisites are XCode and CMake.
Ensure cmake is on the path.
Assuming that LLVM source has been extracted to ``$HOME/llvm-3.7.0.src`` I follow these steps::
Assuming that LLVM source has been extracted to ``$HOME/llvm-8.0.1.src`` I follow these steps::
I am developing Ravi using Visual Studio 2017 Community Edition on Windows 10 64bit, gcc on Unbuntu 64-bit, and clang/Xcode on MAC OS X. I was also able to successfully build a Ubuntu version on Windows 10 using the newly released Ubuntu/Linux sub-system for Windows 10.
I am developing Ravi using Visual Studio 2017 Community Edition on Windows 10 64bit, gcc on Linux 64-bit, and clang/Xcode on MAC OS X. I was also able to successfully build a Ubuntu version on Windows 10 using the newly released Ubuntu/Linux sub-system for Windows 10.
..note:: Location of cmake files prior to LLVM 3.9 was ``$LLVM_INSTALL_DIR/share/llvm/cmake``.
Assuming that LLVM has been installed as described above, then on Windows I invoke the cmake config as follows::
I open the generated project in Xcode and do a build from there. You can also use the command line build tools if you wish - generate the make files in the same way as for Linux.
Building without JIT
====================
You can omit ``-DLLVM_JIT=ON``and ``OMR_JIT=ON``options to build Ravi with a null JIT implementation.
You can omit ``-DLLVM_JIT=ON`` options to build Ravi with a null JIT implementation.
4. Next phase will be linearize the instructions - during this phase the loops, if statements etc will get translated to equivalent of
conditional and unconditional jumps.
5. We could at this stage generate byte code and finish. Initially maybe we will stop here.
5. In the first phase we will stop here, generate byte code and finish.
6. The next phase will be translate into basic blocks.
7. Following that we will construct a CFG, perform dominator analysis and convert to SSA form.
8. Hopefully as a result of above we can do some simple optimizations and then emit the bytecode at the end.
Some other things
-----------------
1. locals that are used as up-values or passed or overwritten by function calls should be marked as having 'escaped'.
1. Locals that are used as up-values or passed or overwritten by function calls should be marked as having 'escaped'.
Having this knowledge will enable backend JIT to use the stack for values that cannot escape.
2. during code generation it will be good to know which registers are type constant - i.e. their types do not change. register allocation
2. During code generation it will be good to know which registers are type constant - i.e. their types do not change. register allocation
should be designed / implemented so that we try to avoid over-writing type data where possible. This will allow backend JIT
to generate more optimized code.
Current Status
--------------
We have a parser implementation that can convert Ravi source to an abstract syntax tree (AST).
Static type checking is not yet implemented.
We have a parser implementation that can convert Ravi source to an abstract syntax tree (AST). Static type checking is being worked on. Progress is very slow but things are moving every now and then when I get time.
For examples of how to call the parser please see `ravi-tests/ravi_test_ast.lua <https://github.com/dibyendumajumdar/ravi/blob/master/ravi-tests/ravi_test_ast.lua>`_.
For examples of how to call the parser please see `ravi-tests/ravi_test_ast.lua <https://github.com/dibyendumajumdar/ravi/blob/master/ravi-tests/ravi_test_ast.lua>`_ and `ravi-tests\ravi_test_ast2.lua <https://github.com/dibyendumajumdar/ravi/blob/master/ravi-tests/ravi_test_ast2.lua>`_. If you run these scripts the parse AST will be dumped to stdout.
failures+=do_asmvm_test("return function (a) for i=1,5 do a=i; end return a end",0,NULL,1,results);// OP_LOADK, OP_MOVE, OP_RETURN, OP_RAVI_FOPREP_I1, OP_RAVI_FORLOOP_I1
results[0].u.i=3;
failures+=do_asmvm_test("return function (a) for i=1,4,2 do a=i; end return a end",0,NULL,1,results);// OP_LOADK, OP_MOVE, OP_RETURN, OP_RAVI_FOPREP_IP, OP_RAVI_FORLOOP_IP
failures+=do_asmvm_test("return function (a) for i=1,5 do a=i; end return a end",0,NULL,1,results);// OP_LOADK, OP_MOVE, OP_RETURN, OP_RAVI_FOPREP_I1, OP_RAVI_FORLOOP_I1
results[0].u.i=3;
failures+=do_asmvm_test("return function (a) for i=1,4,2 do a=i; end return a end",0,NULL,1,results);// OP_LOADK, OP_MOVE, OP_RETURN, OP_RAVI_FOPREP_IP, OP_RAVI_FORLOOP_IP
returnfailures;
}
#endif
#ifndef RAVI_USE_ASMVM
staticinttest_vm()
{
intfailures=0;
//
failures+=test_luacomp1("function x() local t : table = {}; t.name = 'd'; end");
failures+=test_luacompexec1("function test(); local x: integer = 1; return function (j) x = j; return x; end; end; fn = test(); return fn('55')",55);
failures+=test_luacompexec1("ravi.auto(true); function arrayaccess (); local x: integer[] = {5}; return x[1]; end; assert(ravi.compile(arrayaccess)); return arrayaccess()",5);
* Create a dispatch table with pointers to the assembler routines; the dispatch table is stored in the Lua global_State structure
but this could change in future
* Each assembler routine will (after completing its action) fetch the next bytecode instruction and jump to the next
assembler routine using the dispatch table
* The assembler routines are C functions - they are part of one whole program. Hence they make assumptions about
assembler routine using the dispatch table (equivalent to computed goto)
* The assembler routines are **not** C functions - they are part of one whole program. Hence they make assumptions about
register usage which will be documented below. The VM as a whole will have a fixed set of register allocations so that most
important information is held in registers.
@ -33,7 +33,7 @@ Using an assembler like yasm has the problem of computing offsets of C structure
Issues with dynasm
------------------
On Windows 64-bit the generated code requires UNWIND information however the mechanism for this was in LuaJIT specific files (buildvm_peobj) and not fully reusable. I have modified this to decouple from LuaJIT. This took some effort because LuaJIT's code
On Windows 64-bit the generated code requires UNWIND information however the mechanism for this was in LuaJIT specific files (buildvm_peobj) and not fully reusable. I have modified this to de-couple from LuaJIT. This took some effort because LuaJIT's code
has numerous magic numbers with no explanation of what the code is doing. Not very helpful for anyone trying to work out what
the code is doing unless you already know what needs doing.
@ -287,12 +287,11 @@ And the epilogue::
As you can see the unwind information basically tells Windows what the epilogue is supposed to be, and where to find the saved
values of the registers.
Building Ravi With New VM
-------------------------
Building Ravi With New ASM VM
-----------------------------
This is only for the brave who want to hack with the code.
To enable the new VM first build and install VMBuilder as described above.
Then build Ravi using the cmake flags ``-DSTATIC_BUILD=ON`` and ``-DASM_VM=ON`` enabled. Don't enable JIT.
Right now the ASM VM is exercised via the ``test_vm`` sub project. The ASM VM is only invoked in special cases, i.e. a function has small number of instructions and only contains supported instructions, and additionally as OP_CALL is not yet implemented, you can only call the new VM via the Lua C api (see test_asmvm() in test_vm.c).
Right now the ASM VM is exercised via the ``test_asmvm`` sub project. The ASM VM is only invoked in special cases, i.e. a function has small number of instructions and only contains supported instructions, and additionally as OP_CALL is not yet implemented, you can only call the new VM via the Lua C api (see `test_asmvm() in test_asmvm.c <https://github.com/dibyendumajumdar/ravi/blob/master/tests/test_asmvm.c>`_).
Dibyendu Majumdar
5 years ago