# gi **Repository Path**: mirrors_didip/gi ## Basic Information - **Project Name**: gi - **Description**: gijit is a just-in-time trace-compiled golang REPL. Standing on the shoulders of giants (GopherJS and LuaJIT). - **Primary Language**: Unknown - **License**: MIT - **Default Branch**: master - **Homepage**: None - **GVP Project**: No ## Statistics - **Stars**: 0 - **Forks**: 0 - **Created**: 2020-08-08 - **Last Updated**: 2025-09-20 ## Categories & Tags **Categories**: Uncategorized **Tags**: None ## README gijit: a just-in-time trace-compiled golang interpreter ======= `gijit` is a scalable Go REPL for doing interactive coding and data analysis. It is backed by LuaJIT, a Just-in-Time trace compiler. The REPL binary is called simply `gi`, for "go interpreter". NB: `gi` was last built with go1.10.4. It hasn't be updated for go11/12, so please use go1.10.4 with it for now. # the dream Go, if it only had a decent REPL, could be a great language for exploratory data analysis. # the rationale Go has big advantages over Python, R, and Matlab. It has good type checking, reasonable compiled performance, and excellent multicore support. # the aim We want to provide one excellent integrated REPL for Go. Exploratory data analysis should not be hampered by weak type-checking or hard-to-refactor code, and performance should not suffer just because you require interaction with your data. # status and limitations Generally quite usable for small scripting tasks. See the open issues on github for specific open bugs. ~~~ [x] Done: all the basic types and control structures. [x] Done: pointers. [x] Done: structs. [x] Done: interfaces and type assertions. [x] Done: defer and recover. [x] Done: func creates closures. [x] Done: import of binary and source packages. Limitations: _ Paritally done: goroutines, select, channels. What is missing is that Timers and timeouts are not done. Also goroutines are implemented with Lua's coroutines, so they won't interact with the goroutines from a binary Go package. A little elaboration on that last point. I initially implemented goroutines using reflect, but LuaJIT isn't particularly happy about being called from a non-main thread. Most exploratory data analysis works fine, and I figure by the time your code needs goroutines, you probably want to shift to using fully compiled Go code in a library anyway. So there are no immediate plans to put more work into the timers and goroutines. Simply use a compiled library, and call it from `gijit` if you need them. ~~~ quick install ------------- See https://github.com/gijit/gi/releases for binary releases that can be run directly without building from source. To build from source: ~~~ # use go1.10.x We don't support go1.11.y at the moment. $ go get -d github.com/gijit/gi/cmd/gi $ cd $GOPATH/src/github.com/gijit/gi $ (On posix/mac/linux: run `./posix.sh` to build libluajit.a) $ (On windows: run `windows.bat` to build libluajit.a; see https://github.com/gijit/gi/issues/18 for notes on installing both mingw64 and make, which are pre-requisites.) $ make install $ gi ~~~ See also https://github.com/gijit/gi/issues/18 for windows install-from-source help. For various releases there are pre-compiled binaries (https://github.com/gijit/gi/releases). They have batteries included; the prelude is compiled. They run standalone. There is no need to install the `gijit` source with `git`. Importing some source packages works. Importing shadowed (binary Go) packages works. These include the commonly used `fmt`, `os`, `math/rand`, and others listed here https://github.com/gijit/gi/tree/master/pkg/compiler/shadow # Q: Can I embed `gijit` in my app? A: Yes! That's how `gijit` is designed. In fact the `gi` command is just a very thin wrapper around the `pkg/compiler` library. See https://github.com/gijit/gi/blob/master/cmd/gi/repl.go#L9 and https://github.com/gijit/gi/blob/master/pkg/compiler/repl_luajit.go#L63 # LuaJIT did what? LuaJIT is an amazing backend. In our quick and dirty 500x500 random matrix multiplication benchmark, LuaJIT *beat even statically compiled go* code by a factor of 3x. Go's time was 360 msec. LuaJIT's time was 135 msec. Julia uses an optimized BLAS library for this task and beats both Go and LuaJIT by multiplying in 6 msec, but is too immature and too large to be a viable target. Bonus: LuaJIT has Torch7 for machine learning. And, Torch7 has GPU support. [1][2] [1] https://github.com/torch/cutorch [2] https://github.com/torch/torch7/wiki/Cheatsheet#gpu-support Will golang (Go) run on GPUs? It might be possible, someday. # installation from source Release v1.3.0 and later needs go1.10 or later (a critical Windows timezone workaround was provided in go1.10). Works on Mac OSX and Linux and Windows. To build on Windows, you'll need to install mingw64 port of gcc first, if its not already installed, since that is what CGO on windows requires. See the notes in https://github.com/gijit/gi/issues/18 [Update: I put a v1.3.5 binaries here https://github.com/gijit/gi/releases/tag/v1.3.5 ; compiled on Windows10.] ~~~ $ go get -d github.com/gijit/gi/cmd/gi $ cd $GOPATH/src/github.com/gijit/gi $ (On posix/mac/linux: run `./posix.sh` to build libluajit.a) $ (On windows: run `windows.bat` to build libluajit.a ; note pre-reqs below.) $ make install $ gi ~~~ To build from source on windows, see https://github.com/gijit/gi/issues/18 for windows install help. Install both mingw64 and make before building gijit. These are prerequisites for building from source. The released windows binaries will run without them. most recent status ------------------ 2018 Dec 25 update ------------------------------ The REPL is now a better calculator and displayer of answers. Simple math like `1 + 2` and expressions like `math.Exp(4)` will print their answers automatically. Also we slimmed down the repo of history, here at v2.0.2; the original repo will be renamed `gi-large` and we will take over as `gi`. 2018 Dec 14 update ------------------------------ Releaes v1.9.5 brings much improved interaction with native struct values (from imports of compiled Go packages such as those in the standard libary). Native structs can be used as fields in new struct created on the fly in the interpreter. Native methods can be invoked on those native struct values. 2018 March 29 update ------------------------------ Release v1.9.2 includes the `func __callLua(s string) (interface{}, error)` function to call Lua code directly from `gijit` Go code. Release v1.8.0 has much more robust and tested source based imports of packages. Binary packages can be compiled into the `gi` REPL, and then `import`-ed at runtime. Source packages can be read and compiled at runtime. 2018 March 26 update ---------------------------- Release v1.6.5 has reasonable source-based imports of other packages. Tests 1000, 1001, and 1003 are green. All green tests with 1002 commented out. 2018 March 25 update -------------------- Release v1.6.0 brings the initial implementation of import functionality for source packages. 2018 March 6 update ------------------- The release v1.4.x series improves the shadow import system to provide constructors and copy constructors for native Go structs. We pass additional `recover` tests from the Golang compiler test suite. Release v1.3.8 re-enables calls into `fmt` and the other builtin native Go shadowed packages. They now work when called from goroutines. 2018 March 3 update ------------------- Release v1.3.5 fixes an issue with the build on Windows. Release v1.3.4 builds LuaJIT with `XCFLAGS+= -DLUAJIT_ENABLE_GC64` in order to allow more than 4GB of memory use on OSX and Linux. Windows 10 hates that setting, and `luajit.exe` itself crashes with that flag enabled. So it is not enabled on Windows. Release v1.3.3 has internal updates that make progress towards calling native Go functions from coroutines. This is needed to restore binary imports of native Go packages. 2018 March 2 update ------------------- Release v1.3.2 vendors the gonum libraries used, as users were seeing some version skew. There are binaries available. See https://github.com/gijit/gi/releases/tag/v1.3.2 2018 March 1 update ------------------- A major milestone, the v1.2.x series and the latest release bring fully interactive goroutines to the REPL. The REPL can perform receives on unbuffered channels, and interact with background goroutines. A background Lua coroutine runs a scheduler that coordinates. Full blocking at the REPL, on a select or receive that cannot be finished at this time, is not yet implemented. Importantly, native Go imports are turned off while we work on polishing the goroutine system. Hence `import "fmt"` won't work. As of v1.2.4, close() on channels is available. As of v1.2.5, <-ch, for a channel ch, works by itself at the repl and in functions. 2018 Feb 26 update ------------------ With release v1.1.0 we focused on establishing an all-Lua goroutine functionality. Rather than tackle both all-Lua and hybrid Lua/native goroutines all at once, we focused on getting goroutines working completely in Lua land. Release v1.1.0 acheives that goal, making full `go`, `select`, channel send, and channel receive available. v1.1.1 provides a little polish. 2018 Feb 24 update ------------------ Release v1.0.15 brings the ability to make native Go channels via reflection for basic types, and then use those channels in send, receive, and `select` operations to communicate between interpreted and pre-compiled Go code. 2018 Feb 23 update ------------------ Release v1.0.7 fixes a bunch of subtle corner cases in the type system implementation, making it much more robust. 2018 Feb 21 update ------------------ Release v1.0.2 brings a calculator mode, for multiple lines of direct computation. Enter calculator mode with `==` alone on a line, and exit with `:` alone. Release v1.0.0 marks these milestones: * the full program _bench/mul.go (mat_test 501) builds and runs; with no imported packages used. * `gijit` has much improved complex number support. * `gijit` runs on Windows now, as well as OSX and Linux. Limitations not yet addressed: * import of source packages isn't working yet. * imports of binary Go packages don't work, and in general need some help. * chan/select/goroutines are not implemented; init() functions and full ahead-of-time compile of packages are not done. # overview In the form of Q & A: ~~~ On Friday, February 9, 2018 at 12:48:17 PM UTC+7, Christopher Sebastian wrote on golang-nuts: gijit looks really interesting. Do you have a high-level description/diagram anywhere that gives an overview of how the system works? [0] I took a brief look at the git repo, and it seems like you're translating Go into Javascript, and then to Lua, and then using LuaJIT. [1] Is that right? [2] How do you manage the complexity of these seemingly-unwieldy pieces and translation layers? It seems like you have your own 'parser' implementation (maybe copied from the Go src tree? [3]). How will you keep your parser (as well as the rest of the system) up-to-date as Go changes? [4] Is it a very manual process? [5] Thanks for working on this important project! ~Christopher Sebastian ~~~ My reply, condensed: ~~~ [0] Overview/how does the system work? How is `gijit` differ from GopherJS? a) GopherJS does: Go source -> Javascript source. It translates a whole program at a time. while b) `gijit` does: i) Go source -> Lua source. But in small chunks; for example, line-by-line. Instead of whole program, each sequentially entered expression/statement/declaration, (or using :source, a larger fragment if required for mutually recursive definitions) is translated and evaluated. These can be multiple lines, but as soon as the gc syntax front-end tells us we have a complete expression/statement/declaration, then we immediately translate and evaluate it. Yes, the semantics will be subtly different. If you've used R/python/matlab to develop your algorithms before translating them to a compiled language (a highly effective practice that I recommend, and one that motivated gijit's development), you'll know that this is of little concern. ii) Then run that Lua source on LuaJIT. So while the front end of gijit is derived from GopherJS, there is no javascript source generated. There's still quite a bit of javascript remnants in the source, however, as many corner cases haven't been tackled. So that might have misled a cursory inspection. I have adopted the practice, in the GopherJS derived code, of leaving in the javascript generation until I hit the problem with a new test. This lets me visually see, and experience when run, where I haven't handled a case yet. There are many corner cases not yet handled. [2]. Very carefully, with full Test-driven development. :) Seriously though, TDD is fantastic (if not essential) for compiler work. I would add, I'm working with extremely mature building blocks. So if there's a bug, it is almost surely in my code, not the libraries I'm using. The three main libraries are already themselves the highly polished result of very high-quality (German and Swiss!) engineering. a) Mike Pall's LuaJIT is the work of 12 years by a brilliant designer and engineer. Cloud Flare is now sponsoring King's College London to continue LuaJIT's development. b) Richard Musiol's GopherJS is the work of some 5 years. It passes most of the Go standard library tests. c) I don't know how long it took Robert Griesemer and the Go team to write the Go front-end parser and type checker, but there is a ton of time and effort and testing there. And a significant 4th: d) Luar (for Go <-> Lua binary object exchange, so that binary imports of Go libraries work) is a mature library. A great deal of work was done on it by Steve Donovan and contributors. This is all work that doesn't need to be reinvented, and can be leveraged. The slight changes to the above (vendored and modified) libraries in order to work at the REPL are light relaxations of a small set of top-level checks. [3]. I re-use the front-end syntax/parser from the gc compiler only in order to quickly determine if we have a complete expression at the repl, or if we need to ask for another line of input. Then I throw away that parse and use the standard library's go/ast, go/parser, go/types to do the full parse and type-checking. This is necessary because go/parser and go/types are what GopherJS is built around. [4]. Go 1 is very mature, and changes very slowly, if at all, anymore, between releases. [5]. Yes, it is manual. I note the changes to the parser and typechecker are important but quite small, and would rebase easily. Best wishes, Jason ~~~ I would add credit to Lua's originators and contributors from around the world. Roberto Ierusalimschy et al's design and evolution of Lua over the last 25 years make it a (perhaps surprisingly) great tool for this purpose. It certainly surprised me. Lua's primary goal of acting as an embedded scripting language for many disparate host languages with different inheritance/prototype/overloading semantics has sculpted it into a power tool that fits the job like a glove. It is very good at language implementation. status update history ------ See the top of this README for the latest update. 2018 Feb 20 update ------------------ v0.9.19 has great progress getting full program to run. A crude matrix multiplication whole program (mat_test 501) now executes correctly, albeit slowly and untuned. ~~~ `runMultiply(500,9,9) -> 301609258250` from `_bench/mul.go`/`_bench/mul.lua` (a) took 342.456726ms on Go (b) took 5-7 seconds on gijit (about 20x slower) ~~~ See https://github.com/stevedonovan/luar issue #23 to follow some tuning -- we omit Luar's type() override with `ProxyType()` to retain LuaJIT performance on the matrix 501 example. 2018 Feb 18 update ------------------ As of `gigit` v0.9.15, Complex numbers fully supported, with all of the `cmplx` library functions available. The underlying complex.lua library works at LuaJIT natively. There's still some cruft to clear out in the compiler from all the javascript support that is no longer needed. So complex number interactive use is still a little awkward, but this can be quickly improved. The revamped type system supports the matrix multiplication benchmark. Building on Windows, alongside OSX and Linux, now works. 2018 Feb 14 update ------------------ `gijit` was successfully built on Windows10, and this same approach will probably work on earlier Windows versions. See https://github.com/gijit/gi/issues/18 for my notes on getting mingw64 installed, and see the `windows` branch of this repo. OpenBLAS and sci-lua are vendored for matrix operations. The docs are http://scilua.org/. Also from sci-lua, some benchmarks showing LuaJIT can be as fast as C. LuaJIT is typically faster than Julia 0.4.1. 2018 Feb 12 update ------------------ Release v0.9.12 works under both go1.9.4 and go1.9.3, but see the new/revised installation instructions. Actually we recommend avoiding go1.9.4. It is pretty broken/borked. Use go1.9.3 and wait for go1.9.5 before upgrading. The installation instructions are now slightly different, so that all actual building is done under make, where we can set the required environment variables. ~~~ $ go get -d github.com/gijit/gi/cmd/gi $ cd $GOPATH/src/github.com/gijit/gi && make install ~~~ 2018 Feb 09 update ------------------ v0.9.11 allows building under go1.9.4. v0.9.9 restores building under go1.9.3 after attempts to get go1.9.4 to work messed up the build. v0.9.7 attempted to build under the newly release go1.9.4, but failed to do so and has been replaced. 2018 Feb 08 update ------------------ v0.9.6 fixes #20. The type checker now allows cleanly re-defining struct types at the REPL. 2018 Feb 08 update ------------------ `gijit` v0.9.4 feature summary: * interactively code in Go. Just-in-time for Valentine's Day, a Go REPL that is not based on re-compiling everything after every line. * the ability to import binary Go packages. Call into native Go code from the REPL. * use Go as a calculator. Just start the line with `=`. Use `==` to enter calculator mode where multiple direct math expressions can be evaluated. Return to go mode with `:`. * structs, interfaces, pointers, defer are all available. * current limitation: no `go`/`select`/`chan` implementation. * portable. Doesn't depend on Go's plugin system, or on recompiling everything every time. We run on OSX and Linux and Windows. # demo ~~~ $ gi -q gi> import "math" elapsed: '15.155µs' gi> import "fmt" elapsed: '16.507µs' gi> fmt.Printf("hello expressions! %v\n", math.Exp(-0.5) * 3.3 - 2.2) hello expressions! -0.1984488229483099 elapsed: '141.721µs' gi> = 4.0 / 1.3 /* or, with '=', no fmt.Printf needed. */ 3.0769230769231 elapsed: '112.18µs' gi> ~~~ 2018 Feb 07 update ------- In version v0.9.3, defers pass additional tests (that found issues that were fixed), and the repl in raw mode can import Go binary libraries with the call __go_import(path). Raw mode factilitates system debugging and tests. Raw mode isn't needed by end users, unless one is developing gijit. Raw mode allows direct LuaJIT commands to be entered. It is accessed with the ':r' enter-raw-mode command; and ':' returns one to Go mode. ~~~ gi> :r Raw LuaJIT language mode. elapsed: '20.042µs' raw luajit gi> __go_import "fmt" elapsed: '2.048215ms' raw luajit gi> fmt.Printf("hello Go!") hello Go! elapsed: '124.64µs' raw luajit gi> : Go language mode. gi> ~~~ In version v0.9.2, the REPL prints expressions that produce multi-lines of Lua better. We only wrap the final line with a print. This handles expressions that generate anonymous pointer types gracefully. In version v0.9.1, pointers inside structs work. In version v0.9.0, defer handling of named return values received some important correctness fixes. 2018 Feb 06 update ------- In version v0.8.9, the repl received some refactoring to make it easier to test. In version v0.8.8 (quiet) and v0.8.7 (debug prints live), pointer support is much improved. Cloning is restored, and test 028 is green. Basic assignment to pointers, and assignment through pointers, work now. For example, the sequence ~~~ a:= 1 b := &a c := *b *b = 3 ~~~ works as expected (ptr_test.go/Test099). As usual, after the four statements, `c` ends as 1, and `a` ends as 3. Still TODO is supporting pointer members within structs, but that should follow shortly. 2018 Feb 05 update ------- Excellent progress. In version v0.8.6, the majority of the type system from GopherJS was ported over to LuaJIT in the struct.lua file. Type assertions on interfaces are working, cf face_test.go and tests 100, 102, 202. The cloning of structs (test 028 in repl_test.go) is temporarily broken while their infrastructure is being refactored to use the new system. 2018 Feb 02 update ------- In version v0.8.4, we began integrating the GopherJS type system, in order to properly support interfaces. It turns out that, because the Go reflect system is incomplete, GopherJS and now `gijit` both need a complete, stand-alone type system implementation. So adding interface support is a much bigger job than I orignally thought. Nonetheless, with GopherJS lighting the way, we are making progress with the port. Test 100 in face_test.go was red for a long time while we began integrating our updated Lua-metable based object model for types with the GopherJS object model for types. The integration isn't finished yet, as many of the properties that live in the leaf table of a new struct need to be moved up in the properties table, but we've got the basic machinery going and now its simply a matter of fine tuning. Test 100 finally went green, so we felt it was time to mark progress with a release. However other tests remain red, so v0.8.4 will be an internal-only release. In version v0.8.3, an internal release, we restrict `gijit` programs to a subset of legal `go` programs, by imposing some mild restrictions on the names of variables. Minor restriction number one: variable names cannot start with '__' two underscores. Minor restriction number two: in `gijit`, you can't have a variable named `int`, or `float64`, for example. These are names of two of the pre-declared numeric types in Go. So, while ~~~ func main() { var int int _ = int } ~~~ is a legal `Go` program, `gijit` will reject it. `gijit` won't let you re-use any of the basic, pre-declared type names as variables. `uint`, `int`, `int8`, `int16`, `int32`, `int64`, `uint8`, ... etc. are all off-limits. Although in Go this is technically allowed, it can be highly confusing. It is poor practice. The technical reason for this restriction in `gijit` is that otherwise the Go type checker can be corrupted by simple syntax errors involving pre-declared identifiers. That's not an issue for a full-recompile from the scratch each time, but for a continuously online typechecker, it is a problem. To stay online and functional after a syntax error like `var int w`, (where w is unknown, provoking a syntax error yet shadowing the pre-declared type), we disallow such variable names. 2018 Jan 31 update ------- As of release v0.8.2, we support pointers, taking and de-referencing. As of release v0.8.1, `gijit` works as a calculator. It will evaluate expressions at the command line. In order to continue to detect syntax errors in Go code, we adopt the same convention as Lua: the user must prepend an '=' equals sign to the expression. For example: ~~~ gi> = 24/3 8LL elapsed: '117.858µs' gi> ~~~ Aside: the 'LL' suffix indicates a 64-bit, signed integer. Borrowed by LuaJIT from C/C++, it stands for "long long". There's also 'ULL' for uint64. Notice, however, that without the '=', a syntax error is properly detected: ~~~ gi> 24/3 oops: 'problem detected during Go static type checking: '1:1: expected declaration, found 'INT' 24'' on input '24/3' elapsed: '24.166µs' gi> ~~~ Keeping Go's type checking intact at the REPL preserves one of the most important advantages of Go. We catch typos early, at compile time. Multiple expressions at once also work, and each is printed on its own line. ~~~ gi> = 2+4, "gophers" + " " + "rock", 7-3 6LL `gophers rock` 4LL elapsed: '155.55µs' gi> ~~~ 2018 Jan 29 update ------- What is left to do: pointers, interfaces, go-routines, and channels. Release v0.8.0 has command line history available with the up-arrow/down-arrow. This is a nice supplement to the `:h` history listing and replay facility. Like traditional shell editing, we now have ctrl-a (move to beginning of line); ctrl-e (move to end of line); ctrl-k (cut); and ctrl-y (paste) at the REPL. A quick demo of the `:h` history functionality, which is also new (and distinct from the up-arrow/liner functionality). Line history is stored in `$HOME/.gijit.hist`, and is preserved across `gi` restarts. It can be edited by removing sets of lines using the `:rm a-b` command. The `:n` command, where `n` is a number, replays history line `n`. With a `-` dash, a range of commands to be replayed is specified. `:10-` replays from 10 to the end of history, while `:-10` replays everything from the first line in the history, up to and including line 10. `:-` replays everything in the history, because the range endpoints have the intuitive defaults. Commands executed in the current session appear after the `----- current session: -----` delineator. ~~~ $ gi -q ## start quietly, omit the banner for the demo. gi> :h history: 001: greet := "hello gophers!" ----- current session: ----- gi> ## notice that stuff from past sessions is above the `current session:` line. gi> :reset history cleared. gi> :h history: empty ----- current session: ----- gi> a := 1 elapsed: '17.614µs' gi> :h history: ----- current session: ----- 001: a := 1 gi> b := a * 2 elapsed: '50.054µs' gi> :h history: ----- current session: ----- 001: a := 1 002: b := a * 2 gi> :1- replay history 001 - 002: a := 1 b := a * 2 elapsed: '41.932µs' gi> :h history: ----- current session: ----- 001: a := 1 002: b := a * 2 003: a := 1 004: b := a * 2 gi> :3-4 replay history 003 - 004: a := 1 b := a * 2 elapsed: '76.605µs' gi> :-2 replay history 001 - 002: a := 1 b := a * 2 elapsed: '91.664µs' gi> :- ## replay everything in our history replay history 001 - 008: a := 1 b := a * 2 a := 1 b := a * 2 a := 1 b := a * 2 a := 1 b := a * 2 elapsed: '37.896µs' gi> :h history: ----- current session: ----- 001: a := 1 002: b := a * 2 003: a := 1 004: b := a * 2 005: a := 1 006: b := a * 2 007: a := 1 008: b := a * 2 009: a := 1 010: b := a * 2 011: a := 1 012: b := a * 2 013: a := 1 014: b := a * 2 015: a := 1 016: b := a * 2 gi> :rm 3- ## if a range leaves off an endpoint, it defaults to the beginning/end. remove history 003 - 016. gi> :h history: ----- current session: ----- 001: a := 1 002: b := a * 2 gi> :rm - ## same as :reset or :clear remove history 001 - 002. gi> :h history: empty ----- current session: ----- gi> ~~~ Release v0.7.9 has slice `copy` and `append` working. `copy` allows source and destination slices to overlap, adjusting the copy direction automatically. In release v0.7.7, make applied to slices works. For example, `make([]int, 4)` will create a new length four array filled with the zero value for `int`, and then take a slice of that array. In release v0.7.6, taking a slice of an array works. 2018 Jan 28 update ------- # binary imports In release v0.7.5, we added the ability for the `gi` REPL to import "regexp" and "os". More generally, we added the general ability to quickly add existing native Go packages to be imported. The process of making an existing native Go package available is called shadowing. The shadowing process is fairly straightforward. A new utility, `gen-gijit-shadow-import` is run, passing as an argument the package to be shadowed. The utility produces a new directory and file under `pkg/compiler/shadow`. Then a few lines must be added to `pkg/compiler/import.go` so that the package's exported functions will be available to the REPL at runtime, after import. The "regexp" and "os" shadow packages provide examples of how to do this. As an example of shadwoing the io/ioutil package, we ran: ~~~ $ gen-gijit-shadow-import io/ioutil writing to odir '/Users/jaten/go/src/github.com/gijit/gi/pkg/compiler/shadow/io/ioutil' $ ~~~ While shadowing (sandboxing) does not allow arbitrary imports to be called from inside the `gi` without prior preparation, this is often a useful and desirable security feature. Moreover we are able to provide these imports without using Go's linux-only DLL loading system, so we remain portable/more cross-platform compatible. Interfaces are not yet implemented, and so are not yet imported. Update: interfaces are represented in the imports, but are not well tested. Please file issues as you find them. 2018 Jan 27 update ------- In release v0.7.3, arrays are passed to Go native functions, and array copy by value is implemented. Having arrays work well, now we are well-positioned to get slices pointing to arrays working. That will be the next focus. 2018 Jan 25 update ------- In release v0.7.2, for range over strings produces utf8 runes. Release v0.7.1 makes all integers work as map keys. Go integers (int, int64, uint, etc) are represented with cdata in LuaJIT, since otherwise all numbers default to float64/doubles. This was a problem because cdata are boxed, so equality comparison on ints was comparing their addresses. Now we translate all Go integer map keys to strings in Lua, which makes key lookup for a Go integer key work as expected. 2018 Jan 23 update ------ Release v0.7.0 brings local variables inside funtions, by default. The global package name space is not changed by variables declared inside functions. Function code will be much faster. Global level/package level declarations are kept global, and are not prefixed with the 'local' keyword in the Lua translation. Release v0.6.8 brought the ability to update a method definition at the REPL, replacing an old method definition with a new one. Functions and variables could already be re-defined. Now all data types can be updated as you work at the `gi` REPL. 2018 Jan 21 update ------ Release v0.6.7 has working `fmt.Printf`. ~~~ gi> import "fmt" import "fmt" gi> fmt.Printf("Hello World from gi!") Hello World from gi! gi> ~~~ 2018 Jan 20 update ------ Release v0.6.2 has a working `fmt.Sprintf`. This needed handling vararg calls into compiled Go code, which is working now. A quick demo: ~~~ gi> import "fmt" import "fmt" gi> fmt.Sprintf("hi gi") // value not printed at the REPL (at present), so add println: gi> println(fmt.Sprintf("hi gi")) hi gi gi> println(fmt.Sprintf("hi gi %v", 1)) hi gi 1 gi> println(fmt.Sprintf("hi gi %v", "hello")) hi gi hello gi> println(fmt.Sprintf("hi gi %v %v", "hello", 1)) hi gi hello 1 gi> ~~~ 2018 Jan 18 update ------ Release v0.6.0 was aimed at putting the infrastructure in place to support package imports. Specifically, we aimed at getting the first `import "fmt"` and the first use of `fmt.Sprintf` from the REPL. The varargs and int64 handling required in `fmt.Sprintf` made this extra tricky. And so it's not quite done. Nonetheless, I'm releasing v0.6.0 because there were big refactorings that provide significant new internal functionality, and contributors will want to leverage these. The REPL will now accept `import "fmt"` and will wire in the `fmt.Sprintf` for you. It's hardwired for now. Auto loading of functions from already compiled packages will come later. Standing on the shoulders of another giant, the `luar` package, lets us call pre-compiled Go packages from Lua through reflection. A quick demo: ~~~ gi> import "fmt" import "fmt" gi> a:=fmt.Sprintf("hello gi!") a:=fmt.Sprintf("hello gi!") gi> a a hello gi! gi> ~~~ `luar` (https://github.com/stevedonovan/luar) is a mature library that gives us the basis for imports. Since `LuaJIT` provides reasonable int64 handling, we've extended the `luar` functionality to gracefully convey `int64` and `uint64` values. There's more extension to do for `int32`, `int16`, `int8`, `uint32`, etc but this should be straightforward extension of the new functionality. While the vararg handling to make Sprintf actually useful beyond just the format string is missing, this should be done shortly. The red 051 and 052 tests in imp_test.go track the last bits of functionality needed to make Sprintf work. API functions luajit_push_cdata_uint64(), luajit_push_cdata_int64(), and luajit_ctypeid() were added to the luajit API to support passing int64/uint64 values from Go to Lua and back to Go without the loss of data formerly associated with casting to double (float64) and back. 2018 Jan 13 update ------ We've moved within github to make admin easier. We are now at https://github.com/gijit/gi len(a) now displays at the REPL. Fixes #22. This was a minor REPL nit, but well worth addressing. We have a new contributor! Welcome to Malhar Vora. As of v0.5.6, integer modulo and divide now check for divide by zero, and panic like Go when found. The math.lua library is supplemented with math.isnan(), math.finite(), and __truncateToInt(). There's a new section of this readme, https://github.com/gijit/gi#translation-hints that gives specific hints for porting the javascript ternary operator and other constructs. jea: I'll be offline for a day or two. 2018 Jan 12 update ------ Switch statements now work. They work at the top level and inside a function. 2018 Jan 11 update ------ With release v0.5.0 the inital test of the `defer`/ `panic`/`recover` mechanism passes. Woot! There's more to do here, but the design is solid so filling in should be quick. For a stack-unwinding `panic`, we use what Lua offers, the `error` mechanic -- to throw -- combined with the `xpcall` mechanic to catch. The only limitation I found here is on recursive `xpcall`: if you are in a panic stack unwind, and then in a defer function, and your code causes a second error that is *not* a deliberate panic, then that error will be caught but recover won't return that error value to the caller of recover. This is due to a wierd corner case in the implementation of LuaJIT where it does not like recursive `xpcall` invocations, and reports "error in error handling". I asked on the Lua and LuaJIT mailing lists about this, and posted on Stack Overflow. So far no replies. https://stackoverflow.com/questions/48202338/on-latest-luajit-2-1-0-beta3-is-recursive-xpcall-possible. [Update: Mike Pall replied, (yay!) -- but (sadly), it doesn't sound like he'll be fixing this himself. > Error handlers shouldn't throw errors themselves. The semantics would be too messy. -- Mike Pall, on https://github.com/LuaJIT/LuaJIT/issues/383 ] It's a fairly minor limitation, and easy to work around once you notice the bug: just call `panic` directly rather than causing the error. Or don't cause the error at all(!) Or simply use a different side-band to pass around the value. Lots of work arounds. 2018 Jan 10 update ------ With release v0.4.1 we have much improved map support. In `gi`, maps now work as in Go. Multiple-valued queries properly return the zero-value for the value-type when the key is missing, and return the 2nd value correctly. Nil keys and values are handled properly. `delete` on a map works as expected, and maintains the `len` property. 2018 Jan 9 update ------ Functions and methods can now be re-defined at the REPL. The type checker was relaxed to allow this. We changed over from one LuaJIT C binding to anther. The new binding is the same one that LuaR uses, so this enables LuaR exploration. 2018 Jan 8 update ------ Today we landed multiline support. We evalutate Go expressions as they are entered, and these can now span multiple lines. This lifts the prior limitation that meant that functions and types needed to be defined all on one line. This was fun to put together. I used the actual gc front end that parses regular go code. Since gc is written in Go, why not leverage it! The advantage is that we know we are building on correct parsing of the whole language. Of course minor tweaks had to be made to allow statements and expressions at global scope. Happily, from our experience adding the same provisions to GopherJS, we knew these were relatively minor changes. See the updated demo transcript below in this readme for a multi-line function definition taking effect. 2018 Jan 7: latest update ------ Today we acheived passing (light) tests for method definition and invocation! Also a significant discovery for the object system: Steve Donovan's Luar provides object exchange both ways between Go -> Lua and Lua -> Go. That should influence our design of our Go source -> Lua source mapping. If we map in a way that matches what Luar does when it translates from Go binary -> Lua binary, then our objects will translate cleanly into binary Go calls made by reflection. Even more: Luar provides access to the full Go runtime and channels via reflection. Nice! We don't have to reinvent the wheel, and we get to use the high-performance multicore Go scheduler. earlier summary ----- Early stages, work in progress. Contribute! Currently incremental type checking is applied to all code. Slices are bounds-checked at runtime. Functions, closures and slices, as well as basic expressions compile and run. For-loops including for-range loops compile and run. If this is exciting to you, contribute! Existing TODOs/open issues and polite improvement suggestions can be found here https://github.com/gijit/gi/issues However, because we are bulding on the fantastic front end provided by (Gopherjs)[https://github.com/gopherjs/gopherjs], and the fantastic backend provided by (LuaJIT)[http://luajit.org/], progress has been quite rapid. #quick demo session ~~~ $ go get -d github.com/gijit/gi/cmd/gi $ cd $GOPATH/src/github.com/gijit/gi && make install $ $ ... wait for gi build to finish, it builds LuaJIT $ using C, so it takes ~ 20 seconds to install `gi`. $ $ gi # start me up (will be in $GOPATH/bin/gi now). ==================== gi: a go interpreter ==================== https://github.com/gijit/gi Copyright (c) 2018, Jason E. Aten. All rights reserved. License: MIT. See the LICENSE file at https://github.com/gijit/gi/blob/master/LICENSE ==================== [ gi is an interactive Golang environment, also known as a REPL or Read-Eval-Print-Loop ] [ type ctrl-d to exit ] [ type :help for help ] [ gi -h for flag help ] [ gi -q to start quietly ] ==================== built: '2018-01-08T23:46:07-0600' last-git-commit-hash: 'db302d2acb37d3c2ba2a0d376b6f233045928730' nearest-git-tag: 'v0.3.3' git-branch: 'master' go-version: 'go_version_go1.9_darwin/amd64' luajit-version: 'LuaJIT_2.1.0-beta3_--_Copyright_(C)_2005-2017_Mike_Pall._http://luajit.org/' ================== using this prelude directory: '/Users/jaten/go/src/github.com/gijit/gi/pkg/compiler' using these files as prelude: array.lua, map.lua, prelude.lua, slice.lua, struct.lua gi> gi> a := []string{"howdy", "gophers!"} gi> a // ^^ make data using Go's literals. inspect it by typing the variables name. slice of length 2 is _giSlice{[0]= howdy, [1]= gophers!, } gi> a[0] = "you rock" // data can be changed gi> a slice of length 2 is _giSlice{[0]= you rock, [1]= gophers!, } gi> // the Go type checker helps you quickly catch blunders, at compile time. gi> a[-1] = "compile-time-out-of-bounds-access" oops: 'problem detected during Go static type checking: 'where error? err = '1:3: invalid argument: index -1 (constant of type int) must not be negative''' on input 'a[-1] = "compile-time-out-of-bounds-access" ' gi> // runtime bounds checks are compiled in too: gi> a[100] = "runtime-out-of-bounds-access" error from Lua vm.Pcall(0,0,0): 'run time error'. supplied lua with: ' _gi_SetRangeCheck(a, 100, "runtime-out-of-bounds-access");' lua stack: String : ...rc/github.com/gijit/gi/pkg/compiler/prelude.lua:14: index out of range gi> // We can define functions: gi> func myFirstGiFunc(a []string) int { >>> for i := range a { >>> println("our input is a[",i,"] = ", a[i]) >>> }; >>> return 43 >>> } func myFirstGiFunc(a []string) int { for i := range a { println("our input is a[", i, "] = ", a[i]) } return 43 } gi> myFirstGiFunc(a) our input is a[ 0 ] = you rock our input is a[ 1 ] = gophers! gi> // ^^ and call them. They are tracing-JIT compiled on the LuaJIT vm. gi> // more compile time type checking, because it rocks: gi> b := []int{1,1} gi> myFirstGiFunc(b) oops: 'problem detected during Go static type checking: 'where error? err = '1:15: cannot use b (variable of type []int) as []string value in argument to myFirstGiFunc''' on input 'myFirstGiFunc(b) ' gi> ~~~ # Q: can I target non-JIT environments? Yes, LuaJIT can be compiled to run without codegen, using only its interpreter. This can be done at compile time by adjusting the LuaJIT Makefile, or at runtime (http://luajit.org/ext_jit.html). # Q: Can I use PUC Lua instead of LuaJIT? No. We target at 64-bit environment and heavily use the 64-bit integer support and `ffi` functionality of LuaJIT. # editor support An emacs mode `gijit.el` can be found in the `emacs/` subdirectory here https://github.com/gijit/gi/blob/master/emacs/gijit.el M-x `run-gijit` to start the interpreter. Pressing ctrl-n will step through any file that is in `gijit` mode. Other editors: please contribute! # Lua resources - development reference LuaJIT targets Lua 5.1 with some 5.2 extensions. a) main web site https://www.lua.org/ b) Programming in Lua by by Roberto Ierusalimschy, the chief architect of Lua. 1st edition. html format (Lua 5.0) https://www.lua.org/pil/contents.html 2nd edition. pdf format (Lua 5.1) https://doc.lagout.org/programmation/Lua/Programming%20in%20Lua%20Second%20Edition.pdf c) Lua 5.1 Reference Manual, by R. Ierusalimschy, L. H. de Figueiredo, W. Celes Lua.org, August 2006 Lua 5.1 https://www.lua.org/manual/5.1/ # translation hints - developer reference Specific javascript to Lua translation hints are below. Note that `gijit` doesn't generate javascript. However, the compiler package is derived from GopherJS, which did. The job of making the full transition from Javascript to Lua within the GopherJS-derived code base is half-done/still in progress. So when deciding how to change the output in the GopherJS derived code of a particular javascript idiom, we note the following are helpful hints. d1) the ternary operator ~~~ x ? y : z ~~~ should be translated as ~~~ ( x and {y} or {z} )[1] ~~~ d) the comma operator ~~~ x = (a, b, c) // return value of c after executing `a` and `b` ~~~ doesn't have a direct equivalent in Lua. Try to see if you can't define a new function in the prelude to take care of the same processing that a,b,c does. If `b` doesn't refer to `c` directly, and `a` doesn't refer to `b` directly, then ~~~ x = {a, b, c}[3] ~~~ comes close. Rarely does such a construct arise, since `a` and `b` are typically helper computations to compute `c`. However that boxing-unboxing construct is helpful in some tight corners, and may be your fastest alternative. Compared to defining and then calling a new closure, boxing and unboxing is 100x faster. e) join of strings: table.concat({"a", "b", "c"}, ",") f splitting strings, see string:split(sep) NB: Lua has a limit of a few thousand return values. f) s.substr(n): return the substring of s starting at n (0-indexed) Javascript s.substr(4) is a zero-indexed substring from 4 to end of string `s`. The Lua equivalent is string.sub(s, 5) # origin Author: Jason E. Aten License summary: MIT. Credits: some code here is dervied from the Go standard libraries, the Go gc compiler, and from Richard Musiol's excellent Gopherjs project. Those are licensed under the 3-clause BSD license found in the LICENSE file. The LuaJIT vm and compiler are statically linked using CGO, and their MIT license can be found in their sub-directories and online at http://luajit.org/ and https://github.com/LuaJIT/LuaJIT/blob/master/COPYRIGHT See the subdirectories of vendored and utilized libraries for their license details.