Skip to content

Commit

Permalink
Lua via sol2 (#2216)
Browse files Browse the repository at this point in the history
* Add Lua build to matrix

* Add Lua sources

* Add sol2 v3.3.0 sources

* Set up Lua infrastructure

* Run Lua scripts at loading stage

* Prevent scripts from messing with globals table

* Prevent scripts from messing with mod lists

* Fix non-Lua build

* Fix non-Lua build for real

* When checking all mods, check Lua mods only in Lua builds

* Stop world loading in non-Lua build if mod requires Lua

* Fix inverted check

* Astyle

* Add wip docs

* Add a bunch of tests that check Lua error and C++ exception interop

* CMake Lua build

* Lua in CMake build in Github Actions

* Add Lua to msvc build

* Add Lua to android build

* Fix duplicated bindings registration

* Clean up Lua state in exit_handler()

* Bindings for Tripoint

* Add bindings for item and itype_id

* Run Lua functions after game loads

* Add bindings for map and map item iteration

* Add hooks for iuse functions

* Run Lua functions on tinymap after mapgen

* Log lua output to dedicated in-memory log

* Lua console

* Reload some parts of Lua code on demand, reset game.current_mod after load

* Add bindings for getting/setting terrain

* More bindings

* Fix marking tables as readonly

* Hack: pairs() for read-only tables

* MOAR bindings

* Add Smart House Remotes mod

* Fix gcc build

* Fix OSX build

* Fix typos, fix LUA=0 build

* Fix get_all_mods.py grabbing Lua mods for non-Lua tests

* Fix Lua state deallocation after tests has been run

* Simplify addition of new hooks in catalua.cpp

* Implement save/load for global mod state

* Add example mod that saves and loads state

* Fix LUA=0 build again

* Show Lua API version in debug log and in game report

* In LUA=0 builds, prevent loading worlds that depend on Lua and show warning on mods

* Update docs

* Fix some clang lints

* Bind faction_id

* [WiP] Lua doc generator - constructors

* Lua doc generator - member vars and functions

* Fixup constructor

* Lua doc generator - base classes

* Lua doc generator - overloads

* Lua doc generator - prettier docs

* Lua doc generator - tuples

* Lua doc generator - don't show nil return

* Lua doc generator - more docced bindings

* Lua doc generator - fix ref/const in return type

* Lua doc generator - split up bindings to functions

* Encapsulate some code

* Redister std::string_view in docs

* LDC: separate macro for documenting vs values

* make_readonly_table: accept state_view

* LDG: enums and string_ids

* LDG: sol::optional

* LDG: libraries

* Check that userlib/userenum has been finalized

* LDG: library constants

* Bind game api & debug functions

* Switch over to new bindings

* Fix compilation

* Fix LUA=0 build

* Move hooks into a sub-table

* Fix test

* Update SaveLoad Lua Test mod, mark it obsolete

* Update Smart House Remotes mod

* Rename function to better match its purpose

* Register examples for hooks so they show up in docs

* Astyle

* Fix rename

* Fully implement Lua table (de-)serialization

* Update SaveLoad Lua Test mod

* Fix g++ build

* No, it wasn't fine

* Extract lua doc output filename to PATH_INFO

* Update docs

* Warn on accidental direct inclusion of sol/sol.hpp (thanks, vscode)

* Fix invalid int_id definition

* Add string_id(int_id) constructor

* Add bindings for int_id, implement manual conversions

* Fix doc generation for const pointers that are not const char*

* Split up luna into its own header

* Explicit type for color_id enum

* Move luna doc definitions to header

* Allow splitting catalua_bindings.cpp

* Split up catalua_bindings.cpp into 3 files

* Make console fullscreen

* Rename function to match its purpose

* Extract comparison operator hack into header

* Better names + docs

* Mark changes done to sol.hpp

* Fix msvc build

* Build releases with LUA=1

* Fix avatar.get_pos_ms

* Print typeid() if no luna registration found

* Revert "Fix msvc build"

This reverts commit ed46b46.

* Split up into more functions

* Alternative msvc fix

* More msvc fixes

* Add comments to the documentation

* Convert *ter_at to int_id

* GCC fix

* Add regression test

* Fix typo

* More hooks and bindings

* Fix msvc build, again

* Bind l10n api

* Update src/catalua_bindings.cpp

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

* Extract strings from Lua code

* Write Lua backtrace on debugmsg

* Fix: start backtrace with level 0

* Add comment

* Lunadoc: support this_state and function

* Recurrent hooks with customizable interval

* Less calendar checks

* Special fix for special compiler

* Fix LUA=0 build

* Use locale API in SHR mod

* Add docs on binding new stuff to Lua

* Fix typo

* Elaborate on implementing `enum_traits<T>`

* Mark "Smart House Remotes" as obsolete

* Update doc/LUA_SUPPORT.md

* Restrict some of the fun functionality

---------

Co-authored-by: scarf <[email protected]>
Co-authored-by: Zhilkin Serg <[email protected]>
  • Loading branch information
3 people authored Aug 28, 2023
1 parent 6cc49ee commit 90a1e97
Show file tree
Hide file tree
Showing 168 changed files with 59,537 additions and 106 deletions.
19 changes: 17 additions & 2 deletions .github/workflows/matrix.yml
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@ jobs:
cmake: 0
tiles: 0
sound: 0
lua: 0
test-stage: 1
title: GCC 12, Ubuntu, Curses
native: linux64
Expand All @@ -39,14 +40,25 @@ jobs:
cmake: 1
tiles: 1
sound: 1
lua: 1
languages: all
native: linux64
title: GCC 12, Ubuntu, Tiles, CMake, Languages
title: GCC 12, Ubuntu, Tiles, Sound, Lua, CMake, Languages
- compiler: g++-12
os: ubuntu-22.04
cmake: 0
tiles: 1
sound: 1
lua: 1
test-stage: 1
title: GCC 11, Ubuntu, Tiles, Sound, Lua
native: linux64
- compiler: g++-11
os: ubuntu-22.04
cmake: 0
tiles: 1
sound: 0
lua: 0
sanitize: address
native: linux64
title: GCC 12, Ubuntu, Tiles, NoSound, ASan
Expand All @@ -55,6 +67,7 @@ jobs:
cmake: 0
tiles: 1
sound: 0
lua: 0
sanitize: address,undefined
native: linux64
title: Clang 14, Ubuntu, Tiles, NoSound, ASan, UBSan
Expand All @@ -63,10 +76,11 @@ jobs:
cmake: 0
tiles: 1
sound: 1
lua: 1
test-stage: 1
native: osx
grep_clang_version_rxp: "14\\.[0-9]+\\.[0-9]+"
title: Clang 14, macOS 12, Tiles, Localize
title: Clang 14, macOS 12, Tiles, Sound, Localize, Lua
name: ${{ matrix.title }}
runs-on: ${{ matrix.os }}
env:
Expand All @@ -76,6 +90,7 @@ jobs:
OS: ${{ matrix.os }}
TILES: ${{ matrix.tiles }}
SOUND: ${{ matrix.sound }}
LUA: ${{ matrix.lua }}
SANITIZE: ${{ matrix.sanitize }}
TEST_STAGE: ${{ matrix.test-stage }}
LANGUAGES: ${{ matrix.languages }}
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/push-translation-template.yml
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ jobs:
- name: "Install Python3"
run: |
sudo apt install python3-pip
sudo pip3 install polib
sudo pip3 install polib luaparser
- name: "Checkout"
uses: actions/checkout@v3
- name: "Generate translation template"
Expand Down
6 changes: 3 additions & 3 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -224,14 +224,14 @@ jobs:
- name: Build CBN (linux)
if: runner.os == 'Linux' && matrix.mxe == 'none' && matrix.android == 'none'
run: |
make -j$((`nproc`+0)) TILES=${{ matrix.tiles }} SOUND=${{ matrix.tiles }} RELEASE=1 LANGUAGES=all PCH=0 bindist
make -j$((`nproc`+0)) TILES=${{ matrix.tiles }} SOUND=${{ matrix.tiles }} LUA=1 RELEASE=1 LANGUAGES=all PCH=0 bindist
mv cataclysmbn-unstable.tar.gz cbn-${{ matrix.artifact }}-${{ needs.release.outputs.timestamp }}.tar.gz
- name: Build CBN (windows)
if: matrix.mxe != 'none'
env:
PLATFORM: /opt/mxe/usr/bin/${{ matrix.mxe }}-w64-mingw32.static.gcc11-
run: |
make -j$((`nproc`+0)) CROSS="${PLATFORM}" TILES=1 SOUND=1 RELEASE=1 LANGUAGES=all PCH=0 bindist
make -j$((`nproc`+0)) CROSS="${PLATFORM}" TILES=1 SOUND=1 LUA=1 RELEASE=1 LANGUAGES=all PCH=0 bindist
mv cataclysmbn-unstable.zip cbn-${{ matrix.artifact }}-${{ needs.release.outputs.timestamp }}.zip
- name: Build CBN (windows msvc)
if: runner.os == 'Windows'
Expand All @@ -244,7 +244,7 @@ jobs:
- name: Build CBN (osx)
if: runner.os == 'macOS'
run: |
make -j3 TILES=${{ matrix.tiles }} SOUND=${{ matrix.tiles }} RELEASE=1 LANGUAGES=all USE_HOME_DIR=1 OSX_MIN=11 PCH=0 dmgdist COMPILER=clang++
make -j3 TILES=${{ matrix.tiles }} SOUND=${{ matrix.tiles }} LUA=1 RELEASE=1 LANGUAGES=all USE_HOME_DIR=1 OSX_MIN=11 PCH=0 dmgdist COMPILER=clang++
mv CataclysmBN-unstable.dmg cbn-${{ matrix.artifact }}-${{ needs.release.outputs.timestamp }}.dmg
- name: Set up JDK 8 (android)
if: runner.os == 'Linux' && matrix.android != 'none' && matrix.mxe == 'none'
Expand Down
2 changes: 2 additions & 0 deletions CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ SET(CMAKE_TLS_VERIFY ON)
option(TILES "Build graphical tileset version." "OFF")
option(CURSES "Build curses version." "ON")
option(SOUND "Support for in-game sounds & music." "OFF")
option(LUA "Support for in-game scripting with Lua." "OFF")
option(BACKTRACE "Support for printing stack backtraces on crash" "ON")
option(LIBBACKTRACE "Print backtrace with libbacktrace." "OFF")
option(USE_HOME_DIR "Use user's home directory for save files." "ON")
Expand Down Expand Up @@ -201,6 +202,7 @@ ENDIF (CMAKE_BUILD_TYPE STREQUAL Debug)
MESSAGE(STATUS "TILES : ${TILES}")
MESSAGE(STATUS "CURSES : ${CURSES}")
MESSAGE(STATUS "SOUND : ${SOUND}")
MESSAGE(STATUS "LUA : ${LUA}")
MESSAGE(STATUS "BACKTRACE : ${BACKTRACE}")
MESSAGE(STATUS "USE_HOME_DIR : ${USE_HOME_DIR}\n")

Expand Down
4 changes: 4 additions & 0 deletions LICENSE.txt
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,10 @@ PLF List and PLF Colony (src/list.h, src/colony.h) are licensed under the zLib l

getpost (tools/json_tools/format/getpost.h) is licensed under the MIT license, see file for text of license.

Lua (src/lua/*) is licensed under the MIT license, see src_lua/LICENSE.md for text of license.

sol2 (src/sol/*) is licensed under the MIT license, see src/sol/sol.hpp for text of license.



fmtlib (src/fmtlib_*) is licensed under the MIT license (https://github.com/fmtlib/fmt/blob/master/LICENSE.rst). The full license text is as follows:
Expand Down
36 changes: 30 additions & 6 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@
# LTO Set to 1 to enable link-time optimization.
# TILES Set to 1 to enable tiles. Requires SDL.
# SOUND Set to 1 to enable sounds. Requires SDL.
# LUA Set to 1 to enable Lua.
#
# Platforms:
# Linux/Cygwin native
Expand Down Expand Up @@ -166,8 +167,11 @@ CHKJSON_BIN = $(BUILD_PREFIX)chkjson
BINDIST_DIR = $(BUILD_PREFIX)bindist
BUILD_DIR = $(CURDIR)
SRC_DIR = src
LUA_SRC_DIR = $(SRC_DIR)/lua
ASTYLE_BINARY = astyle

CXXFLAGS += -I$(SRC_DIR) -I$(LUA_SRC_DIR)

# Enable astyle by default
ifndef ASTYLE
ASTYLE = 1
Expand Down Expand Up @@ -264,8 +268,10 @@ endif
# when preprocessor defines change, but the source doesn't
ODIR = $(BUILD_PREFIX)obj
ODIRTILES = $(BUILD_PREFIX)obj/tiles
ODIRLUA = $(BUILD_PREFIX)obj/lua
W32ODIR = $(BUILD_PREFIX)objwin
W32ODIRTILES = $(W32ODIR)/tiles
W32ODIRLUA = $(W32ODIR)/lua

ifdef AUTO_BUILD_PREFIX
BUILD_PREFIX = $(if $(RELEASE),release-)$(if $(DEBUG_SYMBOLS),symbol-)$(if $(TILES),tiles-)$(if $(SOUND),sound-)$(if $(BACKTRACE),back-$(if $(LIBBACKTRACE),libbacktrace-))$(if $(SANITIZE),sanitize-)$(if $(MAPSIZE),map-$(MAPSIZE)-)$(if $(USE_XDG_DIR),xdg-)$(if $(USE_HOME_DIR),home-)$(if $(DYNAMIC_LINKING),dynamic-)$(if $(MSYS2),msys2-)
Expand Down Expand Up @@ -573,6 +579,7 @@ ifeq ($(TARGETSYSTEM),WINDOWS)
BINDIST = $(W32BINDIST)
BINDIST_CMD = $(W32BINDIST_CMD)
ODIR = $(W32ODIR)
ODIRLUA = $(W32ODIRLUA)
ifeq ($(DYNAMIC_LINKING), 1)
# Windows isn't sold with programming support, these are static to remove MinGW dependency.
LDFLAGS += -static-libgcc -static-libstdc++
Expand Down Expand Up @@ -794,6 +801,7 @@ TESTSRC := $(wildcard tests/*.cpp)
TESTHDR := $(wildcard tests/*.h)
JSON_FORMATTER_SOURCES := tools/format/format.cpp src/json.cpp
CHKJSON_SOURCES := src/chkjson/chkjson.cpp src/json.cpp
LUA_SOURCES := $(wildcard $(LUA_SRC_DIR)/*.c)
CLANG_TIDY_PLUGIN_SOURCES := \
$(wildcard tools/clang-tidy-plugin/*.cpp tools/clang-tidy-plugin/*/*.cpp)
TOOLHDR := $(wildcard tools/*/*.h)
Expand All @@ -815,6 +823,13 @@ ifeq ($(TARGETSYSTEM),WINDOWS)
endif
OBJS = $(sort $(patsubst %,$(ODIR)/%,$(_OBJS)))

ifeq ($(LUA), 1)
DEFINES += -DLUA
LUA_OBJS = $(sort $(LUA_SOURCES:$(LUA_SRC_DIR)/%.c=$(ODIRLUA)/%.o))
else
LUA_OBJS =
endif

ifdef LANGUAGES
L10N = localization
endif
Expand Down Expand Up @@ -872,12 +887,12 @@ endif
all: version $(CHECKS) $(TARGET) $(L10N) $(TESTS)
@

$(TARGET): $(OBJS)
$(TARGET): $(OBJS) $(LUA_OBJS)
ifeq ($(VERBOSE),1)
+$(LD) $(W32FLAGS) -o $(TARGET) $(OBJS) $(LDFLAGS)
+$(LD) $(W32FLAGS) -o $(TARGET) $(OBJS) $(LUA_OBJS) $(LDFLAGS)
else
@echo "Linking $@..."
@+$(LD) $(W32FLAGS) -o $(TARGET) $(OBJS) $(LDFLAGS)
@+$(LD) $(W32FLAGS) -o $(TARGET) $(OBJS) $(LUA_OBJS) $(LDFLAGS)
@echo Done!
endif

Expand All @@ -892,12 +907,12 @@ endif
$(PCH_P): $(PCH_H)
-$(CXX) $(CPPFLAGS) $(DEFINES) $(subst -Werror,,$(CXXFLAGS)) -c $(PCH_H) -o $(PCH_P)

$(BUILD_PREFIX)$(TARGET_NAME).a: $(OBJS)
$(BUILD_PREFIX)$(TARGET_NAME).a: $(OBJS) $(LUA_OBJS)
ifeq ($(VERBOSE),1)
$(AR) rcs $(BUILD_PREFIX)$(TARGET_NAME).a $(filter-out $(ODIR)/main.o $(ODIR)/messages.o,$(OBJS))
$(AR) rcs $(BUILD_PREFIX)$(TARGET_NAME).a $(filter-out $(ODIR)/main.o $(ODIR)/messages.o,$(OBJS)) $(LUA_OBJS)
else
@echo "Creating $@..."
@$(AR) rcs $(BUILD_PREFIX)$(TARGET_NAME).a $(filter-out $(ODIR)/main.o $(ODIR)/messages.o,$(OBJS))
@$(AR) rcs $(BUILD_PREFIX)$(TARGET_NAME).a $(filter-out $(ODIR)/main.o $(ODIR)/messages.o,$(OBJS)) $(LUA_OBJS)
endif

.PHONY: version
Expand All @@ -911,6 +926,7 @@ version:

# Unconditionally create the object dir on every invocation.
$(shell mkdir -p $(ODIR))
$(shell mkdir -p $(ODIRLUA))

$(ODIR)/%.o: $(SRC_DIR)/%.cpp $(PCH_P)
ifeq ($(VERBOSE), 1)
Expand All @@ -928,6 +944,14 @@ else
@$(RC) $(RFLAGS) $< -o $@
endif

$(ODIRLUA)/%.o: $(LUA_SRC_DIR)/%.c
ifeq ($(VERBOSE), 1)
$(CXX) -xc -std=c11 -c $< -o $@
else
@echo $(@F)
@$(CXX) -xc -std=c11 -c $< -o $@
endif

src/version.h: version

src/version.cpp: src/version.h
Expand Down
8 changes: 7 additions & 1 deletion build-scripts/build.sh
Original file line number Diff line number Diff line change
Expand Up @@ -176,7 +176,13 @@ else
# Use a blacklist of mods that currently fail to load cleanly. Hopefully this list will
# shrink over time.
blacklist=build-scripts/mod_test_blacklist
mods="$(./build-scripts/get_all_mods.py $blacklist)"
if [ "$LUA" == "1" ]
then
do_lua="1"
else
do_lua="0"
fi
mods="$(./build-scripts/get_all_mods.py $blacklist $do_lua)"
run_tests ./tests/cata_test --user-dir=all_modded --mods="$mods" '~*'
fi
fi
Expand Down
7 changes: 6 additions & 1 deletion build-scripts/get_all_mods.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,10 @@
import glob
import json

blacklist_filename, = sys.argv[1:]
assert(len(sys.argv) == 3)
blacklist_filename = sys.argv[1]
do_lua = sys.argv[2] == "1"

with open(blacklist_filename) as blacklist_file:
blacklist = {s.rstrip('\n') for s in blacklist_file.readlines()}

Expand All @@ -26,6 +29,8 @@ def add_mods(mods):
mod_info = json.load(open(info))
for e in mod_info:
if e["type"] == "MOD_INFO":
if not do_lua and "lua_api_version" in e:
continue
ident = e["id"]
if not ident in blacklist:
all_mod_dependencies[ident] = e.get("dependencies", [])
Expand Down
2 changes: 1 addition & 1 deletion build-scripts/requirements.sh
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ if [ -n "$CATA_CLANG_TIDY" ]; then
fi

if [ -n "$LANGUAGES" ]; then
pip install --user polib
pip install --user polib luaparser
fi

# Influenced by https://github.com/zer0main/battleship/blob/master/build/windows/requirements.sh
Expand Down
1 change: 1 addition & 0 deletions data/mods/saveload_lua_test/finalize.lua
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
gdebug.log_info("SLT: finalize")
51 changes: 51 additions & 0 deletions data/mods/saveload_lua_test/main.lua
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
gdebug.log_info("SLT: main")

local mod = game.mod_runtime[ game.current_mod ]
local storage = game.mod_storage[ game.current_mod ]

--[[
If we keep all our data simple and in mod.storage,
we won't even have to register save/load hooks,
it'll be saved/loaded automatically.
]]--
mod.storage = storage

--[[
The following values are default, they'll be assigned on
new game start or on game init before save is loaded.
]]
-- Numeric data
storage.num = 12.3
-- Usertype
storage.tri = Tripoint.new(3, 4, 5)
-- String
storage.tri_as_str = tostring( Tripoint.new(3, 4, 5) )

--[[
If we want to build complex state out of loaded data
we may create a hook that would read loaded data from mod_storage
and create our complex state in the mod_runtime.
]]--
mod.on_game_load_hook = function()
gdebug.log_info("SLT: on_load")

if storage.num then
gdebug.log_info( "Data found! num = ", storage.num )
end
if storage.tri then
gdebug.log_info( "Data found! tri = ", storage.tri )
end
end

--[[
If we have complex enough state (e.g. recursive tables, or with custom metatables)
we may create a hook that would write a simplified version into mod_storage,
so the hardcoded JSON serializer would be able to handle it.
]]--
mod.on_game_save_hook = function()
gdebug.log_info("SLT: on_save")

if storage.num then
gdebug.log_info("Saving NUM value = ", storage.num)
end
end
14 changes: 14 additions & 0 deletions data/mods/saveload_lua_test/modinfo.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
[
{
"type": "MOD_INFO",
"id": "saveload_lua_test",
"name": "SaveLoad Lua Test",
"authors": [ "Olanti" ],
"description": "Mod for testing Lua save/load API.",
"category": "content",
"lua_api_version": 1,
"//": "It's not really obsolete, but it's useful only for devs and modders, and not for player.",
"obsolete": true,
"dependencies": [ "bn" ]
}
]
11 changes: 11 additions & 0 deletions data/mods/saveload_lua_test/preload.lua
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
gdebug.log_info("SLT: preload")

local mod = game.mod_runtime[ game.current_mod ]

game.hooks.on_game_load[ #game.hooks.on_game_load + 1 ] = function( ... )
return mod.on_game_load_hook( ... )
end

game.hooks.on_game_save[ #game.hooks.on_game_save + 1 ] = function( ... )
return mod.on_game_save_hook( ... )
end
4 changes: 4 additions & 0 deletions data/mods/smart_house_remotes/finalize.lua
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
gdebug.log_info("SHR: finalize.")

-- We're not finalizing anything.
-- Besides, there's no finalizing API available yet.
Loading

0 comments on commit 90a1e97

Please sign in to comment.