Skip to content

Commit

Permalink
Port to dune
Browse files Browse the repository at this point in the history
Motivation
==========

With a port to dune, the ctypes library can be embedded in larger dune
projects simply by including it in the directory tree of the bigger
project. This in turn will allow MirageOS unikernels to use Ctypes
seamlessly as part of embedded compilation, using the standard
cross-compilation and library variants support built into Dune. The goal
is to make Ctypes the default FFI interface to MirageOS, and have it
work out of the box on all of the backends. All existing OS platforms
should be supported as well before merge, including Windows.

Porting Approach
================

This PR still installs ocamlfind libraries in the same way as the
previous Makefile infrastructure, so backwards compat is preserved.
However, the findlib schema has been slightly modified to separate out
the foreign library from the core ctypes library. We now install:

- ctypes that contains ctypes.top and ctypes.stubs as before. There are
  aliases to ctypes.foreign that redirect existing uses of those.
- ctypes-foreign contains the foreign library.
  All of the configuration logic for libffi is now in
  src/ctypes-foreign-base, so deleting these directories will remove the
  ffi build logic without touching the core library.

Since dune by default has stricter warnings enabled (the default
--profile=dev mode), the PR currently sprinkles files with [@@@warning
tags. The warnings can be fixed as well if desired, but that would muddy
this PR so not done yet.

Layout
======

The previous packaging would install ctypes and ctypes.stubs in the same
library. Depending on the build system, it means that it's possible to
specify a dependency on ctypes and use ctypes.stubs succesfully. This is
not the case with Dune since it installs these in different directories.
This means that it is necessary to patch these reverse dependencies,
though it can be argued that they were relying on a bug.

Followup work
=============

A test uses OCaml syntax to skip test on windows. Once it's acceptable
to use Dune 3.9, it should be updated to use build_if.

Test and dynamic libraries
==========================

Due to a different linking model, there are some changes in the test
suite regarding dynamic library loading.

Dynamic symbols in ocaml <4.06
------------------------------

The setup for tests is that each test executable tests some properties
using both "stubs" and "foreign" strategies to refer to some symbols
define in `tests/clib/` (the `test_functions` library).

The symbols are accessed through both strategies:
- as a linked symbol (through an `external` via stubs)
- through the dynamic loader (through `dlopen` via foreign)

Dune links the stubs statically, so by default the symbols would not be
visible to the dynamic loader. OCaml sets `-Wl,-E` in `LDFLAGS` to make
these symbols visible at runtime, but until
ocaml/ocaml@edbba02
(between 4.05 and 4.06), this was ignored when linking executables.

So this commit adds these flags when building tests for older versions.
An alternative would be to use `(link_flag)` in an `(env)` stanza but
this requires dune 3.

This issue does not affect the original make-based build because it
assembles the test executables a bit differently: the `test_functions`
library is linked dynamically to the test executable. So the symbols are
already visible to the dynamic loader, even when `-Wl,-E` is not set.

clib loading
------------

This change is specific to windows (but does not change the behavior on
Linux)

It modifies the test setup a bit so that the contents of `clib` are
dynamically loaded at the beginning of each test. This ensures that
later foreign calls (that use the default handle) will succeed.

It was not necessary before because `clib` was linked dynamically. Now
that it is linked statically, dynamic symbols do not have access to the
symbols in the main executable. This is a problem on windows which does
not have the concept of `-Wl,--export-dynamic`. Also, on windows the
stubs DLL can not be loaded directly so we recompile the library using
plain `%{cc}` instead of going through ocamlopt and flexlink.

Instrumentation
===============

This uses dune instrumentation for coverage The instructions are now:

    opam install bisect_ppx
    dune runtest --instrument-with bisect_ppx --force
    bisect-ppx-report html

See <https://github.com/aantron/bisect_ppx#Dune>

Depext handling
===============

This port uses conf-libffi instead of hardcoding depexts in
ctypes-foreign.

Co-authored-by: Anil Madhavapeddy <[email protected]>
  • Loading branch information
2 people authored and yallop committed Jul 7, 2023
1 parent 46cc3ff commit 0f8a92a
Show file tree
Hide file tree
Showing 230 changed files with 2,433 additions and 2,851 deletions.
175 changes: 0 additions & 175 deletions .depend

This file was deleted.

7 changes: 2 additions & 5 deletions .github/workflows/test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -63,10 +63,7 @@ jobs:
opam install -t --deps-only .
- name: Build
run: opam exec -- make
run: opam exec -- dune build

- name: Test
run: opam exec -- make test

- name: Test inverted stubs
run: opam pin add ctypes-inverted-stubs-example https://github.com/yallop/ocaml-ctypes-inverted-stubs-example.git
run: opam exec -- dune runtest
31 changes: 3 additions & 28 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,30 +1,5 @@
.*.swp
_build
_opam
libffi.config
asneeded.config
discover
gen_c_primitives
gen_c_primitives.log
gen_libffi_abi
gen_libffi_abi.log
src/ctypes/ctypes_primitives.ml
src/ctypes_config.h
src/ctypes_config.ml
src/ctypes-foreign/dl_stubs.c
src/ctypes-foreign/dl.ml
src/discover/commands.cm*
src/discover/discover.cm*
src/configure/extract_from_c.cm*
src/configure/gen_c_primitives.cm*
src/configure/gen_libffi_abi.cm*
*~
generated_stubs.c
generated_bindings.ml
generated_struct_bindings.ml
ncurses_generated.ml
ncurses_stubs.c
date_generated.ml
date_stubs.c
fts_generated.ml
fts_stubs.c
libffi_abi.ml
*.install
.merlin
19 changes: 0 additions & 19 deletions .merlin

This file was deleted.

43 changes: 0 additions & 43 deletions META

This file was deleted.

204 changes: 7 additions & 197 deletions Makefile
Original file line number Diff line number Diff line change
@@ -1,200 +1,10 @@
.SECONDEXPANSION:
.PHONY: build clean test

BEST:=$(shell if ocamlopt > /dev/null 2>&1; then echo native; else echo byte; fi)
OPAQUE:=$(shell if ocamlopt -opaque 2>/dev/null; then echo -opaque; fi)
NO_KEEP_LOCS:=$(shell if ocamlopt -no-keep-locs 2>/dev/null; then echo -no-keep-locs; fi)
DEBUG=true
COVERAGE=false
OCAML=ocaml
OCAMLFIND=ocamlfind $(OCAMLFINDFLAGS)
HOSTOCAMLFIND=$(OCAMLFIND)
OCAMLDEP=$(OCAMLFIND) ocamldep
OCAMLMKLIB=$(OCAMLFIND) ocamlmklib
VPATH=src examples
BUILDDIR=_build
BASE_PROJECTS=configure libffi-abigen configured ctypes ctypes-top
FOREIGN_PROJECTS=test-libffi ctypes-foreign
STUB_PROJECTS=cstubs
PROJECTS=$(BASE_PROJECTS) $(FOREIGN_PROJECTS) $(STUB_PROJECTS)
DEP_DIRS=$(foreach project,$(PROJECTS),$($(project).dir))
GENERATED=src/ctypes/ctypes_primitives.ml \
src/ctypes-foreign/libffi_abi.ml \
src/ctypes-foreign/dl.ml \
src/ctypes-foreign/dl_stubs.c \
libffi.config \
asneeded.config \
discover \
gen_c_primitives \
gen_c_primitives.log \
gen_libffi_abi \
src/configure/extract_from_c.cmi \
src/configure/extract_from_c.cmo \
src/configure/gen_c_primitives.cmi \
src/configure/gen_c_primitives.cmo \
src/configure/gen_libffi_abi.cmi \
src/configure/gen_libffi_abi.cmo \
src/discover/commands.cmi \
src/discover/commands.cmo \
src/discover/discover.cmi \
src/discover/discover.cmo
build:
dune build

OCAML_FFI_INCOPTS=$(libffi_opt)
export CFLAGS DEBUG
test:
dune runtest

EXTDLL:=$(shell $(OCAMLFIND) ocamlc -config | awk '/^ext_dll:/{print $$2}')
OSYSTEM:=$(shell $(OCAMLFIND) ocamlc -config | awk '/^system:/{print $$2}')

ifneq (,$(filter mingw%,$(OSYSTEM)))
OS_ALT_SUFFIX=.win
else
OS_ALT_SUFFIX=.unix
endif

# public targets
all: libffi.config $(PROJECTS)

ctypes-base: $(BASE_PROJECTS)
ctypes-foreign: ctypes-base test-libffi
ctypes-stubs: ctypes-base $(STUB_PROJECTS)

clean: clean-examples clean-tests
rm -fr _build
rm -f $(GENERATED)

# ctypes subproject
ctypes.cmi_only = ctypes_static ctypes_primitive_types ctypes_structs cstubs_internals
ctypes.public = lDouble complexL ctypes posixTypes ctypes_types
ctypes.dir = src/ctypes
ctypes.extra_mls = ctypes_primitives.ml
ctypes.deps = bigarray-compat integers
ctypes.linkdeps = integers_stubs
ctypes.install = yes
ctypes.install_native_objects = yes
ifeq ($(XEN),enable)
ctypes.xen = yes
endif

ctypes: PROJECT=ctypes
ctypes: $(ctypes.dir)/$(ctypes.extra_mls) $$(LIB_TARGETS)

# cstubs subproject
cstubs.public = cstubs_structs cstubs cstubs_inverted
cstubs.dir = src/cstubs
cstubs.subproject_deps = ctypes
cstubs.deps = str integers
cstubs.install = yes
cstubs.install_native_objects = yes
cstubs.extra_hs = $(package_integers_path)/ocaml_integers.h

cstubs: PROJECT=cstubs
cstubs: $(cstubs.dir)/$(cstubs.extra_mls) $$(LIB_TARGETS)

# ctypes-foreign subproject
ctypes-foreign.public = dl libffi_abi foreign
ctypes-foreign.dir = src/ctypes-foreign
ctypes-foreign.subproject_deps = ctypes
ctypes-foreign.deps = integers
ctypes-foreign.install = yes
ctypes-foreign.install_native_objects = yes
ctypes-foreign.extra_cs = dl_stubs.c
ctypes-foreign.extra_mls = libffi_abi.ml dl.ml
ctypes-foreign.cmi_opts = $(OPAQUE) $(NO_KEEP_LOCS)
ctypes-foreign.cmo_opts = $(OCAML_FFI_INCOPTS:%=-ccopt %)
ctypes-foreign.cmx_opts = $(OCAML_FFI_INCOPTS:%=-ccopt %)
ctypes-foreign.link_flags = $(libffi_lib) $(lib_process)
ctypes-foreign.threads = yes

ctypes-foreign: PROJECT=ctypes-foreign
ctypes-foreign: $$(LIB_TARGETS)

# ctypes-top subproject
ctypes-top.public = ctypes_printers
ctypes-top.dir = src/ctypes-top
ctypes-top.install = yes
ctypes-top.deps = compiler-libs integers
ctypes-top.subproject_deps = ctypes
ctypes-top.install_native_objects = yes

ctypes-top: PROJECT=ctypes-top
ctypes-top: $$(LIB_TARGETS)

# configuration
configured: src/ctypes/ctypes_primitives.ml src/ctypes-foreign/libffi_abi.ml src/ctypes-foreign/dl.ml src/ctypes-foreign/dl_stubs.c

src/ctypes-foreign/dl.ml: src/ctypes-foreign/dl.ml$(OS_ALT_SUFFIX)
cp $< $@
src/ctypes-foreign/dl_stubs.c: src/ctypes-foreign/dl_stubs.c$(OS_ALT_SUFFIX)
cp $< $@

src/ctypes/ctypes_primitives.ml: src/configure/extract_from_c.ml src/configure/gen_c_primitives.ml
$(HOSTOCAMLFIND) ocamlc -o gen_c_primitives -package str -strict-sequence -linkpkg $^ -I src/configure
./gen_c_primitives > $@ 2> gen_c_primitives.log || (rm $@ && cat gen_c_primitives.log && false)

src/ctypes-foreign/libffi_abi.ml: src/configure/extract_from_c.ml src/configure/gen_libffi_abi.ml
$(HOSTOCAMLFIND) ocamlc -o gen_libffi_abi -package str -strict-sequence -linkpkg $^ -I src/configure
./gen_libffi_abi > $@ 2> gen_c_primitives.log || (rm $@ && cat gen_c_primitives.log && false)

libffi.config: src/discover/commands.mli src/discover/commands.ml src/discover/discover.ml
$(HOSTOCAMLFIND) ocamlc -o discover -package str -strict-sequence -linkpkg $^ -I src/discover
./discover -ocamlc "$(OCAMLFIND) ocamlc" > $@ || (rm $@ && false)

asneeded.config:
./src/discover/determine_as_needed_flags.sh >> $@

# dependencies
depend: configure
$(OCAMLDEP) -one-line $(foreach dir,$(DEP_DIRS),-I $(dir)) \
$(shell find src examples -name '*.mli' -o -name '*.ml') \
| sed "s!src/!_build/src/!g; s!examples/!_build/examples/!g" | sort > .depend

#installation
META-install:
$(OCAMLFIND) install ctypes META CHANGES.md

install-%: PROJECT=$*
install-%:
$(if $(filter yes,$($(PROJECT).install)),\
$(OCAMLFIND) install -add ctypes -optional $^ \
$(LIB_TARGETS) $(LIB_TARGET_EXTRAS) \
$(INSTALL_MLIS) $(INSTALL_CMIS) \
$(INSTALL_CMTS) $(INSTALL_CMTIS) \
$(INSTALL_HEADERS) \
$(if $(filter yes,$($(PROJECT).install_native_objects)),$(NATIVE_OBJECTS)))

$(PROJECTS:%=install-%): META-install

install: META-install $(PROJECTS:%=install-%)

uninstall:
$(OCAMLFIND) remove ctypes

DOCFILES=$(foreach project,$(PROJECTS),\
$(foreach mli,$($(project).public),\
$($(project).dir)/$(mli).mli))
DOCFLAGS=-I $(shell ocamlfind query integers) $(foreach project,$(PROJECTS),-I $(BUILDDIR)/$($(project).dir))

doc:
ocamldoc -html $(DOCFLAGS) $(DOCFILES)


.PHONY: depend clean configure all install doc $(PROJECTS)

include .depend Makefile.rules Makefile.examples Makefile.tests
-include libffi.config
-include asneeded.config

ifeq ($(libffi_available),false)
test-libffi:
@echo "The following required C libraries are missing: libffi."
@echo "Please install them and retry. If they are installed in a non-standard location"
@echo "or need special flags, set the environment variables <LIB>_CFLAGS and <LIB>_LIBS"
@echo "accordingly and retry."
@echo
@echo " For example, if libffi is installed in /opt/local, you can type:"
@echo
@echo " export LIBFFI_CFLAGS=-I/opt/local/include"
@echo " export LIBFFI_LIBS=\"-L/opt/local/lib -lffi\""
@exit 1
else:
test-libffi:
endif
clean:
dune clean
Loading

0 comments on commit 0f8a92a

Please sign in to comment.