forked from ceph/ceph
-
Notifications
You must be signed in to change notification settings - Fork 1
/
README.FreeBSD
127 lines (102 loc) · 4.55 KB
/
README.FreeBSD
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
The FreeBSD build will build most of the tools in Ceph.
Note that the RBD dependant items will not work since FreeBSD does not
have RBD (yet)
I started looking into Ceph, because the HAST solution with CARP and
ggate did not really do what I was looking for. But I'm aiming for
running a Ceph storage cluster of storage nodes that are running ZFS.
End station would be running bhyve on RBD disk that are stored in Ceph.
Progress from last report:
==========================
Most important change:
- All test (but one) run to completion for the current selection of
tools. This is only the case for "My Fork" repository. Some of the
commits need to be pulled into the HEAD
- Cmake is now the only way of building Ceph
- Reworked threading/polling code for the simple socket code.
Now uses a selfpipe, instead of using an odd shutdown() signaling
Linux feature.
- Modified the EventKqueue code to work around the "feature" that
starting threads destroys the the kqueue handles. Just finshed the
code so it is not yet submitted to the main repository
Build Prerequisites
===================
Compiling and building Ceph is tested on 12-CURRENT, but I guess
11-RELEASE will also work. And Clang is at 3.8.0.
It uses the CLANG toolset that is available, 3.7 is not tested, but was
working when that was with 11-CURRENT.
Clang 3.4 (on 10.2-STABLE) does not have all required capabilites to
compile everything
The following setup will get things running for FreeBSD:
- Install bash and link it in /bin
# requires root privileges
sudo pkg install bash
sudo ln -s /usr/local/bin/bash /bin/bash
- Need to add one compatability line to
/usr/include/errno.h
#define ENODATA 87 /* Attribute not found */
(Otherwise some cython compiles will fail.)
Getting the FreeBSD work on Ceph:
=================================
- cd "place to work on this"
git clone https://github.com/wjwithagen/ceph.git
cd ceph
git checkout wip-wjw-freebsd-cmake
Building Ceph
=============
- Go and start building
./do_freebsd.sh
Note that the old way of building using automake is no longer used.
Parts not (yet) included:
=========================
- RBD
Rados Block Devices is implemented in the Linux kernel
It seems that there used to be a userspace implementation first.
And perhaps ggated could be used as a template since it does some of
the same, other than just between 2 disks. And it has a userspace
counterpart.
- BlueStore.
FreeBSD and Linux have different AIO API, and that needs to be made
compatible Next to that is there discussion in FreeBSD about
aio_cancel not working for all devicetypes
- CephFS
Cython tries to access an internal field in dirent which does not
compile
Tests that verify the correct working of the above are also excluded
from the testset
Tests not (yet) include:
=======================
- run-tox-ceph-disk
This is the main test that tests the basic Ceph disk install tool.
One of the next things to complete porting for.
- ceph-detect-init/run-tox.sh
Because the current implementation does not know anything about
FreeBSD rc-init.
- Tests that make use of nosetests.
Calling these does not really work since nosetests is not in
/usr/bin, and /usr/bin/env nosetests does not work on FreeBSD.
test/pybind/test_ceph_argparse.py
test/pybind/test_ceph_daemon.py
Task to do:
===========
- Build an automated test platform that will build ceph/master on
FreeBSD and report the results back to the Ceph developers. This will
increase the maintainability of the FreeBSD side of things.
Developers are signalled that they are using Linux-isms that will not
compile/run on FreeBSD Ceph has several projects for this: Jenkins,
teuthology, pulpito, ...
But even just a while { compile } loop and report the build data on a
static webpage would do for starters.
- Run integration tests to see if the FreeBSD daemons will work with a
Linux Ceph platform.
- Get the python tests that are currently excluded to work, and test OKE
- Compile and test the user space RBD (Rados Block Device).
- Investigate and see if an in-kernel RBD device could be developed a la
'ggate'
- Investigate the keystore, which could be kernel embedded on Linux an
currently prevents building Cephfs and some other parts.
- Scheduler information is not used atm, because the schedulers work
rather different. But at a certain point in time, this would need some
attention:
in: ./src/common/Thread.cc
- Integrate the FreeBSD /etc/rc.d init scripts in the Ceph stack. Both
for testing, but mainly for running Ceph on production machines.