Investigate etcd as an alternative of bootstrap bricks and leader election algorithm

[tatsuya6502]

Investigate etcd to see if it can replace the following areas in admin server:

• bootstrap bricks - admin server's private storage
• leader election - admin server's active/standby management
• brick auto-discovery - find a newly added brick server with zero-conf
• brick scoreboard - keeps brick servers status

etcd is a highly-available key value store for shared configuration and service discovery. It is written in Go language and uses the Raft (a variant of Paxos) consensus algorithm to manage a highly-available replicated log. etcd is a part of CoreOS, a Linux distribution with extensive Docker support for distributed server platform.

Details about etcd:
https://coreos.com/using-coreos/etcd/

[tatsuya6502]

etcd - The Road to 1.0
April 14, 2014 · By Blake Mizerany
http://coreos.com/blog/etcd-The-Road-to-1.0/

[tatsuya6502]

On Jan 30, 2014, @tatsuya6502 wrote:

Investigate etcd to see if it can replace the following areas in admin server:

• bootstrap bricks - admin server's private storage
• leader election - admin server's active/standby management
• brick auto-discovery - find a newly added brick server with zero-conf
• brick scoreboard - keeps brick servers status

etcd is a highly-available key value store for shared configuration and service discovery. It is written in Go language and uses the Raft (a variant of Paxos) consensus algorithm to manage a highly-available replicated log. etcd is a part of CoreOS, a Linux distribution with extensive Docker support for distributed server platform.

I recently started to use etcd v0.4.6 cluster for other project to store application auto-discovery information, and now I'm very confident that etcd will be very good for above tasks in Hibari admin server. It will greatly simplify the admin server implementation and will be much more stable and scalable than the current implementation. I also think I can retire our partition-detector too by using etcd.

A couple of weeks ago, the CoreOS team announced the first release candidate for etcd v2.0.0, and it seems they have brushed up the Raft implementation.

Announcing the etcd 2.0 Release Candidate
December 18, 2014 · By Xiang Li
https://coreos.com/blog/etcd-2-0-release-candidate/

At the heart of this release is a rethink of our Raft implementation. Raft enables etcd to do safe distributed compare-and-swap operations on keys and perform automatic leader election in the face of host failures. As many engineers have observed, implementing a consensus algorithm like Raft is a non-trivial task. But the etcd team has been hard at work. By leveraging all the latest knowledge and best practices in the Raft ecosystem, we’ve developed the new etcd Raft in 600 lines of Go and 2000 lines of tests, covering all cases in the state machine and every case described in the paper.

[tatsuya6502]

I can start to try etcd v2.0.0 RC using pre-built Docker container on CoreOS. Or, build it by myself for SmartOS Zones. I would prefer the latter as I'm running SmartOS on my home server.

etcd is written in Go, so all Go library dependencies are packed into a single binary file. Go's binaries are platform/processor-architecture dependent, but the compiler supports cross-compiling. It covers all possible Hibari platforms:

• illumos amd64 (SmartOS, OmniOS)
• FreeBSD amd64
• Gnu/Linux x86_64
• Gnu/Linux 32-bit ARM

There is an official Docker container with cross-compiling support (the ones with *-cross tag), and I think Gox will be handy for this kind of task. This Dockerfile might be a good example of how to achieve this.

Of course, I can use Go packages in SmartOS, FreeBSD and Arch Linux ARM, but cross-compiling with Gox will be simpler as it can produce binaries for all these targets at once in parallel compiling.

[tatsuya6502]

Too bad. No ARM (and other 32-bit systems) support in etcd.

coreos/etcd/README.md

32-bit systems
etcd has known issues on 32-bit systems due to a bug in the Go runtime. See #358 for more information.

[tatsuya6502]

Tried to build etcd 2.0.0 RC on 64-bit FreeBSD and SmartOS systems.

• FreeBSD 10.1-RELEASE amd64, Go 1.4,1 from pkgng
  • ./build finished witout error, but I haven't ran ./tests yet. (I'm new to Go language, so have to learn how to setup the environment)
• SmartOS (guest zone, standard64 14.3.0), Go 1.3.2 from pkgin
  • ./build failed with the following errors

[root@etcd1 ~/etcd]# go version
go version go1.3.2 solaris/amd64
[root@etcd1 ~/etcd]# ./build 
# github.com/coreos/etcd/pkg/fileutil
gopath/src/github.com/coreos/etcd/pkg/fileutil/lock_unix.go:34: undefined: syscall.Flock
gopath/src/github.com/coreos/etcd/pkg/fileutil/lock_unix.go:34: undefined: syscall.LOCK_EX
gopath/src/github.com/coreos/etcd/pkg/fileutil/lock_unix.go:34: undefined: syscall.LOCK_NB
gopath/src/github.com/coreos/etcd/pkg/fileutil/lock_unix.go:43: undefined: syscall.Flock
gopath/src/github.com/coreos/etcd/pkg/fileutil/lock_unix.go:43: undefined: syscall.LOCK_EX
gopath/src/github.com/coreos/etcd/pkg/fileutil/lock_unix.go:48: undefined: syscall.Flock
gopath/src/github.com/coreos/etcd/pkg/fileutil/lock_unix.go:48: undefined: syscall.LOCK_UN

According to this article "Solaris portability - flock()" by a core SmartOS developer Jonathan Perkin, flock() is of BSD heritage and does not exist on Solaris. He recommends to use POSIX standard fcntl() instead. Go has syscall.FctlFlock, which is a file lock based on fcntl(), and I think etcd should use this for better portability in Unix-like operating systems.

I'll play with Go a bit more (I should start with basic tutorials), and try to send a PR to etcd project.

[tatsuya6502]

etcd.erl - Erlang bindings for etcd key value store
https://github.com/marshall-lee/etcd.erl

[tatsuya6502]

OK. I'm not alone. Other people are making some progress on ARM support 👍
#2308 etcdserver/ARM: starting etcd on crashes the first time, succeeds subsequently

Too bad. No ARM (and other 32-bit systems) support in etcd.

coreos/etcd/README.md

32-bit systems
etcd has known issues on 32-bit systems due to a bug in the Go runtime. See #358 for more information.

[tatsuya6502]

Tried to build etcd 2.0.0 RC on 64-bit FreeBSD and SmartOS systems.
...

• SmartOS (guest zone, standard64 14.3.0), Go 1.3.2 from pkgin
  • ./build failed with the following errors

...
flock() is of BSD heritage and does not exist on Solaris

There are some on-going efforts by other people to make etcd 2 (including bolt) to run on Solaris and illmuos.

• etcd2
  • coreos/bugs #335 -- etc doesn't compile on Solaris
• bolt
  • golang/go #11113 -- x/sys/unix: Solaris lacks Flock (plus related constants)
  • coreos/etcd #3353 -- Recent etcd doesn't compile on Solaris due to bolt dependencies

[akolb1]

@tatsuya6502 The above issues are fixed and now both bolt and etcd work on illumos and should work on Solaris.

[tatsuya6502]

@akolb1 - That's great! I appreciate all your hard work.

The above issues are fixed and now both bolt and etcd work on illumos and should work on Solaris.