native webrtc dependency build script

[keroserene]

Need to make build.sh actually functional, and easy to use.
It should do roughly the following:

• Ensure depot_tools is installed on path (https://www.chromium.org/developers/how-tos/install-depot-tools)
• gclient sync webrtc, and checkout a specific commit hash or release branch
• generate either using gyp or manually the special .ninja build file for libwebrtc_magic, place in out/Release/obj/webrtc, add a couple more lines to out/Release/build.ninja.
• run ninja webrtc_magic
• move the platform specific concat'd archive to lib/ in the correct name
• copy over all necessary .h to include/

This build script will probably take an hour, but will be better than manually doing these steps. Also, this repo can then become much more lightweight by removing the include/ and prebuilt archives, and maybe supplying them elsewhere.

[arlolra]

Yeah, we should definitely pin to a branch head. These can be obtained with gclient sync --with_branch_heads and then gclient sync -r <hash> to get all the deps at the right place (that part is a little tricksy so noting it here for posterity).

[arlolra]

Uh oh, the readme currently says you used a4df27b6713583045e51e20c4eb93718d15ca33e but in 979312a I thought we had decided on bd7d8f7e2b824a887aa12236cb6185d446d7da61.

Can we clarify?

[keroserene]

Ah yes... it should be bd7d8f7e2b824a887aa12236cb6185d446d7da61
I think the other commit hash might be a revert to the same content, but still.

Also, maybe we should try a build from a branch head and see if things are still stable... perhaps branch-head/48?

[arlolra]

Maybe we should try a build from a branch head and see if things are still stable... perhaps branch-head/48?

I had tried that but it's missing a necessary commit as described in #19 (comment) see https://chromium.googlesource.com/external/webrtc/+/ec6b51d318f2cc747446089259495b161db11681/talk/app/webrtc/peerconnection.cc#403

But definitely when the next branch is cut.

[infinity0]

Hey congrats on this! As I see the hardest part getting this into Debian will be to build the webrtc library dependency. Here's my attempt, ignoring the entire depot_tools and using the Debian chromium-browser package instead:

#!/bin/sh
set -x
set -e

git clone https://chromium.googlesource.com/external/webrtc
cd webrtc

# extra stuff listed in DEPS
( cd third_party/gflags && git clone https://chromium.googlesource.com/external/gflags/src )
( cd third_party && git clone https://chromium.googlesource.com/external/webrtc/deps/third_party/junit junit-jar )

# get debian package and use its "configure" settings
# which uses as many system libs as possible
( cd chromium
apt-get source chromium-browser/sid
mv chromium-browser-* src
cd src
debian/rules clean override_dh_auto_configure )

touch ../.gclient # setup_links is overly strict, work around it
./setup_links.py
cp chromium/src/debian/rules .
cat >>rules <<'EOF'
gyp:
{TAB_CHARACTER}GYP_DEFINES="$(defines)" python webrtc/build/gyp_webrtc
EOF
sed -i -e 's/{TAB_CHARACTER}/\t/g' rules
make -f rules gyp

ninja -C out/Release

It's currently failing due to some SSLv3 error, I'm guessing because Debian dropped support for it. I'll have another look when I get some time.

In the meantime you could definitely use the non-Debian parts of the above script, to completely ignore depot_tool which makes you download far more than is necessary. It also makes reproducible builds hard too, you don't want to DL during the build of course. (The Debian package is essentially just a git clone of some of the upstream repos, without any of the gclient sync/config complexity, so you could just expand that out into a series of non-Debian-specific commands.)

Also it would be good if you could figure out what you actually need out of the *.o files. That might make the build easier.

[infinity0]

So I got that script to work. I built it in about 30 minutes (including download times) with webrtc HEAD and chromium 48.0.2564.82. You don't need depot_tools but you do need gsutil which can be installed from pip.

The Debian chromium-browser package is mostly the same tarball as what upstream releases (see debian/rules near the bottom), so you don't need to clone the entire repo as gclient does.

[infinity0]

@arlolra in build.sh you have the line libtool -static -o libwebrtc-magic.a -filelist filelist which AFAICS doesn't work with GNU libtool. Do you know the equivalent for GNU/Linux? I'm using ar rcs but getting some undefined reference errors; not sure if that is a problem with my overall source setup (which is different from yours) or with my use of ar.

[arlolra]

@infinity0 Yeah, that line was for OS X. I think you need to create a script,

create libwebrtc-magic.a
addlib something.a
...
save

end

then, ar -M < script

@keroserene built the magic for GNU/Linux so maybe she can weigh in on what she did.

[keroserene]

@infinity0 @arlolra

For some reason, native webrtc defaults to building thin archives on linux, which I wasn't quite sure how to deal with for concatenation.

So, during my initial attempt to get a working archive at all, I overrode that by removing the -T flag from the main ninja file's ar command, and ended up building a single archive of nested archives. (consisting mostly of the dependencies listed in the example peerconnection_client)

Then, I put together this terrible janky script which takes a single not thin archive argument. It flattens / concats everything within:

#!/bin/bash
echo "Merging archive... $1"
tmp=/tmp/merge-archives
mkdir $tmp
cp $1 $tmp
cd $tmp
files=`ar -t $1`
ar -x $1
all=""
for file in $files; do
  new=`ar -t $file`
  ar -x $file
  echo $new
  all="$all $new"
done
rm $1
ar vrcs $1 $all
ls -al $1
cd -

There's most likely a much simpler way to do this :)

[infinity0]

Ah thanks, I think I can just do ar crs libwebrtc-magic.a $(find . -name '*.o' -not -name '*.main.o') ) after ninja [..].

It turns out my undefined reference errors were because this library forces c++0x; I can do a successful full build (go build) against my own libwebrtc-magic.a by patching away that forcing. Is any particular reason you need C++0x?

[infinity0]

Oh, looks like on my computer I need to explicitly force it to C++11 (which is the system default). I guess cgo must be overriding that setting somehow, which makes it incompatible with external libraries built by a system default compiler. :(

[infinity0]

https://github.com/infinity0/go-webrtc/blob/master/build-webrtc-debian.sh (WIP, but the chat demo already works) in case anyone is interested. Later I will try to do it with webrtc HEAD and also experiment with building less of the objects. (You seem to only be using objects from libjingle_peerconnection, libjingle_peerconnection_unittest, and their dependencies.) It's still a long way to go before an actual Debian package but the way is much clearer now, at least for me.

[arlolra]

@infinity0 see #10, also as described above, branch-heads/49 is preferable to HEAD.

[infinity0]

Ah thanks I had missed that, git clone doesn't fetch it, one needs to add fetch = +refs/branch-heads/*:refs/remotes/origin/branch-heads/* to .git/config then fetch again.

[infinity0]

I'm getting a build error with branch-heads/49 @ 69bd93596468d949b1064d089b0e7e3826b73dbb which succeeds if I cherry-pick 3542013f587f0858fb24fa8e554ec3c01a323da8 on top of it. Such a shame, it's the very first commit on the master branch after 49 forks away from it.

FAILED: c++ -MMD -MF obj/webrtc/base/rtc_base.opensslstreamadapter.o.d -DV8_DEPRECATION_WARNINGS -DCLD_VERSION=2 -D_FILE_OFFSET_BITS=64 -DNO_TCMALLOC -DDISABLE_NACL -DCHROMIUM_BUILD -DCR_CLANG_REVISION=247874-1 -DUI_COMPOSITOR_IMAGE_TRANSPORT -DUSE_AURA=1 -DUSE_ASH=1 -DUSE_PANGO=1 -DUSE_CAIRO=1 -DUSE_DEFAULT_RENDER_THEME=1 -DUSE_LIBJPEG_TURBO=1 -DUSE_X11=1 -DUSE_CLIPBOARD_AURAX11=1 -DENABLE_ONE_CLICK_SIGNIN -DENABLE_PRE_SYNC_BACKUP -DENABLE_WEBRTC=1 -DENABLE_MEDIA_ROUTER=1 -DUSE_PROPRIETARY_CODECS -DENABLE_PEPPER_CDMS -DENABLE_CONFIGURATION_POLICY -DENABLE_NOTIFICATIONS -DENABLE_HIDPI=1 -DENABLE_TOPCHROME_MD=1 -DUSE_UDEV -DDONT_EMBED_BUILD_METADATA -DFIELDTRIAL_TESTING_ENABLED -DENABLE_TASK_MANAGER=1 -DENABLE_EXTENSIONS=1 -DENABLE_PDF=1 -DENABLE_PLUGINS=1 -DENABLE_SESSION_SERVICE=1 -DENABLE_THEMES=1 -DENABLE_AUTOFILL_DIALOG=1 -DENABLE_BACKGROUND=1 -DENABLE_PRINTING=1 -DENABLE_BASIC_PRINTING=1 -DENABLE_PRINT_PREVIEW=1 -DENABLE_SPELLCHECK=1 -DENABLE_CAPTIVE_PORTAL_DETECTION=1 -DENABLE_APP_LIST=1 -DENABLE_SETTINGS_APP=1 -DENABLE_SUPERVISED_USERS=1 -DENABLE_MDNS=1 -DENABLE_SERVICE_DISCOVERY=1 -DV8_USE_EXTERNAL_STARTUP_DATA -DFULL_SAFE_BROWSING -DSAFE_BROWSING_CSD -DSAFE_BROWSING_DB_LOCAL -DEXPAT_RELATIVE_PATH -DWEBRTC_POSIX -DWEBRTC_LINUX -DWEBRTC_INCLUDE_INTERNAL_AUDIO_DEVICE -DFEATURE_ENABLE_SSL -DSSL_USE_OPENSSL -DHAVE_OPENSSL_SSL_H -DLOGGING=1 -DUSE_LIBPCI=1 -DUSE_OPENSSL=1 -DUSE_GLIB=1 -DUSE_NSS_CERTS=1 -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -DNDEBUG -DNVALGRIND -DDYNAMIC_ANNOTATIONS_ENABLED=0 -D_FORTIFY_SOURCE=2 -Igen -I../.. -I../../third_party/jsoncpp/overrides/include -I../../third_party/jsoncpp/source/include -I../../chromium/src/third_party/jsoncpp/overrides/include -I../../chromium/src/third_party/jsoncpp/source/include -I../../chromium/src/third_party/boringssl/src/include -fstack-protector --param=ssp-buffer-size=4  -pthread -fno-strict-aliasing -Wno-unused-parameter -Wno-missing-field-initializers -fvisibility=hidden -pipe -fPIC -Wno-unused-local-typedefs -Wno-unused-parameter -Wno-missing-field-initializers -Wno-strict-overflow -m64 -march=x86-64 -O2 -fno-ident -fdata-sections -ffunction-sections -funwind-tables -fuse-ld=gold -fno-exceptions -fno-rtti -fno-threadsafe-statics -fvisibility-inlines-hidden -Wsign-compare -Woverloaded-virtual -std=gnu++11 -Wno-narrowing -Wno-literal-suffix  -c ../../webrtc/base/opensslstreamadapter.cc -o obj/webrtc/base/rtc_base.opensslstreamadapter.o
../../webrtc/base/opensslstreamadapter.cc:164:27: error: ‘TLS1_CK_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256’ was not declared in this scope
     static_cast<uint16_t>(TLS1_CK_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256);
                           ^
../../webrtc/base/opensslstreamadapter.cc:166:27: error: ‘TLS1_CK_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256’ was not declared in this scope
     static_cast<uint16_t>(TLS1_CK_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256);
                           ^

[arlolra]

Btw, I think that last build error here is because you didn't do gclient sync -r <hash> after checking out the branch head. Need to update the dependencies for it.

But you've moved on from building that way anyways.

[infinity0]

See also https://bugs.chromium.org/p/webrtc/issues/detail?id=5627

[arlolra]

As described, this was implemented in #51

We can open a separate issue for getting it into Debian.