port to Android

[eighthave]

This ports go-webrtc to Android. Most of that was getting a repeatable build process for the libwebrtc component and pointing the go code to that binary. You can find the details of this epic saga here:
https://trac.torproject.org/projects/tor/ticket/28205 #82

In the end, I also got the whole process running in GitLab CI, including building the libwebrtc binaries for GNU/Linux amd64 and ARM. This should close out #32 For example, here is building the libwebrtc.a binary part for Debian and Android (close but not quite working):
https://gitlab.com/eighthave/go-webrtc/pipelines/37556080

And here is building the whole thing using the committed libwebrtc.a binaries, this runs on Debian/stretch using the Go 1.10 and 1.11 binaries, then on Ubuntu/LTS, Ubuntu/cosmic, and Ubuntu/devel using the Go from Ubuntu:
https://gitlab.com/eighthave/go-webrtc/pipelines/37555959

Once this is merged, gitlab will automatically build all commits in this repo here:
https://gitlab.com/torproject/go-webrtc/pipelines

The process for building libwebrtc.a is pretty huge, so that runs just once a month via a schedule:
https://gitlab.com/torproject/go-webrtc/pipeline_schedules

@arlolra @n8fr8 @uniqx

[eighthave]

Also, FYI, someone with admin access to this repo could allow GitLab CI to post the build results like Travis CI does.

[keroserene]

Thank you so much for this effort. Looks fantastic! @eighthave

When I get some time I (or @arlolra ?) can look at this in more depth, but so far... 👍 💯

[arlolra]

This should close out #32

No, this is just building on the work from #23. #32 is more about using the official binaries (as you were trying to do in https://trac.torproject.org/projects/tor/ticket/28205) but from a time before those really even existed.

And here is building the whole thing using the committed libwebrtc.a binaries

The binary is 96.9 MB, which is bigger than the others included here. Are symbols not being stripped? Any idea why that might be?

Also, FYI, someone with admin access to this repo could allow GitLab CI to post the build results like Travis CI does.

@keroserene is the only one with admin privileges because it's a personal repo. While we're admining, it would be great to get @uumaro push/pull rights here.

[eighthave]

I don't know why the libwebrtc.a binary is so large. It could be because Android only provides libc, nothing else. So on Android, it has to link in a lot more libs.

[eighthave]

One thing I do know is that the snowflake _client_ Android binaries that include _libwebrtc.a_ are about 23MB. Also, as far as I can tell, I addressed all the things in the GitHub Review thing, but it seems to think something is still open.

[n8fr8]

23MB! Definitely need to package it as a standalone app/plugin for Orbot somehow.

(Way to go @eighthave !)

[uumaro]

I don't know why the libwebrtc.a binary is so large. It could be because Android only provides libc, nothing else. So on Android, it has to link in a lot more libs.

It looks like the difference is the presence of debugging symbols.

Extract a sample object file:

$ ar x lib/libwebrtc-linux-arm-magic.a peerconnection.o && mv peerconnection.o peerconnection.linux-arm.o
$ ar x lib/libwebrtc-android-arm-magic.a peerconnection.o && mv peerconnection.o peerconnection.android-arm.o

The linux-arm one has no debugging while the android-arm one has:

$ objdump -g peerconnection.linux-arm.o

peerconnection.linux-arm.o:     file format elf32-little

$ objdump -g peerconnection.android-arm.o

peerconnection.android-arm.o:     file format elf32-little

Contents of the .debug_str section:

  0x00000000 636c616e 67207665 7273696f 6e20362e clang version 6.
  0x00000010 302e3020 28747275 6e6b2033 31383636 0.0 (trunk 31866
[147331 more lines]

Side note, neither one is stripped (nor are our other prebuilt libraries).

$ file peerconnection.linux-arm.o peerconnection.android-arm.o 
peerconnection.linux-arm.o:   ELF 32-bit LSB relocatable, ARM, EABI5 version 1 (SYSV), not stripped
peerconnection.android-arm.o: ELF 32-bit LSB relocatable, ARM, EABI5 version 1 (SYSV), not stripped

I would have thought that is_debug=false in build.sh would remove the debugging symbols. @eighthave: what does your gn args --list=is_debug say?

webrtc/src$ gn args --list=is_debug out/Release
is_debug
    Current value = false
      From //out/Release/args.gn:1
    Overridden from the default = true
      From //build/config/BUILDCONFIG.gn:154

[uumaro]

I'd like the Android build to also be covered using the existing .travis.yml. But I don't know how to add it trivially, and I don't want it to block this pull request, so I created a new issue #91.

[uumaro]

Thanks @eighthave for providing all this. It overall looks good.

The part I'm unsure of is the download of the Play Services SDK and automated acceptance of the licenses. My main concerns are:

1. That we don't impose any non-free constraints on users of the library. (I'm going to run a build myself and see what the licenses actually say.)
2. Even if the Play Services library comes with a free license, we need then to make sure to include it among the distributed documentation in e.g. the Tor Browser build (and the go-webrtc lib directory, I guess).
3. I don't think go-webrtc makes use of Play Services in any way, so is there a possibility we can omit it entirely?
4. This overview says "The client library contains the interfaces to the individual Google services and allows you to obtain authorization from users to gain access to these services with their credentials." If part of this library's purpose is authentication (which in the Tor/Snowflake context means deanonymization), we need to be very very careful, and if we can't strip out the library then audit what it is doing and ensure that it never tries to get e.g. oauth credentials nor access the global Google account that seems to exist on an Android phone.

https://bugs.chromium.org/p/webrtc/issues/detail?id=5578#c9

I wonder why the play services are downloaded at all, it makes no sense for the chromium open source project to depend on a package which (according to the licensing prompt) appears to be proprietary. And even less sense for webrtc.

Like, maybe we can comment this out or something?

[eighthave]

What is here is very far from ideal, the Play Services issue is one of many shitty things. Play Services is definitely non-free. What I achieved here is close to two weeks of work. We're already over budget on this work, so this is where it is going to stand until someone else either does the work or finds the budget. Seems to me the ideal outcome would be to free the libwebrtc build process from the vast and horrible chromium build setup, e.g. go more towards the route that @infinity0 did in #32 My guess is that will require two weeks of work by itself.

So for now, I think we need to consider the Android version non-free software, albeit with binaries that we are allowed distribute following the Play Services license. I haven't actually looked though, whether the libwebrtc binaries are free software. They might be, and it could just be the Chromium build system always pulls in Play Services since it also builds Chrome.

FYI, I'm part of the Debian Android Tools Team. The Android SDK is a similar gianormous, ugly build like Chromium. We've build writing our own Makefiles for years now. It is a lot of work to maintain them, but after this experience, I think it might be less worth than wrestling the Google's chromium nightmare.

Issues were addressed

[uumaro]

Just for reference, here's what you get, including the Play Services licence agreement, when you run GOOS=android GOARCH=android ./build.sh:

Click to expand (161 lines)

________ running '/usr/bin/python src/tools/clang/scripts/update.py' in '/home/user/go-webrtc/third_party/webrtc'
Updating Clang to 318667-1...
Downloading prebuilt clang
Downloading https://commondatastorage.googleapis.com/chromium-browser-clang/Linux_x64/clang-318667-1.tgz .......... Done.
Creating directory /home/user/go-webrtc/third_party/webrtc/src/third_party/llvm-build/Release+Asserts
clang 318667-1 unpacked
Hook 'src/build/cipd/cipd_wrapper.py --chromium-root src --ensure-file src/build/cipd/android/android.ensure' took 15.82 secs
Running hooks: 100% (26/26) sdkextras
________ running '/usr/bin/python src/build/android/play_services/update.py download' in '/home/user/go-webrtc/third_party/webrtc'
0> Downloading /tmp/tmpo15uQe/LICENSE...
Downloading 1 files took 3.485948 second(s)
Updating the Google Play services SDK to version 11.2.0.
Terms and Conditions

This is the Android Software Development Kit License Agreement

1. Introduction

1.1 The Android Software Development Kit (referred to in the License Agreement as the "SDK" and specifically including the Android system files, packaged APIs, and Google APIs add-ons) is licensed to you subject to the terms of the License Agreement. The License Agreement forms a legally binding contract between you and Google in relation to your use of the SDK.

1.2 "Android" means the Android software stack for devices, as made available under the Android Open Source Project, which is located at the following URL: http://source.android.com/, as updated from time to time.

1.3 A "compatible implementation" means any Android device that (i) complies with the Android Compatibility Definition document, which can be found at the Android compatibility website (http://source.android.com/compatibility) and which may be updated from time to time; and (ii) successfully passes the Android Compatibility Test Suite (CTS).

1.4 "Google" means Google Inc., a Delaware corporation with principal place of business at 1600 Amphitheatre Parkway, Mountain View, CA 94043, United States.


2. Accepting this License Agreement

2.1 In order to use the SDK, you must first agree to the License Agreement. You may not use the SDK if you do not accept the License Agreement.

2.2 By clicking to accept, you hereby agree to the terms of the License Agreement.

2.3 You may not use the SDK and may not accept the License Agreement if you are a person barred from receiving the SDK under the laws of the United States or other countries, including the country in which you are resident or from which you use the SDK.

2.4 If you are agreeing to be bound by the License Agreement on behalf of your employer or other entity, you represent and warrant that you have full legal authority to bind your employer or such entity to the License Agreement. If you do not have the requisite authority, you may not accept the License Agreement or use the SDK on behalf of your employer or other entity.


3. SDK License from Google

3.1 Subject to the terms of the License Agreement, Google grants you a limited, worldwide, royalty-free, non-assignable, non-exclusive, and non-sublicensable license to use the SDK solely to develop applications for compatible implementations of Android.

3.2 You may not use this SDK to develop applications for other platforms (including non-compatible implementations of Android) or to develop another SDK. You are of course free to develop applications for other platforms, including non-compatible implementations of Android, provided that this SDK is not used for that purpose.

3.3 You agree that Google or third parties own all legal right, title and interest in and to the SDK, including any Intellectual Property Rights that subsist in the SDK. "Intellectual Property Rights" means any and all rights under patent law, copyright law, trade secret law, trademark law, and any and all other proprietary rights. Google reserves all rights not expressly granted to you.

3.4 You may not use the SDK for any purpose not expressly permitted by the License Agreement.  Except to the extent required by applicable third party licenses, you may not copy (except for backup purposes), modify, adapt, redistribute, decompile, reverse engineer, disassemble, or create derivative works of the SDK or any part of the SDK.

3.5 Use, reproduction and distribution of components of the SDK licensed under an open source software license are governed solely by the terms of that open source software license and not the License Agreement.

3.6 You agree that the form and nature of the SDK that Google provides may change without prior notice to you and that future versions of the SDK may be incompatible with applications developed on previous versions of the SDK. You agree that Google may stop (permanently or temporarily) providing the SDK (or any features within the SDK) to you or to users generally at Google's sole discretion, without prior notice to you.

3.7 Nothing in the License Agreement gives you a right to use any of Google's trade names, trademarks, service marks, logos, domain names, or other distinctive brand features.

3.8 You agree that you will not remove, obscure, or alter any proprietary rights notices (including copyright and trademark notices) that may be affixed to or contained within the SDK.


4. Use of the SDK by You

4.1 Google agrees that it obtains no right, title or interest from you (or your licensors) under the License Agreement in or to any software applications that you develop using the SDK, including any intellectual property rights that subsist in those applications.

4.2 You agree to use the SDK and write applications only for purposes that are permitted by (a) the License Agreement and (b) any applicable law, regulation or generally accepted practices or guidelines in the relevant jurisdictions (including any laws regarding the export of data or software to and from the United States or other relevant countries).

4.3 You agree that if you use the SDK to develop applications for general public users, you will protect the privacy and legal rights of those users. If the users provide you with user names, passwords, or other login information or personal information, you must make the users aware that the information will be available to your application, and you must provide legally adequate privacy notice and protection for those users. If your application stores personal or sensitive information provided by users, it must do so securely. If the user provides your application with Google Account information, your application may only use that information to access the user's Google Account when, and for the limited purposes for which, the user has given you permission to do so.

4.4 You agree that you will not engage in any activity with the SDK, including the development or distribution of an application, that interferes with, disrupts, damages, or accesses in an unauthorized manner the servers, networks, or other properties or services of any third party including, but not limited to, Google or any mobile communications carrier.

4.5 You agree that you are solely responsible for (and that Google has no responsibility to you or to any third party for) any data, content, or resources that you create, transmit or display through Android and/or applications for Android, and for the consequences of your actions (including any loss or damage which Google may suffer) by doing so.

4.6 You agree that you are solely responsible for (and that Google has no responsibility to you or to any third party for) any breach of your obligations under the License Agreement, any applicable third party contract or Terms of Service, or any applicable law or regulation, and for the consequences (including any loss or damage which Google or any third party may suffer) of any such breach.


5. Your Developer Credentials

5.1 You agree that you are responsible for maintaining the confidentiality of any developer credentials that may be issued to you by Google or which you may choose yourself and that you will be solely responsible for all applications that are developed under your developer credentials.


6. Privacy and Information

6.1 In order to continually innovate and improve the SDK, Google may collect certain usage statistics from the software including but not limited to a unique identifier, associated IP address, version number of the software, and information on which tools and/or services in the SDK are being used and how they are being used. Before any of this information is collected, the SDK will notify you and seek your consent. If you withhold consent, the information will not be collected.

6.2 The data collected is examined in the aggregate to improve the SDK and is maintained in accordance with Google's Privacy Policy.


7. Third Party Applications

7.1 If you use the SDK to run applications developed by a third party or that access data, content or resources provided by a third party, you agree that Google is not responsible for those applications, data, content, or resources. You understand that all data, content or resources which you may access through such third party applications are the sole responsibility of the person from which they originated and that Google is not liable for any loss or damage that you may experience as a result of the use or access of any of those third party applications, data, content, or resources.

7.2 You should be aware the data, content, and resources presented to you through such a third party application may be protected by intellectual property rights which are owned by the providers (or by other persons or companies on their behalf). You may not modify, rent, lease, loan, sell, distribute or create derivative works based on these data, content, or resources (either in whole or in part) unless you have been specifically given permission to do so by the relevant owners.

7.3 You acknowledge that your use of such third party applications, data, content, or resources may be subject to separate terms between you and the relevant third party. In that case, the License Agreement does not affect your legal relationship with these third parties.


8. Using Android APIs

8.1 Google Data APIs

8.1.1 If you use any API to retrieve data from Google, you acknowledge that the data may be protected by intellectual property rights which are owned by Google or those parties that provide the data (or by other persons or companies on their behalf). Your use of any such API may be subject to additional Terms of Service. You may not modify, rent, lease, loan, sell, distribute or create derivative works based on this data (either in whole or in part) unless allowed by the relevant Terms of Service.

8.1.2 If you use any API to retrieve a user's data from Google, you acknowledge and agree that you shall retrieve data only with the user's explicit consent and only when, and for the limited purposes for which, the user has given you permission to do so.


9. Terminating this License Agreement

9.1 The License Agreement will continue to apply until terminated by either you or Google as set out below.

9.2 If you want to terminate the License Agreement, you may do so by ceasing your use of the SDK and any relevant developer credentials.

9.3 Google may at any time, terminate the License Agreement with you if:
(A) you have breached any provision of the License Agreement; or
(B) Google is required to do so by law; or
(C) the partner with whom Google offered certain parts of SDK (such as APIs) to you has terminated its relationship with Google or ceased to offer certain parts of the SDK to you; or
(D) Google decides to no longer provide the SDK or certain parts of the SDK to users in the country in which you are resident or from which you use the service, or the provision of the SDK or certain SDK services to you by Google is, in Google's sole discretion, no longer commercially viable.

9.4 When the License Agreement comes to an end, all of the legal rights, obligations and liabilities that you and Google have benefited from, been subject to (or which have accrued over time whilst the License Agreement has been in force) or which are expressed to continue indefinitely, shall be unaffected by this cessation, and the provisions of paragraph 14.7 shall continue to apply to such rights, obligations and liabilities indefinitely.


10. DISCLAIMER OF WARRANTIES

10.1 YOU EXPRESSLY UNDERSTAND AND AGREE THAT YOUR USE OF THE SDK IS AT YOUR SOLE RISK AND THAT THE SDK IS PROVIDED "AS IS" AND "AS AVAILABLE" WITHOUT WARRANTY OF ANY KIND FROM GOOGLE.

10.2 YOUR USE OF THE SDK AND ANY MATERIAL DOWNLOADED OR OTHERWISE OBTAINED THROUGH THE USE OF THE SDK IS AT YOUR OWN DISCRETION AND RISK AND YOU ARE SOLELY RESPONSIBLE FOR ANY DAMAGE TO YOUR COMPUTER SYSTEM OR OTHER DEVICE OR LOSS OF DATA THAT RESULTS FROM SUCH USE.

10.3 GOOGLE FURTHER EXPRESSLY DISCLAIMS ALL WARRANTIES AND CONDITIONS OF ANY KIND, WHETHER EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO THE IMPLIED WARRANTIES AND CONDITIONS OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NON-INFRINGEMENT.


11. LIMITATION OF LIABILITY

11.1 YOU EXPRESSLY UNDERSTAND AND AGREE THAT GOOGLE, ITS SUBSIDIARIES AND AFFILIATES, AND ITS LICENSORS SHALL NOT BE LIABLE TO YOU UNDER ANY THEORY OF LIABILITY FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, CONSEQUENTIAL OR EXEMPLARY DAMAGES THAT MAY BE INCURRED BY YOU, INCLUDING ANY LOSS OF DATA, WHETHER OR NOT GOOGLE OR ITS REPRESENTATIVES HAVE BEEN ADVISED OF OR SHOULD HAVE BEEN AWARE OF THE POSSIBILITY OF ANY SUCH LOSSES ARISING.


12. Indemnification

12.1 To the maximum extent permitted by law, you agree to defend, indemnify and hold harmless Google, its affiliates and their respective directors, officers, employees and agents from and against any and all claims, actions, suits or proceedings, as well as any and all losses, liabilities, damages, costs and expenses (including reasonable attorneys fees) arising out of or accruing from (a) your use of the SDK, (b) any application you develop on the SDK that infringes any copyright, trademark, trade secret, trade dress, patent or other intellectual property right of any person or defames any person or violates their rights of publicity or privacy, and (c) any non-compliance by you with the License Agreement.


13. Changes to the License Agreement

13.1 Google may make changes to the License Agreement as it distributes new versions of the SDK. When these changes are made, Google will make a new version of the License Agreement available on the website where the SDK is made available.


14. General Legal Terms

14.1 The License Agreement constitutes the whole legal agreement between you and Google and governs your use of the SDK (excluding any services which Google may provide to you under a separate written agreement), and completely replaces any prior agreements between you and Google in relation to the SDK.

14.2 You agree that if Google does not exercise or enforce any legal right or remedy which is contained in the License Agreement (or which Google has the benefit of under any applicable law), this will not be taken to be a formal waiver of Google's rights and that those rights or remedies will still be available to Google.

14.3 If any court of law, having the jurisdiction to decide on this matter, rules that any provision of the License Agreement is invalid, then that provision will be removed from the License Agreement without affecting the rest of the License Agreement. The remaining provisions of the License Agreement will continue to be valid and enforceable.

14.4 You acknowledge and agree that each member of the group of companies of which Google is the parent shall be third party beneficiaries to the License Agreement and that such other companies shall be entitled to directly enforce, and rely upon, any provision of the License Agreement that confers a benefit on (or rights in favor of) them. Other than this, no other person or company shall be third party beneficiaries to the License Agreement.

14.5 EXPORT RESTRICTIONS. THE SDK IS SUBJECT TO UNITED STATES EXPORT LAWS AND REGULATIONS. YOU MUST COMPLY WITH ALL DOMESTIC AND INTERNATIONAL EXPORT LAWS AND REGULATIONS THAT APPLY TO THE SDK. THESE LAWS INCLUDE RESTRICTIONS ON DESTINATIONS, END USERS AND END USE.

14.6 The rights granted in the License Agreement may not be assigned or transferred by either you or Google without the prior written approval of the other party. Neither you nor Google shall be permitted to delegate their responsibilities or obligations under the License Agreement without the prior written approval of the other party.

14.7 The License Agreement, and your relationship with Google under the License Agreement, shall be governed by the laws of the State of California without regard to its conflict of laws provisions. You and Google agree to submit to the exclusive jurisdiction of the courts located within the county of Santa Clara, California to resolve any legal matter arising from the License Agreement. Notwithstanding this, you agree that Google shall still be allowed to apply for injunctive remedies (or an equivalent type of urgent legal relief) in any jurisdiction.


December 9, 2016

Do you accept the license for version 11.2.0 of the Google Play services client library? [y/n]:

I cheekily tried responding n instead of y to the prompt, and to my surprise, the build kept going! I'll have to let it finish to see what happens, but maybe this could be a solution? We'll have to check that it's not actually getting included (probably with an rm -rf just to be sure).

Do you accept the license for version 11.2.0 of the Google Play services client library? [y/n]:
n
Your version of the Google Play services library is not up to date. You might run into issues building or running the app. Please run `src/build/android/play_services/update.py download` to retry downloading it.
> Hook '/usr/bin/python src/build/android/play_services/update.py download' took 207.17 secs
Running hooks: 100% (26/26), done.
~/go-webrtc
Setting gclient target_os to android

________ running 'git -c core.deltaBaseCacheLimit=2g clone --no-checkout --progress https://chromium.googlesource.com/external/webrtc /home/user/go-webrtc/_gclient_src_cHwETm' in '/home/user/go-webrtc'
Cloning into '/home/user/go-webrtc/_gclient_src_cHwETm'...
remote: Sending approximately 259.97 MiB ...
Receiving objects:  88% (262308/298077), 184.96 MiB | 38.89 MiB/s

[uumaro]

I had to answer n once more to the license prompt (probably because build.sh calls gclient sync twice in the android case), but then it kept building and almost got to the end, only failing at the final linking step.

[3022/3022] AR obj/libwebrtc.a
~/go-webrtc
Copying headers ...
~/go-webrtc/third_party/webrtc/src ~/go-webrtc
~/go-webrtc
~/go-webrtc ~/go-webrtc
~/go-webrtc
Concatenating libraries ...
~/go-webrtc/third_party/webrtc/src/out/Release ~/go-webrtc
./build.sh: line 148: /home/user/go-webrtc/third_party/webrtc/src/third_party/android_ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin/arm-linux-androideabi-ar: No such file or directory
mv: cannot stat 'libwebrtc-magic.a': No such file or directory
Built /home/user/go-webrtc/lib/libwebrtc-android-arm-magic.a
~/go-webrtc
Build complete.

For me, third_party/android_ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin/arm-linux-androideabi-ar didn't exist--the correct path seems to have android_tools/ndk/ instead of android_ndk/. Fixing that, the build goes to completion and leaves me with a lib/libwebrtc-android-arm-magic.a that is of a similar size to eighthave's but is not identical.

$ git diff --stat
 lib/libwebrtc-android-arm-magic.a | Bin 101589420 -> 101549584 bytes
 1 file changed, 0 insertions(+), 0 deletions(-)

(I checked and the large size is due to it being full of debugging information as in #90 (comment), despite is_debug=false being set.)

Anyway, my minor changes on top of eighthave's are at https://github.com/uumaro/go-webrtc/tree/android (currently uumaro@3bbd340). This is still a work in progress as I want to check out the license agreement thing some more.

[keroserene]

@arlolra

@keroserene is the only one with admin privileges because it's a personal repo. While we're admining, it would be great to get @uumaro push/pull rights here.

@uumaro : Invited you to have access to this repo. Thanks

[uumaro]

@uumaro : Invited you to have access to this repo. Thanks

Right on, cheers 😄

[eighthave]

@uumaro great that you got a working build! I'll incorporate your fixes in my merge request. As you can see in my flailing around the license prompt in .gitlab-ci.yml, the issue was about trying to figure out how to skip it in an automated way.

If its possible to build libwebrtc without Play Services being installed, as it seems you did, then it seems feasible to upstream a patch to let that work with the upstream build system.

[eighthave]

Ok, this includes @uumaro's changes, with the binary for now, since it is huge. It seems that now it can at least be manually repeated, so it is not so important to have included in this pull request. Here are all the gitlab-ci runs, it is more or less the same results as before:

• https://gitlab.com/eighthave/go-webrtc/pipelines/38909385
• https://gitlab.com/eighthave/go-webrtc/-/jobs/129525094

[uumaro]

I did two separate runs of gclient sync, one accepting the Google Play service license agreement and one without, in order to see what files get downloaded. Here they are:

$ diff -u <(cd webrtc-no-play && find . -name .git -prune -o -type f) <(cd webrtc-play && find . -name .git -prune -o -type f)
--- /dev/fd/63  2018-12-05 05:36:59.664083603 +0000
+++ /dev/fd/62  2018-12-05 05:36:59.664083603 +0000
@@ -60570,6 +60570,20 @@
 ./src/third_party/android_tools/sdk/extras/google/gcm/gcm-server/src/com/google/android/gcm/server/Message.java
 ./src/third_party/android_tools/sdk/extras/google/gcm/gcm-server/src/com/google/android/gcm/server/Constants.java
 ./src/third_party/android_tools/sdk/extras/google/gcm/gcm-server/src/com/google/android/gcm/server/InvalidRequestException.java
+./src/third_party/android_tools/sdk/extras/google/m2repository/com/google/android/gms/play-services-iid/11.2.0/play-services-iid-11.2.0.aar
+./src/third_party/android_tools/sdk/extras/google/m2repository/com/google/android/gms/play-services-nearby/11.2.0/play-services-nearby-11.2.0.aar
+./src/third_party/android_tools/sdk/extras/google/m2repository/com/google/android/gms/play-services-cast/11.2.0/play-services-cast-11.2.0.aar
+./src/third_party/android_tools/sdk/extras/google/m2repository/com/google/android/gms/play-services-auth-base/11.2.0/play-services-auth-base-11.2.0.aar
+./src/third_party/android_tools/sdk/extras/google/m2repository/com/google/android/gms/play-services-basement/11.2.0/play-services-basement-11.2.0.aar
+./src/third_party/android_tools/sdk/extras/google/m2repository/com/google/android/gms/play-services-base/11.2.0/play-services-base-11.2.0.aar
+./src/third_party/android_tools/sdk/extras/google/m2repository/com/google/android/gms/play-services-gcm/11.2.0/play-services-gcm-11.2.0.aar
+./src/third_party/android_tools/sdk/extras/google/m2repository/com/google/android/gms/play-services-location/11.2.0/play-services-location-11.2.0.aar
+./src/third_party/android_tools/sdk/extras/google/m2repository/com/google/android/gms/play-services-vision/11.2.0/play-services-vision-11.2.0.aar
+./src/third_party/android_tools/sdk/extras/google/m2repository/com/google/android/gms/play-services-tasks/11.2.0/play-services-tasks-11.2.0.aar
+./src/third_party/android_tools/sdk/extras/google/m2repository/com/google/android/gms/play-services-vision-common/11.2.0/play-services-vision-common-11.2.0.aar
+./src/third_party/android_tools/sdk/extras/google/m2repository/com/google/android/gms/play-services-auth/11.2.0/play-services-auth-11.2.0.aar
+./src/third_party/android_tools/sdk/extras/google/m2repository/google_play_services_library.zip.sha1
+./src/third_party/android_tools/sdk/extras/google/m2repository/LICENSE
 ./src/third_party/android_tools/sdk/platform-tools/etc1tool
 ./src/third_party/android_tools/sdk/platform-tools/NOTICE.txt
 ./src/third_party/android_tools/sdk/platform-tools/systrace/NOTICE

So in uumaro@656dca2 in my android branch, I not only auto-decline the license agreement, but also delete the src/third_party/android_tools/sdk/extras/google/m2repository/com/google/android/gms for good measure, just in case it's been accidentally accepted in the same tree in the past. I added a best-effort attempt to check that the download directory looks how we expect it to look, so that the build will break in the case of a future reorganization that downloads the library into a different place.

# "yes n" is to say "no" to the Google Play services license agreement and download.
yes n | gclient sync --with_branch_heads -r $COMMIT || exit 1
# Delete where the Google Play services downloads to, just to be sure.
# First check that an ancestor directory of what we're deleting exists, so we're more likely to notice a source reorganization.
[[ -d "$WEBRTC_SRC/third_party/android_tools/sdk/extras/google/m2repository" ]] || {
	echo "Didn't find Google Play services directory for removal, please check" 1>&2
	exit 1
}
rm -rf "$WEBRTC_SRC/third_party/android_tools/sdk/extras/google/m2repository/com/google/android/gms"

I did some mild refactoring of build.sh to avoid the need for repeated gclient sync.

[uumaro]

My two issues with this port were (1) not to accept a proprietary license and (2) to reduce the size of the output library. Now that both of those are taken care of, is there anything else to do?

@eighthave, does the library in uumaro@c7061bd work for your purposes (i.e. keroserene/snowflake#43)?

I can take care of squashing eighthave's a my commits into a coherent history.

[eighthave]

@n8fr8 since your already in this code at the moment, could you try building snowflake against @uumaro's libwebrtc binary?

[n8fr8]

@eighthave would I need to fork snowflake and go-webrtc to make all the necessary changes to point to the updated "master" repo with the magic binary? Or would this be better somehow built through your gitlab-ci work?

[eighthave]

well, I suck at Go, but I've good luck with symlinking for this kind of thing. I would clone @uumaro's repo to your /go/src, then use symlinks to make Go think that it is building github.com/keroserene/go-webrtc

[uumaro]

I'm just looking for confirmation that uumaro@c7061bd works for @n8fr8 and @eighthave's purposes before merging this branch.

Or @eighthave, did you want to edit .gitlab-ci.yml to match the procedure in build.sh? I can do it, but I can't test it unless there's an easy way for me to run a gitlab-ci job.

[eighthave]

It is very easy to set up your own fork on gitlab.com that will automatically run the gitlab CI jobs:

• log in with github or create account
• create a project under your name, it can either import from a git URL or just be a CI/CD runner for a repo on GitHub
• to get a CI Runner that works for the largedisk jobs, I'll have to set you up with access to our runner. See instructions here https://gitlab.com/fdroid/admin/issues/106

[eighthave]

I rebased your uumaro/android branch on upstream's master and triggered a rebuild on gitlab-CI:

• https://gitlab.com/eighthave/go-webrtc/-/jobs/136625345

Feel free to merge it as it is or rework it as needed

[uumaro]

It is very easy to set up your own fork on gitlab.com that will automatically run the gitlab CI jobs:

Thanks, I started a fork at https://gitlab.com/uumaro/go-webrtc/tree/android and emailed you a token.

I rebased your uumaro/android branch on upstream's master and triggered a rebuild on gitlab-CI:

• https://gitlab.com/eighthave/go-webrtc/-/jobs/136625345

Thanks, this uncovered a bug in one of my changes that manifested when the third_party/webrtc directory did not exist yet.

[uumaro]

I happened to find a way to disable the license prompt. The environment variable CHROME_HEADLESS isn't well documented, but I gather that Google sets it during it automated builds. During gclient sync, it has the effect of not showing the license prompt.

In uumaro@9087925, I set CHROME_HEADLESS=1, but I still kept the echo n, because I'm not sure how future-proof setting CHROME_HEADLESS will be.

[uumaro]

In the end, I also got the whole process running in GitLab CI, including building the libwebrtc binaries for GNU/Linux amd64 and ARM. This should close out #32 For example, here is building the libwebrtc.a binary part for Debian and Android (close but not quite working):
https://gitlab.com/eighthave/go-webrtc/pipelines/37556080

Ok, this includes @uumaro's changes, with the binary for now, since it is huge. It seems that now it can at least be manually repeated, so it is not so important to have included in this pull request. Here are all the gitlab-ci runs, it is more or less the same results as before:

• https://gitlab.com/eighthave/go-webrtc/pipelines/38909385
• https://gitlab.com/eighthave/go-webrtc/-/jobs/129525094

@eighthave, did any of your gitlab CI runs complete successfully, all the way through the gomobile bind command? I've been trying to figure out errors in the branch I made to check that my changes didn't break CI, but it looks like the CI was never completely working? This log and this log both end in ERROR: Job failed: exit code 1. The artifacts specification seems to only contain "include/" and "lib/", nothing that would be produced by gomobile.

I'm just trying to figure out if gomobile bind failing is expected, or if I accidentally broke something.

[eighthave]

I never got a full run of the libwebrtc build to complete successfully in the CI, due to the license prompt and gclient issues. `gomobile bind` here is just a test run anyway, the real `gomobile bind` happens in snowflake.git, and that works provided there is a libwebrtc binary. I think in theory, `gomobile bind` should work for go-webrtc also. The issue in your last build seems to be this, which I have no idea about: ``` # github.com/keroserene/go-webrtc In file included from ctestenums.cc:2: ./include/api/peerconnectioninterface.h:70:10: fatal error: 'memory' file not found rm -r -f "$WORK" gomobile: go build -v -x -buildmode=c-shared -o=/tmp/gomobile-work-646157724/android/src/main/jniLibs/armeabi-v7a/libgojni.so gobind failed: exit status 2 ```

[uumaro]

The issue in your last build seems to be this, which I have no idea about:

# github.com/keroserene/go-webrtc In file included from ctestenums.cc:2:
./include/api/peerconnectioninterface.h:70:10: fatal error: 'memory' file not found

No, ignore that error. That was me randomly trying to add -stdlib=libc++ to try and fix ABI errors in gomobile bind.

Better look at this one instead:
https://gitlab.com/uumaro/go-webrtc/-/jobs/137557604

[eighthave]

I don't know why it would fail to link, my guess it that it is some C++ namespace thing related to `__ndk1`. If I recall correctly, the Android NDK offers 2 or 3 different C++ standard libs, with some options, e.g. https://developer.android.com/ndk/guides/cpp-support Then those config options need to be translated to `gomobile bind` somehow. I guess it would make sense to double check if the config is the same as the snowflake job, which works.

[uumaro]

I spent a few days trying to understand the failure of gomobile bind in .gitlab-ci.yml, and ultimately gave up and removed it from the file. I suppose it's not a problem because it wasn't producing any build artifacts that we were using.

Anyway I've rebased and squashed my and @eighthave's changes that were in my android branch. It's at the point where I want it now. I'd appreciate any review before merging it.

[arlolra]

It's at the point where I want it now. I'd appreciate any review before merging it.

The changes to build.sh seem reasonable to me. Feel free to merge it if you're happy.

[n8fr8]

Super effort, all!