Freenet build 1492: video, diagnostics, pitch black mitigation, and plugin updates

Freenet 0.7.5 build 1492 is now available. [overview]

This release finalizes the mitigation of the pitch black attack
with a test that shows that the mitigation is effective at
reducing the impact of the attack and recovering from it.
This mitigation was sponsored by nlnet as part of the
Horizon 2020 program of the EU: https://nlnet.nl/project/Freenet-Routing/

The video-player was optimized to prevent most visual flickering.

To help performance analysis, there is now a diagnostics component
with thread runtime information. You can activate it in the advanced core settings of your node.
Thanks to DC*.

And several plugins received updates:

• KeyUtils got bugfixes from TheSeeker
• Library now searches all indexes if you just hit search — thanks to redwerk.
• Library now also allows setting a custom link suffic (docname) — thanks to TheSeeker.
• Spider got bugfixes from redwerk and juiceman.
• FlogHelper provides simple clickable audio and video tags.
  Just put a m3u link into the video tag to have video-on-demand.

Included Bugfixes if fred (the Freenet Reference Daemon):

• skip in support.io.SkipShieldingInputStream must return
  non-negative. Thanks to dennisnez.
• NodeConfig was reading a particular config from default on every
  start up. Thanks to desyncr.
• OpenJDK 16+ compat: add wrapper.conf argument to allow access to
  internal openjdk modules.
• Correct "current size" option in Wizard => datastore size.
  Thanks to AC4BB21B.

Sidenote: What is the pitch black attack?

On Freenet, every node has a location on a circle.

To optimize routing, pure friend-to-friend nodes (no opennet)
swap these locations (without changing connections).
The pitch black attack steals the locations one by one
and replaces them with a single one,
so that after a while all nodes are at the same location.

Then no sensible routing is possible anymore.

The mitigation detects when parts of the circle are devoid of routing nodes
and fills up the holes by switching there.

If you want to help us get better, please help us get the current
pending pull-requests in shape for merging, by reviewing and/or by
improving already reviewed but inactive pull-requests
(only on clearnet, because the pull-requests are there):
https://github.com/freenet/wiki/wiki/Large-Pull-requests-that-need-work

We now also have a list of high-impact tasks:
USK@xPkwhS3czAr5oi6yNgLQ7Ld7RbUZuY6eGniS0n5FZtc,WXVIZrZyLlZugJgJKYvbF9SPfKAUAQX9gAX6g7FGNVE,AQACAAE/high-impact-tasks/0

That said: If someone asks you "what is Freenet", you could for example answer:

Freenet is a peer-to-peer platform for
censorship-resistant and privacy-respecting
publishing and communication.

or

Freenet is a communication system that covers the needs for protection
expected from a secure data broker for Multi-Party Data Exchange in IoT for Health.
(source: https://www.igi-global.com/chapter/using-freenet-as-a-broker-for-multi-party-data-exchange-in-iot-for-health/257911 )

This is what Freenet already accomplished.

And we could only get this far together, because without users,
it is hard for sofware to make a difference.

Thank you for using Freenet!

• Arne Babenhauserheide

Developer changelog:

2021-10-27

Changes in 1492:

Fred:

• The diagnostics component provides thread runtime statistics. Thanks to DC*.
  #727

• The multi-node test to show that the mitigation against the pitch
  black attack works.
  #736

• The m3u-player for video on demand over Freenet now prevents most
  visual flickering.
  #734

• Clickable labels for checkboxes. Thanks to AC4BB21B.
  #741

• Fix: skip in support.io.SkipShieldingInputStream must return
  non-negative. Thanks to dennisnez.
  #737

• Fix: NodeConfig was reading a particular config from default on every
  start up. Thanks to desyncr.
  #739

• OpenJDK 16+ compat: add wrapper.conf argument to allow access to
  internal openjdk modules.
  #740

• Fix: Correct "current size" option in Wizard => datastore size. Thanks to
  AC4BB21B.
  #742

Plugins:

• KeyUtils:
  Fix API call to allow compiling, and
  Fix extra bytes display for composed SSK.
  both thanks to TheSeeker.
  hyphanet/plugin-KeyUtils#17
  hyphanet/plugin-KeyUtils#16

• Library:
  search with unchecked boxes thanks to redwerk
  hyphanet/plugin-Library#14 -

  custom index docname thanks to TheSeeker
  hyphanet/plugin-Library#18

• Spider:
  Fix writing to the bucket and Replace SortedIntSet thanks to redwerks
  hyphanet/plugin-Spider#6

  Fix build thanks to Juiceman
  hyphanet/plugin-Spider#5

• FlogHelper
  Add new media tags in tool menu: audio and video thanks to
  AlexandreRio
  hyphanet/plugin-FlogHelper#19
  (you can simply set an uploaded m3u file as as source of a video
  and it will show up on your flog as video on demand)

related: Florent fixed our website build (Thank you!), so we can release
there again. I polished our theme and the messaging on the index-page.
https://freenetproject.org/

• Arne Babenhauserheide

---

Arne Babenhauserheide (20):
Update FlogHelper to version 36
m3u-player: prevent flickering on track-transitions by keeping the size fixed and overlaying an image of the video
Add real node pitch black mitigation test
pitch-black-mitigation: prevent undamped oscillations of the pitch-black-mitigation
document pitch black node test
fix error-prone clock usage
OpenJDK 16+ compat: add wrapper.conf argument to allow access to internal openjdk modules
wrapper.conf: only add --illegal-access=permit on java 9+
refactor (pull out function) and fix the width for even less flickering
make size preservation robust and flexible
use real sizes for canvas
simplify fixing the size during updates
alwag, always divide by explicit floats
add error handling
fix: supports modules should be true when the threshold version (9) is smaller or equal to the current version
Add note about RealNodePitchBlackMitigationTest.java to the simulator/readme.txt
add a SECURITY.md
note that there are no known unfixable identification attacks against F2F-mode, note sybil and restructure a bit
improve SECURITY.md
SECURITY.md: typo: tipps -> tips

Arne Babenhauserheide (freenet releases) (7):
update KeyUtils to 0.5.6 v5028 from commit 4c0b6e11ac66e781dbcb5a9a1ddabf351629c6b6
update Library to 37 from commit cd45f9a5634c82e693d5fe1a870ad6120b9fe980
update Library to 53 from commit c5367b4ceffb2f9a011d68cae0bb205933fe6033
update FlogHelper to actual 36 from commit 085e4edff9d96c344168bf983c27056a869f5770
Update default bookmark editions
Update default bookmark editions
Build 1492

DC* (56):
Add thread cpu to diagnostics
Add NodeDiagnostic module with NodeThreadDiagnostics
Don't use single class imports
Fix indentation
Don't use single class imports
Create Diagnostics component to be able to scale to support multiple diagnostics easily
Add license headers to new files
Move thread info building into its own method
Make NodeDiagnostics field private
Flatten NodeDiagnostics interface
Flatten NodeDiagnostics interface
Create NodeDiagnostics and ThreadDiagnostics interfaces and default implementations
Reduce visibility for fields in ThreadDiagnostics
Use atomicReference for nodeThreadInfo list
Use thread interval to build data points
Fix calculation CPU time percentage
Use NodeDiagnostics type interface rather than default implementation
Use DefaultNodeDiagnostics implementation
Remove unnecessary finals in contructor
Remove unnecessary throw exception
Remove unnecessary copy
Fix CPU time percentaje calculation
Use Comparator class to simplify threads sorting
Use single loop to calculate delta and display
Rename private field to follow convention
Show percentage cpu time between process threads
Separate presentation from actual data for NodeThreadInfo
Compute % CPU by calculate the total CPU time from all threads, not only nodestat's
Renaming internal variables
Re-introduce delta CPU Time and simplify code structure
Output formatting in DiagnosticToadlet
Handle case when thread.getThreadGroup returns null
Remove long -> double coercion Purge inactive threads from threadsCpu list
Use NodeThreadSnapshot to hold thread list, total CPU and interval
Update ConfigToadlet to support enabling/disabling node diagnostics module
Avoid unnecessary casting to double for getCpuTimeDelta
Declare interface rather than implementation
Clean up nodeConfig callback
Update configuration description
Check thread snapshot is available when displaying
Simplify description and normalize names
Clean up unnecessary space
Correct language and simplify terms
Fix grammar mistake on translation for DiagnosticsDescription
Calculate CPU time as % of wall time
Avoid possible race condition on start up
Add docblock to threadStats method
Fix tab vs space mix up
Remove unneccessary code style fixes
newline at end of line
Create threadSnapshot inner class to avoid pooler executor messing thread'...

build01492-pre1

update FlogHelper to actual 36 from commit 085e4edff9d96c344168bf983c…

build01491: 2021-05-02

Freenet 0.7.5 build 1491 is now available.

This build is a bugfix release.
It ships cleanup to the pitch black defense and to streaming video and audio in the browser.

For the pitch black defense, it disables defensive swapping
if swapping is disabled in general because the node uses opennet.
This should resolve problems to the network structure and data persistence
introduced during the pitch-black fix because
opennet nodes started swapping when they had network-issues (Sorry for that :-().
Also it delays the pitch black defense to 10 minutes after startup instead of 1 minute after startup
to further reduce the probability of stray swapping.
Mitigating the Pitch Black Attack is funded by NLnet through the NGIO PET fund with financial support from the European Commission's Next Generation Internet programme.

Sidenote: If you have a Freenet project, that might be a good match for the Assure grant or the Search and Discovery grant, have a look at their site! The two application periods are open until first of June and a good match for individuals who want to get funding for concrete improvements: https://nlnet.nl/themes/

For streaming, this release fixes operation over proxies that provide a http node via https,
and it adds m3u compability for mobile devices - thanks to DC*
(some mobile browsers answer the query "supports m3u" with "maybe" O_o).

Further improvments in this release:

• Improve OpenBSD support thanks to jv@D7b92AQB2664AkrR.
• Fix plugin compatibility problems from refactoring.
• More cleanup of old code thanks to Jan Gerritsen.
• Fix label for logged attribute RoutingDisabled - thanks to Jan Gerritsen

Thank you for using Freenet!

• Arne Babenhauserheide

Developer changelog:

2021-05-02

Changes in 1491:

• disable pitch black defense if swapping is disabled because the node uses opennet
  this should resolve problems to the network structure
  introduced during the pitch-black fix because
  opennet nodes started swapping when they had network-issues.
  Sorry for that :-(

• start pitch black defense 10 minutes after startup, not 1 minute after startup
  further reduce the probability of stray swapping

• fix streaming: deal with proxies providing a http node via https
  streaming should now work over proxies.

• m3u compability for mobile devices - thanks to DC*
  some mobile browsers answer the query "supports m3u" with "maybe" O_o

• add OpenBSD support thanks to jv@D7b92AQB2664AkrR

• Add compatibility method for plugins (SCGIPublisher needs it)

• recover constructor for compatibility with plugins

• Fixed several unused variables issues - thanks to Jan Gerritsen

• Fix label for logged attribute RoutingDisabled - thanks to Jan Gerritsen

• Arne Babenhauserheide

---

Arne Babenhauserheide (6):
start pitch black defense 10 minutes after startup, not 1 minute after startup
fix: deal with proxies providing a http node via https
disable pitch black defense if swapping is disabled
add OpenBSD support thanks to jv@D7b92AQB2664AkrR
Add compatibility method for plugins (SCGIPublisher needs it)
recover constructor for compatibility with plugins

Arne Babenhauserheide (freenet releases) (3):
Update default bookmark editions
Update default bookmark editions
Build 1491

DC* (1):
m3u compability for mobile devices

Jan Gerritsen (2):
Fixed several unused variables issues
Fix label for logged attribute RoutingDisabled

testing-build-1491-pre1

testing

build01491-pre1

Merge branch 'pitch-black-defense-after-10-minutes' into next

build01490: 2021-03-14

Freenet 0.7.5 build 1490 is now available.

This release provides three different changes:

• streaming in the browser,
• mitigating the pitch black attack, and
• providing the windows-installer from the node again.

For streaming, it builds on the m3u-filter to provide
direct in-browser streaming. Currently
it works with audio (mp3 and ogg vorbis)
and video (ogg theora) in sufficient quality for
music playlists and specially crafted video-streams.

With this change, adding a media-tag that references an m3u-list like

<audio src="stream.m3u" controls="controls">
not supported?
</audio>

or

<video src="stream.m3u" controls="controls">
<a href="stream.m3u">stream.m3u</a>, starting with <a href="sff-001.ogv">sff-001.ogv</a>.
    Putting the first video here causes freesitemgr to more likely
    put the first video into the container, giving faster initial startup.
</video>

In Sharesites you need to add at least one space before the tag.

There are three example sites:

• Audio-Playlist: USK@1wpsnrzb9fiFmmq6OozoZ0Mnk376AlizUKjKPh3TXwM,CwfRO8X0xaCUzH28eoHigxUV-5TkRV8hCoQdO2kmsQM,AQACAAE/streaming-over-fproxy/3/
• Streaming Video: USK@UGh1rxaHczJVr4k4LwxWrxFBc-Dt5P0F3IuPONpp8ZQ,klWF5g9B2PLst8tRO8c9tzk6XvisGynFVJRzM8-9718,AQACAAE/stream-36c3-sff/3/
• Radio Interview: USK@4DQ15JpGlVGDdyXvQE3Egz7SLK2TzMAUmp~aptnwyt4,ljFASreV8AHaQhscfrNLuVyl3qksltgP9sndtLuUHB8,AQACAAE/stream-radiocc-freenet/1/

This follows an old idea to enhance freesites with strictly checked interactivity,
similar to userscripts but without the security implications of adding a script
that might have a very different threat model than Freenet itself.

Also this release ships a mitigation for the pitch black attack.
For details see https://www.mail-archive.com/[email protected]/msg55182.html

Mitigating the pitch black attack is supported by NLnet
as part of the NGI0 PET Fund.
For details see https://nlnet.nl/project/Freenet-Routing/
Thank you!

There are still more tests needed to ensure that the change suffices
to resolve the main objection that existed among people well-versed
in the technical background of Freenet, GNUnet, and others
against using the friend-to-friend mode of Freenet since 2007.

It used to be far too easy to break a pure friend-to-friend net
Arne Babenhauserheide
(all nodes using high-security mode). This should be resolved now.

And finally the UOM transfer max sizes were tripled,
so the windows-installer provided by Freenet nodes gets updated again,
and the annoying warning about freenet-latest-installer-windows.exe
during install and update disappears.

Thank you for using Freenet!

Arne Babenhauserheide

Developer changelog:

2021-03-14

Changes in 1490:

• inject vanilla Javascript m3u-player into Freesites
  to get in-browser playlist streaming (with config to disable if needed).
  Thanks to nextgens and desyncr for the review!
• mitigate the pitch black attack following
  https://www.mail-archive.com/[email protected]/msg55182.html
  Thanks to nextgens and x for the review!
• triple the UOM transfer max sizes
  so the windows-installer provided by nodes gets updated again.
  Thanks to desyncr for the review!

---

  [m3u-player] inject vanilla Javascript m3u-player into Freesites
  [m3u-player] adjust ContentFilterTest: header with script-source
  experiments in location manager, do not compile
  start experimenting with highlevelsimpleclient
  Implement initial untested pitch-black defense
  [m3u-player] prefetch the next three tracks to smooth segment-sizes
  pitchBlack: add intra-day persistency and cleanup
  pitchBlack: fix: decode yesterdays file content as base64
  improve stats (missed commit)
  pitchBlack: also secure the CHK store
  Choose between SSK und Pubkey at random
  create a local variable to hold LocalDateTime.now() and queue next job at beginning to avoid try-block
  use secure random (CSPRNG) for the KSK filename
  cleanup leftover removal: if deletion failed, use deleteOnExit, and only listFiles once.
  replace base64 encoding of file contents by plain UTF-8 encoding.
  only interpret DNF as isRequestExceptionBecauseUriIsNotAvailable (= must swap)
  change prefix of status files
  fix imports
  skip UTF_8 and bufferedWriter: we’re dealing with raw bytes.
  fix imports
  add config option for the m3u-player embedding
  add l10n for config option for the m3u-player embedding
  triple max size for update file transfers to avoid errors due to the larger windows installer
  use constants for the max sizes over update
  Update default bookmark editions
  Update default bookmark editions
  Build 1490

testing-build-1490-pre1

• Add inline m3u-player for streaming over Freenet right from the browser
• Add fix for the pitch-black attack.

Install via new_installer_offline_1490-pre1.jar (on GNU/Linux) or FreenetInstaller-1490-pre1.exe (Windows).

You can also test it, by replacing freenet.jar and freenet.jar.new by the released freenet-testing-build-1490-pre1.jar in an existing Freenet install (stop the node first!).

build01489: 2020-12-15

Freenet 0.7.5 build 1489 is now available.

This release adds an m3u-filter that enables safe access to m3u-playlists.

Together with the existing support for media files
(Audio using mp3, Ogg Vorbis, or Flac, and video using Ogg Theora),
safe playlists make Freenet viable as streaming platform:
You can create an m3u-file and others can
play it right from their media-players (i.e. mpv or vlc).

On the network layer this release adopts
the one safe release from the let-it-burn patchset:
Maximum backoff-times are reduced from 3 hours to 8 minutes.
This might increase bandwidth available.

Also progresspage.js now has a declared license for Libre-JS
and the checkall script was rewritten.

We hope you enjoy using Freenet!

• Arne Babenhauserheide

Developer changelog:

2020-12-15

Changes in 1489:

• Add m3u support with mp3, ogg, oga, ogv, and flac. Thanks to Bombe for reviewing!

• Reduce max backoff times from 3 hours to 8 minutes;
  the one safe change from the let-it-burn patchset.
  This might increase bandwidth available. Thanks to TheSeeker.

• add explicit license for Libre-JS to progresspage.js

• rewrite checkall to match the class for all input elements,
  add explicit license for Libre-JS

• Arne Babenhauserheide

---

Arne Babenhauserheide (46):
prefix links in m3u-files with host+port of the node to allow for local playback with arbitrary keys.
rewrite checkall, match the class for all input elements, and add explicit license for Libre-JS
add explicit license for Libre-JS to progresspage.js
cleanup whitespace
use host and port from request, storing it in the fetch context
FIX: did not skip empty lines terminated with CRLF.
preserve final linebreak on Windows
actually fix merge errors
fix merge error
host→private hostAndPort, otherParams→otherMimeTypeParams, inline
Allow prefetching of inline mp3 and ogg
Remove unused constants
get scheme, host, and port for proxies.
Fix readability
adjust test to use scheme host and port
Fail m3u test on missing files
add spaces around +
remove unnecessary casts
remove single-use constructor again
remove outdated TODO
Use the right header and header-field for host
First split Forwarded header by , then parse only the first entry.
Remove non-existent parameter
fix comment for schemeHostAndPort
check headers against whitelist from fproxy.bindTo and proxy.port
move schemeHostAndPort prefix parsing to easily testable class
add tests and correctly treat default values
simplify: filename in exception
check against null
code-style: braces and spaces in FetchContext
license: GPLv3+ → GPLv2+ (what it should have been)
kill stray newlines
hostAndPort → schemeHostAndPort
ContentFilter: Initialize earlier to simplify code
kill unused variables and imports, outdated or misleading comments
parametrized tests to named test methods
Get Config settings as Options to benefit from the defaults
Do not break line unnecessarily
Move scheme host and port parsing to static parse method.
Fix tests: Still need port fallback.
simplify UriFilterProxyHeaderParser: simple static method
bubble exceptions to test failure
provide default implementation for the previous readFilter interface
deprecated compatibility interface and moved from HashMap to Map
also migrate JPEGFilter.readFilter from HashMap to Map
Add one more compatibility function and deprecate it.

Arne Babenhauserheide (freenet releases) (8):
Update default bookmark editions
update NEWS.md
style: space-comma to comma-space
limit travis checks to Java in debian stable, Java LTS and the most recent GA version.
Update default bookmark editions
Update default bookmark editions
Update default bookmark editions
Build 1489

DC* (1):
Add support for Android VM

TheSeeker (1):
Reduce max backoff times.

babenhauserheide (4):
Add failover for FetchContext reading of old saved requests
Remove parsing of Forwarded header again to reduce the scope
Fail test with exception on first missing file
Remove unused imports

testing-build-1489-pre1

1489 test release

build01488: 2020-11-08

Freenet 0.7.5 build 1488 is now available.

In this build we only updated translations to make Freenet more inviting:

• add Hungarian
• update French

A big thank you to our team at Transifex!

Thank you for using Freenet!

• Arne Babenhauserheide

Developer changelog:

2020-11-08

Changes in 1488:

In 1488 we only updated translations to make Freenet more inviting:

• add Hungarian
• update French

A big thank you to our team at Transifex!

• Arne Babenhauserheide

---

Arne Babenhauserheide (freenet releases) (5):
update french translations — thanks to our transifex team
add hungarian translation, thanks to the transifex team
update NEWS
Update default bookmark editions
Build 1488