Re-upstreaming and maintainership of the KGPE-D16 - Cost: 30 000E and counting

[tlaurion]

There was a proposition to do required work under refused grant application. If there is interest from third party to fund conjointly needed work, board could be maintained and reupstreamed, under u-bmc, coreboot and under Heads.

Edit:

• Proposition from 3mdeb here

• History here

• Possible funding path here

  • EDIT: Funded by Immunefi Re-upstreaming and maintainership of the KGPE-D16 - Cost: 30 000E and counting #719 (comment)
  • Work plan: Re-upstreaming and maintainership of the KGPE-D16 - Cost: 30 000E and counting #719 (comment)

• Trail of 4.11 reported problems: Re-upstreaming and maintainership of the KGPE-D16 - Cost: 30 000E and counting #719 (comment)

Last updates, newsletter, code source drop : #719 (comment)

[tlaurion]

Traces:
u-bmc
kgpe-d16 actual board status

This board works as a QubesOS server while work is still needed.
Freshly landed QubesOs-network-server code

A followup on #717 (comment)

[Tonux599]

@tlaurion did 3mdeb decide not to take it on? It would be good to have as it's the only completely blob free board I can think of that's qubes compatible. Provided a 6200 series is used.

[tlaurion]

@Tonux599 3mbeb is absolutely willing to do the work.

[tlaurion]

@Tonux599 @mikebdp2

NlNet decided to not take it on (by funding the project's work).
Q: Will we decide to take it on and make it happen? Let's see. :)

[tlaurion]

Applicable?

[tlaurion]

@pietrushnic

[pietrushnic]

@tlaurion sorry, for not replying, recently we have quite a lot on my plate. Definitely, we would like to publish here documents that we sent to NLNet while applying for a grant. There is also a slightly modified version that we provided to Insurgo on Slack. Both documents rely on coreboot state in December 2019. @miczyg I think you can push grant content to 3mdeb GitHub repo as PR and link here for review.

[miczyg1]

This Pull Request contains the application 3mdeb sent to NLNet to obtain the grant:
3mdeb/kgpe-osf#1
Feel free to comment and review.

[Tonux599]

@pietrushnic @miczyg1 When should we expect an outcome? Also could you briefly outline any ways that members of the community could support you if or if not you get funding?

[pietrushnic]

@Tonux599 there is no timeline since there is no official commitment from anyone to fully found the effort. We get some good words from various individuals and organizations, but that's all. @tlaurion mentioned that he can sponsor part of the effort, but we are not sure anyone would be happy with half-finished stuff. Vikings Gmbh sent us hardware so we have something to work on. NLNet did not approve of this grant, so I would not expect it will change in the future - I may be the wrong here. To sum the status of this project is "not founded" so we even do not put that on schedule and worry more about founded effort e.g. TrenchBoot and fwupd for Qubes OS.

How the community can help? We are very open here, whatever works for the community.

1. We can split this project and drive as an open and free contribution if there are developers and testers who willing to support the development and debugging. In such a situation we will support whoever will do that as much we can including, coordination review and help in merging code upstream.
2. We can open any channel that the community is comfortable with to try to gather remaining founds. @tlaurion suggested OpenCollective, we can align probably to any suggestion - we have no problem in handling the administrative part.
3. We have a PayPal donation button on our website, although this is probably one of the worst options.
4. We are fine with commercial assignments if anyone is willing to go that path. It can be even a very small part of the task, but at least it will move us forward.
5. If we can sell a platform in some form or bundle services, which will generate margin, then we can use it to bring back the platform to the mainline.

We would like to help, but without funding, we will do the best effort and it may take very long to bring back this platform. Please note our goal is not just to bring back the platform to mainline, but create a structure in which long term maintenance would not cost a lot. We did that for PC Engines routers for over 4 years.

[tlaurion]

@Tonux599 : For the moment, step would be to get people interested in the port. Let's remember that the KGPE-D16:

• was funded by Libreboot and put upstream by RaptorEngineering but previous work did not receive enough attention to be maintained and got dropped in coreboot 4.11 release for lack of maintainership.
• the community crowdfunded the port of the BMC chip to OpenBMC and that work wasn't maintained.
• NlNet refused to fund the work because they didn't understand the reasoning of funding old platform.

So the point here would be first to get enough attention so that the story doesn't repeat itself.
Insurgo is interested to fund a big part of the work, but we concluded that we would not do this alone.

Some history:

• FOSDEM talk by 3mdeb on AMD coreboot status
• Reddit thread on announced drop of the KGPE-d16

Putting a bounty tag for this.

[tlaurion]

Something really interesting was proposed on this QubesOS ticket:
QubesOS/qubes-issues#2809 (comment)

@pietrushnic, I would love to hear on your reception.
This would basically mean that the independent tasks could be independently funded upon PR acceptation. Not sure how to orchestrate this.

We are currently experimenting with OpenCollective, while the fees are really high, which makes the move less interesting (a lot of money being lost in translation).

Other options might be more interesting, like Whonix did.

So the question here being mainly:

• How to raise attention toward the need of this board across technical communities who would need this RYF platform internally. Examples: self-hosters (Ex: younohost clients)
• How to manage donation and reduce its processing fees.
• How to organise those tickets (metatickets with daughters tickets for tasks? Broader?), and where so that money is raised for work to be done directly.

See this ticket to see how chicken/egg problems can last for a year even when money is available without a clear plan.

This is the inverse problem here, just like funded work normally work.
Tasks here are defined, costs attached for work to be done is quantified with a fair price.

[pietrushnic]

@tlaurion XVilka suggestion looks interesting, but definitely key problem is orchestration. We would like to avoid the situation when 3 or more teams work on a given feature in parallel and then post results claiming the founds. There should be serious commitment based on proven reputation and trust, otherwise, we have public bidding problem where lower price wins with quality.

Whonix stuff is even better, but we would have to research how it looks from the taxation and administration perspective.

[tlaurion]

@pietrushnic I'm not sure there is conflict between the two ideas with XVilka avenue.
If the ticket is clear saying that 3mdeb realizes the work in question (milestones) and funds are released upon proof of work (PR).

But you are better placed then I am to see if this would work or not for 3mdeb.

[mikebdp2]

@tlaurion and @pietrushnic : Have you contacted a Free Software Foundation? Since they have a lot of interest in RYF devices and are probably using these KGPE-D16 servers by themselves, if they'd learn about this critical situation - maybe they'll decide to co-fund?

P.S. also, some notes at #717 may be relevant.

[pietrushnic]

@mikebdp2 we (as 3mdeb) never contacted FSF. I'm not sure if the situation is critical but it will get worst over time. From experience longer platform lays unmaintained bigger would be effort to bring it back. Do you know anyone who we should reach?

@tlaurion I believe you are in a better position to talk with FSF, then 3mdeb.

[pkubaj]

22000E isn't that much different from what Raptor wanted for OpenBMC port (at https://www.raptorengineering.com/coreboot/kgpe-d16-bmc-port-offer.php). Maybe you should set up similar crowdfounding? Currently you only say "we are ready to work" etc. Raptor did the same and received no money, but when they officially started crowdfounding, it turned out there were enough people interested in it to gather the money.

[pietrushnic]

@pkubaj very good point. It looks like Martin was intermediary gating the process. In case of difference in pricing, please note this was a different time and they are in the US not in PL, their costs are way higher than ours. More to that we are Yocto Participants and have Embedded Linux people on board, OpenBMC uses Yocto/OE build system, so this also simplifies the situation for 3mdeb.

[tlaurion]

@mikebdp2 we (as 3mdeb) never contacted FSF. I'm not sure if the situation is critical but it will get worst over time. From experience longer platform lays unmaintained bigger would be effort to bring it back. Do you know anyone who we should reach?

@tlaurion I believe you are in a better position to talk with FSF, then 3mdeb.

@pietrushnic writing to them.

[alexmaloteaux]

@Tonux599 : For the moment, step would be to get people interested in the port. Let's remember that the KGPE-D16:

* was funded by [Libreboot and put upstream by RaptorEngineering](https://libreboot.org/docs/hardware/kgpe-d16.html) but previous work did not receive enough attention to be maintained and got dropped in coreboot 4.11 release for lack of maintainership.

* the community crowdfunded the port of the[ BMC chip to OpenBMC ](https://www.raptorengineering.com/coreboot/kgpe-d16-bmc-port-offer.php)and that work wasn't maintained.

* NlNet refused to fund the work because they didn't understand the reasoning of funding old platform.

So the point here would be first to get enough attention so that the story doesn't repeat itself.
Insurgo is interested to fund a big part of the work, but we concluded that we would not do this alone.

Some history:

* [FOSDEM talk by 3mdeb on AMD coreboot status](https://fosdem.org/2020/schedule/event/coreboot_amd/)

* [Reddit thread on announced drop of the KGPE-d16](https://www.reddit.com/r/linux/comments/dz2hlt/boards_like_asus_kgped16_the_most_powerful/)

Putting a bounty tag for this.

Hi is there a list of the bugs/linux crash that Michal talk about in his fosdem talk ? Do they only happen when iommu is enabled ?

Btw i have a kgpe-d16 with jtag header soldered and an olimex. So if you are starting any group for dev , put me in, free time is limited but i can always help a bit and test.

[tlaurion]

@alexmaloteaux asked:

Hi is there a list of the bugs/linux crash that Michal talk about in his fosdem talk ? Do they only happen when iommu is enabled ?
@miczyg1 ?

[miczyg1]

Simply have a look at the coreboot mailing list:

• https://mail.coreboot.org/hyperkitty/list/[email protected]/thread/ME4TULTDZGIYJEMNR2IB2W6P7ONG76EU/#GCJF7CIWGCKZCDWYPWAM2LKQW6APHZRJ (memory problems)
• https://mail.coreboot.org/hyperkitty/list/[email protected]/thread/ZR7LMBXOFKR5FWVQDI22UB6O3LO5WGQ3/#TH57VEGCY357ZH534PU5PKSDSMYLTLFH (S3 suspend/resume problems)
• https://mail.coreboot.org/hyperkitty/list/[email protected]/thread/VWFKH7FDWLO5KGZMIJ7VM67SIUWJRVZL/#OCHSI6UL2XZDSA3HCHOONNN5EX4HLUGE (kernel panic from new microcode)
• https://mail.coreboot.org/hyperkitty/list/[email protected]/thread/H5KHOUVDPQA2NSUE43IBRKSYBZQK4DWN/#H5KHOUVDPQA2NSUE43IBRKSYBZQK4DWN (SLUB memory allocator in Linux panics)
• https://mail.coreboot.org/hyperkitty/list/[email protected]/thread/RDHEDOTBYVD74NWQWYYNLAJWPD7CMDRZ/#RDHEDOTBYVD74NWQWYYNLAJWPD7CMDRZ (KVM/SMV not working)
• https://mail.coreboot.org/hyperkitty/list/[email protected]/thread/AW4DF557YDTTJGKDFRTHMZJBTF65ZJZT/#VOFCIRV24HT3OIG5IDYH4GEXY5VSS3W5 (kernel panics on kernel newer than 4.9)

[pietrushnic]

In case of what have to be done I believe Arthur email is quite good. @tlaurion you replied to it here

[lss4]

Ugh, that's really strange. I use USB input devices just fine and can boot from USB just fine (but I don't use Linux, maybe that's why). The coreboot-provided Memtest86+ does report errors right after starting, but AFAIK this is known issue on this board and if you use Memtest86+ booted from another source, it's apparently fine.

I also don't use Heads, since it only supports Linux.

Something else to mention.

1. USB worked while in SeaBIOS so I could use it to select a boot option. It also worked in boot menus like GRUB. However, that's all where it could work. After I booted into something (like MenuetOS), USB input devices stop working completely, so I have to use PS/2 input devices from that point on.
2. I tried booting some USB sticks containing Linux distros. They all failed before reaching the phase to load the kernel.
3. Memtest86+ would freeze regardless of the form: as a coreboot payload, as a floppy image payload, or from USB stick (such as those shipped with Linux distros).
4. I also tried booting some DOS floppies images as payloads (with the help of Mike Banon's patches), and they didn't work either (the system would hang at early stages, before being usable). Don't know if the issue is caused by memory not being initialized properly, or that SeaBIOS is not behaving in a way some DOS kernels expect (so far no DOS images are working). I'm using a 16MB (128Mbit) flash chip for coreboot by the way.
5. So far I only managed to get MenuetOS (as floppy image payload) booted to a usable state, just without USB keyboard/mouse.

I don't know about your use case though, and I'm not really sure about the status of USB. For me it definitely was problematic.

[pietrushnic]

Hi @pietrushnic, what I meant was: if you are looking for a baseline version to base the ram code off, make sure you choose a commit that was boot tested on actual hw otherwise you may introduce a bug that will be hard to find and may have been already fixed in a different coreboot commit.

Yes, this is fair point. I believe @miczyg1 can say more about his approach to selecting correct code base that will be bring up upstream. There is some limit to number of memory configurations we can test, but we will get to call for beta testers at some point I believe.

[pietrushnic]

I also don't use Heads, since it only supports Linux.

@pkubaj petitiboot is able to kexec FreeBSD so why not Heads :)

[pkubaj]

petitboot is able to kexec FreeBSD kernel, but this still has some issues.

1. Skiroot uses kexec-lite from https://github.com/open-power/kexec-lite which is not the kexec most distros use, so there may be some differences,
2. FreeBSD uses its own loader on amd64, but Petitboot can't load loader. It loads just the kernel, which means we can't easily get ZFS on root on powerpc64* systems (unless one builds zfs module in the FreeBSD kernel). It's also not possible to load any other module at boot time (from /boot/loader.conf) or change kernel variables writable at boot-time (one can work it around by adding them to the kernel command line, but I wouldn't call that the FreeBSD way).
   Having to load directly the kernel has one big disadvantage - it's not possible to have FDE. /boot needs to be unencrypted, to read kernel. When using FreeBSD loader, it's possible to have the 1st stage loader in the MBR, which then decrypt the HDD and chainload another loader. This is not possible with loading kernel directly.
   On POWER systems, there's also the issue that /boot needs to be FAT32, because Skiroot's Linux can't read FreeBSD's UFS or ZFS...

[pietrushnic]

@macpijan cc

@pkubaj those are interesting insights, but it seem so far there was no commercial interest in fixing above issues. We are little bit off-topic here, but maybe 3mdeb at some point could fix those issues.

[pkubaj]

@pietrushnic
The Talos board I have at home was bought (along with the CPU and some RAM) by FreeBSD Foundation for me to work on FreeBSD Ports on POWER.
I explicitly requested a grant from them and it got approved. It took quite some time to get it, but I THINK it would be possible for you to request a grant to work on FreeBSD on POWER (kernel / OS side). It's another thing whether that gets approved, though. But I'd be happy if there were more OS devs for POWER on FreeBSD, since while on the ports side we're pretty good (Firefox, Libreoffice etc.), we're pretty much behind in terms of OS / kernel (no DRI, no bhyve, no loader and performance is quite behind Linux - even my KGPE-D16 can compile things faster than Talos on FreeBSD).
Anyway, the point was that you MAY be able to get some sponsorship from FreeBSD Foundation regarding POWER.

[pietrushnic]

@pkubaj thank you, we will look into sponsorship of FreeBSD Foundation.

[SergiiDmytruk]

Request for anyone interested in using KGPE-D16 with write-protection of flash chip: if your chip is not already listed in flashrom/flashrom#185, please post a comment there stating chip's model.

[macpijan]

In terms of the October deliverables:

• we plan to deliver Phase 2 as planned
• we plan to deliver Phase 3 except for the IMM1.6 - KGPE-D16 firmware shall support resource allocator v4 task - we are still working on that and plan to move it to Phase 4 for November instead

We plan the Phase 2 + Phase 3 release for this week.

[pietrushnic]

@macpijan do you think it make sense to ask Libreboot community for testing?

[githubisnonfree]

hi. leah here. i'm currently in the process of building myself a d16 again, precisely because of your efforts. i will be available for testing. Tashtari in libreboot IRC also has D16 hardware and is quite enthusiastic about your work.

When your work is ready, I wish to integrate it in lbmk, which is libreboot's build system. It can download coreboot, applying any number of custom patches desired. lbmk currently uses standard coreboot 4.11

edit: iirc Bardon also has D16 hardware. also, the FSF has a testing station.

also:

FSF deploys all their hosting on a D16, but has a special D16 test stand. An entire spare D16 just for testing configurations on. Contact the sysadmin team at the FSF. Ian Kelling (iank on #fsfsys IRC) is very interested in your work, and is a current sysadmin at the FSF.

[tlaurion]

Will buy a tpm2 module and can test myself Heads for tpm1 and tpm2 measured boot configuration support, for both workstation board with AMD graphics and BMC for server board.
PS: Would love to see the u-bmc needed work being a stretch funding goal? @pietrushnic: How much would that u-bmc for ast2050 chip additional work cost?

I will also try my best to bring a u-root + Heads board under Heads (being coreboot based as opposed to linuxboot based), encompassing the cpu toolset (@rminnich: amazing work!!! https://github.com/linuxboot/book/blob/master/cpu/README.md#the-u-root-cpu-command ), since I intend to use that board as my local build system and use those 32 cores and 64gb of ram collecting dust.... And generate heat in the cold weather here while doing so.

The kgpe-d16 in current state works already as a Qubes 4.1 platform, while some pushing would be needed to have network-server project cascade forwarding of external traffic to specific qubes(@Rudd-O) coming from hidden services down to their actual offered service provided in services qubes, as well as being able to provide some yunohost or equivalent services locally. That's where I was in the past, and where I personally want this to go forward.

I will reenable that machine that sleeps for way too long.

[macpijan]

do you think it make sense to ask Libreboot community for testing?

It makes perfect sense, but maybe more after a fourth phase (somewhere early December) when we can actually boot something.

How much would that u-bmc for ast2050 chip additional work cost?

I haven't really used u-bmc, but if it also needs U-Boot and Linux for AST2050 (which I assume it does) then most of the work between OpenBMC and u-bmc would be common. We have presented our view on that in this presentation: #719 (comment) (starting with slide 13).

I guess you could also try to use the existing kernel, but you would be stuck with the 12 years old kernel running on your BMC then.

[pietrushnic]

FYI v0.1.0 was released: https://docs.dasharo.com/variants/asus_kgpe_d16/releases/

Please note you can join newsletter to get notifications.

Please let us know what we can improve.

[pkubaj]

Are there plans to push this work to upstream or will it be maintained as a fork?

[pietrushnic]

@pkubaj please take a look at mileston 7. In short we definitely plan to upstream all code that would be accepted by upstream, but let's be honest timeline of that is unknown.

[owlshrimp]

A heads up: Over in the mainline they're looking at possibly dropping quite a large number of AMD platforms (fam 14h, 15h, 16h?) if they can't move over to the newer resource allocater (though for 15h there may be some early PoC?). It looks like this work will need to support the new resource allocator in order to be merged.

[macpijan]

The allocator v4 was in scope of this project and is being worked on already. There are still a couple of more features required in this last deprecation call (like the PARALLEL_MP) which were not part of the initial plan. We will do our best to implement it as part of the upstream effort, though.

[awokd]

Is there a separate issue somewhere for LEGACY_SMP_INIT removal/PARALLEL_MP? I've been looking into it a bit as it relates to fam14-16 deprecation.

[pietrushnic]

@awokd we all have to be professional in light of what is going on mailing list 1 2. We tried to present 3mdeb position at coreboot leadership. We will work closely with community to avoid what was proposed. We also consider various backup plans, if you are intrested in discussing those please join Dasharo Matrix workspace

[Rudd-O]

The kgpe-d16 in current state works already as a Qubes 4.1 platform, while some pushing would be needed to have network-server project cascade forwarding of external traffic to specific qubes(@Rudd-O) coming from hidden services down to their actual offered service provided in services qubes, as well as being able to provide some yunohost or equivalent services locally. That's where I was in the past, and where I personally want this to go forward.

I believe Qubes network server is already available for r4.1 and I know I tested it on my throwaway laptop before releasing it on Github:

https://github.com/Rudd-O/qubes-network-server

[tlaurion]

@Rudd-O I just saw new commits since last time I checked. Didn't know cascading was implemented, will test and sorry for the noise if it was implemented !

[Rudd-O]

Cascading is not explicitly implemented in Qubes 4.1, although the current implementation may just support it. Please feel free to try!

[tlaurion]

Some trails on 4.11 problems :

• gathering of bug reports/limitations from miczyg Re-upstreaming and maintainership of the KGPE-D16 - Cost: 30 000E and counting #719 (comment)
• Paul Menzel: Re-upstreaming and maintainership of the KGPE-D16 - Cost: 30 000E and counting #719 (comment) "Random boot failures, all of memory not supported, higher energy usage compared to vendor firmware, problems loading latest microcode"
• Recap on current status by myself on 4.11: Re-upstreaming and maintainership of the KGPE-D16 - Cost: 30 000E and counting #719 (comment), when TPM and coreboot patches were added to Heads by tonux : Re-upstreaming and maintainership of the KGPE-D16 - Cost: 30 000E and counting #719 (comment)
• Ram init issues: Re-upstreaming and maintainership of the KGPE-D16 - Cost: 30 000E and counting #719 (comment)
• Maximum memory support being theoretically 256GB, but tested? https://www.raptorengineering.com/coreboot/kgpe-d16-status.php https://libreboot.org/docs/hardware/kgpe-d16.html#memory-compatibility-with-libreboot

That is what I could grab from skimming both my own memory and traces.
Anyone has other problems still current on 4.11 to report, or other places we might have missed at this point?
@pietrushnic @Th3Fanbus @githubisnonfree ?

[githubisnonfree]

neox on irc reported raminit issues on their board; works in coreboot 4.6 but not coreboot 4.11. i'm having that person do a git bisect to find when coreboot broke, between 4.6 and 4.11, and will report this to you all when it is found

…

On Thu, 16 Dec 2021 06:40:48 -0800 tlaurion ***@***.***> wrote: Some trails on 4.11 problems : - gathering from miczyg #719 (comment) - Paul Menzel: #719 (comment) "Random boot failures, all of memory not supported, higher energy usage compared to vendor firmware, problems loading latest microcode" - Recap on current status by myself on 4.11: #719 (comment), when TPM and coreboot patches were added to Heads by tonux : #719 (comment) - Ram init issues: #719 (comment) - Maximum memory support being theoretically 256GB, but tested? https://www.raptorengineering.com/coreboot/kgpe-d16-status.php https://libreboot.org/docs/hardware/kgpe-d16.html#memory-compatibility-with-libreboot That is what I could grab from skimming both my own memory and traces. Anyone has other problems still current on 4.11 to report, or other places we might have missed at this point? @pietrushnic @Th3Fanbus @githubisnonfree ? -- Reply to this email directly or view it on GitHub: #719 (comment) You are receiving this because you were mentioned. Message ID: ***@***.***>

-- Leah Rowe ***@***.***> Company Director & Libreboot developer Do you know you have rights? The right to privacy, free speech, the right to read and the right to learn. Defend freedom. Use free (free as in freedom) software. Spread freedom. Tell everyone you know about it! https://www.gnu.org/philosophy/free-sw.html Minifree Ltd, trading as Ministry of Freedom. Registered in England, registration No. 9361826 VAT Registration No. GB202190462 Minifree Ltd, 19 Hilton Road, Canvey Island Essex SS8 9QA, United Kingdom United Kingdom

[githubisnonfree]

by the way i'm not sure if this is addressed but:

is the re-upstreaming work also being done on kcma-d8? the d8/d16 codebases are very similar in coreboot, and i think it would be worthwhile.

edit:
also, in coreboot 4.11 there are other fam15h/10h boards. what will become of those? i think those other ones lack onboard graphics chips but they could be nice to re-integrate, potentially

[macpijan]

being done on kcma-d8? the d8/d16 codebases are very similar in coreboot, and i think it would be worthwhile.

It is not part of this effort. But once KGPE-D16 code is refreshed, it would be much easier.
It was evaluated, but did not fit into the scope. See slide 23 in the presentation: #719 (comment)

[tlaurion]

@Tonux599 @macpijan should board configs and coreboot point to latest stable branches of 3mdeb as of now?