Proposal for a Deluge companian tool platform

[PaulFreund]

In order to give users a good out of the box experience and at the same time giving tool developers a toolbox that makes it easy to add features I propose an application platform to interact with the Deluge that can be developed as a joint effort by the community.

Requirements

• Works on all major platforms: Windows, Linux, OSX, Android, iOS
• Works with an USB cable only
• Must allow at least the following features
  • Deluge related
    • Update the firmware based on stored images
    • Transferring files from and to the Deluge
    • Show display state and switch between 7SEG and OLED
    • Interacting with the project state (value transfer etc.)
    • Display and capture of Debug information
    • Interaction automation of the deluge inputs
  • Platform related
    • User Login
    • Load and Store of preset/song files
    • Store and display of debug logs
    • Accept new firmware builds and metadata from CI system
    • Provide analytic information about images like code size / performance over time
• Low level of entry for developers
  • As few technology stacks as possible
  • Simple and easy to use toolchain
  • Provide high level APIs and abstraction of complex concepts
  • Development can be done on any major PC platform (Windows, Linux, OSX)
  • Development and testing can be done on dev PC and does not require additional infrastructure

Proposal

My proposal is to base the platform on web technology as that can be adapted to a lot of situations and works out of the box on most. I have looked at a lot of the currently available solutions and have built similar applications in the past so I'm very confident this will work well.

• On Desktop all requirements can be fulfilled out of the box with a server side REST API, WebMIDI and Filesystem API in supporting browsers
• On Mobile for all features to work we will most likely require an app host like Capacitor but the rest of the application should be able to handle the platform difference transparently
• Backend
  • Built with Node.js, modern JavaScript with ESM. As few dependencies as possible
  • SQL based database backend supporting both sqlite files and different options of hosting providers like AWS
  • Serves frontend files
  • Authentication provided by GitHub since most open source users might already have that
  • Reasoning: Node.js uses the same language as the frontend, the same code could be packaged as an Electron application and it is easy to setup and deploy. SQL is the most universal in terms of databases and there is a huge list of servers/implementations to pick from. The data will mostly work in all of them
• Frontend
  • Modern JavaScript with ESM and CDN-able libraries, no compilation required
  • Vue.js as frontend library with inline templates and without compilation (example)
  • PrimeVue as a component library as they are extensive and well supported
  • Deluge interaction in a separate Javascript library as a submodule from the SynthstromAudible organization
  • Reasoning: I evaluated most recent frameworks and Vue.js can be used without any compilation, has two way data bindings, the code mostly looks like normal HTML and JS and is also used by many big players. I created a very simple of the proposed structure here, of course the application will be split in multiple files.
• Hosting
  • Repository in the SynthstromAudible organization
  • Domain provided by Synthstrom, for example https://deluge.synthrstrom.com/
  • Hosting provided by Synthstrom as they already have an AWS account, see Node.js on AWS
  • Deployed via GitHub Actions CI
  • Reasoning: Mostly work with what we already have, switching should be very easy if required

Implementation

I have made this proposal because I think an application like this will be good for the project and I have chosen the technologies I think are easy to approach, modern and a good fit to our requirements.

I am willing to put in the effort and work on the application and platform as long there is broad support for this proposal by people wanting to work on tooling and there won't be a major rewrite to change the technology a few months in. If the majority of the community wants to go into another direction I am happy to drop this plan and participate in something else.

[m-m-adams]

For deluge interaction, what about rust+wasm? I've been looking for a reason to try it for a while

[sapphire-arches]

I have a fair amount of experience using Rust+WASM for building web interfaces for debug tools. It's a bit rougher around the edges than the JS based flows but is somewhat serviceable, and I would probably have more fun hacking on that than javascript =P.

[PaulFreund]

While it is a cool idea it would significantly increase the barrier if entry. Right now the only thing that is needed is a text editor and the vanilla text stack knowledge. Rust+WASM introduces a whole additional stack

[sapphire-arches]

I would strongly recommend we implement a build step - it can be annoying but similar to the reasons we have automatic build and test checks for the main firmware, having some amount of automatic QA on the site and JS library is invaluable. A build step also lets us use typescript which is IMO a significant upgrade from Javascript (slightly fewer weird behaviors, more warnings when you're about to blow your feet off).

[PaulFreund]

I am fully fine with contributing to a read made environment where I know the general technology and there is example code so I know what is idiomatic but I don't have enough time to really understand all those stacks and make the right decisions. Since there is so much consensus on fatter Toolchains I will wait for someone else to pick up the task and contribute where possible, especially general architecture. I'm on Discord in case someone wants to discuss something :)

[MrHaila]

One new release that might be of interest is Bun, a new Node.js runtime alternative: https://bun.sh/

The relevance here is that the Bun runtime absorbed the most commonly used dependencies from the Node toolchain and made them faster/cleaner/zero config. Using Bun instead of Node would make it feel like there are fewer tools and complexity. I've poked it a bit, and it works as advertised.

Not that the Node toolchain is terrible, but it's nice to have simpler options ✌️

[MrHaila]

@PaulFreund I feel you on the pain of keeping up with the JS ecosystem. I certainly wouldn't if it wasn't part of my day job. I played around a bit and ended up with this as something concrete to propose: https://github.com/MrHaila/deluge-sysex-tools

I think it's pretty neat and hits most of the requirements. Any thoughts?

[PaulFreund]

Thank you for the example @MrHaila.

I think this comes down to a difference in philosophy and I might just be on the little extreme side of using the simplest solution that can possibly work. If I had to work with 10-20 people building a major web application with hundreds of concurrent thoudsands of users I might consider the benefits of TypeScript and solutions like Bun but for this project I don't see the value added to justify the added complexity. I would have used node for the simple reason that it is battle tested over years and I can one click host it in a lot of environments which is no longer possible with Bun.

As I said before, if you are interrested to contribute please feel free to just start and build something, I can add value to it. If after some time nobody picked up site development and also set up the CI and hosting for it I will probably start with the initial proposal anyway and see where it leads us,

[trappar]

As I said before, if you are interrested to contribute please feel free to just start and build something, I can add value to it. If after some time nobody picked up site development and also set up the CI and hosting for it I will probably start with the initial proposal anyway and see where it leads us,

That's why I set this up. It's ready for contribution and already has CI and hosting/deployments setup: https://github.com/trappar/deluge-app

[sichtbeton]

From a user perspective the following would be good and some of these points are already possible with existing 3rd party apps like deluge commander and the like:

• safely delete samples without destroying songs
• import presets with samples in one go
• Incremental sd card backup
• Rename samples and update songs accordingly
• Preview songs
• Collect and save songs

[MrHaila]

I like this line of thinking quite a lot. Happy to also help in some capacity. A few notes based on working with https://github.com/MrHaila/delugr

1. Is a backend actually needed?

I don't see that the core requirements really depend on any server state or storage, but the complexity of managing user accounts in a GDPR-compliant way and working with a full-stack project roughly doubles the scope of the proposal, no joke. the technologies are not hard, but the sheer amount of work and UX consideration goes up by a lot.

My counter-proposal is to treat the Deluge as the backing store and the USB interface as the API. Other non-deluge-dynamic-things, such as available binaries, could be directly pulled from the public APIs like this Github repo. Local song storage, backups and sample libraries (= file management) would also neatly happen on the user's local machine. Device logs and such could also either be stored as files or simply in the browser's own offline store which is quite good for this stuff.

As an upside, the project could then become an amazingly portable single-page-app that works equally well locally vs hosted from the web.

2. UI state and business logic should be completely separate as they are such different projects and probably of interest to different authors.

It might make the most sense to have a WebMidi TS module published to NPM with it's own set of CI/CD automation to keep the core working and then consume that in another web project. This would, at the same time, facilitate the community making their own spinoff projects (CLI tools, apps, ...) based on the same core implementation. TS is an obvious choice for providing type-safe apis nowadays.

3. Pre-built UI components will not be enough.

PrimeVue looks cool, but the most important aspect of it is the Tailwind CSS compatibility: https://primevue.org/tailwind

This mostly means that, for the price of initial setup complexity, it will be possible to productively make completely new stuff that looks and feels the same as existing stuff. My argument is that making a very information-dense UI means taking opinions about how the data is laid out, and and that point, it makes all the difference to have the right tool for the job. I've gone through a bit of this pain with the Delugr song and sample lists, where it's critical to have fine-grained control over the layout components.

So the ideal mix here is something like 90% UI lib and 10% hand-crafted components for where it makes all the difference.

4. Avoiding dependencies is not necessarily the right core value to take.

A good-quality implementation from someone else is probably better than whatever you wrote, for reasons that will become apparent in a year. All code has bugs, and it's a good idea to use code that's gone through the ringer and back already (lodash, vueuse, Luxon, etc.)...

...but everyone hates updating dependencies and managing version conflicts. It's a better idea to just not implement complex things unless it's a core requirement. Regardless of whether it pulls a dependency or not. Tree-shaking helps with this problem a lot as having dependencies available during editor time doesn't mean they end up bloating the build.

Hopefully, there's something there that helps 🙂

[PaulFreund]

Thank you for the kind words :)

Regarding 1 - My intent was to design the application in a modular way so it can work with or without the online components and let different modules communicate in an event driven way (e.g. via an Event Bus). The public APIs of GitHub are limited enough that we need some external infrastructure to properly collect firmware artifacts and metadata (like graphing firmware size over time). I would not want to implement the user authentication etc. at the beginning but only once we feel it is needed. The proposed Node.js backend can be implemented and extended very easily while not limiting us in our possibilities. The application shall always be able to run without the online infrastructure components, they shall just be unavailable (in a similar project where the same UI is used in an Electron application and delivered by an embedded webserver I made the backend report the available capabilities and let the UI show functions depending on that.

Regarding 2 - As stated in the comments above I would use vanilla ESM but if someone takes the time to do it with TS I can probably extend on that.

Regardin 3 - So far I have mainly written components with Vue components and styled them afterwards with a global css file but I only heard good things about it

Regarding 4 - As with everything I think it highly depends. All the implementations we use today exist because someone decided to not use the prior art but do it themselves. It's always about making your own decision vs. relying on the decisions of others and how much we trust in a projects decisions to be the right ones for us.

All of that being said I made this proposal based on my own experience designing and implementing very similar applications in industrial environments but coming from a systems level and embedded background and without work experience as a dedicated web developer so I understand people might want to use different technologies. I'm happy if someone steps in and works on this, I just wanted provide a starting point and this is what I already know, can contribute and at least for me adheres to the KISS principle.

I think regardless of stack it makes sense to fulfill or make it possible to fulfill the requirements state above but of course we can discuss about all of them :) Thanks for your feedback!

[m-m-adams]

Regarding one I think the use for a backend would largely be preset sharing. If we don't store any actual user data is gdpr still a problem? It could just be an email/password to login and post your presets

[MrHaila]

Good question. Privacy laws aren't such a big deal in a very lightweight use case like that. They tell you there must be a way to manage and delete user accounts, so a bit of UX work. A bigger concern is the storage of login methods in the first place, as that's a high-value attack vector. An easy workaround is only to support passkeys and/or social logins like Github or Google or Discord.

Supabase seems to be the product in vogue for rolling out these simple backends with user accounts and a managed database.

[jamiefaye]

There are two ends to this where standardizing is very important. On the backend, it is the schema of the databases that will be used to store presets, samples, profiles, and the like. On the frontend, it is the SysEx / USB protocols and particularly the API used to address the Deluge's internal capability. Everything in the middle will vary widely. It does make sense to avoid gratuitous complication.

The key issue is the Deluge-side implementation as it not only has to be useful and sensible, but it must also be minimal, since we don't have much extra space and time to devote to it, considering how highly-tuned the Deluge has always been.

[MrHaila]

What do you think is the best way forward here? A library to talk SysEx and some UX around it to make it easier to work on and test the Deluge-side implementation?

I guess my question is: what is the minimum feature set for such a library to be useful and helpful for further Deluge development?

At a guess:

• A library to handle Deluge communication with a few placeholder commands that allow for easily adding & removing new ones.
• Basic unit testing setup for the library to make TDD possible in the future.
• A super thin CLI tool to invoke the above library for easy testing. Wizards know how to invoke code directly from raw libraries, but that's no way to live.
• A rough web page to also invoke the above library for easy testing. This is way more work, but it makes the thing more accessible and demonstrates that it works in a browser end-to-end.

Anything else?

I could whip something up pretty quickly if that helps. It doesn't matter much if the whole thing gets re-written five times over as long as it helps people work on the really important stuff (Deluge) easier.

[TCPderp]

I may be willing to work on components of this. Especially if it means we can create an easier way to import midi clips to the deluge. I would love to store midi clips like I do samples and be able to just load them into a clip. Being able to manage midi clips and samples in the companion app would be wonderful.