add support for iOS

[prabirshrestha]

Would be great if iOS was supported.

[woodworthkyle]

I've been looking into this on and off for a few months now. My main goal is to get Lapce buildable for Mac Catalyst (which includes iOS in some/all capacity). And in order to do this, I require floem to be compiled for the Mac Catalyst target (x86_64-apple-ios-macabi). I'm not sure if this warrants the creation of a separate issue, but I suppose my efforts are still relevant to this issue.

The rfd package currently does not compile for Mac Catalyst/iOS, but temporary workarounds can allow it to compile. Further work needs to be done for this to be fully compatible. Specifically, a closer look at obtaining the current app's window via winit in order to present the Mac Catalyst file dialog.

The forked winit package currently does not compile for Mac Catalyst. Again, with some workarounds, this can be compiled for Mac Catalyst. There are some subtleties that need to be worked out in order for this package to be fully compatible. From what I remember this is mostly concerned with Mac Catalyst specific app level features like UIMenu.

My current progress is attempting to get the softbuffer package to compile for Mac Catalyst. There is support for MacOS in this package, which can be used as somewhat of a template to get it to compile for Mac Catalyst. However, the main blocking issue I have right now is there are no convenient packages that allow access to UIKit bindings in a similar fashion to the way the cocoa package does for AppKit. So my next step is to implement such bindings for UIKit.

It's unclear whether or not further packages will need modifications once I am through with softbuffer, but I have a sense that I am getting close to having floem being fully compiled for Mac Catalyst target.

[woodworthkyle]

Some good news! I was able to get the counter example to render! I can’t interact with it, but I can see the window, text, and buttons.

It took quite a bit to get to this point. Once I was able to get softbuffer to compile, floem was able to fully compile with the example. However, the executable panicked because there was no fonts found. The error mentioned something about the cosmic-text package. I realized that cosmic-text had no iOS support, so i had to clone and modify the package. Still was getting the same error, so I dove deeper into the cosmic-text code only to find out that the fonts weren’t being properly loaded. I had to dive into the fontdb package code to find the issue. It seemed as though the filesystem calls that are part of std::fs don’t quite work as expected for Mac Catalyst targets. I ran into an issue where the read_dir().flatten() returns the correct number of contents of a given directory, but the paths for each entry were blank. Another issue I ran into was related to std::fs::File::open() and the memmap2 package. My original thought was that since there was odd behavior with read_dir, then open may not be working correctly either. I haven’t verified this yet. Another reason could be that memmap2 doesn’t have iOS or macOS support. Which confused me because how could floem work for macOS correctly if there is no support for macOS all the way down here at memmap2?

I digress. My initial guess as to why filesystem calls don’t work is that this is being built for Mac Catalyst and these apps are more strictly confined by their app bundle structures (you can’t just run a mac catalyst app from a cli, it requires an entire app bundle). So we need to use NSFileManager to get any information about the file system. Unfortunately, cocoa-foundation package does not have this implemented, so I partially added this feature just so I could get passed this hurdle. After some debugging, I was able to read the raw bytes of a file!

I was still getting no fonts found, but there was another instance in fontdb that was using fs and memmap2 calls that was preventing the data from loading properly. I replaced this code with pretty much the exact same code as I used before. And it was at this moment that the counter example launched and I was able to see a GUI!

I'm going to look a little deeper at why I can't interact with the example. I remember something about when the window gets created with respect to the event loops that is important for iOS applications. I'll probably start here.

[jrmoulton]

This is awesome!

[woodworthkyle]

Some bad news, I can't get this app bundle to run on an iOS simulator. The app fails to launch for some watchdog reason. I guess it 0x8BADF00D and got food poisoning (https://developer.apple.com/documentation/xcode/addressing-watchdog-terminations). It's hard to debug this issue with my current workflow. I'm doing most of my dev work with the remote dev feature in the CodeApp (https://github.com/thebaselab/codeapp) on my iPad in conjunction with VNCing into my 2015 MacBook Pro (I just realized how old this laptop is). I've been attempting to capture debug info with the xctrace tool, but it fails to write to disk because I have less than 5GB available. I can't be bothered (but probably should now) to dig through a decade old filesystem and purge unwanted files. Ugh.

Irrelevant question

Do I:
A. Purge old files
B. Back-up file system and re-install OS
C. Just replace the laptop with the Mac Mini you've been eyeing for this exact side project

In other news. Turns out the reason I cannot interact with anything is because the mouse clicks are being registered as touches. And floem does not have touches implemented at the moment:

floem/src/app_handle.rs

Line 205 in ac05875

WindowEvent::Touch(_) => {}

I will look at how touches can be implemented for floem. It seems winit supports touches with iOS via UITouch, so that's a good sign.

[woodworthkyle]

Some mock code to add to window_handle.rs to handle touches. This basically treats a single taps as a single LMB click. I'll test this out later, but also I'm not sure if this is the best way to handle touches.

pub(crate) fn touch_input(&mut self, touch: Touch) {
        let phase: TouchPhase = touch.phase;
        let num_touches : u64 = /* somehow get multiple touches if possible? */
        // Handle single touch as if it were an LMB click
        mut button: PointerButton = PointerButton::Primary
        let count = if touch.phase == TouchPhase::Started && num_touches == 1 {
            if let Some((count, last_pos, instant)) = self.last_pointer_down.as_mut() {
                if *count == 4 {
                    *count = 1;
                } else if instant.elapsed().as_millis() < 500
                    && last_pos.distance(self.cursor_position) < 4.0
                {
                    *count += 1;
                } else {
                    *count = 1;
                }
                *instant = Instant::now();
                *last_pos = self.cursor_position;
                *count
            } else {
                self.last_pointer_down = Some((1, self.cursor_position, Instant::now()));
                1
            }
        } else {
            0
        };
        // Handle 2-touch tap as if it were a single RMB click
        if (num_touches == 2) {
            // Handle right click
            button = PointerButton::Secondary;
        }
        let event = PointerInputEvent {
            pos: self.cursor_position,
            button,
            modifiers: self.modifiers,
            count,
        };
        match phase {
            TouchPhase::Started => {
                self.event(Event::PointerDown(event));
            }
            TouchPhase::Ended => {
                self.event(Event::PointerUp(event));
            }
        }
    }

winit claims to support multitouch, but its unclear right now how this works. At least with respect to iOS. The implementation utilizes the UIResponder touch events API (https://developer.apple.com/documentation/uikit/uiresponder#1653447) but then proceeds to essentially serialize the set of UITouches thereby converting them into single touch events with no way of determining how the touches are related.

Seems like there is an open issue in the main winit branch with regards to multiple touches (rust-windowing/winit#2379), but is specified for touchpads.

A potential workaround would be to add additional debouncing code to check how far in distance and how close in time a second touch is with respect to a first touch and use this as a way to get number of touches on a screen. I think this would require some modification to the last_pointer_down method to keep track of touches.

There are some subtleties here that will require problem solving. On Mac Catalyst targets, we have 3 potential platforms: Mac, iPad, iPhone.

• A Mac will have a trackpad/mouse that responds to touch.
• An iPad/iPhone will have a screen that responds to touch and sometimes a trackpad/mouse that also responds to touch.
• A Mac will always have a cursor, but iPad/iPhone will only have a "cursor" when there is a trackpad/mouse attached as an input device.

I think it's best for now to convert these touch events to pointer events and deal with the nuances at a later time. And there will be many more nuances.

[woodworthkyle]

So I spent some time trying to do this the correct way for pull request. I forked all the repositories I made changes to and created a branch for each one named ios_macabi. I also added some more changes to app_handle.rs and window_handle.rs to handle touches. And it works!

Here is a gif of the counter example.

I'll see if I can make progress towards running this app bundle on an iPhone simulator next.

[tdomhan]

Great to see progress. Do you have your version where softbuffer compile errors are fixed available somewhere?

[woodworthkyle]

I have them available in a fork of softbuffer under the ios_macabi branch: https://github.com/woodworthkyle/softbuffer/tree/ios_macabi

It’s been a while since I’ve touched this, but I believe the fixes are mostly to do with some additions to the cocoa package: https://github.com/woodworthkyle/core-foundation-rs/tree/ios_macabi

[woodworthkyle]

I have some updates!

Originally, I was targeting Mac Catalyst. However, my interpretation of this target may have been misconstrued. I thought it was a sort of cross-platform target for the Apple ecosystem (which is somewhat true). It turns out Apple Silicon devices can actually run iOS binaries natively. When it comes to x86 Apple devices, this is where Mac Catalyst comes into play somehow. I still don't quite understand this. Mac Catalyst seems like a way to utilize AppKit with UIKit. So Mac Catalyst binaries will not run on iOS, but iOS binaries will run on Apple Silicon MacOS devices (aarch64).

Anyways, I stopped at a point where I was trying to run a Mac Catalyst binary on an iOS Simulator. This led me down the path to try and run the Mac Catalyst binary on an iOS device. Along this path I discovered that iOS binaries can just run on Apple Silicon. So for now, I will be pursuing this effort (essentially just compiling iOS binaries now).

I had to go down the whole rabbit-hole of learning how to properly package an app bundle for iOS (basically what needs to be in the Info.plist file). As well as learning how to code-sign (provision profiles, certificates, entitlements) and package the entire bundle (zip to .ipa file) all from command line. There was a lot of debugging that entailed reverse engineering a working example of the rust based game engine bevy for iOS (https://dev.to/wadecodez/exploring-rust-for-native-ios-game-development-2bna).

I'm at the point where debugging is a bit of a hassle, but somehow I am still making progress. Since these binaries have to be bundled inside a sandbox, we can't just run the application from a terminal and see it's stdout. Especially since it's an iOS app running on MacOS. Luckily we can forward the stdout to a file from a terminal

open -o <stdout_file> <App Bundle Path>

However, debugging on an iOS device is a showstopper...but not for long. xcrun has a set of sub-utilities that allow us to install and launch application bundles.

# Install on iOS device
xcrun devicectl device install app --device <device_id> <path to app bundle we have built on dev machine>
# Launch on iOS device
xcrun devicectl device process launch --device <device_id> <path to app bundle on iOS device; found from the output of the previous command>

There is a --console feature that can be enabled while launching the app, but right now does not work properly. But will with Xcode 16! (https://forums.developer.apple.com/forums/thread/756393). This feature will allow us to view standard output while the app is running on the iOS device itself (I think).

So the good news is this: I am able to install and launch the app (without it crashing) on my iOS device. The same goes for my development MacOS machine. I cannot attach lldb or view the standard output of the app running on my iOS device...yet. I can, however, view the standard output as well as attach lldb to the same exact binary running on MacOS. And as far as I can tell, they behave the same (does not crash). The bad news: the UI still does not appear. I get a black screen on my iOS device and a blank window on MacOS.

The lldb debugger seems to point out that we are in fact in the floem event-loop. Here is a snippet of the backtrace (frames 32 and 33 are of interest):

* thread #1, name = 'main', queue = 'com.apple.main-thread', stop reason = signal SIGSTOP
...  
CountUpDowniOS`objc2::message::MessageReceiver::send_message::hb723ff597cecd9df(self=0x00000001ed85c9b8, sel=Sel @ 0x000000016f541580, args=<unavailable>) at mod.rs:231:13
    frame #15: 0x00000001016dbbfc CountUpDowniOS`icrate::Foundation::generated::__NSThread::NSThread::isMainThread_class::he4c4e6b51394888b at extern_methods.rs:266:14
    frame #16: 0x00000001016a7cac CountUpDowniOS`icrate::Foundation::additions::thread::MainThreadMarker::new::hf163ad99fb599a49 at thread.rs:88:12
    frame #17: 0x00000001016bb668 CountUpDowniOS`floem_winit::platform_impl::platform::event_loop::setup_control_flow_observers::control_flow_main_end_handler::h313df66958c569f9((null)=0x0000600000d540a0, activity=32, (null)=0x0000000000000000) at event_loop.rs:288:23
    frame #18: 0x0000000185ad487c CoreFoundation`__CFRUNLOOP_IS_CALLING_OUT_TO_AN_OBSERVER_CALLBACK_FUNCTION__ + 36
    frame #19: 0x0000000185ad4768 CoreFoundation`__CFRunLoopDoObservers + 536
    frame #20: 0x0000000185ad3e90 CoreFoundation`__CFRunLoopRun + 1028
    frame #21: 0x0000000185ad3434 CoreFoundation`CFRunLoopRunSpecific + 608
    frame #22: 0x000000019027d19c HIToolbox`RunCurrentEventLoopInMode + 292
    frame #23: 0x000000019027cfd8 HIToolbox`ReceiveNextEventCommon + 648
    frame #24: 0x000000019027cd30 HIToolbox`_BlockUntilNextEventMatchingListInModeWithFilter + 76
    frame #25: 0x0000000189332cc8 AppKit`_DPSNextEvent + 660
    frame #26: 0x0000000189b294d0 AppKit`-[NSApplication(NSEventRouting) _nextEventMatchingEventMask:untilDate:inMode:dequeue:] + 700
    frame #27: 0x0000000189325ffc AppKit`-[NSApplication run] + 476
    frame #28: 0x00000001892fd240 AppKit`NSApplicationMain + 880
    frame #29: 0x0000000189550654 AppKit`_NSApplicationMainWithInfoDictionary + 24
    frame #30: 0x000000019edb1f50 UIKitMacHelper`UINSApplicationMain + 972
    frame #31: 0x00000001b4f887bc UIKitCore`UIApplicationMain + 148
    frame #32: 0x0000000100977c4c CountUpDowniOS`floem_winit::platform_impl::platform::event_loop::EventLoop$LT$T$GT$::run::h449db51c5458c4e4(self=EventLoop<floem::app::UserEvent> @ 0x000000016f542f30, event_handler={closure_env#0} @ 0x000000016f543088) at event_loop.rs:164:13
    frame #33: 0x0000000100a4c5cc CountUpDowniOS`floem_winit::event_loop::EventLoop$LT$T$GT$::run::hc5f2088b04f9527d(self=<unavailable>, event_handler=<unavailable>) at event_loop.rs:249:9
    frame #34: 0x0000000100a99888 CountUpDowniOS`floem::app::Application::run::hc924b2ac6a7da0e8(self=Application @ 0x000000016f543138) at app.rs:133:17
    frame #35: 0x00000001008cefb0 CountUpDowniOS`floem::app::launch::h8b5dfb71b39929bc(app_view=0x0000016f54343800) at app.rs:29:5
    frame #36: 0x00000001008ce8d4 CountUpDowniOS`counter::main::hc58597ec8044d3e1 at main.rs:84:5
    frame #37: 0x00000001008d1db4 CountUpDowniOS`core::ops::function::FnOnce::call_once::h59771150cf6d8fb3((null)=(CountUpDowniOS`counter::main::hc58597ec8044d3e1 at main.rs:83), (null)=<unavailable>) at function.rs:250:5
...

I'm hoping this isn't a red herring and actually means we can run a floem based app on iOS. There is still work to be done as far as getting something to be rendered though. I will press forward and act as if debugging an iOS App running on MacOS will yield hopeful results. This is really the only way forward unless I want to mess around with Xcode 16 beta. But I think I'll wait until it's full release in a month or so.

[jrmoulton]

Super cool!! Thanks for the update and the work!

I'm definitely interested in running floem apps on iOS.