Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WIP: ARCore-based position tracking #395

Draft
wants to merge 3 commits into
base: master
Choose a base branch
from
Draft

Conversation

knuxify
Copy link

@knuxify knuxify commented Nov 17, 2024

Partially fixes #93, fixes #311.

Enable HMD position tracking based on ARCore's motion tracking capabilities. Note that this only works on devices that support ARCore; see the official list for more information. (All devices on this list are supported, regardless of depth API support/camera resolution. Tracking accuracy may vary.)

Performance looks to be pretty good, though I am testing this on a Samsung Galaxy S23 which is pretty much a best-case scenario. The device does heat up after a while, but from what I've heard that's pretty standard for ARCore. However, as of writing, it's not stable enough yet to be actually used, which is why this PR is marked as a draft.

I'm opening this PR to get some feedback and perhaps some help with development. Here's a general TODO list:

  • Add user-side toggle for enabling/disabling ARCore-based tracking (right now it's enabled by default)
  • Add user-side settings for ARCore (whether to use orientation data from Cardboard or ARCore directly (not recommended unless I can work out the delay, see later comment), camera FPS limit, filtering to camera configs without depth support/with lower resolution for processing power/accuracy tradeoff, etc.)
  • Figure out orientation drift. The orientation (rotation) that we get from the Cardboard API drifts away from the ARCore tracking orientation over time, so e.g. walking forwards starts moving sideways instead. Using the orientation data from ARCore itself solves this, but also introduces a ~0.5s delay which is very noticeable. I don't think there's a way to reduce this delay - it's just a side effect of how ARCore works, and is probably going to be worse on weaker devices. The solution would likely be to somehow adjust the orientation to match the one that ARCore reports...?
    • Occasionally when ARCore loses tracking it will "teleport", i.e. the position will be shifted; I think this could be related to the repositioning behavior described in the Pose docs. This completely breaks orientation when it occurs, so fixing the above would probably fix this too (in fact, I'm willing to believe that it's the same issue).
  • Figure out pose scaling. Currently the actual speed of movement is a little out of wack, we will probably need to scale the values from ARCore. I've done some more testing and I think it's actually fine, but someone with more VR experience would need to test.
  • Figure out altitude. Right now it seems to be wrong (maybe it just needs scaling? or maybe it's picked up from the first plane it identifies), so you'll usually end up clipping through the floor or just generally being very low. The ARCore geospatial API has the ability to get altitude data, but it must use Google's location APIs which is privacy-intrusive and has a ratelimit. I was also considering using the phone's barometer sensor and a short calibration process before each play session to determine the floor position and what scaling to use for anything higher than it, but I'm not sure if that's actually possible + wouldn't work on phones that don't have a barometer/pressure sensor.
    • This is done! The solution I went with is to find the lowest-placed plane and place an anchor on it; this way we can track the Y position of the floor, then subtract it from the headset position. While working on this, I also found out that ARCore's world space is based on the position of the device when the session is set up, so placing the device on the floor while turning on PhoneVR works best (but this also works very well once it gains tracking, so the UX is at least somewhat nice).
    • I did have a WIP patch for barometer-based tracking, but the barometer values were very jittery, and to smooth them out accurately and maybe include data from other sensors for extra accuracy, I'd probably need something like a Kalman filter. Unfortunately I do not know enough maths to actually implement one, so that's left as an exercise to someone more knowledgeable than me.
    • We could probably add a setting to force a specific altitude.

I saw somebody say that iVRy supports ARCore-based tracking (personally I didn't know that, I'd have to check) - if that's true, it'd be interesting to see how they tackle some of these problems.

@knuxify knuxify marked this pull request as draft November 17, 2024 20:38
@@ -51,3 +51,7 @@ unzip download.zip
rm download.zip
fi

rm -r "arcore-android-sdk"
curl -sLS https://github.com/google-ar/arcore-android-sdk/archive/refs/heads/main.zip > download.zip
Copy link
Author

@knuxify knuxify Nov 17, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

note to self: we should probably tie this to the same version we pull in app/build.gradle.

CTX.lastOrientation = pose.orientation;
}

if (CTX.arcoreEnabled && CTX.arSession != nullptr) {
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this pattern kinda sucks; ideally we'd set CTX.arcoreEnabled to false if arSession fails to initialize

@knuxify knuxify force-pushed the arcore branch 3 times, most recently from a7635e5 to efdafc2 Compare November 18, 2024 17:27
@ShootingKing-AM
Copy link
Member

Nice initiative, I didnt check the code yet, also i dont have a arCore supported device :c
I was actually thinking of removing the hard coded device fingerprint gating in official arcoreApp to see if i can run this on my phone. I actually dont see the that some devices can use arcore and some cannot other than google marketing effort.

@knuxify
Copy link
Author

knuxify commented Nov 19, 2024

I was actually thinking of removing the hard coded device fingerprint gating in official arcoreApp to see if i can run this on my phone.

IIRC this isn't as simple as removing a check, each device has calibration data specific to it (and of course there's no public documentation on how to add that calibration data yourself). Could work to just copy the calibration data from another device and live with the inaccuracy (quick googling shows that people have spoofed the device information to report the Pixel 2 way back in 2019 and got it working, but that was a long time ago...)

@ShootingKing-AM
Copy link
Member

ShootingKing-AM commented Nov 19, 2024

True that too, but whats the use of development or including arcore if only some set of devices can support it ? We need to have something for the others who dont have arcore too, i only think only 10% (assumptions) of devices in market support arcore

@knuxify
Copy link
Author

knuxify commented Nov 20, 2024

True that too, but whats the use of development or including arcore if only some set of devices can support it ?

The use is that the devices that support it can make use of it :D

On a more serious note though - I can certainly see that this is a bit unfair, and would probably cause added burden to you as the maintainer (since you don't have a device to test this). However, I think having at least some devices able to use this is still better than not having this feature at all. I also hope that supporting one method for getting position can help with adding support for more methods in the future).

(The only other currently-available method I know for tracking headset position is using an external method like PSMoveServiceEX + a LED bulb mounted to the HMD, but that's more involved than just running an app, and is difficult to do homebrew from my own experience.)

Figuring out alternatives to ARCore is likely out-of-scope for this PR, though.

@ShootingKing-AM
Copy link
Member

I mean i dont have an issue adding this, but i dont have a way to test it (as of now) as long as you are okay with future pings for issues related to ARCore :p

Also i dont think its that hard to add 6DOF by directly getting accelerometer reading, thought you have some knowhow regarding that.

GLuint arTexture = 0;

AlvrQuat lastOrientation = {0.f, 0.f, 0.f, 0.f};
float lastPosition[3] = {0.f, 0.f, 0.f};

float currentMagnetometerValues[4] = {0.f, 0.f, 0.f, 0.f};
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oops, leftover from trying to fix orientation

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

ARCore support? Vertical Movement detection
2 participants