Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Search improvements #35

Open
wants to merge 7 commits into
base: main
Choose a base branch
from
Open

Search improvements #35

wants to merge 7 commits into from

Conversation

simonwiles
Copy link
Member

This is a bunch of stuff that goes back a month or so, that I'd like to PR to create a baseline for some other branches. No rush to review/merge or deploy, I suppose.

  • Crop images for matches to the bbox rather than the pose extents
  • Rewrite the pose drawing code to take the above into account
  • Redo some of the UI controls as overlays

We can see actual actors / people now 🎉
This took rather more false starts that I'd care to admit -- in the end I have done away with previous code and reconsidered from first principles -- I think scaling to fit the canvas is the wrong approach.  This is more complicated than scaling a full frame, since we don't know the aspect ratio ahead of time, but I think this approach is the cleanest.
This is a bit of a cop-out; it might be better to do this in the db query instead, but I didn't want to rewrite the API endpoint again.
@broadwell
Copy link
Contributor

This is all really great, though there are a few issues with the search result renderings that I'll outline below. Those could either be addressed prior to merging or handled later. And I'll just skate over a few features that might be good to add next, which could have knock-on effects later: 3D pose matching, pose editor, ability to turn off camera when not capturing image. Also maybe a pager for the search results?

Anyway, regarding the pose cutouts/renderings, incorporating the bounding boxes is a major improvement. But there are just a few teething issues and one big-picture (no pun intended) question hanging around, which become more apparent when the Show Pose box is selected:

  1. The overlay is drawn out of place horizontally when the bbox is wider than it is tall:
    Screen Shot 2024-09-16 at 11 34 50 AM
    Screen Shot 2024-09-16 at 11 35 16 AM
  2. Sometimes the pose keypoints extend beyond the bbox but are still located within the image, probably because the actual body points are obscured and are being extrapolated by the model. Should we extend the match cutout to include them (this would involve taking the max of the keypoint coords and the boox coords, presumably)?
    Screen Shot 2024-09-16 at 11 32 57 AM
    Screen Shot 2024-09-16 at 11 33 13 AM
  3. What to do when the pose extends beyond the image boundaries? This is an example in which the pose extends both vertically and horizontally beyond the frame, and it's perhaps more troublesome than the common situation of the lower body being out of frame, because the most salient aspect of the query pose is the raised left arm, which obviously has been extrapolated by the model but is cut out of the match display.
    Screen Shot 2024-09-16 at 11 32 20 AM
    Screen Shot 2024-09-16 at 11 32 32 AM

Issues 2 and 3 were previously handled by extending the pose match viz with blank space up to the max coords of the keypoints to enable drawing the extrapolated keypoints over it. This adds more complexity to the excerpt/drawing code but isn't super challenging. There are however other ways to handle it, though, some of which could lead to a better UX, including maybe a toggle option to "show keypoints that are out of frame" or something. Introducing 3D renderings of the full extrapolated pose also would be one way to convey this information, but those can be pretty heavyweight and maybe should only be shown when/if it's possible to "drill down" to view the details of a single matched pose.

I'm happy to take a swing at the above as well, but figured I'd flag it here given that you've been much working more directly with this code lately and might have some quick fixes in mind.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants