-
Notifications
You must be signed in to change notification settings - Fork 6
2024 Testing OS specific accessibility APIs in WPT
- GitHub issue: https://github.com/Igalia/webengineshackfest/issues/
Valerie introduces the topic
Valerie: Slides are available in the notes for this topic: https://notes.igalia.com/p/pgqfgU8DV#/
Valerie presents the slides
- This presentation is about testing the parts of the browser that translate browser stuff into platform accessibility API.
- 4 accessibility APIs for 3 platforms
- Mapping between DOM element and API node is standardized
- Core Accessibility API Mappings
- There are mappings, states, and events.
Luke: I noticed there was no mention of Android nor in the spec. Can you say more on that?
Valerie: It's just a sorely needed area. We should have Android mappings, but it hasn't been done yet. We should look into finding funding or people for that.
Dan: There is a system, it's just not in the AAM standards?
Valerie: One interesting thing about the AAM is that when we do have new features, that's where we show them. AAM is more like recording decisions that have been made a long time ago, but no one has recorded them yet.
Alice: The a11y API is there on Android, web content is accessible; it hasn't been written up in the standard. No one has done it yet.
- The kinds of tests that exist for mappings / ARIA. You can check internal role concepts, but not platform ones.
Valerie: There is also the concept of the computed role. The computed role doesn't have all the information about the internal role of the element.
- Some mappings link back to the mappings used in the WAI spec.
- There are more mappings (MathML, Graphics, SVG, etc). They also don't have tests.
- Manual or browser-specific tests are currently the main way to test these features.
sideshowbarker: In practice how I do this is I ask a practicioner and have them test it for me. You gotta have someone who knows what they are doing and can test on multiple screen readers / multiple browsers.
- We have a prototype of testdriver extension that can do these things.
- Platform-specific Python code which examines the application's exposed a11y surface.
- Some piping required to get PID / Application name, needed to examine applicationw with platform a11y APIs.
Questions: - Does this make sense for WPT? - Is this the right way to do it? - How to find the browser for a11y API purposes? PID or name? - How should tests look? Assertions for all platforms or assertions for only one platform?
Anne: There might be another alternative where the test convey the expectation and each specific platform has a mapping. Whenever there is a new platform you just update the mapping file.
Valerie: That sounds great.
Alice: I'm not entirely clear on how that would work... If you look at the mapping, there are many different pieces of it. it would have to be a big mapping table.
Anne: Maybe there is some mapping possible.
Valerie: You can see based on the mapping that each one is a couple assertions, but if there was some structured data format way of running those...
Alice: If we add the Android mapping, is that just android.accessibility or whatever the format is.
Anne: I imagine it is similar to how we have test harness report where each implementation supplies the format.
Alice: Sounds a bit like dump accessibility test tree.
Valerie: That's worth thinking about.
Anne: It might also make it easier if people don't have to think about the mappings, but instead the high-level semantic concepts of the specs. How these map to platform a11y APIs is up to implementation.
Valerie: Not sure that someone who isn't familiar with the platforms could write these tests.
Anne: To some extent. If you are familiar with ARIA and HTML.
Dan: asks question clarifying question about test in slides
Valerie: Something with a computed role button could still be used in different ways.
Dan: What if we every scenario had a name which had a different platform implementation.
Anne: The web driver binding would take care of making sure the exposed mapping matches the expected platform. It could compare against the value exposed by the platform APIs.
Valerie: It would have to skip the a11y API later.
Dan: Is it possible to have a test that goes through the whole stack, but another thing that could be good would if tests could be written against the thing that's more specific than the computed roles.
Valerie: Maybe there's something to explore there, but I do want to focus on the platform accessibility APIs.
Valerie: Final question: There will be new concepts that don't exist in wpt.fy currently. wpt.fyi already shows results as browser / OS pairs. shows mockup of new wpt.fyi view
Luke: I guess to answer the question from my perspective. Having every browser / OS pair on wpt.fyi is needed regardless of this. There are rendering related failures (scrollbar gutter bugs on certain platforms). From a developers perspective this is not interopable. I think that it's something that's useful regardless of this.
Ms2ger: You're presenting a table with rows per assertions. That's not really how wpt.fyi works. It shows results per subtest. clarifies how subtets and assertions worked
Valerie: Thanks for the clarification. I still think this will have non-applicable spots in the view because some browser / platform pairs are not relevant.
Anne: At some point you're going to run a test which will have a result, which will have a result for each pair unless you don't run the test at all. Er. Subtest in this case.
Valerie: A better example is a test that is a single test file that only tests Windows-related subtests. On other platforms, what's in the subtest column?
Anne: At the start of the subtest that has a precondition assertion, but in the other ones the pre-condition failed? If the condition fails, it fails the test, but in a non-failing way.
Valerie: We want to fail the test in a non-failing way or not run the test.
Luke: Could you not have subtest that has multiple assertions? Then every subtest would run in every platform pair.
Alice: Sounds like in that view we don't have subtests?
Luke: You still have subtests, but they are concepts and not platform mappings.
Keith: Don't you lose visibility when one platform is failing?
Anne: this project is already kind of incorrect.
Valerie: We would have to know the subtest...for it to map to assertions. I want to know if you are passing the IA2 role, but no the IA2 attribute. They are both important.
Anne: We don't really do that for other tests. In the detail you can click down to see where the test started failing.
Keith: I think there's another issue demonstrated table. There is more than one API on Windows, so there's a combinatorial explosion.
Valerie: I want to respond to Anne, there is a lot of inconsistency in WPT tests.
Alice: in our wone example example we didn't use subtests, but we'd probably switch.
Valerie: In this individual files per platform. In this case we need to track progress switching from one API to another.
Ms2ger: Looking at this table, IA2 and UIA are conceptually different rows versus different columns.
Anne: I also would expect them to be columns.
Valerie: When I originally made this table, I did split them. When in reality it would be run all tests that you can run on this platform.
Anne: it runs them on a single client right?
Valerie: The a11y APIs are queried directly by the APIs. On Windows you can say give me all the applications that expose IA2 or give me all applications that expose IA2.
Anne: clarifies
Valerie: We're leaving out the AT, but it's the end of the browser.
Keith: The point is to capture the discrepency betwen the a11y node in the browser and the OS APIs...these aren't screen reader tests though. We just want to verify what the screen reader gets.
Valerie: And there is a test suite for that (screen reader behavior?)
Alice: It's a browser bug if the OS mapping isn't exposed properly. It isn't a browser bug if the AT doesn't behave properly.
Valerie: If something works for Chrome, but not for Firefox. Those are the bugs we are talking about.
Luke: Did you say there's a separate test suite for the for ....
Valerie: There's an inititiave to test whether or not screen readers expose ARIA properly to screen reader users?
Luke: Can they understand which app they are coming from?
Valerie: In that test suite they are doing things like firefox on windows, chrome on windows. They limited because there isn't a lot of automation available and ATs are very opinionated about how these are exposed to users. There's lots of manual testing.
Luke: I guess you can't rely on the output of the screen reader
Valerie: They do. They record the output.
Anne: It still seems like a downside that you have to modify the tests. If you had a new mapping
Luke: Cause we have have
Valerie: You are still writing a JSON file.
Anne: All the "mappings" to the high level concepts are in a new place.
Luke: If you could do the mapping to JSON as a common subset.
Alice: I think I can kind of imagine it...if you are getting a bag of properties that have the mappings. I still don't get how it's not almost as much work.
Anne: You have to do it once for each mapping but the tests have the same outcome.
Valerie: Just in this case. The assertions for...
Anne: I'm not sure exactly how you would organize it. Do you want to duplicate it in each test or have it all in one place?
Luke: You'd have input type button, etc, in the file and then what they map to?
Anne: I think it's a lot more because role="button" applies to many HTML elements could
Luke: If we just say push button is a concept, you could just look up the mapping for that concept in the JSON file. Then the test file would just be doing the HTML aspect of it. If I proposed a new pushbutton, I wouldn't need to expose the new platform concepts.
Valerie: You would have to konw for you concept that this matches.
Luke: If Android came along, you could just fill out that mapping. I don't know how how that would work for events.
Anne: If I imagine people at Apple working on that, they only need to write the high-level mapping for the test. It will automatically work for the Apple people and the Windows people.
Valerie: I guess I'm thinking about concepts about adding mappings and role.
Keith: You would effectively define Core AAM and ?? as libraries that other things ingest.
sideshowbarker: I have a clarification about number 2: how to find the browser? You mentioned needing parallelization. Is this parallelization about running multiple tests in multple browser instances. WPT does this already. It launches multiple browser instance. The other browser-specific test automatimation runs one test at a time with multiple instances.
Valerie: Yeah, that's what I mean. We can't do that with the prototype because each one as the same name.
Alice: The other thing that immediately is a pain is that we can't our instance of Chrome running while runnigng tests.
sideshowbarker: I don't see any good reason not to expose the PID. For instance when debugging, when you attach to browser process you need to use the PID. You need the PID so expose the PID.
Luke: I think one of the other benefits of using the PID is you don't need update test infrastruture when you are running something like MiniBrowser or content_shell, you'll need the PID or an exhaustive list. PID solves the paralellization problem. minibrowser even gives you the PID in the title bar of the window, because it's such a common thing to need it to attach.
Javier: Maybe I haven't understood correctly the need for the PID. I implemented a new webdriver extension command that requires putting chrome in a specific state. I defined a new command precisely for that. As far as I understood. Once you define the command in the webdriver, you can use the devtools protocol for communication with the browser.
Valerie: We can't use that protocol.
Javier: I have to implement the same mapping ... the communication between the web driver and browser is transparent.
Luke: Still requiring on the browser giving you the information correctly.
Valerie: I think what you are trying to explain isn't possible. The a11y API isn't exposed through the devtools protocol.
Alice: We did have a slide about this. Could we add a million more things to webdriver to expose all of these things? I didn't like that idea very much because it's not an integration test. You are not testing the a11y API, it's almost just a webdriver protocol API.
Valerie: It would mean editing all the browsers.
Keith: It would mean a whole set of tests itself.
Javier: Maybe we discuss this offline?
Martin: Is there any downside?
Valerie: I opened a bug and they said no.
Alice: The endpoint could be anywhere and you could hop through multiple nodes. It might on a different machine. It might not be quite as straight-forward. It might be them revealing this thing that they are trying not to reveal?
Anne: Why can't webdriver call these APIs that you want to test. It seems that webdriver is the starting point. It sounds like a webdriver problem. I'm not sure how much API you need to add, but...
Luke: It's not calling a browser API.
Anne: It still needs to be mediated somehow.
Valerie: I think it's worth exploring more.
Luke: I don't know how going through webdriver helps things too much. You can't rely on the browser to give you the information. My understanding is the webdriver.
Anne: What is the alternative for the bridging?
Luke: The PID is kind of separate to webdriver...
Valerie: webdriver is also what starts the browser.
Anne: The browser calling the OS API to set the information.
Keith: Are you familiar with dbus or any of the buses? The browser just dumps information to the OS and then you pick it up somehow. The browser sets stuff in this bus and then it picks it up.
Valerie: We're testing this stuff gestures to area between AX API and browser accessibility
Luke: ... missed this Webdriver is just a handy way to launch the browser and can do other things like clicking. Biggest reason why it should be aprt of WPT
sideshowbarker: There is this work for an AT driver for webdriver.
Valerie: We discussed that a moment ago. That's about if the AT is properly interpreting the ARIA propery.
Keith: In this graphic it's very important to call out. Very specifically we are talking about the button that says AX API. That's what we want to test. We have to test that bit. Valerie: We don't want to go to browser internals and say 'what would you send to ax api' we want to test 'what did you send to ax api'
sideshowbarker: This is a really interesting engineering challenge. It would be a fun thing to work on for people so I am optimistic for us coming up with something.
Valerie: That is the true spirit of people who come to the web engines hackfest. WPT sounds fun let's do it.
Luke: Sounds feasible. That's the really important part. Especially in tandem with making sure that AT interpet roles properly. If that fails then everything else has failed. The more testing, the more information you can give to web developers. Things not exposed properly are not supported. MDN has no way to expose that information. Maybe MDN isn't the right place, but some sort of caniuse.
Valerie: I don't think the regular web author would do these things (if not exposed)?
Alice: That is part of the a11y practice on the frontend -- that's why they burn out. It's really complex and underrecognized. You have to keep it in your head because it's not documented.
Luke: You have to have access to every single platform or you have to ask someone who knows. When I was a web developer we could say things were accessible button couldn't actually test it. Some sort of caniuse would be great.