textOutput() and gridOuput() are incomplete and inaccurate in WEBGL

[calebfoss]

Most appropriate sub-area of p5.js?

• Accessibility
• Color
• Core/Environment/Rendering
• Data
• DOM
• Events
• Image
• IO
• Math
• Typography
• Utilities
• WebGL
• Build Process
• Unit Testing
• Internalization
• Friendly Errors
• Other (specify if possible)

p5.js version

1.6.0

Web browser and version

No response

Operating System

No response

Steps to reproduce this

I first identified this issue in the discussion for PR #6122

When using WEBGL mode, textOutput() and gridOutput() only produce descriptions of 2D shapes, and the descriptions are inaccurate because they do not take into account the difference in origin point or the camera settings.

In the example below, the textOutput describes a square in the upper left corner (rather than lower right where it appears on the canvas) and does not include the sphere.

function setup() {
  createCanvas(400, 400, WEBGL);
  textOutput();
}

function draw() {
  background(220);
  sphere();
  square(100, 100, 100);
}

Live in p5 editor

I think producing accurate descriptions in WEBGL would be a big undertaking, and in the meantime, I think it would be best for textOuput() and gridOutput() to throw a friendly error when called in WEBGL mode. I'm happy to open a PR for this.

[davepagurek]

If anyone is interested in taking this on, the way I've calculated screen-space positions out of WebGL coordinates in the past is by doing this:

// Assuming `this` is a `RendererGL`:
const toClipSpace = new DOMMatrixReadOnly(this.uPMatrix.mat4).multiply(
  new DOMMatrixReadOnly(this.uMVMatrix.mat4)
);
const clip = toClipSpace.transformPoint(new DOMPoint(0,0,0));
const screenX = (clip.x / clip.w + 1) / 2 * this.width;
const screenY = (clip.y / clip.w + 1) / 2 * this.height;

[Gaurav-1306]

i would like to contribute

[davepagurek]

Thanks @Gaurav-1306! I'll assign it to you. Let me know if I can help clarify anything!

[Gaurav-1306]

p5.js/src/accessibility/outputs.js

Lines 544 to 550 in cea9e09

	p5.prototype._getPos = function (x, y) {
	const untransformedPosition = new DOMPointReadOnly(x, y);
	const currentTransform = this._renderer.isP3D ?
	new DOMMatrix(this._renderer.uMVMatrix.mat4) :
	this.drawingContext.getTransform();
	const { x: transformedX, y: transformedY } = untransformedPosition
	.matrixTransform(currentTransform);

@davepagurek
in this code, I think the values of x and y are wrong, and solving that could remove the error(for textoutput()). So after the condition check that the renderer is 3d or not what should I do? should i make a new function that calculates the correct x and y position(using your suggestion for 3d and checking what is wrong with 2d) or do I have to change this._renderer.uMVMatrix.mat4 and this.drawingContext.getTransform() to find the correct value of x and y.

[Gaurav-1306]

hey @davepagurek

//gets position of shape in the canvas
p5.prototype._getPos = function (x, y) {
  const untransformedPosition = new DOMPointReadOnly(x, y);

  if (this._renderer.isP3D) {
    const toClipSpace = new DOMMatrixReadOnly(this._renderer.uPMatrix.mat4)
      .multiply(
        new DOMMatrixReadOnly(this.uMVMatrix.mat4)
      );
    const clip = toClipSpace.transformPoint(new DOMPoint(0, 0, 0));
    const transformedX = (clip.x / clip.w + 1) / 2 * this.width;
    const transformedY = (clip.y / clip.w + 1) / 2 * this.height;

    return { x: transformedX, y: transformedY };
  } else {
    const currentTransform = this.drawingContext.getTransform();
    const { x: transformedX, y: transformedY } = untransformedPosition
      .matrixTransform(currentTransform);

    console.log('the diff');
    console.log(transformedX, transformedY);

i tried solving the problem using this, but i am unable to do it. can you help!

[davepagurek]

That looks like a good start! It looks like the function currently returns some strings for the location:

p5.js/src/accessibility/outputs.js

Lines 553 to 559 in cea9e09

	if (transformedX < 0.4 * canvasWidth) {
	if (transformedY < 0.4 * canvasHeight) {
	return 'top left';
	} else if (transformedY > 0.6 * canvasHeight) {
	return 'bottom left';
	} else {
	return 'mid left';

so maybe rather than returning an object with an x and y position, your code should be something like this, where both branches set transformedX and transformedY, to be used later on when returning a string:

//gets position of shape in the canvas
 p5.prototype._getPos = function (x, y) {
   const untransformedPosition = new DOMPointReadOnly(x, y);
+  let transformedX = 0;
+  let transformedY = 0;

   if (this._renderer.isP3D) {
     const toClipSpace = new DOMMatrixReadOnly(this._renderer.uPMatrix.mat4)
       .multiply(
         new DOMMatrixReadOnly(this.uMVMatrix.mat4)
       );
     const clip = toClipSpace.transformPoint(new DOMPoint(0, 0, 0));
-     const transformedX = (clip.x / clip.w + 1) / 2 * this.width;
+     transformedX = (clip.x / clip.w + 1) / 2 * this.width;
-     const transformedY = (clip.y / clip.w + 1) / 2 * this.height;
+     transformedY = (clip.y / clip.w + 1) / 2 * this.height;

-     return { x: transformedX, y: transformedY };
   } else {
     const currentTransform = this.drawingContext.getTransform();
-     const { x: transformedX, y: transformedY } = untransformedPosition
+     { x: transformedX, y: transformedY } = untransformedPosition
       .matrixTransform(currentTransform);

     console.log('the diff');
     console.log(transformedX, transformedY);

[Forchapeatl]

We can close this issue , textOutput and gridOutput works perfectly with the latest version of p5.js <script src="https://cdnjs.cloudflare.com/ajax/libs/p5.js/1.10.0/p5.js"></script>

[calebfoss]

Updated example with v1.10

The label now correctly describes the square's position as the bottom right! That's awesome!

Unfortunately, the sphere is still not described, and now the label says that the square takes up 15625% of the canvas, whereas before it said 6%.

[Forchapeatl]

Thank you for the update @calebfoss . We can Identify 2 issues here. Unreported shapes and inaccurate percentage. Please take a look at my code . It seems seems all 3D shapes are not reported by the function.

function setup() {
  createCanvas(400, 400, WEBGL);
  textOutput(LABEL);
}

function draw() {
  background(220);
  sphere();
  circle(50, 50, 100);
  square(100, 100, 100);
}

[calebfoss]

@Forchapeatl, exactly! 3D shape descriptions were never implemented. That's what I was referring to here:

When using WEBGL mode, textOutput() and gridOutput() only produce descriptions of 2D shapes

In this accessibility features proposal, I go into more detail on the future of these functions.

I'm glad you're looking into improving the accessibility features!