Suggestion of a function to get screen coordinates

[inaridarkfox4231]

Increasing access

Proposal to implement functions like screenX(), screenY(), screenZ() in processing.

I often see suggestions that p5.js also needs a function to calculate screen coordinates.
I thought it would be good to have such a function, even if it's not the method I proposed here.

Most appropriate sub-area of p5.js?

• Accessibility
• Color
• Core/Environment/Rendering
• Data
• DOM
• Events
• Image
• IO
• Math
• Typography
• Utilities
• WebGL
• Build process
• Unit testing
• Internationalization
• Friendly errors
• Other (specify if possible)

Feature request details

I thought of a function like this:

p5.prototype.getNDC = function (v){
  const _gl = this._renderer;
  // Transfer to camera coordinate system using uMVMatrix
  const camCoord = _gl.uMVMatrix.multiplyPoint(v);
  // Calculate ndc using uPMatrix
  const ndc = _gl.uPMatrix.multiplyAndNormalizePoint(camCoord);
  // Drop into canvas coordinates.
  // The depth value is converted so that near is 0 and far is 1.
  const _x = (0.5 + 0.5 * ndc.x) * this.width;
  const _y = (0.5 - 0.5 * ndc.y) * this.height;
  const _z = (0.5 + 0.5 * ndc.z);
  // Output in vector form.
  return createVector(_x, _y, _z);
}

This is a function that calculates screen coordinates and depth values ​​from 3D coordinates for the currently set camera.
Screen coordinates are values ​​based on the coordinate system of the 2D canvas, and depth values ​​are 0 for near and 1 for far.

This kind of technique is actually already used in the p5.js library.

In #6116, when I improved orbitControl, it became necessary to obtain normalized device coordinates, so I implemented functions by suggestion of dave pagurek.
The method used here is based on that method.

"getNDC" is a temporary name and has no particular meaning. I think it needs to be changed with a proper name.

getNDC demo

let gr;

function setup() {
  createCanvas(400, 400, WEBGL);
  gr = createGraphics(400, 400);
  gr.textSize(20);
  gr.textAlign(CENTER,CENTER);
  noStroke();
}

function draw() {
  background(220);
  orbitControl();
  lights();
  fill(255);
  sphere(40);

  const ndc = getNDC(createVector(0,0,0));
  gr.clear();
  gr.text(ndc.z.toFixed(2), ndc.x, ndc.y);

  push();
  noLights();
  camera(0,0,1,0,0,0,0,1,0);
  ortho(-1,1,-1,1,0,1);
  translate(0,0,1);
  texture(gr);
  plane(2);
  pop();
}

p5.prototype.getNDC = function (v){
  const _gl = this._renderer;
  // Transfer to camera coordinate system using uMVMatrix
  const camCoord = _gl.uMVMatrix.multiplyPoint(v);
  // Calculate ndc using uPMatrix
  const ndc = _gl.uPMatrix.multiplyAndNormalizePoint(camCoord);
  // Drop into canvas coordinates.
  // The depth value is converted so that near is 0 and far is 1.
  const _x = (0.5 + 0.5 * ndc.x) * this.width;
  const _y = (0.5 - 0.5 * ndc.y) * this.height;
  const _z = (0.5 + 0.5 * ndc.z);
  // Output in vector form.
  return createVector(_x, _y, _z);
}

2024-05-19.09-02-24.mp4

[davepagurek]

I think this could make sense to add, anecdotally it's probably the question I've been asked the most about p5, although not exclusively to WebGL, so 2D mode could feasibly implement something like this too.

@limzykenneth any thoughts on naming? Flash used to have methods like localToGlobal(coord) and globalToLocal(coord), for inspiration.

(Also, I reopened the issue, was it intended to be closed?)

[ffd8]

FYI - I brought up this issue with @bohnacker years ago (while attempting to port a Processing lib to p5.js that depended on screenX/Y/Z) and they came up with the following solution (I'm not sure if it's been submitted for the upcoming new website (of which all existing ones should be auto-ported incase authors didn't see message to submit) – would of course be great to have them built in!

https://github.com/bohnacker/p5js-screenPosition

[inaridarkfox4231]

After I came up with the idea, I lost motivation, so I closed it.
I'll leave it up to contributors to decide what to do with this discussion. Personally, I think it would be nice to have a function that simply returns screen coordinates and depth values ​​for 3D coordinates.

[Garima3110]

Personally, I think it would be nice to have a function that simply returns screen coordinates and depth values ​​for 3D coordinates.

I too agree to this , it would be nice to have such a function.
Just a suggestion ,
How about naming the function worldToScreen(coord) ?
This name is descriptive and indicates that the function converts 3D world coordinates to 2D screen coordinates.

[inaridarkfox4231]

Having a function like this will help contributors create new methods, like I did in #6116. I also think it is useful because it can be used to place visual information in 3D space.

2024-06-03.21-46-57.mp4

[ffd8]

For reference, two additional issue threads that dove into this topic:
Add screenX, screenY and screenZ
modelX/Y/Z() and screenX/Y/Z() for p5.js (WebGL)

[davepagurek]

Personally, I think it would be nice to have a function that simply returns screen coordinates and depth values ​​for 3D coordinates.

I too agree to this , it would be nice to have such a function.
Just a suggestion ,
How about naming the function worldToScreen(coord) ?
This name is descriptive and indicates that the function converts 3D world coordinates to 2D screen coordinates.

I like worldToScreen, it also leaves open a possibility of screenToWorld in the future too.

[ffd8]

I didn't understand the demo referenced above by @inaridarkfox4231 which had a sphere with lighting changing, but upon digging into your sketches, found a very relevant and well working example: ndc4_depthValue by dark_fox - amazing snippet!

+1 for worldToScreen() (perhaps something like aliases could then be built around it that mimic Processing's screenX()... for added compatibility)

[ffd8]

One small thing to add – Processing's screenX() are also so nice in that they allow 2D canvas x/y vector to screen capture.. so if one uses translate() or rotate(), it still knows where the coordinate is.. it would be ideal if said function could also handle Canvas2D rendering, as at the moment, it's explicitly WEBGL.

[inaridarkfox4231]

In the sketch introduced in the comment above, validation is applied using the depth value so that it is not displayed if it is on the back side.
icosahedron

  // Validate using the depth value of the origin
  const ndc0 = getNDC(createVector(0,0,0));
  for(let i=0; i<12; i++){
    const v = icosaV[i];
    const ndc = getNDC(v);
    if(ndc.z>ndc0.z)continue;
    if(showType===0)guide.text(vertexDataIcosa[i], ndc.x, ndc.y);
    if(showType===1)guide.text("["+i+"]",ndc.x, ndc.y);
  }

(After that, I made various changes for that sketch, and removed the ball because it was unnecessary.)

[davepagurek]

One small thing to add – Processing's screenX() are also so nice in that they allow 2D canvas x/y vector to screen capture.. so if one uses translate() or rotate(), it still knows where the coordinate is.. it would be ideal if said function could also handle Canvas2D rendering, as at the moment, it's explicitly WEBGL.

I left some code for world to local over here in a comment a while ago: https://www.reddit.com/r/p5js/s/6co56yZ78u

The reverse, local to screen, would be something like:

    const matrix = drawingContext.getTransform();
    const localCoord = matrix
      .scale(1 / pixelDensity())
      .transformPoint(
        new DOMPoint(x, y)
      );

[ffd8]

@davepagurek Wow that's awesome! For whatever/obvious reason the pixelDensity() does mess things up, especially since p5 has 2 by default, everything was doubled.. making pixelDensity(1) in the setup solved it. Any idea how/why that is and if it can be made flexible to whatever pixelDensity() is given? I found a solution for pixelDensity(2) using translate(), however it breaks on any other value than 1 or 2...:

const matrix = drawingContext.getTransform();
let pd = pixelDensity();
let scl = 1 / pd;
let transform = createVector(0, 0);
if(pd != 1){
	transform = createVector(-width * scl, -height * scl);
}

return matrix
.scale(scl)
.translate(transform.x, transform.y)
.transformPoint(
	new DOMPoint(v.x, v.y)
);

[davepagurek]

Oops, my fault, I think that's an order of operations problem. So p5 deals with pixel density by applying an initial scale(). This means that drawingContext.getTransform() is equivalent to a multiplication of that density first (call that D), and then multiplication by whatever other transforms you have call that (T): D × T. We just want to get T since that's the coordinate space p5 functions work in. To undo the density, we have to pre-multiply the division by density (D^-1 × D × T = T), but I was accidentally doing it afterwards (D × T × D^-1) which doesn't simplify to the same thing.

Here's a sketch that I think should work:

function setup() {
  createCanvas(200, 200)
  pixelDensity(3)
  
  translate(width/2, height/2)

  const matrix = new DOMMatrix()
    .scale(1 / pixelDensity())
    .multiply(drawingContext.getTransform());
  
  console.log(matrix
    .transformPoint(
        new DOMPoint(0, 0)
    ));
  // Logs 100, 100
}

[ffd8]

@davepagurek This all goes wooosh over my head, but very interesting to follow and worked perfect for all big/tiny densities, thanks for having another look at it.

Here's my implementation for a library I'm in early stages of sketching, where it's great to 'flatten' any transformed points in 2D/3D space and this combined with @inaridarkfox4231 example for WEBGL work amazingly, so many thanks for figuring out these complex issues!

Already flagged to do so, and hope this finds its way into v2.0, since it's something I've wished for many times in the past (having been so used to it from Processing world). I think across the various issues on the topic, there have been multiple folks interested to jump in for a contribution. Here's a basic combination of both – of course a fully fledged out implementation would also need to take into account createGraphics() layers, instanceMode, be ready to accept the vector or any combo of x, y, [z] coords, ideally also with aliases that match Processing's set of functions for additional smooth crossover.

function screenVector(v) {
	if(_renderer.drawingContext instanceof CanvasRenderingContext2D) {
		const matrix = new DOMMatrix()
		.scale(1 / pixelDensity())
		.multiply(drawingContext.getTransform());

		return matrix
		.transformPoint(
			new DOMPoint(v.x, v.y)
			);
	} else {
		const _gl = _renderer;
		const camCoord = _gl.uMVMatrix.multiplyPoint(v);
		const ndc = _gl.uPMatrix.multiplyAndNormalizePoint(camCoord);
		const _x = (0.5 + 0.5 * ndc.x) * width;
		const _y = (0.5 - 0.5 * ndc.y) * height;
		const _z = (0.5 + 0.5 * ndc.z);
		return createVector(_x, _y, _z);
	}
}

[davepagurek]

Btw I've assigned this to me to indicate that I'll be the point person for getting this into p5.js 2.0, but it's still open for contributors to implement if anyone is interested!

[Garima3110]

Btw I've assigned this to me to indicate that I'll be the point person for getting this into p5.js 2.0, but it's still open for contributors to implement if anyone is interested!

I would like to implement this! ,would try submitting the PR to this maybe by the end of this week or max next week due to some university stuff going on currently.

[Garima3110]

Since the PR #7113 solves this issue, I am closing this.
Thanks a lot everyone for your suggestions!

[ffd8]

@davepagurek – any ideas what specifically changed between v1.9.4 to 1.10.0 that would have broken that screenVector() function above? Not sure if it's connected to PR #7113 or if it ocured much earlier? In the changes to 1.10.0? Everything worked fine using p5.js v1.9.0 (up to 1.9.4) and now causes a website crash on any of my previous sketches which used WEBGL (this is for an upcoming XYscope JS port, drawing on oscilloscopes). The issue is isolated to when WEBGL is activated, so if I made the following tests using the same function as above:

function screenVector(v) {
	const _gl = _renderer
	const camCoord = _gl.uMVMatrix.multiplyPoint(v)
	const ndc = _gl.uPMatrix.multiplyAndNormalizePoint(camCoord)
	const _x = (0.5 + 0.5 * ndc.x) * width
	const _y = (0.5 - 0.5 * ndc.y) * height
	const _z = (0.5 + 0.5 * ndc.z)
	return createVector(_x, _y, _z)
}

And send the following vector in it (why did it produce negative values?):

v = createVector(50, 50, 50)
print(screenVector(v))
// 1.9.4 returns: {"isPInst":true,"x":501.3333328247071,"y":479.3333317756653,"z":0.9023569102287292}
// 1.10.0+ returns: {"isPInst":true,"x":-351.99999237060547,"y":-373.99997663497925,"z":2.626262671947479}

strangely if I don't give a Z value or have it at zero, it now produces a null:

v = createVector(0, 0, 0) // same without Z
print(screenVector(v))
// 1.9.4 returns: {"isPInst":true,"x":448,"y":426,"z":0.9090909171104431}
// 1.10.0+ returns: {"isPInst":true,"x":448,"y":426,"z":null}

Aha, isolated 2 issues.. if I change one of those values from 0 to something else, it becomes null??

v = createVector(10, 0, 0) // same without Z
print(screenVector(v))
// 1.9.4 returns: {"isPInst":true,"x":457.9999999046326,"y":426,"z":0.9090909171104431}
// 1.10.0+ returns: {"isPInst":true,"x":null,"y":426,"z":null}  = why did the x-axis become null??

v = createVector(0, 10, 0) // same without Z
print(screenVector(v))
// 1.9.4 returns: {"isPInst":true,"x":448,"y":435.99999970793726,"z":0.9090909171104431}
// 1.10.0+ returns: {"isPInst":true,"x":448,"y":null,"z":null}

Which makes me believe the whole thing is crashing any time the incoming Z axis value is zero (which in my case, would be any 2D graphics being drawn in WEBGL that don't use the z-axis).. with any neg/pos value, it doesn't produce null – although it produces way different values than it previously did...

v = createVector(-10, 10, 10)
print(JSON.stringify(screenVector(v)))
// 1.9.4 returns: {"isPInst":true,"x":437.8734178180936,"y":436.12658198272123,"z":0.9078123082088518}
// 1.10.0+ returns: {"isPInst":true,"x":1247.9999923706055,"y":-373.99997663497925,"z":9.090909278392791}

v = createVector(-10, 10, 10)
print(JSON.stringify(screenVector(v)))
// 1.9.4 returns: {"isPInst":true,"x":438.1234568843135,"y":435.8765429214195,"z":0.9103379554218716}
// 1.10.0+ returns: {"isPInst":true,"x":-351.99999237060547,"y":1225.9999766349792,"z":-7.070707237720489}

In v1.9.4, those values stayed relatively similar when changing the Z-axis from +/- 10, however since v1.10.0, the coordinates seem somehow extreme (1200+ on a ~890px canvas?) and are producing negative screen coords.

Am I doing something wrong since the updated version or did a bug get introduced?

[davepagurek]

We separated the internal uModelMatrix and uViewMatrix values and only lazily multiply them together for rendering, so you'll probably want to define your own by doing: const uMVMatrix = _renderer.uModelMatrix.copy().mult(_renderer.uViewMatrix)

[ffd8]

@davepagurek that goes woooosh over my head, but thankyouverymuch for this snippet const uMVMatrix = _renderer.uModelMatrix.copy().mult(_renderer.uViewMatrix)! Returns results as previously expected and previous sketches using WEBGL now work fine with latest p5.js version. Guessing this function used to exist in the _renderer, as it's referenced at the top and worked for many version of p5.js. Should it be re-implemented or it changed for a specific reason elsewhere?

[davepagurek]

Glad it's working for you! This one was an intentional change -- uMVMatrix combines two things: the transform applied by the camera position, and the transform of the object applied via translate(), rotate(), scale(), etc. For a number of reasons it makes sense to not combine all of those, e.g. to do view-dependent lighting effects, or to be able to switch cameras without losing the current transforms or vice versa.

But we're also noticing that a lot of custom code was relying on this as a way to peek under the hood. So I think our more long term solution is to actually provide a real, guaranteed-not-to-change API for this. @Garima3110 has been doing some great work on that front (worldToScreen will be in p5 2.0! #7113), and you can track progress on more APIs that do similar things in this issue: #7342