I want to know some functions of handling received data

[suoni]

recently I study the medical 3D brower,I'm excited to find out your project,when the client receive the message,after reading the bufferData,I want to know what are some simply funcitons to construct the model,looking forward to your replay,thank you

[yngve-sk]

Hi, I'm not sure what you mean by "the model", you mean the 3D rendering itself, or the underlying in-memory in-browser dataset used to render the 3D model?

It's a while since I worked on this, but what happens in broad terms is this:

1. The dataset is loaded as a VolumeDataset (see volume-dataset.js)
2. The isovalues are loaded into GPU memory
3. The rendering then only uses isovalues in the GPU memory (see glsl shader code for exactly how it is done), and rendering the detail is controlled by propagating uniforms to GPU memory.

So there's mainly 3 layers:

1. The isovalue dataset (stored in GPU memory)
2. The count-map histogram (stored in regular memory)
3. Selections and interaction settings (simply some uniforms, controlling camera position, light parameters etc (propagated to GPU memory when changed)

Hope this helps

[suoni]

Hi! thank you for your reply,I understand.but I want to know something else,when the client receives the isovalues.what is the next? which function perform next?

I want to learn your project to build my demo which just shows 3D. I got to know that your server how to handle the datasets.but I do not know the client how to do this,I need't to know the basic principle of the rendering. I just hope you could tell me something about your project,for example some functions code or articles.

thank you very much! haha

[yngve-sk]

Aha, check out the angular view controller:
client/src/angular-assets/ng-controllers/dataset-view-controller.js

The logic for loading a list of datasets, + what is done when the dataset is received is there. Check the
$scope.loadSelectedDataset method, it is called when a dataset is selected and loaded.

For the GLSL code, if you search the repo for .glsl you'll get the results.
For 3D volume rendering, academic literature is your best source but there's also some good blog posts, mainly stuff on raycasting, lighting, how shaders work and so on.

Here's what is done
For each pixel (fragment shader), shoot a ray through the isovalues.
For each isovalue the ray steps through, apply lighting, shadows and whatever other effects you want. These effects are calculated mathematically, some are really simple, some are more complicated.

Real Time Volume Graphics is a good resource on this, but you could also just search up some blog posts on ray casting and basic 3D stuff like lighting, shadows etc. If you're at a university you can look up papers on it and implement those.

You do need some existing knowledge to understand this. Like, how to get the data to the shaders, what will be in your vertex shaders, fragment shaders, lighting, some simple 3D vector math, looking up values from 2D / 3D arrays as textures in shaders, light / shadow compositing and so on. If none of this makes sense to you right now you'll probably need to read up quite a bit.

As for finding out what happens with code, it'll probably be more helpful for you to simply download and run my code, and use the chrome debugger and step through to the code, then you can see exactly what happens.

[suoni]

I will read your reply carefully，thank you for your help indeed;

[suoni]

Hi! thank you very much. your advice give me a lot of help. recently I have read your code，I'd like to ask you a few questions, if you don't mind.

1. what's the role of the VolumeDataset[header]
2. whats the mean of the 'histogram'
3. the server first sends the VolumeDataset[header], and then sends the Int16Array, why does send it the Int16Array directly

Could you tell me the reason ? thank you

[yngve-sk]

Hi, sure

1. The header contains some info about how the data is stored, cell spacing and so on, sampling rates may vary in the x, y, z direction, so for example a 10x10x10 cube may represent a 10x10x30 dataset. It was the most convenient way to send it since otherwise you'd have to send it somehow combined with the Int16Array. Read up on websockets, you can send strings, array buffers etc, if you have an array buffer, prefer just sending the buffer rather than a stringified version of the buffer.
2. it is the count-map of isovalues. It counts how many times each isovalue (between 0 and 4096 i think) occurs
3. answered in 1)

There probably is better ways but this was my first go with websockets. Originally the plan was to run heavier computations on the server and serve the results to the browser so a websocket connection made sense.

[suoni]

Haha, Your reply is very fast! I am trying to build the 3D model that only depends on JS, not NodeJs. I read the 'volume-dataset.js' file, how to turn the '.dat' file into the theDatasets object. I read the algorithm codedull and dry.Could you recommend some articles or blogs? thank you

[yngve-sk]

Hi, can't say I know of it other than what you can look up on google. You can try load the .dat file into the browser and run the same routines as is run on the nodejs server. Though you need to read the .ini file because it has the header information with sampling size etc, or you need to specify that manually. The .ini file contains

[DatFile]
oldDat Spacing X=1
oldDat Spacing Y=1
oldDat Spacing Z=1

And this information varies per dataset, if you are to build something you should consider the dataset type etc, and format.

That's about all I can say without going into excruciating detail on stuff you'll find just from googling around (which I sadly have no time for :D). What you're trying to do is absolutely possible though

good luck!

[suoni]

Hi, now I have already put the volume-dataset.js on the chrome. haha! when the chrome runs this file, it parses and handles the .dat files those are above 200M, the chrome runs very slowly, that's the reason why you use the Nodejs ? so help me to solve some problems:

1. Absolutely, this project can run in the brower, not depends on Nodejs, why do you use it?
2. I am a front-end developer, for some reasons, I only use WebGL2.0, your project has a certain refenence value, how can I improve the performance on the chrome with your algorithms? I can not use Nodejs or else.
3. I hope it can deal with the files those are above 200M, or it dose not make sense in the medical, so I want to optimize this project in the brower, that's what I want to do

thank you very much

[yngve-sk]

Hi, yes that is why!

2. Hmh not sure, I used webGL + some experimental extensions if I remember right. Improving performance... See next answer:

1+3)There's only so much you can do as far as creating the isovalues, gradient etc since it is a traversal of a typed 3D array represented as a 1D array. Though, after you have done the traversal, your best bet would be to load it into GPU memory and delete it from regular memory, if you can somehow store all the info in the GPU, as well as query your GPU for information you'd be good to go performance wise. This is why ideally, that kind of processing is written in C/C++ or some other highly performant language, you can achieve similar speed with JS using the typed arrays, but if you're gonna run it in the browser you are going to have worse performance so that is a challenge.

TL:DR Do as much as you can on the GPU, it may be tricky and may not be possible, I don't know, but that's probably your best bet. There are in-GPU databases and whatnot existing. If you do manage to find a way I'd be very interested in seeing how you did it! :D

[suoni]

haha, thank you, You are so friendly, now I am excited to share some results with you:

1. when the Chrome deals with the 220M file, It takes 10000 - 12000ms to produce the dataset, when it shows 3D model, It takes about 500 -1000ms , My laptop configuration is a bit poor, So I think it will perform better when it run on the hight configuration
2. So the brower takes longer time to produces the dataset , I am trying to optimize the dateset code in order to bear the 800M file, If it can do this, it's wonderful !
   3.Do you know the code which manges the color schemes，such as 'X-Ray', 'Brain' etc? I read angular.js a bit hard.

I am from china, Nice to make friend with you, haha !

[yngve-sk]

Hello. It is fun when I see the ones asking for help is putting in effort. Yeah you will run into trouble with the browser regardless... To edit color scheme, look at the transfer function. The transfer function maps isovalues, or permutations of isovalues+gradient magnitude + curvature, or only iso + gmag to color and opacity values. This mapping can be done in many ways, look up academic articles / blog posts for how to do it, there are many ways. If I recall correctly, the premade color schemes are simply saved states of transfer function output from interaction.

Good luck!

[suoni]

Hello! Thank for your reply, I consider using three.js. In your code, What gets the data to the shaders and renders module is some tools, for example, 'glslify', 'twgl.js' and so on. I don't know how much the risk which uses three.js, I think you thought about it when you studied this project. Could you give me some advice?

thank you

[yngve-sk]

Sure, threeJS will enable you to things faster but you'll also lose some low level access, and possibly take a performance hit.

glslify basically just allows you to do some importing with glsl, purely a convenience thing.
twgl.js wraps around WebGL2 and helps you easier get your data to your GPU.

[suoni]

Oh, what's the mean of the low level access? ThreeJs takes a performance hit? why? I don't understand this

[suoni]

Hello, Yeah, I get threeJS in order to make it easily indeed, I read your code. There is a lot of theoretical knowledge, I am a Web developer, It's beyond my ability though I study these carefully, But I really want to make it better, At the same time, I find few persons to do this in the brower, I think it's the future of showing medical image, So I hope we can work together to make it fun, haha ,By the way, Do you know how to import the isovalues in three? or threeJs can't do it?

Good luck!