[SYSTEMDS-3546] Push down image pre-processing to blas/mkl #1843

anishsapkota · 2023-06-13T14:58:04Z

This is where i intend to add the native bindings for intel MKL image processing.

Baunsgaard

Hi anishsapkota

Welcome to Opensource,

I suggest you start with doing the following:

Add the .h file with a function definition.
Add the native function in NativeHelper.java
Add a test verifying behavior of your new function, you do this by adding a new folder in src/test/java/org/apache/sysds/test/functions and call it native, inside this make a java file named something appropriate.

Furthermore, I suggest initially to make something that just hooks up and call the cpp without actually doing much to verify the binding and then after this make the MKL work.

src/main/cpp/libAnish.cpp

Baunsgaard

This is nice progress,

I understand based on your comment that commenting out other code this was the only way for you to make it work.

But we should strive to find a way to make all of it work (your point as well)
perhaps the limitation is that you do not have BLAS installed as well, and maybe there is a requirement that you need to install both to compile the library?

Alternatively we can add a new cpp file called systemds_img.cpp and header file that you hook into. But then you need to make a new NativeHelper class instance that then hooks up to the new file. In general i would be interested if there was a nice way of doing such a thing.

Thanks

src/main/java/org/apache/sysds/utils/NativeHelper.java

src/test/java/org/apache/sysds/test/functions/native/NativeBindingTestWithDgemm.java

src/main/cpp/libAnish.cpp

Baunsgaard · 2023-07-07T10:10:46Z

Hi @anishsapkota,

How is it going? is there uncommitted progress or do you need any further comments?
Also, feel free if any of the comments are resolved, to click resolved yourself.

…pen blas, img_translate implemented

…ingTestWithDgemm.java this was just for initial testing to verify the binding, can be deleted now.

anishsapkota · 2023-09-09T15:57:48Z

Hi @Baunsgaard ,
I have been loading the shared libraries manually on my tests as the Native Helper currently supports LINUX and Windows. I have been working using MAC_OS and the libraries are therefore not loaded automatically. Could you please suggest me something on this?

Apart from this, I have created a new file systemds_img.cpp and add the methods that I have implemented there, added nativ function tests to JavaTest.yaml and created a new Java Class ImgNativeHelper that extends NativeHelper.

src/test/java/org/apache/sysds/test/functions/nativ/ImgUtilsTest.java

anishsapkota · 2023-09-11T12:01:42Z

Hi @Baunsgaard ,
I am waiting for the tests to run, so that we can merge.

regarding the performance results, should I be comparing the performance between OpenBlas and Intel MKL ? or with the existing DML implementation?

Baunsgaard · 2023-09-11T12:27:45Z

Hi @Baunsgaard , I am waiting for the tests to run, so that we can merge.

regarding the performance results, should I be comparing the performance between OpenBlas and Intel MKL ? or with the existing DML implementation?

You need to compare the results between all possible ones you have.
But most importantly between DML Java version and your MKL.

Baunsgaard · 2023-09-11T12:29:39Z

Regarding the tests, I need to allow them to run since this is your first commit it does not allow them to automatically run.
I will keep an eye on it and run the tests when you commit, if you feel like you are waiting to long just write a comment.

anishsapkota · 2023-09-11T13:07:38Z

java.lang.UnsatisfiedLinkError: /github/workspace/src/main/cpp/lib/libsystemds_openblas-Linux-x86_64.so: libopenblas.so.0: cannot open shared object file: No such file or directory

I still can't figure it out why it can't find the lib. could you please review the static block from my last commit ?

Baunsgaard · 2023-09-15T08:58:54Z

Hi @Baunsgaard

System.IO.IOException: No space left on device : '/home/runner/runners/2.309.0/_diag/Worker_20230915-060835-utc.log'
Unhandled exception. System.IO.IOException: No space left on device : '/home/runner/runners/2.309.0/_diag/Worker_20230915-060835-utc.log'

https://github.com/apache/systemds/actions/runs/6191936104

well, i guess we need to cleanup some of the resources.
I think we only have 8GB disk space, but i might remember it wrong.

what we need to do is to remove cached files from install of our docker image to reduce its size via apt-get clean

docker/testsysds.Dockerfile

Then include MKL and OpenBLAS in the image rather than installing them in the build script of cpp and make sure we do not keep around anything not needed in our install.

But this is out of scope of this PR, i would prefer if we make it pass as it is and do the Docker refinements some other time.

anishsapkota · 2023-09-15T09:11:43Z

@Baunsgaard @j143
Hi,
after some intensive trying using gitpod I could conclude following things:

libmkl_rt.so cannot open the file: no such file or directory this error was because ldconfig could not find the exported LD_LIBRARY_PATH. I could solve this problem by adding a .conf file with /opt/intel/oneapi/mkl/latest/lib/intel64 in /etc/ld.so.conf.d directory and reloading ldconfig.
Next Thanks to @j143 I stumbled across his video about Intel MKL installation on ubuntu. If I would do it just like he mentioned using a offline package installer which has CLI based UI and requires user interaction ( I could not find a way to do this using shell script), everything would work fine after doing what I mentioned in 1.
If we were to use apt package installer instead of offline one, somehow the include folder containing all the necessary header file like "mkl.h" is missing in mkl root folder. I tried using "locate mkl.h" no result.
I could not find the solution to this.
Next option would be to install whole intel-basekit which contains MKL as well. It is unfortunately too large ~13 GB I think and could have led to the not enough space error.
Regarding "mkl_dnn.h" .I have mentioned here [SYSTEMDS-3546] Push down image pre-processing to blas/mkl #1843 (comment), if we download it unzip it, we have 2 folders in it include and lib, then copy "mkl_dnn.h" and "mkl_dnn_types.h"(second name is something similar) for its include folder to /opt/intel/oneapi/mkl/latest/include (if present) and the libs to /opt/intel/oneapi/mkl/latest/lib/intel64. Using this trick I could compile the cpp files with "mkl_dnn.h"header file as well. But not recommended it was just an experiment.
Nevertheless, I could run the benchmarks using both mkl and openblas, also DML implementations using -stats flag in gitpod. Although I could not install mkl properly in GitHub test docker container. I will process the results and post it.

@Baunsgaard should I comment out "mkl test case" so that we can merge? or how should I proceed now?

Baunsgaard · 2023-09-15T09:35:16Z

@Baunsgaard @j143 Hi, after some intensive trying using gitpod I could conclude following things:

1. **libmkl_rt.so cannot open the file: no such file or directory** this error was because ldconfig could not find the exported LD_LIBRARY_PATH. I could solve this problem by adding a **.conf file** with **/opt/intel/oneapi/mkl/latest/lib/intel64** in **/etc/ld.so.conf.d** directory and reloading **ldconfig.**

2. Next Thanks to @j143 I stumbled across his video about Intel MKL installation on ubuntu. If I would do it just like he mentioned using a **offline package installer** which has CLI based UI and requires user interaction ( _I could not find a way to do this using shell script_), everything would work fine after doing what I mentioned in 1.
   If we were to use **apt package installer** instead of offline one, somehow the include folder containing all the necessary header file like "mkl.h" is missing in mkl root folder. I tried using "locate mkl.h" no result.
   I could not find the solution to this.
   Next option would be to install whole intel-basekit which contains MKL as well. It is unfortunately too large ~13 GB I think and could have led to the not enough space error.

3. Regarding "mkl_dnn.h" .I have mentioned here [[SYSTEMDS-3546] Push down image pre-processing to blas/mkl #1843 (comment)](https://github.com/apache/systemds/pull/1843#issuecomment-1718879255), if we download it unzip it, we have 2 folders in it include and lib, then copy "mkl_dnn.h" and "mkl_dnn_types.h"(second name is something similar) for its include folder to /opt/intel/oneapi/mkl/latest/include (if present) and the libs to /opt/intel/oneapi/mkl/latest/lib/intel64. Using this trick I could compile the cpp files with "mkl_dnn.h"header file as well. **But not recommended it was just an experiment.**

4. Nevertheless, I could run the benchmarks using both mkl and openblas, also DML implementations using -stats flag in gitpod. Although I could not install mkl properly in GitHub test docker container. I will process the results and post it.

@Baunsgaard should I comment out "mkl test case" so that we can merge? or how should I proceed now?

lets go with the Blas test then and add some TODO that reference this PR about the MKL tests, then I will try to make it work at a later time, since i need it for other things.

Install OneAPi Test

Baunsgaard

I like it, nice implementation.

My main comment is the comparison is not fair. There are some things we can do to improve it if you want (read the detailed comments for more).

Before we merge it we need to address all the comments, but you do not need to do this before your presentation Monday, and it is up to you if you even want to.

Thanks!

Baunsgaard · 2023-09-15T15:04:47Z

.github/workflows/javaTests.yml

-        path: target/jacoco.exec
-        retention-days: 1
+      - name: Checkout Repository
+        uses: actions/checkout@v4


we need to revert here, since the change is only syntax.

Baunsgaard · 2023-09-15T15:05:48Z

src/main/cpp/build.sh

+
+  apt-get update
+  apt-get install libopenblas-dev -y
+fi


add comment on MKL install.

Baunsgaard · 2023-09-15T15:06:37Z

src/main/cpp/imgUtils.cpp

+        cout <<  endl;
+    }
+    cout << "\n"<< endl;
+}


i suggest removing this debugging printImage.

Baunsgaard · 2023-09-15T15:07:07Z

src/main/cpp/imgUtils.cpp

+               cblas_dcopy(end_x - start_x, &img_in[(x_in) + (y_in) * in_w],
+                           1, &img_out[y * out_w + start_x + static_cast<int>(offset_x)], 1);
+           }
+       }


indentation.

Baunsgaard · 2023-09-15T15:07:54Z

src/main/cpp/imgUtils.cpp

+    return img_out;
+}
+
+void img_translate(double* img_in, double offset_x, double offset_y,


cast the offsets in the first line rather than every call to offset_x and y, and maybe consider changing the API to use int offsets.

Baunsgaard · 2023-09-15T15:15:36Z

src/main/java/org/apache/sysds/utils/ImgNativeHelper.java

+
+import java.io.File;
+
+public class ImgNativeHelper extends NativeHelper {


you can add doc on the Helper class itself.
i would suggest a hint at where to find the CPP files.

Baunsgaard · 2023-09-15T15:16:47Z

src/main/java/org/apache/sysds/utils/ImgNativeHelper.java

+                System.load(System.getProperty("user.dir")+"/src/main/cpp/lib/libsystemds_" + blasType + "-Darwin-x86_64.dylib");
+            }
+        } catch (Exception e) {
+            e.printStackTrace();


i would like this to throw the exception, not just print stack trace.

you can wrap the exception in a throw new RuntimeException(e);

Baunsgaard · 2023-09-15T15:21:05Z

src/test/java/org/apache/sysds/test/functions/nativ/ImgUtilsTest.java

+        assertArrayEquals(expectedOutput, img_out, 1e-9); // Compare arrays with a small epsilon
+    }
+
+    public void runTests(String blasType) {


i understand why you designed it this way, but please make the test integrate with JUnit,

To do this remove the runTests method and make all your static calls annotated with @ Test.
Then make the imgNativeHelper a field in this class that the constructor instantiate.
Then finally the class needs to be changed into a parameterized class, that calls the constructor selecting either BLAS or MKL.

Hi @anishsapkota , there are a plenty of tests in the neighboring folders you can take inspiration from!

Baunsgaard · 2023-09-15T15:22:57Z

src/test/java/org/apache/sysds/test/functions/nativ/PerformanceComparison.java

+
+import java.util.Random;
+
+import static org.apache.sysds.utils.ImgNativeHelper.*;


no wildcard imports allowed.

Baunsgaard · 2023-09-15T15:23:27Z

src/test/java/org/apache/sysds/test/functions/nativ/PerformanceComparison.java

+    }
+
+    public void runBlasTests(boolean sparse,int n, int seed) {
+    double spSparse = 0.1;


indentation

anishsapkota · 2023-09-15T21:29:37Z

My main comment is the comparison is not fair. There are some things we can do to improve it if you want (read the detailed comments for more).

Hi @Baunsgaard ,
thank your for reviewing the code and the suggestions. Rest assured, I will address all of them after my presentation.

could you be more specific, how can I make the comparison fairer? I will be incorporating your previous suggestions of calculating std and analysing dml execution time using -stats flag. Do you have any further suggestions?
It would be interesting for the presentation to address this issue before that.

Baunsgaard · 2023-09-16T09:41:13Z

My main comment is the comparison is not fair. There are some things we can do to improve it if you want (read the detailed comments for more).

Hi @Baunsgaard , thank your for reviewing the code and the suggestions. Rest assured, I will address all of them after my presentation.

could you be more specific, how can I make the comparison fairer? I will be incorporating your previous suggestions of calculating std and analysing dml execution time using -stats flag. Do you have any further suggestions? It would be interesting for the presentation to address this issue before that.

The fairest comparison would be to integrate the native instruction as a DML supported command. It does not take long to make. But requires you to know where to do it.
Once integrated we can compare with the overhead of the system on both executions and thereby get a fair comparison.

For now the stats comparison is close enough.

anishsapkota · 2023-09-17T22:31:23Z

The fairest comparison would be to integrate the native instruction as a DML supported command. It does not take long to make. But requires you to know where to do it.

could you please let me know which files I should be looking at, so that I can implement this as well ?

anishsapkota · 2023-09-17T22:31:34Z

anishsapkota · 2023-09-17T22:31:52Z

anishsapkota · 2023-09-17T22:32:03Z

anishsapkota · 2023-09-17T22:32:11Z

anishsapkota · 2023-09-17T22:35:33Z

DML Implementations Benchmark sample with -stats flag

image_rotate

SystemDS Statistics:
Total elapsed time: 1,842 sec.
Total compilation time: 0,042 sec.
Total execution time: 1,800 sec.
Number of compiled Spark inst: 1.
Number of executed Spark inst: 0.
Cache hits (Mem/Li/WB/FS/HDFS): 1835090/0/0/0/1.
Cache writes (Li/WB/FS/HDFS): 3/14/0/1.
Cache times (ACQr/m, RLS, EXP): 0,946/0,352/1,060/0,058 sec.
HOP DAGs recompiled (PRED, SB): 0/0.
HOP DAGs recompile time: 0,167 sec.
Spark ctx create time (lazy): 0,000 sec.
Spark trans counts (par,bc,col):0/0/0.
Spark trans times (par,bc,col): 0,000/0,000/0,000 secs.
Spark async. count (pf,bc,op): 0/0/0.
ParFor loops optimized: 1.
ParFor optimize time: 0,001 sec.
ParFor initialize time: 0,004 sec.
ParFor result merge time: 0,028 sec.
ParFor total update in-place: 0/262144/262168
Total JIT compile time: 36.02 sec.
Total JVM GC count: 175.
Total JVM GC time: 0.374 sec.
Heavy hitter instructions:
Instruction Time(s) Count
1 rightIndex 3,288 786443
2 m_img_rotate 1,711 1
3 m_img_transform 1,711 1
4 rmvar 1,434 2359350
5 leftIndex 1,213 262168
6 createvar 0,997 1048631
7 castdts 0,846 524294
8 && 0,544 786432
9 mvvar 0,392 524300
10 < 0,355 524288

image_cutout

SystemDS Statistics:
Total elapsed time: 0,108 sec.
Total compilation time: 0,042 sec.
Total execution time: 0,065 sec.
Number of compiled Spark inst: 1.
Number of executed Spark inst: 0.
Cache hits (Mem/Li/WB/FS/HDFS): 2/0/0/0/1.
Cache writes (Li/WB/FS/HDFS): 0/1/0/1.
Cache times (ACQr/m, RLS, EXP): 0,032/0,000/0,000/0,033 sec.
HOP DAGs recompiled (PRED, SB): 0/0.
HOP DAGs recompile time: 0,000 sec.
Spark ctx create time (lazy): 0,000 sec.
Spark trans counts (par,bc,col):0/0/0.
Spark trans times (par,bc,col): 0,000/0,000/0,000 secs.
Spark async. count (pf,bc,op): 0/0/0.
Total JIT compile time: 13.577 sec.
Total JVM GC count: 176.
Total JVM GC time: 0.368 sec.
Heavy hitter instructions:
Instruction Time(s) Count
1 write 0,033 1
2 sp_rblk 0,032 1
3 rmvar 0,001 2
4 leftIndex 0,000 1
5 rand 0,000 1
6 createvar 0,000 4
7 + 0,000 2
8 < 0,000 1
9 mvvar 0,000 8
10 - 0,000 2

image_crop

New w/h=409.0/409.0
SystemDS Statistics:
Total elapsed time: 0,134 sec.
Total compilation time: 0,029 sec.
Total execution time: 0,105 sec.
Number of compiled Spark inst: 1.
Number of executed Spark inst: 0.
Cache hits (Mem/Li/WB/FS/HDFS): 9/0/0/0/1.
Cache writes (Li/WB/FS/HDFS): 1/8/0/1.
Cache times (ACQr/m, RLS, EXP): 0,037/0,000/0,000/0,063 sec.
HOP DAGs recompiled (PRED, SB): 0/0.
HOP DAGs recompile time: 0,000 sec.
Spark ctx create time (lazy): 0,000 sec.
Spark trans counts (par,bc,col):0/0/0.
Spark trans times (par,bc,col): 0,000/0,000/0,000 secs.
Spark async. count (pf,bc,op): 0/0/0.
Total JIT compile time: 36.052 sec.
Total JVM GC count: 176.
Total JVM GC time: 0.367 sec.
Heavy hitter instructions:
Instruction Time(s) Count
1 write 0,063 1
2 sp_rblk 0,037 1
3 m_img_crop 0,005 1
4 - 0,002 4
5 rmempty 0,001 1
6 rshape 0,000 3
7 rmvar 0,000 8
8 + 0,000 8
9 leftIndex 0,000 1
10 rand 0,000 2

image_translate

SystemDS Statistics:
Total elapsed time: 0,166 sec.
Total compilation time: 0,049 sec.
Total execution time: 0,117 sec.
Number of compiled Spark inst: 1.
Number of executed Spark inst: 0.
Cache hits (Mem/Li/WB/FS/HDFS): 3/0/0/0/1.
Cache writes (Li/WB/FS/HDFS): 0/3/0/1.
Cache times (ACQr/m, RLS, EXP): 0,039/0,000/0,000/0,077 sec.
HOP DAGs recompiled (PRED, SB): 0/0.
HOP DAGs recompile time: 0,000 sec.
Spark ctx create time (lazy): 0,000 sec.
Spark trans counts (par,bc,col):0/0/0.
Spark trans times (par,bc,col): 0,000/0,000/0,000 secs.
Spark async. count (pf,bc,op): 0/0/0.
Total JIT compile time: 13.584 sec.
Total JVM GC count: 176.
Total JVM GC time: 0.367 sec.
Heavy hitter instructions:
Instruction Time(s) Count
1 write 0,077 1
2 sp_rblk 0,039 1
3 m_img_translate 0,001 1
4 leftIndex 0,000 1
5 rmvar 0,000 12
6 rand 0,000 1
7 rightIndex 0,000 1
8 createvar 0,000 5
9 < 0,000 8
10 mvvar 0,000 10

Baunsgaard · 2023-09-18T11:27:59Z

In the results you have above the time of the Heavy hitter instructions: is what i would use to compare. The rest of the time reported is all overhead of Java , I/O and other compilation related elements.
But i would modify the script you use. A example of a measuring script:

res = read($1)
print(sum(res))
for(i in 1:$2) {
    res2 = OPERATION(res)
}
print(sum(res2))

This reads the file in the first argument and then repeats the operation the number of times specified in the second argument. Because of the way we execute in SystemDS the sum before the loop make sure res is actually read from disk.

The downside to this approach is that it is impossible/hard for you to get the individual calls execution time.

j143 · 2023-09-24T08:45:13Z

Hi @anishsapkota -- let's move this forward, request the reviewer to commit this much by addressing the comments (which seem minor).

could you please let me know which files I should be looking at, so that I can implement this as well ?

after that you can add further functionality. This would remove the burden on you & merge conflicts.

Thanks, Janardhan

j143 · 2023-11-06T04:04:40Z

Hi @anishsapkota -- thanks for your contribution. do you need some help here?

j143 · 2023-12-04T03:11:46Z

Hi @Baunsgaard , this PR does seems fine. only minor comments are there. Is it okay to take this in?

Baunsgaard · 2023-12-04T14:50:23Z

Hi @Baunsgaard , this PR does seems fine. only minor comments are there. Is it okay to take this in?

Hi @j143 , it can be taken in but we need to be careful, and go through the code, since some of the changes (for instance GitHub actions) contain changes that should not be made.

corepointer · 2023-12-14T11:40:09Z

Hi!

Sorry that I'm a bit late to the discussion.

Our current Intel MKL support ends at version 2019.5 (you noticed the problems with dnn ops not compiling). I would be surprised if all magically worked by just using OneMKL and OneDNN.

We should probably put the native ops testing into a separate test suite and use a Docker container that supports it.

Furthermore, moving to OneAPI would change external behavior and require a major version bump.

Regards, Mark

j143 · 2024-02-08T13:58:44Z

marking this PR for next release because there are some concerns pointed by @corepointer

Our current Intel MKL support ends at version 2019.5 (you noticed the problems with dnn ops not compiling). I would be surprised if all magically worked by just using OneMKL and OneDNN.

We should probably put the native ops testing into a separate test suite and use a Docker container that supports it.

Furthermore, moving to OneAPI would change external behavior and require a major version bump.

[ÂSYSTEMDS-3546] good start

4430848

Baunsgaard reviewed Jun 14, 2023

View reviewed changes

src/main/cpp/libAnish.cpp Outdated Show resolved Hide resolved

[SYSTEMDS-3546] initial mkl native binding tested

b07c498

Baunsgaard reviewed Jun 16, 2023

View reviewed changes

Baunsgaard changed the title ~~[SYSTEMDS-3546] good start~~ [SYSTEMDS-3546] Push down image pre-processing to blas/mkl Jun 16, 2023

anishsapkota and others added 9 commits July 14, 2023 13:43

image rotation using mkl lib initial implementation

a222abb

image rotation using mkl lib initial implementation

4ce1d42

image rotation using mkl lib initial implementation

7698dfe

image rotation using mkl lib initial implementation

a8934f3

Delete src/test/java/org/apache/sysds/test/functions/native directory

8f96891

image cutout and crop initial implementation / img rotate bug fixed

ce2cd82

image cutout and crop initial implementation / img rotate bug fixed

ac0bc85

OpenBlas Bug fixed, now all implementations works with both mkl and o…

7da4708

…pen blas, img_translate implemented

Delete src/test/java/org/apache/sysds/test/functions/nativ/NativeBind…

495ea9f

…ingTestWithDgemm.java this was just for initial testing to verify the binding, can be deleted now.

Baunsgaard mentioned this pull request Sep 6, 2023

[SYSTEMDS-2839] IntelMKL Integration Extension #1887

Closed

anishsapkota added 3 commits September 9, 2023 17:27

implemented native methodes moved to new file systemds_img.cpp

2ef022d

ImgNativeHelper created

73b9fa2

License added

59a7bf8

Âadded the missing compiled libs

85284e0

Baunsgaard reviewed Sep 9, 2023

View reviewed changes

src/test/java/org/apache/sysds/test/functions/nativ/ImgUtilsTest.java Outdated Show resolved Hide resolved

anishsapkota added 2 commits September 9, 2023 21:12

static block added to ImgNativeHelper

aa81d73

adjusted native lib path, another try

46481b4

anishsapkota closed this Sep 11, 2023

anishsapkota reopened this Sep 11, 2023

anishsapkota and others added 8 commits September 15, 2023 10:37

test

1243e98

Merge branch 'main' into systemds-3546

b8fd318

removed mkl test

6f4d069

added exception if non square images

6fc5bad

removed

2a0bc24

Merge pull request #2 from anishsapkota/systemds-3546

3862c5b

Install OneAPi Test

corrected javaTest.yaml

c978a62

Merge branch 'main' into main

b43c633

Baunsgaard reviewed Sep 15, 2023

View reviewed changes

j143 added this to the systemds-3.2.0 milestone Dec 4, 2023

j143 modified the milestones: systemds-3.2.0, next-release Feb 8, 2024


		import java.io.File;

		public class ImgNativeHelper extends NativeHelper {


		import java.util.Random;

		import static org.apache.sysds.utils.ImgNativeHelper.*;

[SYSTEMDS-3546] Push down image pre-processing to blas/mkl #1843

Are you sure you want to change the base?

[SYSTEMDS-3546] Push down image pre-processing to blas/mkl #1843

Conversation

anishsapkota commented Jun 13, 2023

Baunsgaard left a comment

Choose a reason for hiding this comment

Baunsgaard left a comment

Choose a reason for hiding this comment

Baunsgaard commented Jul 7, 2023

anishsapkota commented Sep 9, 2023

anishsapkota commented Sep 11, 2023 • edited Loading

Baunsgaard commented Sep 11, 2023

Baunsgaard commented Sep 11, 2023

anishsapkota commented Sep 11, 2023

Baunsgaard commented Sep 15, 2023 • edited Loading

anishsapkota commented Sep 15, 2023 • edited Loading

Baunsgaard commented Sep 15, 2023

Baunsgaard left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

anishsapkota commented Sep 15, 2023

Baunsgaard commented Sep 16, 2023

anishsapkota commented Sep 17, 2023

anishsapkota commented Sep 17, 2023

anishsapkota commented Sep 17, 2023

anishsapkota commented Sep 17, 2023

anishsapkota commented Sep 17, 2023

anishsapkota commented Sep 17, 2023 • edited Loading

DML Implementations Benchmark sample with -stats flag

Baunsgaard commented Sep 18, 2023 • edited Loading

j143 commented Sep 24, 2023 • edited Loading

j143 commented Nov 6, 2023

j143 commented Dec 4, 2023

Baunsgaard commented Dec 4, 2023

corepointer commented Dec 14, 2023

j143 commented Feb 8, 2024

anishsapkota commented Sep 11, 2023 •

edited

Loading

Baunsgaard commented Sep 15, 2023 •

edited

Loading

anishsapkota commented Sep 15, 2023 •

edited

Loading

anishsapkota commented Sep 17, 2023 •

edited

Loading

Baunsgaard commented Sep 18, 2023 •

edited

Loading

j143 commented Sep 24, 2023 •

edited

Loading