v2.0

[decahedron1]

• Refactor session inputs/outputs to allow specifying inputs/retrieving outputs by name.
• Accept more ndarray types for better ergonomics & performance: no more into_dyn() or CowArray::from in most cases
• Make IoBinding more clear and easier to use by merging it into Session::run.
• Make SessionBuilder cloneable (How to clone a SessionBuilder? #70)
• Address unsoundness issues (ort::value::Value is unsound since it exposes raw pointers as public fields #69)
• Fix Value lifetimes (Output goes wrong after being returned by a function #74)
• Add YOLOv8 example.
• Follow SemVer convention.

Fixes #97, #80, #70, #69, #66

TODO

• Add logic to convert between Values and HashMap/Vec for map/sequence types as discussed in Support for sequence/map types #30
• Identify & fix any remaining memory leaks or performance pitfalls.
• Ensure new input/output system is not flaky, make sure dropping Values isn't bug prone. Related: Change field order to drop Environment last #105
• See if we can further reduce memory allocations in hot paths Avoid unnecessary memory allocation in run  #80
• Create a docs site and add more examples for common models, both simple & complex.
• Add tests/examples for IoBinding and ensure it's usable and working as intended HowTo - Notes to use pinned host buffers for cuda and tensorrt #88 Issues using IoBindings #66
• See if I can get my hands on hardware to test NN-API and ROCm EPs NN-API #45 rocm support #16
• Should pyke provide prebuilt static libraries, WASM, DirectML? Provide prebuilt ONNX Runtime binaries with more execution providers support #28

[codecov]

Codecov Report

Attention: 714 lines in your changes are missing coverage. Please review.

Comparison is base (6a1c19d) 51.88% compared to head (34f8161) 37.99%.
Report is 3 commits behind head on main.

❗ Current head 34f8161 differs from pull request most recent head a4e731c. Consider uploading reports for the commit a4e731c to get more accurate results

Files	Patch %	Lines
src/value.rs	48.71%	100 Missing ⚠️
src/session/mod.rs	54.64%	83 Missing ⚠️
src/execution_providers/tensorrt.rs	0.00%	70 Missing ⚠️
src/lib.rs	47.11%	55 Missing ⚠️
src/execution_providers/rocm.rs	0.00%	38 Missing ⚠️
src/execution_providers/cuda.rs	0.00%	35 Missing ⚠️
src/execution_providers/qnn.rs	0.00%	35 Missing ⚠️
src/io_binding.rs	0.00%	31 Missing ⚠️
src/tensor/types.rs	27.50%	29 Missing ⚠️
src/execution_providers/cann.rs	0.00%	28 Missing ⚠️
... and 16 more

Additional details and impacted files

@@             Coverage Diff             @@
##             main      #78       +/-   ##
===========================================
- Coverage   51.88%   37.99%   -13.89%     
===========================================
  Files          13       28       +15     
  Lines        1089     1479      +390     
===========================================
- Hits          565      562        -3     
- Misses        524      917      +393     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[cpgddln]

Thanks for the work on this new version.

The Session::run method returns an ort::Value<'static>. Despite the 'static lifetime, these require the Session to exist to refer to valid memory. The documentation notes:

Note that ORT-owned values will prevent the session from being dropped until all values are dropped; to prevent this behavior, extract the data with [Value::extract_tensor].

Similarly, the OrtOwnedTensor<'static> require the session to exist, and their drop method also seems to be doing some reference counting.

However, the following segfaults in the gpt example:

let outputs = session.run(inputs![&array])?;
let generated_tokens: OrtOwnedTensor<f32, _> = outputs["output1"].extract_tensor()?;
drop(outputs);
drop(session);
println!("{:?}", generated_tokens);

The array is properly displayed when removing the drop.

Either:

• It was assumed that the C-side ORT keeps the memory alive until all Value and OrtOwnedTensor have been dropped, even if the rust Session gets dropped. This does not seem to be the case, and the output Value should be tied to the lifetime of the Session.
• Only the Value does reference counting for the Session, in which case extract_tensor(&'a self) on Value<'static> should return a OrtOwnedTensor<'a>, not a OrtOwnedTensor<'static>

[bharrisau]

Bit of an issue with the SessionOutputs. IoBinding::bind_input takes a Value, but SessionOutputs can only give an &Value. So not possible to use an output as a subsequent input.

Current workaround

let outs = bind.run().unwrap();
assert!(outs.len() == 1);
let out_copy = unsafe { std::ptr::read(&outs["output0"]) };
std::mem::forget(outs); // Because we copied an Arc
bind.bind_input("images", out_copy).unwrap();

[decahedron1]

@bharrisau Thank you, just implemented DerefMut for SessionOutputs which allows you to move the Values out like so:

let output1: Value<'_> = outputs.remove("output1").unwrap();

[bharrisau]

As I mentioned in #88 - there is a significant performance penalty from the CowArray doing an extra copy in Value::from_array. First CowArray.as_standard_layout() which either copies the array if layout is wrong, or else returns a CowRepr::View of the input array. Then CowArray.as_mut_ptr() which will result in a copy if the input is CowRepr::View.

Probably need another way of creating the Value, either taking an &mut ArcArray, or else creating from a slice+shape. With the &mut ArcArray, it looks like you could call .as_mut_ptr() which will skip the copy if the Arc is unique. Then call .clone() and save the clone until the call to run(), then drop the clone and it would be unique again. Should ensure only 1 mutable reference at a time.

[JewishLewish]

Referencing to e694ebc Commit.

symlink works smoothly on Linux 👍
Windows users require for Developer Mode to be turned on.