Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SHM subsystem: Rust #823

Merged
merged 118 commits into from
Apr 19, 2024
Merged
Show file tree
Hide file tree
Changes from 117 commits
Commits
Show all changes
118 commits
Select commit Hold shift + click to select a range
20d5b26
add watchdog codebase
yellowhatter Nov 14, 2023
9bee417
integrated and tested POC
yellowhatter Nov 20, 2023
b78336b
[skip ci] Merge commit '1daaaf512c575b55042b88aa76be94205a93611c'
yellowhatter Nov 21, 2023
d521c05
Merge commit '780ec606ba4000a17b2da0e2b0b668e11d37c65e'
yellowhatter Dec 15, 2023
f643f93
Update shm.rs
yellowhatter Dec 15, 2023
f7d053c
WIP: implemented buffer headers in separate SHM segment, made buffer …
yellowhatter Dec 21, 2023
5523201
- refactored POSIX shm wrapper
yellowhatter Dec 21, 2023
bc202a3
- use posix_shm module for data segment in SharedMemoryManager
yellowhatter Dec 22, 2023
26e0ac5
Merge commit 'd8c27a6e017938f4435d5abea4a0142c9d23ff59'
yellowhatter Dec 22, 2023
0c1e385
WIP on SHM
yellowhatter Jan 11, 2024
43b5391
Fix clippy, additional improvements
yellowhatter Jan 11, 2024
fcc064f
Implement watchdog periodic tasks with optional realtime scheduling
yellowhatter Jan 16, 2024
00a4d1d
[skip ci] WIP on SHM provider\client API
yellowhatter Jan 18, 2024
3ccab12
[skip ci] WIP on SHM API
yellowhatter Jan 22, 2024
2fa05c6
Merge commit 'b669489bc814a758741f09c671ecc3a0683697a0'
yellowhatter Jan 22, 2024
eb608c9
[skip ci] Big WIP on SHM API integration
yellowhatter Jan 24, 2024
5ee84e7
[skip ci] working SHM (tests passing)
yellowhatter Jan 26, 2024
e234699
Merge commit '0943243da1a9eb14644897b13934be25610c4a94'
yellowhatter Jan 26, 2024
db4036a
[skip ci] WIP on API
yellowhatter Feb 5, 2024
6e07611
Merge commit 'e8dca1e0ac7036ee04ab024115d418224492343d'
yellowhatter Feb 5, 2024
6d05776
[skip ci] WIP on SHM API
yellowhatter Feb 8, 2024
fc31e7d
Merge commit '2bd4518334d0a9e7e2703c7fd57dfa4547a96819'
yellowhatter Feb 8, 2024
2dc3991
WIP on API
yellowhatter Feb 16, 2024
b65a575
Merge commit '7ebdb3cab481609989f1e9d3124e75d2851458e5'
yellowhatter Feb 19, 2024
743b081
- added SharedMemoryReader to Session and Runtime building API
yellowhatter Feb 21, 2024
0067f63
[skip ci] correct shm establish logic
yellowhatter Feb 22, 2024
1f1fabd
Remove SharedMemoryFactory
yellowhatter Feb 22, 2024
3f2e8db
[skip ci]
yellowhatter Feb 23, 2024
149ba73
[skip ci] Final updates to SHM provider API
yellowhatter Feb 27, 2024
0acd83d
Merge commit '90617ff606b6d4c89d1a3f166dcee6a2b7d5461a'
yellowhatter Feb 27, 2024
4e3a3d0
ZSlice: safe mutable acces to inner contents and support for copy-on-…
yellowhatter Mar 1, 2024
e6025ec
[skip ci]
yellowhatter Mar 5, 2024
9d69725
Merge commit 'f2e99b6fe07bf7899a458e0a16bfe85e3cdebca6'
yellowhatter Mar 6, 2024
c9ef86e
[skip ci]
yellowhatter Mar 11, 2024
8328652
[skip ci]
yellowhatter Mar 11, 2024
9177fb3
[skip ci]
yellowhatter Mar 12, 2024
f4b86f0
Merge commit '1c8ed89b659a2eb9561684602a03390c4f46e279'
yellowhatter Mar 13, 2024
4dd2611
Solved additive feature problem
yellowhatter Mar 13, 2024
469ff3d
- remove dependency in zenoh-buffers
yellowhatter Mar 13, 2024
a6fcdaf
Merge commit '61838d57b88ba2d06f8e16720d6020a80502a905'
yellowhatter Mar 14, 2024
1d19d7c
fix tests
yellowhatter Mar 14, 2024
f32eb00
- refine buld system to optimize workspace dependencies
yellowhatter Mar 14, 2024
7d3e8af
Merge commit 'c401e3a8e02fec0699030086fc751235203fb0ed'
yellowhatter Mar 14, 2024
0ac16af
Update shm.rs
yellowhatter Mar 14, 2024
98bbf1c
- replace async-std with tokio in zenoh-shm
yellowhatter Mar 14, 2024
25fed05
fix examples
yellowhatter Mar 14, 2024
4e86ea9
ooops
yellowhatter Mar 14, 2024
bbd0074
ignore test with too long SHM segment id for MacOS
yellowhatter Mar 14, 2024
3867fad
lower test complexity to adopt runner capabilities
yellowhatter Mar 14, 2024
0e57da2
ignore test with too long SHM segment id for MacOS
yellowhatter Mar 14, 2024
84d3481
- use md5 hashes to shorten SHM segments OS ids (need for MacOS)
yellowhatter Mar 15, 2024
6bfb777
use crc instead of md5
yellowhatter Mar 15, 2024
6d481a2
- get rid of allocator-api2 as work for Allocator is postponed for a …
yellowhatter Mar 15, 2024
6307a66
move from 'const ID: ProtocolID' to trait interface to support both s…
yellowhatter Mar 18, 2024
f24ac21
support compile-time and runtime ProtocolID setting
yellowhatter Mar 18, 2024
b863763
Add more tests to dedicated execution
yellowhatter Mar 18, 2024
57f1f88
[skip ci] add more *concurrent tests to dedicated execution
yellowhatter Mar 18, 2024
f0453f2
- more SHM API docs
yellowhatter Mar 19, 2024
278940c
Merge commit '0bdd13606f639ca273baf7640dd6773fa021a1db'
yellowhatter Mar 19, 2024
683b2bb
exclude test_default_features for SHM tests
yellowhatter Mar 19, 2024
834b023
Move test_helpers.rs into tests to follow the guideline of integratio…
YuanYuYuan Mar 25, 2024
d26b518
Merge pull request #7 from YuanYuYuan/feat/zenoh-shm/integration-test
YuanYuYuan Mar 25, 2024
ddfb70f
update doc
yellowhatter Mar 25, 2024
bbb3083
Eliminate zenoh-buffers -> zenoh-shm dependency to illustarte the pro…
yellowhatter Mar 25, 2024
4e2e295
fix: Add disabled by default `shared-memory` feature in zenoh-shm
fuzzypixelz Mar 27, 2024
a53042d
[skip ci] client storage interface fix
yellowhatter Mar 28, 2024
8dfb1fa
[skip ci] fix map method interface for SharedMemoryProvider
yellowhatter Mar 29, 2024
89ea297
Merge commit '4e2e295f875b1c8192d70b60ada1f00a297f57fd'
yellowhatter Mar 29, 2024
62d695e
brush-up some things after merge
yellowhatter Mar 29, 2024
d5e09ba
PR review fixes
yellowhatter Mar 29, 2024
3c7a349
Merge commit 'e04c8613d3c33472edcbde6fde5bd4beeb24f2bf'
yellowhatter Mar 29, 2024
69e7f6d
[skip ci] add shm feature to zenoh-ext
yellowhatter Apr 1, 2024
2752a93
[skip ci] oops
yellowhatter Apr 1, 2024
b77d2a7
[skip ci] add shared-memory feature traversing for zenoh-shm dependants
yellowhatter Apr 1, 2024
2b2b150
rename ZSliceShm to ZSliceShmMut
yellowhatter Apr 3, 2024
9342ce5
- fix build
yellowhatter Apr 5, 2024
4264059
[skip ci]
yellowhatter Apr 5, 2024
cec36d5
[skip ci]
yellowhatter Apr 5, 2024
862acad
- remove ZSliceMut API
yellowhatter Apr 8, 2024
ca08987
Merge commit 'b2ef3ddcbd13779553b038b10576e9a761fd015d'
yellowhatter Apr 8, 2024
24fecd6
Merge commit '0926dd36bf81cb56729af63ea39c2c4389bfbad4'
yellowhatter Apr 9, 2024
ba62fe6
- stabby fixed for 1.72
yellowhatter Apr 9, 2024
9e0b980
fix doctests
yellowhatter Apr 9, 2024
89eb8b1
Merge commit '9ba3eb9782bb1f6f32312fa4f633eab51c337340'
yellowhatter Apr 10, 2024
24cd36e
Merge commit '66f46814aa7f24681938dfbe37fdf466fad918bd'
yellowhatter Apr 11, 2024
3fc76f8
- ZSliceShm
yellowhatter Apr 12, 2024
bd6e636
Support ZSliceShm and ZSliceShmMut in Payload
yellowhatter Apr 12, 2024
68212ac
- optimize ZSlicBuffer trait
yellowhatter Apr 12, 2024
62368c3
fix code format errors
yellowhatter Apr 12, 2024
95fdc58
- SHM buffer API evolution
yellowhatter Apr 16, 2024
25f115d
[skip ci] polish Payload API for SHM
yellowhatter Apr 16, 2024
eae6cd9
move SHM Buffer API to separate "slice"module
yellowhatter Apr 16, 2024
134afb4
Improve SHM Buffer API concept
yellowhatter Apr 17, 2024
a6beeb2
Update payload.rs test
yellowhatter Apr 17, 2024
6ea0aff
Merge commit 'a675130d99d5da5dcbd54b915f45e5d35c581322'
yellowhatter Apr 17, 2024
53a7009
fixes after merge
yellowhatter Apr 17, 2024
0ad392f
build fixes
yellowhatter Apr 17, 2024
18706fc
- fix recursion error for in SHM buffer API
yellowhatter Apr 17, 2024
e8fe54f
- implement trait API for SHM buffers
yellowhatter Apr 17, 2024
b4807b7
Merge protocol_changes
Mallets Apr 18, 2024
8723bc2
Fix merge
Mallets Apr 18, 2024
c0af61c
Rework Serialize trait
Mallets Apr 18, 2024
f41b54b
Impl &mut T for Serialize/Deserialize. Fix valgrind CI.
Mallets Apr 18, 2024
1221a98
Update commons/zenoh-shm/src/lib.rs
Mallets Apr 18, 2024
3df753b
Revert wrong change on log
Mallets Apr 18, 2024
0ffad29
Merge branch 'shm_watchdog_poc' of github.com:ZettaScaleLabs/zenoh in…
Mallets Apr 18, 2024
5ffadfc
Merge protocol_changes
Mallets Apr 18, 2024
0903424
Update zenoh/src/bytes.rs
Mallets Apr 18, 2024
71f1b02
Fix use
Mallets Apr 18, 2024
48979a2
Merge branch 'shm_watchdog_poc' of github.com:ZettaScaleLabs/zenoh in…
Mallets Apr 18, 2024
a1cd67e
Fix use
Mallets Apr 18, 2024
1dd2c0c
Review fixes
yellowhatter Apr 18, 2024
54691f6
fix recursive call
yellowhatter Apr 18, 2024
e79bd02
Update commons/zenoh-shm/src/api/provider/types.rs
Mallets Apr 18, 2024
94f37d8
unstable for shm slice traits
yellowhatter Apr 18, 2024
39ee7f2
Add more #[zenoh_macros::unstable_doc]
yellowhatter Apr 18, 2024
cb0908e
SHM establishment reorg
Mallets Apr 19, 2024
aed5e14
add missing ztimeout! in tests
yellowhatter Apr 19, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 7 additions & 1 deletion .config/nextest.toml
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,15 @@ slow-timeout = { period = "60s", terminate-after = 2 }
filter = """
test(=zenoh_session_unicast) |
test(=zenoh_session_multicast) |
test(=zenoh_unicity_p2p) |
test(=zenoh_unicity_brokered) |
test(=transport_tcp_intermittent) |
test(=transport_tcp_intermittent_for_lowlatency_transport) |
test(=three_node_combination)
test(=three_node_combination) |
test(=watchdog_alloc_concurrent) |
test(=header_check_memory_concurrent) |
test(=header_link_concurrent) |
test(=header_link_failure_concurrent)
"""
threads-required = 'num-cpus'
slow-timeout = { period = "60s", terminate-after = 6 }
9 changes: 8 additions & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,9 @@ jobs:
- name: Clippy unstable targets
run: cargo +stable clippy --all-targets --features unstable -- --deny warnings

- name: Clippy shared memory without unstable
run: cargo +stable clippy --all-targets --features shared-memory -- --deny warnings

- name: Clippy all features
if: ${{ matrix.os == 'ubuntu-latest' || matrix.os == 'macOS-latest' }}
run: cargo +stable clippy --all-targets --all-features -- --deny warnings
Expand Down Expand Up @@ -92,8 +95,12 @@ jobs:
run: cargo nextest run --exclude zenoh-examples --exclude zenoh-plugin-example --workspace

- name: Run tests with SHM
if: ${{ matrix.os == 'macOS-latest' || matrix.os == 'windows-latest' }}
run: cargo nextest run -F shared-memory -F unstable -E 'not (test(test_default_features))' --exclude zenoh-examples --exclude zenoh-plugin-example --workspace

- name: Run tests with SHM + unixpipe
if: ${{ matrix.os == 'ubuntu-latest' }}
run: cargo nextest run -F shared-memory -F transport_unixpipe -p zenoh-transport
run: cargo nextest run -F shared-memory -F unstable -F transport_unixpipe -E 'not (test(test_default_features))' --exclude zenoh-examples --exclude zenoh-plugin-example --workspace

- name: Check for feature leaks
if: ${{ matrix.os == 'ubuntu-latest' }}
Expand Down
132 changes: 129 additions & 3 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 4 additions & 0 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -112,9 +112,11 @@ lazy_static = "1.4.0"
libc = "0.2.139"
libloading = "0.8"
tracing = "0.1"
lockfree = "0.5"
lz4_flex = "0.11"
nix = { version = "0.27", features = ["fs"] }
num_cpus = "1.16.0"
num-traits = { version = "0.2.17", default-features = false }
ordered-float = "4.1.1"
panic-message = "0.3.0"
paste = "1.0.12"
Expand Down Expand Up @@ -146,6 +148,7 @@ serde_cbor = "0.11.2"
serde_json = "1.0.114"
serde-pickle = "1.1.1"
serde_yaml = "0.9.19"
stabby = "4.0.5"
sha3 = "0.10.6"
shared_memory = "0.12.4"
shellexpand = "3.0.0"
Expand All @@ -159,6 +162,7 @@ tokio-util = "0.7.10"
tokio-tungstenite = "0.21"
tokio-rustls = "0.25.0"
# tokio-vsock = see: io/zenoh-links/zenoh-link-vsock/Cargo.toml (workspaces does not support platform dependent dependencies)
thread-priority = "0.15"
console-subscriber = "0.2"
typenum = "1.16.0"
uhlc = { version = "0.7.0", default-features = false } # Default features are disabled due to usage in no_std crates
Expand Down
9 changes: 3 additions & 6 deletions ci/valgrind-check/src/queryable_get/bin/z_queryable_get.rs
Original file line number Diff line number Diff line change
Expand Up @@ -22,22 +22,19 @@ async fn main() {

let _z = zenoh_runtime::ZRuntimePoolGuard;

let queryable_key_expr = KeyExpr::try_from("test/valgrind/data").unwrap();
let queryable_key_expr = keyexpr::new("test/valgrind/data").unwrap();
let get_selector = Selector::try_from("test/valgrind/**").unwrap();

println!("Declaring Queryable on '{queryable_key_expr}'...");
let queryable_session = zenoh::open(Config::default()).res().await.unwrap();
let _queryable = queryable_session
.declare_queryable(&queryable_key_expr.clone())
.declare_queryable(queryable_key_expr)
.callback(move |query| {
println!(">> Handling query '{}'", query.selector());
let queryable_key_expr = queryable_key_expr.clone();
zenoh_runtime::ZRuntime::Application.block_in_place(async move {
query
.reply(
queryable_key_expr,
query.value().unwrap().payload().clone(),
)
.reply(queryable_key_expr, query.value().unwrap().payload().clone())
.res()
.await
.unwrap();
Expand Down
Loading