Skip to content

Commit

Permalink
Misc. cleanup
Browse files Browse the repository at this point in the history
- Clean up dependencies
- Propagate errors from `FrameStackForWriting.write_frame` closure
  instead of panicking
- Remove some unused code
  • Loading branch information
sk1p committed May 16, 2023
1 parent a0ff7c1 commit b47f5db
Show file tree
Hide file tree
Showing 9 changed files with 21 additions and 204 deletions.
6 changes: 0 additions & 6 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

6 changes: 0 additions & 6 deletions libertem_asi_mpx3/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -18,17 +18,11 @@ crossbeam = "0.8.2"
crossbeam-channel = "0.5.6"
env_logger = "0.9.3"
log = "0.4.17"
memmap2 = "0.5.6"
numpy = "0.17.0"
pyo3 = { version = "0.17", features = ["abi3-py37"] }
serde = { version = "1.0.143", features = ["derive"] }
spin_sleep = "1.1.1"
ipc-test = { path = "../ipc_test" }
serval-client = { path = "../serval-client" }
stats = { path = "../stats" }
nix = "0.26.1"
zerocopy = "0.6.1"
md5 = "0.7.0"

[features]
extension-module = ["pyo3/extension-module"]
Expand Down
154 changes: 0 additions & 154 deletions libertem_asi_mpx3/README.md
Original file line number Diff line number Diff line change
@@ -1,154 +0,0 @@
# LiberTEM-dectris-rs

[![LiberTEM-dectris-rs on GitHub](https://img.shields.io/badge/GitHub-MIT-informational)](https://github.com/LiberTEM/LiberTEM-dectris-rs)

This is a Python package for efficiently receiving data from DECTRIS detectors
with [the zeromq interface](https://media.dectris.com/210607-DECTRIS-SIMPLON-API-Manual_EIGER2-chip-based_detectros.pdf).
The low-level, high-frequency operations are performed in a background thread
implemented in rust, and multiple frames are batched together for further
processing in Python.

## Usage

```python
import numpy as np
import libertem_dectris
from libertem_live.detectors.dectris.DEigerClient import DEigerClient

# trigger acquisition via the REST API, needs `libertem-live`
nimages = 256 * 256
ec = DEigerClient('localhost', 8910) # hostname and port of the DCU REST API
ec.setDetectorConfig('ntrigger', 1)
ec.setDetectorConfig('nimages', 1)
ec.setDetectorConfig('trigger_mode', 'exte')
ec.setDetectorConfig('ntrigger', nimages)
result = ec.sendDetectorCommand('arm')
sequence_id = result['sequence id']

conn = libertem_dectris.DectrisConnection(
uri="tcp://localhost:9999",
handle_path="/tmp/dectris_shm",
frame_stack_size=32,
num_slots=2000,
bytes_per_frame=512*512,
huge=False,
)

# as we have armed the detector above, we know the sequence number
# that we should expect:
# (in other cases, can also call
# `conn.start_passive` and `conn.wait_for_arm` to passively wait for
# the detector to be armed)
conn.start(sequence_id)

# any other process can use a `CamClient` to use data
# stored in the SHM:
cam_client = libertem_dectris.CamClient(conn.get_socket_path())

try:
while True:
# get at most `max_size` frames as a stack
# (might get less at the end of the acquisition)
stack_handle = conn.get_next_stack(max_size=32)

# if the receiver is idle, stack_handle will be None here:
if stack_handle is None:
break

# the expected shape and data type:
frame_shape = tuple(reversed(stack_handle.get_shape()))
dtype = np.dtype(stack_handle.get_pixel_type()).newbyteorder(
stack_handle.get_endianess()
)

# pre-allocate some memory for the pixel data:
# (would be pulled out of the loop in real code)
buf = np.zeros((len(stack_handle),) + frame_shape, dtype=dtype)

# decompress into the pre-allocated buffer
cam_client.decompress_frame_stack(stack_handle, out=buf)

# free up the shared memory slot for this frame stack:
cam_client.done(stack_handle)

# we can still use the decompressed data:
buf.sum()
finally:
conn.close() # clean up background thread etc.
cam_client.close()

```

## Changelog

### v0.2.10

- Add functionality to generate mock data for sending via the simulator
- Add missing numpy dependency
- Make debug output a bit more reliable

### v0.2.9

- Add more debug and trace output

### v0.2.7

- Log more details in `DectrisConnection.log_shm_stats` and change log level to
`INFO`
- Increase timeout for sending headers and frames

### v0.2.4 - v0.2.6

- Updated examples and CI configuration

### v0.2.3

- Add `env_logger`: set environment variable `LIBERTEM_DECTRIS_LOG_LEVEL` to e.g. `'INFO'` to enable logging
- Improved error handling: raise an exception instead of panicing on serialization errors
- Ignore messages with mismatching series ID
- Add explicit checks for the correct `header_detail` levels
- Move code into monorepo

### v0.2.2

- Vendor `bitshuffle` and add `Frame.decompress_into` method, PR [#10](https://github.com/LiberTEM/LiberTEM-dectris-rs/pull/10)

### v0.2.1

- Catch frame ID mismatch, PR [#9](https://github.com/LiberTEM/LiberTEM-dectris-rs/pull/9)

### v0.2.0

- Added `libertem_dectris.headers` submodule that exports header classes
- Added ways to create `libertem_dectris.Frame` and `libertem_dectris.FrameStack`
objects from Python, mostly useful for testing
- Added binding to random port for the simulator
- Properly parametrize with zmq endpoint URI
- Fix many clippy complaints

### v0.1.0

Initial release!

## Development

This package is using [pyo3](https://pyo3.rs/) with
[maturin](https://maturin.rs/) to create the Python bindings. First, make sure
`maturin` is installed in your Python environment:

```bash
(venv) $ pip install maturin
```

Then, after each change to the rust code, run `maturin develop -r` to build and
install a new version of the wheel.

As we vendor `bitshuffle`, make sure to clone with `git clone --recursive ...`, or manually
[take care of initializing and updating submodules](https://github.blog/2016-02-01-working-with-submodules/).

## Release

- update changelog above
- bump version in Cargo.toml if not already bumped, and push
- create a release from the GitHub UI, creating a new tag vX.Y.Z
- done!
6 changes: 4 additions & 2 deletions libertem_asi_mpx3/examples/mpx3_example_03_acquisition.py
Original file line number Diff line number Diff line change
Expand Up @@ -94,7 +94,9 @@ def init_acquisition(serverurl, detector_config):
detector_config["TriggerMode"] = "CONTINUOUS"

# 0.1 second = 10fps
dwell_time = 0.0005
# 0.0005 second = 2000fps
dwell_time = 0.0001
# dwell_time = 0.0005

# Sets the trigger period (time between triggers) in seconds.
detector_config["TriggerPeriod"] = dwell_time
Expand Down Expand Up @@ -177,7 +179,7 @@ def acquisition_test(serverurl):
# What (image) format to write the files in.
"Format": "pgm",
# What data to build a frame from (tot, toa, tof, count)
"Mode": "count"
"Mode": "count",
}]
}

Expand Down
8 changes: 4 additions & 4 deletions libertem_asi_mpx3/examples/simple.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,9 +6,9 @@
data_uri="localhost:8283",
api_uri="http://localhost:8080",
handle_path="/tmp/asi_mpx3_shm",
frame_stack_size=32,
frame_stack_size=16,
num_slots=2000,
bytes_per_frame=512*512,
bytes_per_frame=512*512*2,
huge=False,
)

Expand All @@ -28,7 +28,7 @@
# stored in the SHM:
cam_client = libertem_asi_mpx3.CamClient(conn.get_socket_path())

tq = tqdm.tqdm(total=config.get_n_triggers())
tq = tqdm.tqdm(total=config.get_n_triggers() * 512 * 512 * 2, unit='B', unit_scale=True, unit_divisor=1024)

while True:
# get at most `max_size` frames as a stack
Expand All @@ -44,7 +44,7 @@

frames = cam_client.get_frames(stack_handle)

tq.update(len(frames))
tq.update(len(frames) * 512 * 512 * 2)

del frames # let's hope no-one else keeps a reference, as it will be invalid after `done` is called

Expand Down
1 change: 0 additions & 1 deletion libertem_asi_mpx3/pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,6 @@ classifiers = [
"Programming Language :: Python :: Implementation :: PyPy",
]
dependencies = [
"numpy>=1.16.0",
]

[project.urls]
Expand Down
24 changes: 0 additions & 24 deletions libertem_asi_mpx3/src/cam_client.rs
Original file line number Diff line number Diff line change
Expand Up @@ -85,27 +85,3 @@ impl Drop for CamClient {
trace!("CamClient::drop");
}
}

#[cfg(test)]
mod tests {
use std::path::PathBuf;

use numpy::PyArray;
use tempfile::tempdir;

use ipc_test::SharedSlabAllocator;
use pyo3::{prepare_freethreaded_python, Python};

use crate::{
cam_client::CamClient,
frame_stack::{FrameStackForWriting, FrameStackHandle},
};
use tempfile::TempDir;

fn get_socket_path() -> (TempDir, PathBuf) {
let socket_dir = tempdir().unwrap();
let socket_as_path = socket_dir.path().join("stuff.socket");

(socket_dir, socket_as_path)
}
}
10 changes: 8 additions & 2 deletions libertem_asi_mpx3/src/frame_stack.rs
Original file line number Diff line number Diff line change
Expand Up @@ -60,16 +60,22 @@ impl FrameStackForWriting {

/// Temporarily take mutable ownership of a buffer
/// for a single frame, and receive the data into it.
pub fn write_frame(&mut self, meta: &FrameMeta, mut fill_buffer: impl FnMut(&mut [u8])) {
pub fn write_frame<E>(
&mut self,
meta: &FrameMeta,
mut fill_buffer: impl FnMut(&mut [u8]) -> Result<(), E>,
) -> Result<(), E> {
// FIXME: `fill_buffer` should return a `Result`
let start = self.cursor;
let stop = start + meta.data_length_bytes;
let dest = &mut self.slot.as_slice_mut()[start..stop];
fill_buffer(dest);
fill_buffer(dest)?;

self.meta.push(meta.clone());
self.offsets.push(self.cursor);
self.cursor += meta.data_length_bytes;

Ok(())
}

pub fn writing_done(self, shm: &mut SharedSlabAllocator) -> FrameStackHandle {
Expand Down
10 changes: 5 additions & 5 deletions libertem_asi_mpx3/src/receiver.rs
Original file line number Diff line number Diff line change
Expand Up @@ -263,12 +263,12 @@ fn recv_frame(

let dest_rest = &mut dest_buf[head_src.len()..];

// FIXME: error handling (can we pass the error through as result of the closure?)
// FIXME: this blocks - we need to check for control messages every now and then
stream.read_exact(dest_rest).unwrap();

// trace!("{dest_rest:?}");
});
match stream.read_exact(dest_rest) {
Ok(_) => Ok(()),
Err(e) => Err(AcquisitionError::ConnectionError { msg: e.to_string() }),
}
})?;

Ok(meta)
}
Expand Down

0 comments on commit b47f5db

Please sign in to comment.