Add flux normaliser processor #1878

hrobarts · 2024-07-25T11:12:14Z

Description

New FluxNormaliser processor to normalise data

to a float value
to an array of floats with size matching the number of angles in the data
to the mean value calculated from a specified region of interest in the data itself

Example Usage

Changes

Testing you performed

Demo scripts in https://github.com/TomographicImaging/CIL-Demos/blob/flux_normaliser_demo/misc/flux_normaliser.ipynb
Tests in test_DataProcessor.py TestFluxNormaliser class

Related issues/links

#1877

Checklist

I have performed a self-review of my code
I have added docstrings in line with the guidance in the developer guide
I have updated the relevant documentation
I have implemented unit tests that cover any new or modified functionality
CHANGELOG.md has been updated with any functionality change
Request review from all relevant developers
Change pull request label to 'Waiting for review'

Contribution Notes

Please read and adhere to the developer guide and local patterns and conventions.

The content of this Pull Request (the Contribution) is intentionally submitted for inclusion in CIL (the Work) under the terms and conditions of the Apache-2.0 License
I confirm that the contribution does not violate any intellectual property rights of third parties

hrobarts · 2024-08-01T14:16:22Z

Some examples:
Normalise all data with a scalar value

processor = FluxNormaliser(flux=10)
processor.set_input(data)
data_norm = processor.get_output()
show2D([data, data_norm], ['Data','Normalised data'])

Normalise data with a scalar value per projection

processor = FluxNormaliser(flux=np.arange(1,2,(2-1)/(data.get_dimension_size('angle'))))
processor.set_input(data)
data_norm = processor.get_output()
show2D([data, data_norm], slice_list=('vertical',80))

Normalise data using a region in the background of each projection.

roi = {'vertical':(0,14), 'horizontal':(0,14)}
processor = FluxNormaliser(roi=roi)
processor.set_input(data)
processor.show_roi()

The default output of show_roi() shows the roi plotted on the dataset for the first and last projection and the projection which has the maximum variation from the mean intensity (Z-score) in the roi, as well as a plot of the mean intensity per projection.

In this case we can see that although the roi is empty for most projections, there are some cases where there is material in the roi like index 176.
We can also use processor.show_roi(angle_index=250) with a specific angle_index if we want to check a different projection which looks like an outlier

In this case, if we change the roi we can make sure it's empty for all projections

roi = {'vertical':(0,10), 'horizontal':(0,10)}
processor = FluxNormaliser(roi=roi)
processor.set_input(data)
processor.show_roi()

data_norm = processor.get_output()

In this example with synchrotron data, the background has a gradient so the Z-score might not be the best metric to use.

roi = {'vertical':(10,135), 'horizontal':(125,150)}
processor = FluxNormaliser(roi=roi)
processor.set_input(data)
processor.show_roi()

roi = {'vertical':(10,135), 'horizontal':(135,150)}
processor = FluxNormaliser(roi=roi)
processor.set_input(data)
processor.show_roi()

data_norm = processor.get_output()
show2D([data, data_norm], slice_list=('vertical',80))

The gradient is opposite if we use the roi on the other side

roi = {'vertical':(10,135), 'horizontal':(10,32)}
processor = FluxNormaliser(roi=roi)
processor.set_input(data)
processor.show_roi()

data_norm = processor.get_output()
show2D([data, data_norm], slice_list=('vertical',80))

hrobarts · 2024-08-14T16:33:08Z

Updated to plot the mean, minimum and maximum in the roi

hrobarts · 2024-08-19T13:00:29Z

Updated so preview_configuration() still works on data with one angle

Also works on 2D data where the roi is specified with one of horizontal or vertical. Plot the roi on the sinogram

Also works if the dataorder is different

Also works if the data has multiple channels, the flux to normalise to is the mean in the roi averaged over the channels. preview_configuration() shows the central channel by default

or the channel can be specified processor.preview_configuration(channel=0)

Signed-off-by: Hannah Robarts <[email protected]>

hrobarts · 2024-08-20T17:22:26Z

@gfardell we discussed whether the preview_configuration should just be an option in logging. At the moment I've left it as a function the users can access - what do you think?

Wrappers/Python/cil/processors/FluxNormaliser.py

lauramurgatroyd

This is looking great and super useful! Excellent documentation in this PR to go along with it as well. as I have said in my comments, we need to make sure it lives somewhere useful!

I added some comments, mostly to improve the unit tests a bit and add some docs

Wrappers/Python/cil/processors/FluxNormaliser.py

Wrappers/Python/test/test_DataProcessor.py

Wrappers/Python/cil/processors/FluxNormaliser.py

Wrappers/Python/test/test_DataProcessor.py

Signed-off-by: Hannah Robarts <[email protected]>

…nto flux_normaliser

hrobarts · 2024-10-16T13:35:02Z

Hi @lauramurgatroyd thank you for your review! I have made lots of updates

More efficiently access data in process (following conversation with @gfardell )
Error if horizontal and vertical are not in the last two dimensions of the data
Change norm_value to target and allow it to be passed as a number or string mean or first which gets the target from the mean or first value of the flux array (e.g. this could be the mean value in the roi across all projections)
Moved _calculate_flux and _calculate_target to separate functions out of check_input
New tests to test this functionality

I have added the notebook where I created the examples above to CIL-Demos/misc in this PR TomographicImaging/CIL-Demos#187

CHANGELOG.md

Wrappers/Python/cil/processors/FluxNormaliser.py

lauramurgatroyd · 2024-10-16T16:40:17Z

Wrappers/Python/cil/processors/FluxNormaliser.py

+        equal to the number of projections in the dataset. If flux=None, calculate
+        flux from the roi, default is None.
+
+    roi: dict (optional)


Suggested change

roi: dict (optional)

roi: dict (optional), default=None

Please add defaults like this, as in docs guidance: https://tomographicimaging.github.io/CIL/nightly/developer_guide/

lauramurgatroyd · 2024-10-16T16:40:29Z

Wrappers/Python/cil/processors/FluxNormaliser.py

+
+    Parameters:
+    -----------
+    flux: float or list of floats (optional)


Suggested change

flux: float or list of floats (optional)

flux: float or list of floats (optional), default=None

lauramurgatroyd · 2024-10-16T16:40:50Z

Wrappers/Python/cil/processors/FluxNormaliser.py

+        and/ or 'horizontal'. If an axis is not specified in the roi dictionary, 
+        the full range will be used, default is None.
+
+    target: string or float


Suggested change

target: string or float

target: string or float, default='mean'

please list string options like: {'hi', 'ho'} in this example https://numpydoc.readthedocs.io/en/latest/example.html#example

Wrappers/Python/cil/processors/FluxNormaliser.py

lauramurgatroyd · 2024-11-19T15:42:58Z

Wrappers/Python/cil/processors/FluxNormaliser.py

+                    if d in self.roi:
+                        # check indices are ints
+                        if not all(isinstance(i, int) for i in self.roi[d]):
+                            raise TypeError("roi values must be int, found {} and {}"
+                            .format(str(type(self.roi[d][0])), str(type(self.roi[d][1]))))
+                        # check indices are in range
+                        elif (self.roi[d][0] >= self.roi[d][1]) or (self.roi[d][0] < 0) or self.roi[d][1] > dataset.get_dimension_size(d):
+                            raise ValueError("roi values must be start > stop and between 0 and {}, found start={} and stop={} for direction '{}'"
+                            .format(str(dataset.get_dimension_size(d)), str(self.roi[d][0]), str(self.roi[d][1]), d ))
+                        # create slice


I wonder if we can make this some utility for validating roi throughout CIL

(could make a separate issue)

Hi Laura, this is a good point, I'll make an issue. I think the requirement for the ROI can vary between processors but there are definitely some common things.

lauramurgatroyd · 2024-11-19T15:45:04Z

Wrappers/Python/cil/processors/FluxNormaliser.py

-
-        return True
+        # check if flux array contains 0s
+        if 0 in self.flux:


does this catch 0.0

Yes, I've added some tests.

lauramurgatroyd

Looks great and nice examples in CIL-Demos misc.
As discussed this needs:

an example of 'first' in the demo
the demo being merged to main

gfardell

This is looking great. I'd be happy to merge it in as it is. But I think it would be quick to add a numba implementation of the actual calculation. We can look at it together this week.

gfardell · 2024-11-25T08:56:27Z

Wrappers/Python/cil/processors/FluxNormaliser.py

+    roi: dict, optional
+        Dictionary describing the region of interest containing the background
+        in the image. The roi is specified as `{'axis_name1':(start,stop), 
+        'axis_name2':(start,stop)}`, where the key is the axis name 'vertical'
+        and/ or 'horizontal'. If an axis is not specified in the roi dictionary, 
+        the full range will be used.


If this is just acquisition data then we know the axis names so I would remove the axis_name1 axis_name2

gfardell · 2024-11-25T08:58:25Z

Wrappers/Python/cil/processors/FluxNormaliser.py

+
+        if 'horizontal' in dataset.dimension_labels:
+            self.h_axis = dataset.get_dimension_axis('horizontal')
+            self.h_size = dataset.get_dimension_size('horizontal')


I didn't know these methods existed, I'd always just found the index of dimension labels and then used that in shape!

gfardell · 2024-11-25T09:03:51Z

Wrappers/Python/cil/processors/FluxNormaliser.py

+        angle: float, optional
+            Angle to plot, default=None displays the data with the minimum
+            and maximum pixel values in the roi, otherwise the angle to display 
+            can be specified as a float and the closest angle will be displayed.
+            For 2D data, the roi is plotted on the sinogram.


Elsewhere I think we use 'angle' as the name of the axis, then the argument is the index of that projection in a list, not the value. It might be worth just discussing this further, not because it's wrong but it could just be a change of expected behaviour. Would it cause any issues if they specified an index?

It would also have to be careful about units - especially if we decide to convert things in the future and store in a defined way.

gfardell · 2024-11-25T09:08:33Z

Wrappers/Python/cil/processors/FluxNormaliser.py

+        for i in range(num_proj):
+            arr_proj = data.array.flat[i*proj_size:(i+1)*proj_size]
+            if len(self.flux.flat) > 1:
+                f = self.flux.flat[i] 
+            arr_proj *= self.target_value/f
+            out.array.flat[i*proj_size:(i+1)*proj_size] = arr_proj


This would be trivial to add a numba implementation, and here it would speed things up a lot. So I think we should look at that before we merge.

Signed-off-by: Hannah Robarts <[email protected]>

Add flux normaliser processor, init tests

c477096

gfardell assigned lauramurgatroyd and hrobarts Jul 25, 2024

hrobarts added 2 commits July 26, 2024 09:16

Add roi check input logic

45cb0a5

Add roi functionality

d9cc167

hrobarts and others added 4 commits August 5, 2024 09:29

Add show_roi and deal with no roi specified

fd53e4d

Merge branch 'master' into flux_normaliser

1459eb9

Merge branch 'master' into flux_normaliser

2394e0e

Preview min and max, log plot, plot angle

36cfa78

hrobarts and others added 6 commits August 19, 2024 13:01

Update preview_configuration() to deal with different data shapes

07964e9

Make process work with one angle

e724a3a

Update tests

b4f516c

FluxNormaliser unit tests

c4b39cc

Merge branch 'master' into flux_normaliser

105574c

Signed-off-by: Hannah Robarts <[email protected]>

Docs update

ece454f

hrobarts linked an issue Aug 20, 2024 that may be closed by this pull request

Add a FluxNormaliser processor #1877

Open

hrobarts added 3 commits August 20, 2024 13:11

Remove redundant dependency

19f494a

Remove plotting in tests

608ffee

Mock plot

2985e62

hrobarts marked this pull request as ready for review August 20, 2024 14:15

hrobarts added 2 commits August 20, 2024 14:20

Changelog and test format update

cf4b56f

Update mock

3a0f7bb

hrobarts requested a review from gfardell August 20, 2024 14:50

hrobarts added the Waiting for review label Aug 20, 2024

paskino reviewed Aug 23, 2024

View reviewed changes

Wrappers/Python/cil/processors/FluxNormaliser.py Outdated Show resolved Hide resolved

paskino reviewed Aug 23, 2024

View reviewed changes

Wrappers/Python/cil/processors/FluxNormaliser.py Outdated Show resolved Hide resolved

paskino reviewed Aug 23, 2024

View reviewed changes

Wrappers/Python/cil/processors/FluxNormaliser.py Outdated Show resolved Hide resolved

casperdcl force-pushed the master branch 3 times, most recently from a910556 to f8b513b Compare September 2, 2024 13:11

lauramurgatroyd requested changes Sep 11, 2024

View reviewed changes

hrobarts and others added 13 commits September 13, 2024 10:30

Merge branch 'master' into flux_normaliser

0403fde

Signed-off-by: Hannah Robarts <[email protected]>

Merge branch 'master' into flux_normaliser

fa16eb8

Add colorbar to plots and warning based on data range

787b6bd

Review changes

21e773c

Merge branch 'master' into flux_normaliser

4e610a5

Merge branch 'flux_normaliser' of github.com:TomographicImaging/CIL i…

3126cc1

…nto flux_normaliser

Fix tests after removing tolerance

e8423bb

Update tests

1d7e157

Add _calculate_flux function

ce2fe22

Change target calculation

db715e6

Require h and v data order at the end

69e4466

New tests for calculate_target

da5f977

Merge branch 'master' into flux_normaliser

dc4f22d

hrobarts mentioned this pull request Oct 16, 2024

New flux normaliser examples TomographicImaging/CIL-Demos#187

Merged

hrobarts requested a review from lauramurgatroyd October 16, 2024 13:35

hrobarts and others added 2 commits October 16, 2024 14:00

Remove unused import

5780dac

Merge branch 'master' into flux_normaliser

64485a9

lauramurgatroyd reviewed Nov 19, 2024

View reviewed changes

lauramurgatroyd approved these changes Nov 19, 2024

View reviewed changes

Review updates

eb9db89

gfardell requested changes Nov 25, 2024

View reviewed changes

Merge branch 'master' into flux_normaliser

14b2fd1

Signed-off-by: Hannah Robarts <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add flux normaliser processor #1878

Add flux normaliser processor #1878

hrobarts commented Jul 25, 2024 •

edited

Loading

hrobarts commented Aug 1, 2024

hrobarts commented Aug 14, 2024

hrobarts commented Aug 19, 2024

hrobarts commented Aug 20, 2024

lauramurgatroyd left a comment

hrobarts commented Oct 16, 2024

lauramurgatroyd Oct 16, 2024

lauramurgatroyd Oct 16, 2024

lauramurgatroyd Oct 16, 2024

lauramurgatroyd Nov 19, 2024

lauramurgatroyd Nov 19, 2024

lauramurgatroyd Nov 19, 2024

hrobarts Nov 22, 2024

lauramurgatroyd Nov 19, 2024

hrobarts Nov 22, 2024

lauramurgatroyd left a comment •

edited by hrobarts

Loading

gfardell left a comment

gfardell Nov 25, 2024

gfardell Nov 25, 2024

gfardell Nov 25, 2024

gfardell Nov 25, 2024

	flux: float or list of floats (optional)
	flux: float or list of floats (optional), default=None

	target: string or float
	target: string or float, default='mean'

Add flux normaliser processor #1878

Are you sure you want to change the base?

Add flux normaliser processor #1878

Conversation

hrobarts commented Jul 25, 2024 • edited Loading

Description

Example Usage

Changes

Testing you performed

Related issues/links

Checklist

Contribution Notes

hrobarts commented Aug 1, 2024

hrobarts commented Aug 14, 2024

hrobarts commented Aug 19, 2024

hrobarts commented Aug 20, 2024

lauramurgatroyd left a comment

Choose a reason for hiding this comment

hrobarts commented Oct 16, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lauramurgatroyd left a comment • edited by hrobarts Loading

Choose a reason for hiding this comment

gfardell left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hrobarts commented Jul 25, 2024 •

edited

Loading

lauramurgatroyd left a comment •

edited by hrobarts

Loading