feat(bop): Adds multiprocessing to the bop writer

[cornerfarmer]

calc_gt_mask and calc_gt_info can now be calculated in parallel across multiple cpu processes, which should reduce the execution time significantly when having a scene with many objects. For 300 objects, the bop writer execution time reduces from 60s to 12s with 8 cores

Resolves: #994

[cornerfarmer]

BlenderProcHelper result for run 543:

Done with: [amass_human_poses_python 38s, auto_shading_python 9s, basic_python 13s, blenderkit_python 19s, bop_object_on_surface_sampling_python 73s, bop_object_physics_positioning_python 98s, bop_object_pose_sampling_python 74s, bop_scene_replication_python 143s, camera_depth_of_field_python 18s, camera_object_pose_python 23s, camera_random_trajectories_python 49s, camera_sampling_python 18s, coco_annotations_python 13s, diffuse_color_image_python 19s, dust_python 29s, entity_displacement_modifier_python 16s, entity_manipulation_python 10s, front_3d_object_sampling_python 84s, front_3d_python 102s, front_3d_with_improved_mat_python 104s, git_log 1s, haven_python 22s, ikea_python 17s, kinect_azure_noise_python 13s, lens_distortion_python 15s, lens_distortion_python 10s, light_sampling_python 11s, material_manipulation_python 10s, material_randomizer_python 13s, matterport3d_python 67s, motion_blur_rolling_shutter_python 28s, motion_blur_rolling_shutter_python 21s, multi_render_python 247s, nocs_python 18s, object_pose_sampling_python 10s, on_surface_object_sampling_python 12s, optical_flow_python 23s, physics_convex_decomposition_python 129s, physics_positioning_python 23s, pix3d_python 16s, random_backgrounds_python 17s, random_room_constructor_python 81s, replica_python 338s, scenenet_python 50s, scenenet_with_cctextures_python 51s, semantic_segmentation_python 13s, shapenet_python 18s, shapenet_with_scenenet_python 51s, shapenet_with_suncg_python 68s, spotlight_python 43s, stereo_matching_python 41s, stereo_matching_with_projector_python 133s, suncg_basic_python 31s, suncg_with_cam_sampling_python 56s, suncg_with_improved_mat_python 72s, suncg_with_object_replacer_python 57s, tests_python 15s, urdf_loading_and_manipulation_python 50s]
Done with 58 of 58 tests.
There were errors: ❌
Unit tests output:

${INTERNAL_NODES_DISC_SPACE}/Dominik_Winkelbauer/blender/blender-3.5.1-linux-x64/3.5/python/lib/python3.10/site-packages/numpy/core/getlimits.py:500: UserWarning: The value of the smallest subnormal for <class 'numpy.float64'> type is zero.
  setattr(self, word, getattr(machar, word).flat[0])
${INTERNAL_NODES_DISC_SPACE}/Dominik_Winkelbauer/blender/blender-3.5.1-linux-x64/3.5/python/lib/python3.10/site-packages/numpy/core/getlimits.py:89: UserWarning: The value of the smallest subnormal for <class 'numpy.float64'> type is zero.
  return self._float_to_str(self.smallest_subnormal)
${INTERNAL_NODES_DISC_SPACE}/Dominik_Winkelbauer/blender/blender-3.5.1-linux-x64/3.5/python/lib/python3.10/site-packages/numpy/core/getlimits.py:500: UserWarning: The value of the smallest subnormal for <class 'numpy.float32'> type is zero.
  setattr(self, word, getattr(machar, word).flat[0])
${INTERNAL_NODES_DISC_SPACE}/Dominik_Winkelbauer/blender/blender-3.5.1-linux-x64/3.5/python/lib/python3.10/site-packages/numpy/core/getlimits.py:89: UserWarning: The value of the smallest subnormal for <class 'numpy.float32'> type is zero.
  return self._float_to_str(self.smallest_subnormal)
...........
----------------------------------------------------------------------
Ran 11 tests in 9.891s

OK

urdf_loading_and_manipulation (main_python)

The output of urdf_loading_and_manipulation:main_python:000000_scene_gt_info_json has changed:

Expand

+ root['1'][3]['bbox_visib'][3]: 78
- root['1'][3]['bbox_visib'][3]: 79
+ root['4'][6]['bbox_visib'][3]: 17
- root['4'][6]['bbox_visib'][3]: 16
+ root['9'][2]['bbox_visib'][3]: 13
- root['9'][2]['bbox_visib'][3]: 14

stereo_matching (main_python)

The output of stereo_matching:main_python:disparity has changed: Diff: 0.0194, Pixel diff: 0.0031

Compare image	Newly generated image

The output of stereo_matching:main_python:disparity has changed: Diff: 0.0104, Pixel diff: 0.0036

Compare image	Newly generated image

Pylint output

Expand

Messages

Module blenderproc.python.loader.CCMaterialLoader (blenderproc/python/loader/CCMaterialLoader.py)

Line	Col.	Type	Symbol	ID	Obj	Message
72	0	convention	line-too-long	C0301		Line too long (152/120)

Module blenderproc.python.types.LightUtility (blenderproc/python/types/LightUtility.py)

Line	Col.	Type	Symbol	ID	Obj	Message
125	12	refactor	consider-using-augmented-assign	R6104	Light.setup_as_projector	Use '/=' to do an augmented assign directly
216	0	convention	trailing-whitespace	C0303		Trailing whitespace

Module blenderproc.python.writer.BopWriterUtility (blenderproc/python/writer/BopWriterUtility.py)

Line	Col.	Type	Symbol	ID	Obj	Message
9	0	convention	wrong-import-order	C0411		standard import "from typing import List, Optional, Dict, Tuple" should be placed before "import trimesh"
10	0	convention	wrong-import-order	C0411		standard import "import warnings" should be placed before "import trimesh"
11	0	convention	wrong-import-order	C0411		standard import "import datetime" should be placed before "import trimesh"
18	0	convention	wrong-import-order	C0411		standard import "import sys" should be placed before "import trimesh"
62	0	convention	line-too-long	C0301		Line too long (131/120)
79	8	warning	redefined-outer-name	W0621	write_bop	Redefining name 'dataset_objects' from outer scope (line 523)
165	44	warning	protected-access	W0212	write_bop	Access to a protected member _pyrender_init of a client class
165	15	refactor	consider-using-with	R1732	write_bop	Consider using 'with' for resource-allocating operations
169	0	convention	trailing-whitespace	C0303		Trailing whitespace
294	21	warning	redefined-outer-name	W0621	_BopWriterUtility.get_frame_gt	Redefining name 'dataset_objects' from outer scope (line 523)
379	38	warning	redefined-outer-name	W0621	_BopWriterUtility.write_frames	Redefining name 'dataset_objects' from outer scope (line 523)
505	0	convention	trailing-whitespace	C0303		Trailing whitespace
517	78	convention	trailing-whitespace	C0303		Trailing whitespace
521	8	warning	global-variable-undefined	W0601	_BopWriterUtility._pyrender_init	Global variable 'renderer' undefined at the module level
521	8	warning	global-variable-undefined	W0601	_BopWriterUtility._pyrender_init	Global variable 'renderer_large' undefined at the module level
521	8	warning	global-variable-undefined	W0601	_BopWriterUtility._pyrender_init	Global variable 'dataset_objects' undefined at the module level
539	0	convention	line-too-long	C0301		Line too long (173/120)
554	78	convention	trailing-whitespace	C0303		Trailing whitespace
560	8	warning	global-variable-not-assigned	W0602	_BopWriterUtility._calc_gt_masks_iteration	Using global for 'renderer' but no assignment is done
560	8	warning	global-variable-not-assigned	W0602	_BopWriterUtility._calc_gt_masks_iteration	Using global for 'dataset_objects' but no assignment is done
567	0	convention	trailing-whitespace	C0303		Trailing whitespace
657	0	convention	line-too-long	C0301		Line too long (160/120)
658	0	convention	trailing-whitespace	C0303		Trailing whitespace
659	0	convention	trailing-whitespace	C0303		Trailing whitespace
662	0	convention	line-too-long	C0301		Line too long (196/120)
675	11	convention	trailing-whitespace	C0303		Trailing whitespace
677	78	convention	trailing-whitespace	C0303		Trailing whitespace
683	8	warning	global-variable-not-assigned	W0602	_BopWriterUtility._calc_gt_info_iteration	Using global for 'renderer_large' but no assignment is done
683	8	warning	global-variable-not-assigned	W0602	_BopWriterUtility._calc_gt_info_iteration	Using global for 'dataset_objects' but no assignment is done
683	8	warning	global-variable-not-assigned	W0602	_BopWriterUtility._calc_gt_info_iteration	Using global for 'renderer' but no assignment is done
687	11	warning	protected-access	W0212	_BopWriterUtility._calc_gt_info_iteration	Access to a protected member _renderer of a client class
688	12	warning	protected-access	W0212	_BopWriterUtility._calc_gt_info_iteration	Access to a protected member _renderer of a client class
689	12	warning	protected-access	W0212	_BopWriterUtility._calc_gt_info_iteration	Access to a protected member _renderer of a client class
696	0	convention	trailing-whitespace	C0303		Trailing whitespace
824	0	convention	line-too-long	C0301		Line too long (202/120)
825	0	convention	trailing-whitespace	C0303		Trailing whitespace
833	44	warning	redefined-outer-name	W0621	_BopWriterUtility.calc_gt_coco	Redefining name 'dataset_objects' from outer scope (line 523)

Metrics

Count per types

Name	Count
convention	22
refactor	2
warning	16

Count per messages

Name	Count
line-too-long	6
trailing-whitespace	12
consider-using-augmented-assign	1
redefined-outer-name	4
protected-access	4
consider-using-with	1
global-variable-undefined	3
global-variable-not-assigned	5
wrong-import-order	4

Count per modules

Name	Count
blenderproc.python.loader.CCMaterialLoader	1
blenderproc.python.types.LightUtility	2
blenderproc.python.writer.BopWriterUtility	37

Count per path

Name	Count
blenderproc/python/loader/CCMaterialLoader.py	1
blenderproc/python/types/LightUtility.py	2
blenderproc/python/writer/BopWriterUtility.py	37

***The tests are done.*** Check the log:

Last update: 10/30/2023, 18:20:51