Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Error while training the model "ValueError: num_samples should be a positive integer value, but got num_samples=0" #116

Open
prabhata07 opened this issue Nov 29, 2022 · 2 comments
Open

Error while training the model "ValueError: num_samples should be a positive integer value, but got num_samples=0" #116

prabhata07 opened this issue Nov 29, 2022 · 2 comments

Comments

@prabhata07
Copy link

prabhata07 commented Nov 29, 2022 •
edited
Loading

Code we used for training:
!python /content/gdrive/MyDrive/IITH_VCA/CaDDN/tools/train.py --cfg_file /content/gdrive/MyDrive/IITH_VCA/CaDDN/tools/cfgs/kitti_models/CaDDN.yaml

Log we got:
2022-11-19 21:23:27,419 INFO Start logging
2022-11-19 21:23:27,420 INFO CUDA_VISIBLE_DEVICES=ALL
2022-11-19 21:23:27,420 INFO cfg_file /content/gdrive/MyDrive/IITH_VCA/CaDDN/tools/cfgs/kitti_models/CaDDN.yaml
2022-11-19 21:23:27,420 INFO batch_size 4
2022-11-19 21:23:27,420 INFO epochs 80
2022-11-19 21:23:27,420 INFO workers 8
2022-11-19 21:23:27,420 INFO extra_tag default
2022-11-19 21:23:27,421 INFO ckpt None
2022-11-19 21:23:27,421 INFO pretrained_model None
2022-11-19 21:23:27,421 INFO launcher none
2022-11-19 21:23:27,421 INFO tcp_port 18888
2022-11-19 21:23:27,421 INFO sync_bn False
2022-11-19 21:23:27,421 INFO fix_random_seed False
2022-11-19 21:23:27,421 INFO ckpt_save_interval 1
2022-11-19 21:23:27,421 INFO local_rank 0
2022-11-19 21:23:27,421 INFO max_ckpt_save_num 30
2022-11-19 21:23:27,421 INFO merge_all_iters_to_one_epoch False
2022-11-19 21:23:27,421 INFO set_cfgs None
2022-11-19 21:23:27,421 INFO max_waiting_mins 0
2022-11-19 21:23:27,422 INFO start_epoch 0
2022-11-19 21:23:27,422 INFO save_to_file False
2022-11-19 21:23:27,422 INFO cfg.ROOT_DIR: /content/gdrive/MyDrive/IITH_VCA/CaDDN
2022-11-19 21:23:27,422 INFO cfg.LOCAL_RANK: 0
2022-11-19 21:23:27,422 INFO cfg.CLASS_NAMES: ['Car', 'Pedestrian', 'Cyclist']
2022-11-19 21:23:27,422 INFO
cfg.DATA_CONFIG = edict()
2022-11-19 21:23:27,422 INFO cfg.DATA_CONFIG.DATASET: KittiDataset
2022-11-19 21:23:27,422 INFO cfg.DATA_CONFIG.DATA_PATH: /gdrive/MyDrive/3D_Obj/CaDDN/data/kitti
2022-11-19 21:23:27,422 INFO cfg.DATA_CONFIG.POINT_CLOUD_RANGE: [2, -30.08, -3.0, 46.8, 30.08, 1.0]
2022-11-19 21:23:27,422 INFO
cfg.DATA_CONFIG.DATA_SPLIT = edict()
2022-11-19 21:23:27,422 INFO cfg.DATA_CONFIG.DATA_SPLIT.train: train
2022-11-19 21:23:27,423 INFO cfg.DATA_CONFIG.DATA_SPLIT.test: test
2022-11-19 21:23:27,423 INFO
cfg.DATA_CONFIG.INFO_PATH = edict()
2022-11-19 21:23:27,423 INFO cfg.DATA_CONFIG.INFO_PATH.train: ['kitti_infos_trainval.pkl']
2022-11-19 21:23:27,423 INFO cfg.DATA_CONFIG.INFO_PATH.test: ['kitti_infos_test.pkl']
2022-11-19 21:23:27,423 INFO cfg.DATA_CONFIG.FOV_POINTS_ONLY: True
2022-11-19 21:23:27,423 INFO
cfg.DATA_CONFIG.DATA_AUGMENTOR = edict()
2022-11-19 21:23:27,424 INFO cfg.DATA_CONFIG.DATA_AUGMENTOR.DISABLE_AUG_LIST: ['placeholder']
2022-11-19 21:23:27,424 INFO cfg.DATA_CONFIG.DATA_AUGMENTOR.AUG_CONFIG_LIST: [{'NAME': 'random_image_flip', 'ALONG_AXIS_LIST': ['horizontal']}]
2022-11-19 21:23:27,424 INFO
cfg.DATA_CONFIG.POINT_FEATURE_ENCODING = edict()
2022-11-19 21:23:27,424 INFO cfg.DATA_CONFIG.POINT_FEATURE_ENCODING.encoding_type: absolute_coordinates_encoding
2022-11-19 21:23:27,424 INFO cfg.DATA_CONFIG.POINT_FEATURE_ENCODING.used_feature_list: ['x', 'y', 'z', 'intensity']
2022-11-19 21:23:27,424 INFO cfg.DATA_CONFIG.POINT_FEATURE_ENCODING.src_feature_list: ['x', 'y', 'z', 'intensity']
2022-11-19 21:23:27,424 INFO cfg.DATA_CONFIG.DATA_PROCESSOR: [{'NAME': 'mask_points_and_boxes_outside_range', 'REMOVE_OUTSIDE_BOXES': True}, {'NAME': 'calculate_grid_size', 'VOXEL_SIZE': [0.16, 0.16, 0.16]}]
2022-11-19 21:23:27,424 INFO cfg.DATA_CONFIG.BASE_CONFIG: cfgs/dataset_configs/kitti_dataset.yaml
2022-11-19 21:23:27,425 INFO
cfg.DATA_CONFIG.IMAGE = edict()
2022-11-19 21:23:27,425 INFO cfg.DATA_CONFIG.IMAGE.ENABLED: True
2022-11-19 21:23:27,425 INFO
cfg.DATA_CONFIG.IMAGE.COLLATE = edict()
2022-11-19 21:23:27,425 INFO cfg.DATA_CONFIG.IMAGE.COLLATE.MODE: Pad
2022-11-19 21:23:27,425 INFO cfg.DATA_CONFIG.IMAGE.COLLATE.CONSTANT_VALUES: [0.485, 0.456, 0.406]
2022-11-19 21:23:27,425 INFO
cfg.DATA_CONFIG.DEPTH_MAP = edict()
2022-11-19 21:23:27,425 INFO cfg.DATA_CONFIG.DEPTH_MAP.ENABLED: True
2022-11-19 21:23:27,425 INFO cfg.DATA_CONFIG.DEPTH_MAP.DOWNSAMPLE_FACTOR: 4
2022-11-19 21:23:27,426 INFO
cfg.DATA_CONFIG.CALIB = edict()
2022-11-19 21:23:27,426 INFO cfg.DATA_CONFIG.CALIB.ENABLED: True
2022-11-19 21:23:27,426 INFO
cfg.MODEL = edict()
2022-11-19 21:23:27,426 INFO cfg.MODEL.NAME: CaDDN
2022-11-19 21:23:27,426 INFO
cfg.MODEL.FFE = edict()
2022-11-19 21:23:27,426 INFO cfg.MODEL.FFE.NAME: DepthFFE
2022-11-19 21:23:27,426 INFO
cfg.MODEL.FFE.DDN = edict()
2022-11-19 21:23:27,426 INFO cfg.MODEL.FFE.DDN.NAME: DDNDeepLabV3
2022-11-19 21:23:27,427 INFO cfg.MODEL.FFE.DDN.BACKBONE_NAME: ResNet101
2022-11-19 21:23:27,427 INFO
cfg.MODEL.FFE.DDN.ARGS = edict()
2022-11-19 21:23:27,427 INFO cfg.MODEL.FFE.DDN.ARGS.feat_extract_layer: layer1
2022-11-19 21:23:27,427 INFO cfg.MODEL.FFE.DDN.ARGS.pretrained_path: ../checkpoints/deeplabv3_resnet101_coco-586e9e4e.pth
2022-11-19 21:23:27,427 INFO
cfg.MODEL.FFE.CHANNEL_REDUCE = edict()
2022-11-19 21:23:27,427 INFO cfg.MODEL.FFE.CHANNEL_REDUCE.in_channels: 256
2022-11-19 21:23:27,427 INFO cfg.MODEL.FFE.CHANNEL_REDUCE.out_channels: 64
2022-11-19 21:23:27,427 INFO cfg.MODEL.FFE.CHANNEL_REDUCE.kernel_size: 1
2022-11-19 21:23:27,427 INFO cfg.MODEL.FFE.CHANNEL_REDUCE.stride: 1
2022-11-19 21:23:27,428 INFO cfg.MODEL.FFE.CHANNEL_REDUCE.bias: False
2022-11-19 21:23:27,428 INFO
cfg.MODEL.FFE.DISCRETIZE = edict()
2022-11-19 21:23:27,428 INFO cfg.MODEL.FFE.DISCRETIZE.mode: LID
2022-11-19 21:23:27,428 INFO cfg.MODEL.FFE.DISCRETIZE.num_bins: 80
2022-11-19 21:23:27,428 INFO cfg.MODEL.FFE.DISCRETIZE.depth_min: 2.0
2022-11-19 21:23:27,428 INFO cfg.MODEL.FFE.DISCRETIZE.depth_max: 46.8
2022-11-19 21:23:27,428 INFO
cfg.MODEL.FFE.DDN_LOSS = edict()
2022-11-19 21:23:27,428 INFO cfg.MODEL.FFE.DDN_LOSS.weight: 3.0
2022-11-19 21:23:27,428 INFO cfg.MODEL.FFE.DDN_LOSS.alpha: 0.25
2022-11-19 21:23:27,429 INFO cfg.MODEL.FFE.DDN_LOSS.gamma: 2.0
2022-11-19 21:23:27,429 INFO cfg.MODEL.FFE.DDN_LOSS.fg_weight: 13
2022-11-19 21:23:27,429 INFO cfg.MODEL.FFE.DDN_LOSS.bg_weight: 1
2022-11-19 21:23:27,429 INFO
cfg.MODEL.F2V = edict()
2022-11-19 21:23:27,429 INFO cfg.MODEL.F2V.NAME: FrustumToVoxel
2022-11-19 21:23:27,429 INFO
cfg.MODEL.F2V.SAMPLER = edict()
2022-11-19 21:23:27,429 INFO cfg.MODEL.F2V.SAMPLER.mode: bilinear
2022-11-19 21:23:27,429 INFO cfg.MODEL.F2V.SAMPLER.padding_mode: zeros
2022-11-19 21:23:27,429 INFO
cfg.MODEL.MAP_TO_BEV = edict()
2022-11-19 21:23:27,430 INFO cfg.MODEL.MAP_TO_BEV.NAME: Conv2DCollapse
2022-11-19 21:23:27,430 INFO cfg.MODEL.MAP_TO_BEV.NUM_BEV_FEATURES: 64
2022-11-19 21:23:27,430 INFO
cfg.MODEL.MAP_TO_BEV.ARGS = edict()
2022-11-19 21:23:27,430 INFO cfg.MODEL.MAP_TO_BEV.ARGS.kernel_size: 1
2022-11-19 21:23:27,430 INFO cfg.MODEL.MAP_TO_BEV.ARGS.stride: 1
2022-11-19 21:23:27,430 INFO cfg.MODEL.MAP_TO_BEV.ARGS.bias: False
2022-11-19 21:23:27,430 INFO
cfg.MODEL.BACKBONE_2D = edict()
2022-11-19 21:23:27,430 INFO cfg.MODEL.BACKBONE_2D.NAME: BaseBEVBackbone
2022-11-19 21:23:27,431 INFO cfg.MODEL.BACKBONE_2D.LAYER_NUMS: [10, 10, 10]
2022-11-19 21:23:27,431 INFO cfg.MODEL.BACKBONE_2D.LAYER_STRIDES: [2, 2, 2]
2022-11-19 21:23:27,431 INFO cfg.MODEL.BACKBONE_2D.NUM_FILTERS: [64, 128, 256]
2022-11-19 21:23:27,431 INFO cfg.MODEL.BACKBONE_2D.UPSAMPLE_STRIDES: [1, 2, 4]
2022-11-19 21:23:27,431 INFO cfg.MODEL.BACKBONE_2D.NUM_UPSAMPLE_FILTERS: [128, 128, 128]
2022-11-19 21:23:27,431 INFO
cfg.MODEL.DENSE_HEAD = edict()
2022-11-19 21:23:27,431 INFO cfg.MODEL.DENSE_HEAD.NAME: AnchorHeadSingle
2022-11-19 21:23:27,431 INFO cfg.MODEL.DENSE_HEAD.CLASS_AGNOSTIC: False
2022-11-19 21:23:27,431 INFO cfg.MODEL.DENSE_HEAD.USE_DIRECTION_CLASSIFIER: True
2022-11-19 21:23:27,432 INFO cfg.MODEL.DENSE_HEAD.DIR_OFFSET: 0.78539
2022-11-19 21:23:27,432 INFO cfg.MODEL.DENSE_HEAD.DIR_LIMIT_OFFSET: 0.0
2022-11-19 21:23:27,432 INFO cfg.MODEL.DENSE_HEAD.NUM_DIR_BINS: 2
2022-11-19 21:23:27,432 INFO cfg.MODEL.DENSE_HEAD.ANCHOR_GENERATOR_CONFIG: [{'class_name': 'Car', 'anchor_sizes': [[3.9, 1.6, 1.56]], 'anchor_rotations': [0, 1.57], 'anchor_bottom_heights': [-1.78], 'align_center': False, 'feature_map_stride': 2, 'matched_threshold': 0.6, 'unmatched_threshold': 0.45}, {'class_name': 'Pedestrian', 'anchor_sizes': [[0.8, 0.6, 1.73]], 'anchor_rotations': [0, 1.57], 'anchor_bottom_heights': [-0.6], 'align_center': False, 'feature_map_stride': 2, 'matched_threshold': 0.5, 'unmatched_threshold': 0.35}, {'class_name': 'Cyclist', 'anchor_sizes': [[1.76, 0.6, 1.73]], 'anchor_rotations': [0, 1.57], 'anchor_bottom_heights': [-0.6], 'align_center': False, 'feature_map_stride': 2, 'matched_threshold': 0.5, 'unmatched_threshold': 0.35}]
2022-11-19 21:23:27,432 INFO
cfg.MODEL.DENSE_HEAD.TARGET_ASSIGNER_CONFIG = edict()
2022-11-19 21:23:27,432 INFO cfg.MODEL.DENSE_HEAD.TARGET_ASSIGNER_CONFIG.NAME: AxisAlignedTargetAssigner
2022-11-19 21:23:27,432 INFO cfg.MODEL.DENSE_HEAD.TARGET_ASSIGNER_CONFIG.POS_FRACTION: -1.0
2022-11-19 21:23:27,432 INFO cfg.MODEL.DENSE_HEAD.TARGET_ASSIGNER_CONFIG.SAMPLE_SIZE: 512
2022-11-19 21:23:27,433 INFO cfg.MODEL.DENSE_HEAD.TARGET_ASSIGNER_CONFIG.NORM_BY_NUM_EXAMPLES: False
2022-11-19 21:23:27,433 INFO cfg.MODEL.DENSE_HEAD.TARGET_ASSIGNER_CONFIG.MATCH_HEIGHT: False
2022-11-19 21:23:27,433 INFO cfg.MODEL.DENSE_HEAD.TARGET_ASSIGNER_CONFIG.BOX_CODER: ResidualCoder
2022-11-19 21:23:27,433 INFO
cfg.MODEL.DENSE_HEAD.LOSS_CONFIG = edict()
2022-11-19 21:23:27,433 INFO
cfg.MODEL.DENSE_HEAD.LOSS_CONFIG.LOSS_WEIGHTS = edict()
2022-11-19 21:23:27,433 INFO cfg.MODEL.DENSE_HEAD.LOSS_CONFIG.LOSS_WEIGHTS.cls_weight: 1.0
2022-11-19 21:23:27,433 INFO cfg.MODEL.DENSE_HEAD.LOSS_CONFIG.LOSS_WEIGHTS.loc_weight: 2.0
2022-11-19 21:23:27,433 INFO cfg.MODEL.DENSE_HEAD.LOSS_CONFIG.LOSS_WEIGHTS.dir_weight: 0.2
2022-11-19 21:23:27,433 INFO cfg.MODEL.DENSE_HEAD.LOSS_CONFIG.LOSS_WEIGHTS.code_weights: [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0]
2022-11-19 21:23:27,434 INFO
cfg.MODEL.POST_PROCESSING = edict()
2022-11-19 21:23:27,434 INFO cfg.MODEL.POST_PROCESSING.RECALL_THRESH_LIST: [0.3, 0.5, 0.7]
2022-11-19 21:23:27,434 INFO cfg.MODEL.POST_PROCESSING.SCORE_THRESH: 0.1
2022-11-19 21:23:27,434 INFO cfg.MODEL.POST_PROCESSING.OUTPUT_RAW_SCORE: False
2022-11-19 21:23:27,434 INFO cfg.MODEL.POST_PROCESSING.EVAL_METRIC: kitti
2022-11-19 21:23:27,434 INFO
cfg.MODEL.POST_PROCESSING.NMS_CONFIG = edict()
2022-11-19 21:23:27,434 INFO cfg.MODEL.POST_PROCESSING.NMS_CONFIG.MULTI_CLASSES_NMS: False
2022-11-19 21:23:27,434 INFO cfg.MODEL.POST_PROCESSING.NMS_CONFIG.NMS_TYPE: nms_gpu
2022-11-19 21:23:27,434 INFO cfg.MODEL.POST_PROCESSING.NMS_CONFIG.NMS_THRESH: 0.01
2022-11-19 21:23:27,435 INFO cfg.MODEL.POST_PROCESSING.NMS_CONFIG.NMS_PRE_MAXSIZE: 4096
2022-11-19 21:23:27,435 INFO cfg.MODEL.POST_PROCESSING.NMS_CONFIG.NMS_POST_MAXSIZE: 500
2022-11-19 21:23:27,435 INFO
cfg.OPTIMIZATION = edict()
2022-11-19 21:23:27,435 INFO cfg.OPTIMIZATION.BATCH_SIZE_PER_GPU: 4
2022-11-19 21:23:27,435 INFO cfg.OPTIMIZATION.NUM_EPOCHS: 80
2022-11-19 21:23:27,435 INFO cfg.OPTIMIZATION.OPTIMIZER: adam_onecycle
2022-11-19 21:23:27,435 INFO cfg.OPTIMIZATION.LR: 0.001
2022-11-19 21:23:27,435 INFO cfg.OPTIMIZATION.WEIGHT_DECAY: 0.01
2022-11-19 21:23:27,436 INFO cfg.OPTIMIZATION.MOMENTUM: 0.9
2022-11-19 21:23:27,436 INFO cfg.OPTIMIZATION.MOMS: [0.95, 0.85]
2022-11-19 21:23:27,436 INFO cfg.OPTIMIZATION.PCT_START: 0.4
2022-11-19 21:23:27,436 INFO cfg.OPTIMIZATION.DIV_FACTOR: 10
2022-11-19 21:23:27,436 INFO cfg.OPTIMIZATION.DECAY_STEP_LIST: [35, 45]
2022-11-19 21:23:27,436 INFO cfg.OPTIMIZATION.LR_DECAY: 0.1
2022-11-19 21:23:27,436 INFO cfg.OPTIMIZATION.LR_CLIP: 1e-07
2022-11-19 21:23:27,436 INFO cfg.OPTIMIZATION.LR_WARMUP: False
2022-11-19 21:23:27,436 INFO cfg.OPTIMIZATION.WARMUP_EPOCH: 1
2022-11-19 21:23:27,437 INFO cfg.OPTIMIZATION.GRAD_NORM_CLIP: 10
2022-11-19 21:23:27,437 INFO cfg.TAG: CaDDN
2022-11-19 21:23:27,437 INFO cfg.EXP_GROUP_PATH: content/gdrive/MyDrive/IITH_VCA/CaDDN/tools/cfgs/kitti_models
2022-11-19 21:23:28,525 INFO Loading KITTI dataset
2022-11-19 21:23:28,526 INFO Total samples for KITTI dataset: 0
Traceback (most recent call last):
File "/content/gdrive/MyDrive/IITH_VCA/CaDDN/tools/train.py", line 198, in
main()
File "/content/gdrive/MyDrive/IITH_VCA/CaDDN/tools/train.py", line 112, in main
total_epochs=args.epochs
File "/content/gdrive/MyDrive/IITH_VCA/CaDDN/pcdet/datasets/init.py", line 69, in build_dataloader
drop_last=False, sampler=sampler, timeout=0
File "/usr/local/lib/python3.7/dist-packages/torch/utils/data/dataloader.py", line 213, in init
sampler = RandomSampler(dataset)
File "/usr/local/lib/python3.7/dist-packages/torch/utils/data/sampler.py", line 94, in init
"value, but got num_samples={}".format(self.num_samples))
ValueError: num_samples should be a positive integer value, but got num_samples=0

Screen shots of the same:
image

Error screen shot:
image
@Gang-Chen-China
Copy link

2022-11-19 21:23:28,526 INFO Total samples for KITTI dataset: 0,

It seems you don't load the data.

@zs9797
Copy link

zs9797 commented Aug 14, 2023

I have encountered the same problem. Have you resolved it?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@prabhata07 @Gang-Chen-China @zs9797