2048-Homog-11-08-distributedTraining-newScenes-11-08-fullDepth-newScenes-32-grayscale.log

Starting the distributed training at Sun 11 Aug 15:32:09 BST 2024
Command running in screen session distributed_depth-newScenes_11-08-fullDepth-newScenes_0_20240811-153209 on ray25
Command running in screen session distributed_depth-newScenes_11-08-fullDepth-newScenes_1_20240811-153209 on ray03
Command running in screen session distributed_depth-newScenes_11-08-fullDepth-newScenes_2_20240811-153209 on ray07
Command running in screen session distributed_depth-newScenes_11-08-fullDepth-newScenes_3_20240811-153209 on ray04
Learning rate is now 1e-5 nto 1e-4
Distributed training completed at Sun 11 Aug 15:32:18 BST 2024
/homes/tp4618/Documents/bitbucket/miniconda3/envs/redoGlue/lib/python3.12/site-packages/kornia/feature/lightglue.py:44: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
  @torch.cuda.amp.custom_fwd(cast_inputs=torch.float32)
/homes/tp4618/Documents/bitbucket/miniconda3/envs/redoGlue/lib/python3.12/site-packages/kornia/feature/lightglue.py:44: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
  @torch.cuda.amp.custom_fwd(cast_inputs=torch.float32)
/vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/gluefactory/models/matchers/loadedLightGlue.py:25: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
  @torch.cuda.amp.custom_fwd(cast_inputs=torch.float32)
[08/11/2024 15:32:59 gluefactory INFO] Starting experiment 11-08-fullDepth-newScenes
/vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/gluefactory/models/matchers/loadedLightGlue.py:25: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
  @torch.cuda.amp.custom_fwd(cast_inputs=torch.float32)
outputs saved to /homes/tp4618/Documents/bitbucket/SuperGlueThesis/external/glue-factory/outputs/training/11-08-fullDepth-newScenes/log.txt
[08/11/2024 15:33:00 gluefactory INFO] Will fine-tune from weights of pretrain_lightglue
/vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/gluefactory/train.py:1724: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
  init_cp = torch.load(str(init_cp), map_location="cpu")
/homes/tp4618/Documents/bitbucket/miniconda3/envs/redoGlue/lib/python3.12/site-p[08/11/2024 15:33:34 gluefactory INFO] Using device cuda:0
[08/11/2024 15:33:35 gluefactory.datasets.base_dataset INFO] Creating dataset TreeDepth
[08/11/2024 15:33:35 gluefactory.datasets.treedepth INFO] Initialized TreeDepth dataset with configuration: {'name': 'treedepth', 'num_workers': 1, 'train_batch_size': '???', 'val_batch_size': '???', 'test_batch_size': '???', 'shuffle_training': False, 'batch_size': 8, 'num_threads': 1, 'seed': 0, 'prefetch_factor': 2, 'data_dir': 'syntheticForestData/', 'depth_subpath': 'depthData/', 'image_subpath': 'imageData/', 'info_dir': 'fileLists(newScenes)', 'train_split': 'train_scenes_clean.txt', 'train_num_per_scene': 300, 'val_split': 'valid_scenes_clean.txt', 'val_num_per_scene': None, 'val_pairs': 'valid_pairs.txt', 'test_split': 'test_scenes_clean.txt', 'test_num_per-factory/gluefactory/train.py:1724: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
  init_cp = torch.load(str(init_cp), map_location="cpu")
[08/11/2024 15:33:19 gluefactory INFO] Training in distributed mode with 4 GPUs
[08/11/2024 15:33:35 gluefactory INFO] Using device cuda:0
[08/11/2024 15:33:35 gluefactory.datasets.base_dataset INFO] Creating dataset TreeDepth
[08/11/2024 15:33:35 gluefactory.datasets.treedepth INFO] Initialized TreeDepth dataset with configuration: {'name': 'treedepth', 'num_workers': 1, 'train_batch_size': '???', 'val_batch_size': '???', 'test_batch_size': '???', 'shuffle_training': False, 'batch_size': 8, 'num_threads': 1, 'seed': 0, 'prefetch_factor': 2, 'data_dir': 'syntheticForestData/', 'depth_subpath': 'depthData/', 'image_subpath': 'imageData/', 'info_dir': 'fileLists(newScenes)', 'train_split': 'train_scenes_clean.txt', 'train_num_per_scene': 300, 'val_split': 'valid_scenes_clean.txt', 'val_num_per_scene': None, 'val_pairs': 'valid_pairs.txt', 'test_split': 'test_scenes_clean.txt', 'test_num_per_scene': None, 'test_pairs': None, 'views': 2, 'min_overlap': 0.1, 'max_overlap': 0.7, 'num_overlap_bins': 3, 'sort_by_overlap': False, 'triplet_enforce_overlap': False, 'read_depth': True, 'read_image': True, 'grayscale': False, 'preprocessing': {'resize': None, 'edge_divisible_by': None, 'side': 'long', 'interpolation': 'bilinear', 'align_corners': None, 'antialias': True, 'square_pad': False, 'add_padding_mask': False}, 'p_rotate': 0.0, 'reseed': False, 'load_features': {'do': False, 'path': 'exports/closed-treedepth-rgb-undist-depthr_SP-k2048-nms3/{scene}.h5', 'data_keys': None, 'device': None, 'trainable': False, 'add_data_path': True, 'collate': False, 'scale': ['keypoints', 'lines', 'orig_lines'], 'padding_fn': 'pad_local_features', 'padding_length': 2048, 'numeric_type': 'float32'}}
[08/11/2024 15:33:35 gluefactory.datasets.treedepth INFO] Sampling new items for train with seed 0.
[08/11/2024 15:33:36 gluefactory.datasets.treedepth INFO] Sampling new items for val with seed 0.
[08/11/2024 15:33:36 gluefactory INFO] Training loader has 243 batches
[08/11/2024 15:33:36 gluefactory INFO] Validation loader has 324 batches
/vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/gluefactory_nonfree/superpoint.py:426: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
  scaler = GradScaler()
/vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/gluefactory/models/matchers/lightglue.py:646: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
  @torch.cuda.amp.custom_fwd(cast_inputs=torch.float32)
/vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/gluefactory/models/matchers/depth_matcher.py:40: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
  @torch.cuda.amp.custom_fwd(cast_inputs=torch.float32)
[08/11/2024 15:33:45 gluefactory INFO] Parameters with scaled learning rate:
{}
/vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/gluefactory/train.py:1889: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
  scaler = GradScaler(enabled=args.mixed_precision is not None)
[08/11/2024 15:33:45 gluefactory INFO] Training with mixed_precision=None
[08/11/2024 15:33:45 gluefactory INFO] Starting training with configuration:
data:
  name: treedepth
  train_split: train_scenes_clean.txt
  train_num_per_scene: 300
  val_split: valid_scenes_clean.txt
  val_pairs: valid_pairs.txt
  min_overlap: 0.1
  max_overlap: 0.7
  num_overlap_bins: 3
  read_depth: true
  read_image: true
  batch_size: 32
  num_workers: 4
  load_features:
    do: false
    path: exports/closed-treedepth-rgb-undist-depthr_SP-k2048-nms3/{scene}.h5
    padding_length: 2048
    padding_fn: pad_local_features
model:
  name: two_view_pipeline
  extractor:
    name: gluefactory_nonfree.superpoint
    max_num_keypoints: 2048
    force_num_keypoints: true
    detection_threshold: 0.0
    nms_radius: 3
    trainable: false
  ground_truth:
    name: matchers.depth_matcher
    th_positive: 3
    th_negative: 5
    th_epi: 3
  matcher:
    name: matchers.lightglue
    filter_threshold: 0.3
    flash: true
    checkpointed: true
  allow_no_extract: true
train:
  seed: 42
  epochs: 50
  optimizer: adam
  opt_regexp: null
  optimizer_options: {}
  lr: 1.0e-05
  lr_schedule:
    type: exp
    start: 10
    exp_div_10: 45
    on_epoch: true
    factor: 1.0
    options: {}
  lr_scaling:
  - - 100
    - - dampingnet.const
  eval_every_iter: 500
  save_every_iter: 5000
  log_every_iter: 100
  log_grad_every_iter: null
  test_every_epoch: 1
  keep_last_checkpoints: 50
  load_experiment: pretrain_lightglue
  median_metrics: []
  recall_metrics: {}
  pr_metrics: {}
  best_key: loss/total
  dataset_callback_fn: sample_new_items
  dataset_callback_on_val: false
  clip_grad: null
  pr_curves: {}
  plot:
  - 5
  - gluefactory.visualization.visualize_batch.make_match_figures
  submodules: []
benchmarks:
  treeEval1:
    data:
      preprocessing:
        side: long
        resize: 1600
    eval:
      estimator: poselib
      ransac_th: 0.5
  finnEval:
    data:
      preprocessing:
        side: long
        resize: 1600
    eval:
      estimator: poselib
      ransac_th: 0.5
image_mode: null

[08/11/2024 15:33:45 gluefactory INFO] Starting epoch 0
getting last checkpoint pretrain_lightglue
getting best checkpoint pretrain_lightglue
Checkpoint path used is /vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/outputs/training/pretrain_lightglue/checkpoint_best.tar 
model conf after setting, loaded from checkpoint!!! {'name': 'two_view_pipeline', 'extractor': {'name': 'gluefactory_nonfree.superpoint', 'max_num_keypoints': 2048, 'force_num_keypoints': True, 'detection_threshold': 0.0, 'nms_radius': 3, 'trainable': False}, 'ground_truth': {'name': 'matchers.depth_matcher', 'th_positive': 3, 'th_negative': 5, 'th_epi': 3}, 'matcher': {'name': 'matchers.lightglue', 'filter_threshold': 0.3, 'flash': True, 'checkpointed': True}, 'allow_no_extract': True}
the rank is 2 and world size is 4
cuda:0
23 initpy data sets path gluefactory.datasets.treedepth
205 tools.py utils the classes are  [('TreeDepth', <class 'gluefactory.datasets.treedepth.TreeDepth'>)]
136 in treedepth
conf keys: dict_keys(['name', 'num_workers', 'train_batch_size', 'val_batch_size', 'test_batch_size', 'shuffle_training', 'batch_size', 'num_threads', 'seed', 'prefetch_factor', 'data_dir', 'depth_subpath', 'image_subpath', 'info_dir', 'train_split', 'train_num_per_scene', 'val_split', 'val_num_per_scene', 'val_pairs', 'test_split', 'test_num_per_scene', 'test_pairs', 'views', 'min_overlap', 'max_overlap', 'num_overlap_bins', 'sort_by_overlap', 'triplet_enforce_overlap', 'read_depth', 'read_image', 'grayscale', 'preprocessing', 'p_rotate', 'reseed', 'load_features'])
Loaded training dataset: treedepth
calling get_data_loader on dataset object - dataset.get_data_loader('train', distributed=args.distributed, shuffle=True)


scene_lists_path: /homes/tp4618/Documents/bitbucket/SuperGlueThesis/external/glue-factory/gluefactory/datasets/tartanSceneLists(newScenes)


root anrs', 'views', 'min_overlap', 'max_overlap', 'num_overlap_bins', 'sort_by_overlap', 'triplet_enforce_overlap', 'read_depth', 'read_image', 'grayscale', 'preprocessing', 'p_rotate', 'reseed', 'load_features'])
Loaded training dataset: treedepth
calling get_data_loader on dataset object - dataset.get_data_loader('train', distributed=args.distributed, shuffle=True)


scene_lists_path: /homes/tp4618/Documents/bitbucket/SuperGlueThesis/external/glue-factory/gluefactory/datasets/tartanSceneLists(newScenes)


root and info /vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/data/syntheticForestData fileLists(newScenes)
!!!!!sample_new_items: num_per_scene: 300
did a random shuffle!!!


scene_lists_path: /homes/tp4618/Documents/bitbucket/SuperGlueThesis/external/glue-factory/gluefactory/datasets/tartanSceneLists(newScenes)


root and info /vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/data/syntheticForestData fileLists(newScenes)
!!!!!sample_new_items: num_per_scene: None
did a random shuffle!!!
205 tools.py utils the classes are  [('TwoViewPipeline', <class 'gluefactory.models.two_view_pipeline.TwoViewPipeline'>)]
205 tools.py utils the classes are  [('SuperPoint', <class 'gluefactory_nonfree.superpoint.SuperPoint'>)]
None 



ModuleNotFoundError: matchers.lightglue
205 tools.py utils the classes are  []
400 lightglue conf flash is True
ModuleNotFoundError: matchers.depth_matcher
205 tools.py utils the classes are  [('DepthMatcher', <class 'gluefactory.models.matchers.depth_matcher.DepthMatcher'>)]
93 base_model.py DepthMatcher is trainable
93 base_model.py TwoViewPipeline is trainable




conf.model.name two_view_pipeline 




replaced!! the weights using two_view_pipeline



args distibuted
device ids are [device(type='cuda', index=0)]
after mpodel weight sub in
591 train.py, conf.train.lr_schedule.on_epoch is: True
/homes/tp4618/Documents/bitbucket/miniconda3/envs/redoGlue/lib/python3.12/site-packages/kornia/feature/lightglue.py:44: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
  @torch.cuda.amp.custom_fwd(cast_inputs=torch.float32)
/vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/gluefactory/models/matchers/loadedLightGlue.py:25: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
  @torch.cuda.amp.custom_fwd(cast_inputs=torch.float32)
/vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/gluefactory/train.py:2098: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  with autocast(enabled=args.mixed_precision is not None, dtype=mp_dtype):
/vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/gluefactory_nonfree/superpoint.py:647: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  with autocast():
ray25:852539:852803 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eno1
ray25:852539:852803 [0] NCCL INFO NET/IB : No device found.
ray25:852539:852803 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eno1
ray25:852539:852803 [0] NCCL INFO NET/Socket : Using [0]eno1:146.169.52.25<0>
ray25:852539:852803 [0] NCCL INFO Using non-device net plugin version 0
ray25:852539:852803 [0] NCCL INFO Using network Socket
ray25:852539:852803 [0] NCCL INFO comm 0x5f1c60c21980 rank 0 nranks 4 cudaDev 0 nvmlDev 0 busId 1000 commId 0x10d43582167b166 - Init START
ray25:852539:852803 [0] NCCL INFO comm 0x5f1c60c21980 rank 0 nRanks 4 nNodes 4 localRanks 1 localRank 0 MNNVL 0
ray25:852539:852803 [0] NCCL INFO Channel 00/02 :    0   1   2   3
ray25:852539:852803 [0] NCCL INFO Channel 01/02 :    0   1   2   3
ray25:852539:852803 [0] NCCL INFO Trees [0] 2/-1/-1->0->-1 [1] -1/-1/-1->0->1
ray25:852539:852803 [0] NCCL INFO P2P Chunksize set to 131072
ray25:852539:852803 [0] NCCL INFO Channel 00/0 : 3[0] -> 0[0] [receive] via NET/Socket/0
ray25:852539:852803 [0] NCCL INFO Channel 01/0 alRank 0 MNNVL 0
ray07:1162307:1162484 [0] NCCL INFO Trees [0] 1/3/-1->2->0 [1] -1/-1/-1->2->1
ray07:1162307:1162484 [0] NCCL INFO P2P Chunksize set to 131072
ray07:1162307:1162484 [0] NCCL INFO Channel 00/0 : 1[0] -> 2[0] [receive] via NET/Socket/0
ray07:1162307:1162484 [0] NCCL INFO Channel 01/0 : 1[0] -> 2[0] [receive] via NET/Socket/0
ray07:1162307:1162484 [0] NCCL INFO Channel 00/0 : 2[0] -> 3[0] [send] via NET/Socket/0
ray07:1162307:1162484 [0] NCCL INFO Channel 01/0 : 2[0] -> 3[0] [send] via NET/Socket/0
ray07:1162307:1162484 [0] NCCL INFO Connected all rings
ray07:1162307:1162484 [0] NCCL INFO Channel 00/0 : 0[0] -> 2[0] [receive] via NET/Socket/0
ray07:1162307:1162484 [0] NCCL INFO Channel 00/0 : 2[0] -> 0[0] [send] via NET/Socket/0
ray07:1162307:1162484 [0] NCCL INFO Channel 00/0 : 3[0] -> 2[0] [receive] via NET/Socket/0
ray07:1162307:1162484 [0] NCCL INFO Channel 00/0 : 2[0] -> 1[0] [send] via NET/Socket/0
ray07:1162307:1162484 [0] NCCL INFO Channel 01/0 : 2[0] -> 1[0] [send] via NET/Socket/0
ray07:1162307:1162484 [0] NCCL INFO Connected all trees
ray07:1162307:1162484 [0] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 512 | 512
ray07:1162307:1162484 [0] NCCL INFO 2 coll channels, 0 collnet channels, 0 nvls channels, 2 p2p channels, 2 p2p channels per peer
ray07:1162307:1162484 [0] NCCL INFO comm 0x6451259eabb0 rank 2 nranks 4 cudaDev 0 nvmlDev 0 busId 1000 commId 0x10d43582167b166 - Init COMPLETE
/homes/tp4618/Documents/bitbucket/miniconda3/envs/redoGlue/lib/python3.12/site-packages/torch/_dynamo/eval_frame.py:600: UserWarning: torch.utils.checkpoint: the use_reentrant parameter should be passed explicitly. In version 2.4 we will raise an exception if use_reentrant is not passed. use_reentrant=False is recommended, but if you need to preserve the current default behavior, you can pass use_reentrant=True. Refer to docs for more details on the differences between the two variants.
  return fn(*args, **kwargs)
/homes/tp4618/Documents/bitbucket/miniconda3/envs/redoGlue/lib/python3.12/site-packages/torch/utils/checkpoint.py:295: FutureWarning: `torch.cpu.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cpu', args...)` instead.
  with torch.enable_grad(), device_autocast_ctx, torch.cpu.amp.autocast(**ctx.cpu_autocast_kwargs):  # type: ignore[attr-defined]
[08/11/2024 15:34:18 gluefactory INFO] \Training intermediate DataFrame saved to: /homes/tp4618/Documents/bitbucket/SuperGlueThesis/external/glue-factory/outputs/training/11-08-fullDepth-newScenes/train_11-08-fullDepth-newScenes_intermediate_0_0_20240811_153418.pkl

[08/11/2024 15:34:18 gluefactory INFO] [E 0 | it 0] loss {total 3.334E-01, last 1.885E-01, assignment_nll 1.885E-01, nll_pos 3.103E-01, nll_neg 6.665E-02, num_matchable 2.108E+02, num_unmatchable 1.492E+02, confidence 8.901E-02, row_norm 2.182E-01}
Current iteraiton: 0
Data logged to training DataFrame: {'epoch': 0, 'iteration': 0, 'total': 0.33342868089675903, 'last': 0.18847325444221497, 'assignment_nll': 0.18847325444221497, 'nll_pos': 0.3102923035621643, 'nll_neg': 0.06665418297052383, 'num_matchable': 210.8125, 'num_unmatchable': 149.15625, 'confidence': 0.08900631964206696, 'row_norm': 0.21816596388816833}
line 813in train.py doing validation
/homes/tp4618/Documents/bitbucket/miniconda3/envs/redoGlue/lib/python3.12/site-packages/kornia/feature/lightglue.py:44: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
  @torch.cuda.amp.custom_fwd(cast_inputs=torch.float32)
/vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/gluefactory/models/matchers/loadedLightGlue.py:25: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
  @torch.cuda.amp.custom_fwd(cast_inputs=torch.float32)
/vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/gluefactory_nonfree/superpoint.py:647: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  witp4618/SuperGlueThesis/external/glue-factory/gluefactory/models/matchers/loadedLightGlue.py:25: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
  @torch.cuda.amp.custom_fwd(cast_inputs=torch.float32)
/vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/gluefactory_nonfree/superpoint.py:647: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  with autocast():
/vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/gluefactory/train.py:2295: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  results_df = pd.concat([results_df, new_entry], ignore_index=True)
[08/11/2024 15:45:38 gluefactory INFO] VALIDATION Intermediate DataFrame saved to: /homes/tp4618/Documents/bitbucket/SuperGlueThesis/external/glue-factory/outputs/training/11-08-fullDepth-newScenes/validation_11-08-fullDepth-newScenes_20240811_154538.pkl

[08/11/2024 15:45:38 gluefactory INFO] [Validation] {match_recall 8.497E-01, match_precision 9.198E-01, accuracy 8.838E-01, average_precision 7.862E-01, loss/total 6.356E-01, loss/last 6.356E-01, loss/assignment_nll 6.356E-01, loss/nll_pos 1.056E+00, loss/nll_neg 2.147E-01, loss/num_matchable 1.001E+03, loss/num_unmatchable 4.911E+02, loss/row_norm 8.848E-01}
[08/11/2024 15:45:38 gluefactory.utils.experiments INFO] Saving checkpoint checkpoint_best.tar
[08/11/2024 15:45:40 gluefactory INFO] New best val: loss/total=0.6355505599938764
/vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/gluefactory/train.py:2098: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  with autocast(enabled=args.mixed_precision is not None, dtype=mp_dtype):
/vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/gluefactory_nonfree/superpoint.py:647: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  with autocast():
/homes/tp4618/Documents/bitbucket/miniconda3/envs/redoGlue/lib/python3.12/site-packages/torch/_dynamo/eval_frame.py:600: UserWarning: torch.utils.checkpoint: the use_reentrant parameter should be passed explicitly. In version 2.4 we will raise an exception if use_reentrant is not passed. use_reentrant=False is recommended, but if you need to preserve the current default behavior, you can pass use_reentrant=True. Refer to docs for more details on the differences between the two variants.
  return fn(*args, **kwargs)
/homes/tp4618/Documents/bitbucket/miniconda3/envs/redoGlue/lib/python3.12/site-packages/torch/utils/checkpoint.py:295: FutureWarning: `torch.cpu.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cpu', args...)` instead.
  with torch.enable_grad(), device_autocast_ctx, torch.cpu.amp.autocast(**ctx.cpu_autocast_kwargs):  # type: ignore[attr-defined]
operation.
  results_df = pd.concat([results_df, new_entry], ignore_index=True)
[08/11/2024 15:46:51 gluefactory INFO] VALIDATION Intermediate DataFrame saved to: /homes/tp4618/Documents/bitbucket/SuperGlueThesis/external/glue-factory/outputs/training/11-08-fullDepth-newScenes/validation_11-08-fullDepth-newScenes_20240811_154651.pkl

[08/11/2024 15:46:51 gluefactory INFO] [Validation] {match_recall 8.497E-01, match_precision 9.199E-01, accuracy 8.839E-01, average_precision 7.863E-01, loss/total 6.355E-01, loss/last 6.355E-01, loss/assignment_nll 6.355E-01, loss/nll_pos 1.056E+00, loss/nll_neg 2.147E-01, loss/num_matchable 1.002E+03, loss/num_unmatchable 4.911E+02, loss/row_norm 8.849E-01}
[08/11/2024 15:46:51 gluefactory.utils.experiments INFO] Saving checkpoint checkpoint_best.tar
[08/11/2024 15:46:53 gluefactory INFO] New best val: loss/total=0.6355035161419725
/vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/gluefactory/train.py:2098: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  with autocast(enabled=args.mixed_precision is not None, dtype=mp_dtype):
/vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/gluefactory_nonfree/superpoint.py:647: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  with autocast():
/homes/tp4618/Documents/bitbucket/miniconda3/envs/redoGlue/lib/python3.12/site-packages/torch/_dynamo/eval_frame.py:600: UserWarning: torch.utils.checkpoint: the use_reentrant parameter should be passed explicitly. In version 2.4 we will raise an exception if use_reentrant is not passed. use_reentrant=False is recommended, but if you need to preserve the current default behavior, you can pass use_reentrant=True. Refer to docs for more details on the differences between the two variants.
  return fn(*args, **kwargs)
/homes/tp4618/Documents/bitbucket/miniconda3/envs/redoGlue/lib/python3.12/site-packages/torch/utils/checkpoint.py:295: FutureWarning: `torch.cpu.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cpu', args...)` instead.
  with torch.enable_grad(), device_autocast_ctx, torch.cpu.amp.autocast(**ctx.cpu_autocast_kwargs):  # type: ignore[attr-defined]
[08/11/2024 16:01:04 gluefactory INFO] \Training intermediate DataFrame saved to: /homes/tp4618/Documents/bitbucket/SuperGlueThesis/external/glue-factory/outputs/training/11-08-fullDepth-newScenes/train_11-08-fullDepth-newScenes_intermediate_0_100_20240811_160104.pkl

[08/11/2024 16:01:04 gluefactory INFO] [E 0 | it 100] loss {total 2.178E-01, last 1.186E-01, assignment_nll 1.186E-01, nll_pos 1.655E-01, nll_neg 7.170E-02, num_matchable 2.643E+02, num_unmatchable 1.227E+02, confidence 6.974E-02, row_norm 2.061E-01}
[08/11/2024 16:14:46 gluefactory INFO] \Training intermediate DataFrame saved to: /homes/tp4618/Documents/bitbucket/SuperGlueThesis/external/glue-factory/outputs/training/11-08-fullDepth-newScenes/train_11-08-fullDepth-newScenes_intermediate_0_200_20240811_161446.pkl

[08/11/2024 16:14:46 gluefactory INFO] [E 0 | it 200] loss {total 2.672E-01, last 1.926E-01, assignment_nll 1.926E-01, nll_pos 2.712E-01, nll_neg 1.140E-01, num_matchable 2.495E+02, num_unmatchable 1.219E+02, confidence 6.311E-02, row_norm 1.904E-01}
Data logged to validation: {'epoch': 0, 'iteration': 0, 'match_recall': 0.8496696976621179, 'match_precision': 0.9198328379038218, 'accuracy': 0.8838065611349569, 'average_precision': 0.786243470571216, 'loss/total': 0.6355512776430049, 'loss/last': 0.6355512776430049, 'loss/assignment_nll': 0.6355512776430049, 'loss/nll_pos': 1.0563756478799355, 'loss/nll_neg': 0.21472690287243906, 'loss/num_matchable': 1001.4895752895753, 'loss/num_unmatchable': 491.1065637065637, 'loss/row_norm': 0.8848498335216036}
comparing to see if it hte best experiment 875 train.py
key used was loss/total
 FROM RESULTS [conf.train.best_key] 0.6355512776430049
 best_eval inf
Current iteraiton: 100
Data logged to training DataFrame: {'epoch': 0, 'iteration': 100, 'total': 0.28327101469039917, 'last': 0.1744787096977234, 'assignment_nll': 0.1744787096977234, 'nll_pos': 0.25368261337280273, 'nll_neg': 0.09527481347322464, 'num_matchable': 233.1875, 'num_unmatchable': 131.609375, 'confidence': 0.07920143008232117, 'row_norm': 0.1946549415588379}
Current iteraiton: 200
Data logged to training DataFrame: {'epoch': 0, 'iteration': 200, 'total': 0.26929816603660583, 'last': 0.20449212193489075, 'assignment_nll': 0.20449212193489075, 'nll_pos': 0.31590431928634644, 'nll_neg': 0.09307988733053207, 'num_matchable': 236.5, 'num_unmatchable': 138.046875, 'confidence': 0.05306059867143631, 'row_norm': 0.1925976276397705}
line 813in train.py doing validati/homes/tp4618/Documents/bitbucket/miniconda3/envs/redoGlue/lib/python3.12/site-packages/kornia/feature/lightglue.py:44: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
  @torch.cuda.amp.custom_fwd(cast_inputs=torch.float32)
2)
/vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/gluefactory/models/matchers/loadedLightGlue.py:25: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
  @torch.cuda.amp.custom_fwd(cast_inputs=torch.float32)
/vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/gluefactory/models/matchers/loadedLightGlue.py:25: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
  @torch.cuda.amp.custom_fwd(cast_inputs=torch.float32)
[08/11/2024 16:25:50 gluefactory INFO] VALIDATION Intermediate DataFrame saved to: /homes/tp4618/Documents/bitbucket/SuperGlueThesis/external/glue-factory/outputs/training/11-08-fullDepth-newScenes/validation_11-08-fullDepth-newScenes_20240811_162550.pkl

[08/11/2024 16:25:50 gluefactory INFO] [Validation] {match_recall 7.835E-01, match_precision 9.072E-01, accuracy 8.382E-01, average_precision 7.153E-01, loss/total 7.838E-01, loss/last 7.838E-01, loss/assignment_nll 7.838E-01, loss/nll_pos 1.127E+00, loss/nll_neg 4.409E-01, loss/num_matchable 1.001E+03, loss/num_unmatchable 4.911E+02, loss/row_norm 7.610E-01}
[08/11/2024 16:25:55 gluefactory.utils.experiments INFO] Saving checkpoint checkpoint_0_968.tar
[08/11/2024 16:25:56 gluefactory INFO] Starting epoch 1
[08/11/2024 16:25:56 gluefactory INFO] lr changed from 1e-05 to 1e-05
[08/11/2024 16:25:56 gluefactory.datasets.treedepth INFO] Sampling new items for train with seed 43.
Data logged to validation: {'epoch': 0, 'iteration': 242, 'match_recall': 0.7834804135399895, 'match_precision': 0.9072480330596099, 'accuracy': 0.8382000208821536, 'average_precision': 0.7153278805574395, 'loss/total': 0.7837678291622736, 'loss/last': 0.7837678291622736, 'loss/assignment_nll': 0.7837678291622736, 'loss/nll_pos': 1.126635963852341, 'loss/nll_neg': 0.44089969438015264, 'loss/num_matchable': 1001.4895752895753, 'loss/num_unmatchable': 491.1065637065637, 'loss/row_norm': 0.7610156418273808}
comparing to see if it hte best experiment 875 train.py
key used was loss/total
 FROM RESULTS [conf.train.best_key] 0.7837678291622736
 best_eval 0.6355512776430049
591 train.py, conf.train.lr_schedule.on_epoch is: True
!!!!!sample_new_items: num_per_scene: 300
did a random shuffle!!!
/homes/tp4618/Documents/bitbucket/miniconda3/envs/redoGlue/lib/python3.12/site-packages/kornia/feature/lightglue.py:44: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
  @torch.cuda.amp.custom_fwd(cast_inputs=torch.float32)
/vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/gluefactory/models/matchers/loadedLightGlue.py:25: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
  @torch.cuda.amp.custom_fwd(cast_inputs=torch.float32)
[08/11/2024 16:28:09 gluefactory INFO] \Training intermediate DataFrame saved to: /homes/tp4618/Documents/bitbucket/SuperGlueThesis/external/glue-factory/outputs/training/11-08-fullDepth-newScenes/train_11-08-fullDepth-newScenes_intermediate_1_0_20240811_162809.pkl

[08/11/2024 16:28:09 gluefactory INFO] [E 1 | it 0] loss {total 2.472E-01, last 1.815E-01, assignment_nll 1.815E-01, nll_pos 2.526E-01, nll_neg 1.104E-01, num_matchable 2.494E+02, num_unmatchable 1.251E+02, confidence 5.438E-02, row_norm 1.912E-01}
neg 4.409E-01, loss/num_matchable 1.002E+03, loss/num_unmatchable 4.911E+02, loss/row_norm 7.610E-01}
[08/11/2024 16:27:41 gluefactory.utils.experiments INFO] Saving checkpoint checkpoint_0_968.tar
[08/11/2024 16:27:43 gluefactory INFO] Starting epoch 1
[08/11/2024 16:27:43 gluefactory INFO] lr changed from 1e-05 to 1e-05
[08/11/2024 16:27:43 gluefactory.datasets.treedepth INFO] Sampling new items for train with seed 43.
Data logged to validation: {'epoch': 0, 'iteration': 242, 'match_recall': 0.7835347309996262, 'match_precision': 0.907277858395374, 'accuracy': 0.8382223020649324, 'average_precision': 0.7153900846091016, 'loss/total': 0.7837689918901009, 'loss/last': 0.7837689918901009, 'loss/assignment_nll': 0.7837689918901009, 'loss/nll_pos': 1.1266661997467395, 'loss/nll_neg': 0.44087178108775016, 'loss/num_matchable': 1001.5061776061776, 'loss/num_unmatchable': 491.1034749034749, 'loss/row_norm': 0.7609892198938201}
comparing to see if it hte best experiment 875 train.py
key used was loss/total
 FROM RESULTS [conf.train.best_key] 0.7837689918901009
 best_eval 0.6355035161419725
591 train.py, conf.train.lr_schedule.on_epoch is: True
!!!!!sample_new_items: num_per_scene: 300
did a random shuffle!!!
/homes/tp4618/Documents/bitbucket/miniconda3/envs/redoGlue/lib/python3.12/site-packages/kornia/feature/lightglue.py:44: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
  @torch.cuda.amp.custom_fwd(cast_inputs=torch.float32)
/vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/gluefactory/models/matchers/loadedLightGlue.py:25: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
  @torch.cuda.amp.custom_fwd(cast_inputs=torch.float32)
[08/11/2024 16:28:09 gluefactory INFO] \Training intermediate DataFrame saved to: /homes/tp4618/Documents/bitbucket/SuperGlueThesis/external/glue-factory/outputs/training/11-08-fullDepth-newScenes/train_11-08-fullDepth-newScenes_intermediate_1_0_20240811_162809.pkl

[08/11/2024 16:28:09 gluefactory INFO] [E 1 | it 0] loss {total 2.452E-01, last 1.794E-01, assignment_nll 1.794E-01, nll_pos 2.620E-01, nll_neg 9.684E-02, num_matchable 2.712E+02, num_unmatchable 1.093E+02, confidence 5.552E-02, row_norm 1.907E-01}
[08/11/2024 16:40:07 gluefactory INFO] \Training intermediate DataFrame saved to: /homes/tp4618/Documents/bitbucket/SuperGlueThesis/external/glue-factory/outputs/training/11-08-fullDepth-newScenes/train_11-08-fullDepth-newScenes_intermediate_1_100_20240811_164007.pkl

[08/11/2024 16:40:07 gluefactory INFO] [E 1 | it 100] loss {total 2.943E-01, last 2.399E-01, assignment_nll 2.399E-01, nll_pos 3.566E-01, nll_neg 1.232E-01, num_matchable 2.262E+02, num_unmatchable 1.492E+02, confidence 4.811E-02, row_norm 1.793E-01}
[08/11/2024 16:52:05 gluefactory INFO] \Training intermediate DataFrame saved to: /homes/tp4618/Documents/bitbucket/SuperGlueThesis/external/glue-factory/outputs/training/11-08-fullDepth-newScenes/train_11-08-fullDepth-newScenes_intermediate_1_200_20240811_165205.pkl

[08/11/2024 16:52:05 gluefactory INFO] [E 1 | it 200] loss {total 3.558E-01, last 2.913E-01, assignment_nll 2.913E-01, nll_pos 4.377E-01, nll_neg 1.449E-01, num_matchable 2.027E+02, num_unmatchable 1.446E+02, confidence 5.716E-02, row_norm 1.765E-01}
Current iteraiton: 0
Data logged to training DataFrame: {'epoch': 1, 'iteration': 0, 'total': 0.26364800333976746, 'last': 0.19980350136756897, 'assignment_nll': 0.19980350136756897, 'nll_pos': 0.2807256579399109, 'nll_neg': 0.11888135969638824, 'num_matchable': 233.0625, 'num_unmatchable': 136.234375, 'confidence': 0.053060829639434814, 'row_norm': 0.19053345918655396}
Current iteraiton: 100
Data logged to training DataFrame: {'epoch': 1, 'iteration': 100, 'total': 0.2943170964717865, 'last': 0.2399119734764099, 'assignment_nll': 0.2399119734764099, 'nll_pos': 0.356620728969574, 'nll_neg': 0.12320323288440704, 'num_matchable': 226.15625, 'num_unmatchable': 149.171875, 'confidence': 0.04811403155326843, 'row_norm': 0.1793297827243805}
Current iteraiton: 200
Data logged to training DataFrame: {'epoch': 1, 'iteration': 200, 'total': 0.3558315932750702, 'last': 0.2913365960121155, 'assignment_nll': 0.2913365960121155, 'nll_pos': 0.4377361536026001, 'nll_neg': 0.14493706822395325, 'num_matchable': 202.6875, 'num_unmatchable': 144.5625, 'confidence': 0.057158052921295166, 'row_norm': 0.17654776573181152}
line 813in train.py doing validation
/homes/tp4618/Documents/bitbucket/miniconda3/envs/redoGlue/lib/python3.12/site-packages/kornia/feature/lightglue.py:44: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
  @torch.cuda.amp.custom_fwd(cast_inputs=torch.float32)
/vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/gluefactory/models/matchers/loadedLightGlue.py:25: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
  @torch.cuda.amp.custom_fwd(cast_inputs=torch.float32)
2)
[08/11/2024 17:02:45 gluefactory INFO] VALIDATION Intermediate DataFrame saved to: /homes/tp4618/Documents/bitbucket/SuperGlueThesis/external/glue-factory/outputs/training/11-08-fullDepth-newScenes/validation_11-08-fullDepth-newScenes_20240811_170245.pkl

[08/11/2024 17:02:45 gluefactory INFO] [Validation] {match_recall 7.492E-01, match_precision 9.183E-01, accuracy 8.205E-01, average_precision 6.919E-01, loss/total 8.630E-01, loss/last 8.630E-01, loss/assignment_nll 8.630E-01, loss/nll_pos 1.288E+00, loss/nll_neg 4.383E-01, loss/num_matchable 1.001E+03, loss/num_unmatchable 4.911E+02, loss/row_norm 7.329E-01}
[08/11/2024 17:02:49 gluefactory.utils.experiments INFO] Saving checkpoint checkpoint_1_1940.tar
[08/11/2024 17:02:50 gluefactory INFO] Starting epoch 2
[08/11/2024 17:02:50 gluefactory INFO] lr changed from 1e-05 to 1e-05
[08/11/2024 17:02:50 gluefactory.datasets.treedepth INFO] Sampling new items for train with seed 44.
Data logged to validation: {'epoch': 1, 'iteration': 242, 'match_recall': 0.7491856755437077, 'match_precision': 0.9182787138522822, 'accuracy': 0.8204663228804541, 'average_precision': 0.6918574044142911, 'loss/total': 0.8629782140945376, 'loss/last': 0.8629782140945376, 'loss/assignment_nll': 0.8629782140945376, 'loss/nll_pos': 1.2877037613548368, 'loss/nll_neg': 0.4382526745667329, 'loss/num_matchable': 1001.488416988417, 'loss/num_unmatchable': 491.1061776061776, 'loss/row_norm': 0.7329432342043254}
comparing to see if it hte best experiment 875 train.py
key used was loss/total
 FROM RESULTS [conf.train.best_key] 0.8629782140945376
 best_eval 0.6355505599938764
591 train.py, conf.train.lr_schedule.on_epoch is: True
!!!!!sample_new_items: num_per_scene: 300
did a random shuffle!!!
/homes/tp4618/Documents/bitbucket/miniconda3/envs/redoGlue/lib/python3.12/site-packages/kornia/feature/lightglue.py:44: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
  @torch.cuda.amp.custom_fwd(cast_inputs=torch.float32)
/vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/gluefactory/models/matchers/loadedLightGlue.py:25: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
  @torch.cuda.amp.custom_fwd(cast_inputs=torch.float32)
[08/11/2024 17:05:25 gluefactory INFO] \Training intermediate DataFrame saved to: /homes/tp4618/Documents/bitbucket/SuperGlueThesis/external/glue-factory/outputs/training/11-08-fullDepth-newScenes/train_11-08-fullDepth-newScenes_intermediate_2_0_20240811_170525.pkl

[08/11/2024 17:05:25 gluefactory INFO] [E 2 | it 0] loss {total 2.665E-01, last 2.149E-01, assignment_nll 2.149E-01, nll_pos 3.249E-01, nll_neg 1.049E-01, num_matchable 2.475E+02, num_unmatchable 1.257E+02, confidence 4.972E-02, row_norm 1.843E-01}
neg 4.382E-01, loss/num_matchable 1.002E+03, loss/num_unmatchable 4.911E+02, loss/row_norm 7.330E-01}
[08/11/2024 17:04:56 gluefactory.utils.experiments INFO] Saving checkpoint checkpoint_1_1940.tar
[08/11/2024 17:04:58 gluefactory INFO] Starting epoch 2
[08/11/2024 17:04:58 gluefactory INFO] lr changed from 1e-05 to 1e-05
[08/11/2024 17:04:58 gluefactory.datasets.treedepth INFO] Sampling new items for train with seed 44.
Data logged to validation: {'epoch': 1, 'iteration': 242, 'match_recall': 0.7492625389320049, 'match_precision': 0.9183533108832753, 'accuracy': 0.8205370186838864, 'average_precision': 0.6919793893931916, 'loss/total': 0.8629715747354574, 'loss/last': 0.8629715747354574, 'loss/assignment_nll': 0.8629715747354574, 'loss/nll_pos': 1.287767657534036, 'loss/nll_neg': 0.43817548558518693, 'loss/num_matchable': 1001.5061776061776, 'loss/num_unmatchable': 491.1034749034749, 'loss/row_norm': 0.7330007809009331}
comparing to see if it hte best experiment 875 train.py
key used was loss/total
 FROM RESULTS [conf.train.best_key] 0.8629715747354574
 best_eval 0.6355035161419725
591 train.py, conf.train.lr_schedule.on_epoch is: True
!!!!!sample_new_items: num_per_scene: 300
did a random shuffle!!!
/homes/tp4618/Documents/bitbucket/miniconda3/envs/redoGlue/lib/python3.12/site-packages/kornia/feature/lightglue.py:44: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
  @torch.cuda.amp.custom_fwd(cast_inputs=torch.float32)
/vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/gluefactory/models/matchers/loadedLightGlue.py:25: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
  @torch.cuda.amp.custom_fwd(cast_inputs=torch.float32)
[08/11/2024 17:05:25 gluefactory INFO] \Training intermediate DataFrame saved to: /homes/tp4618/Documents/bitbucket/SuperGlueThesis/external/glue-factory/outputs/training/11-08-fullDepth-newScenes/train_11-08-fullDepth-newScenes_intermediate_2_0_20240811_170525.pkl

[08/11/2024 17:05:25 gluefactory INFO] [E 2 | it 0] loss {total 2.416E-01, last 1.928E-01, assignment_nll 1.928E-01, nll_pos 2.769E-01, nll_neg 1.087E-01, num_matchable 2.524E+02, num_unmatchable 1.293E+02, confidence 5.051E-02, row_norm 1.840E-01}
[08/11/2024 17:17:23 gluefactory INFO] \Training intermediate DataFrame saved to: /homes/tp4618/Documents/bitbucket/SuperGlueThesis/external/glue-factory/outputs/training/11-08-fullDepth-newScenes/train_11-08-fullDepth-newScenes_intermediate_2_100_20240811_171723.pkl

[08/11/2024 17:17:23 gluefactory INFO] [E 2 | it 100] loss {total 3.152E-01, last 2.557E-01, assignment_nll 2.557E-01, nll_pos 4.033E-01, nll_neg 1.082E-01, num_matchable 2.372E+02, num_unmatchable 1.263E+02, confidence 5.254E-02, row_norm 1.779E-01}
[08/11/2024 17:29:20 gluefactory INFO] \Training intermediate DataFrame saved to: /homes/tp4618/Documents/bitbucket/SuperGlueThesis/external/glue-factory/outputs/training/11-08-fullDepth-newScenes/train_11-08-fullDepth-newScenes_intermediate_2_200_20240811_172920.pkl

[08/11/2024 17:29:20 gluefactory INFO] [E 2 | it 200] loss {total 2.417E-01, last 1.859E-01, assignment_nll 1.859E-01, nll_pos 2.849E-01, nll_neg 8.679E-02, num_matchable 2.676E+02, num_unmatchable 1.133E+02, confidence 5.747E-02, row_norm 1.870E-01}
Current iteraiton: 0
Data logged to training DataFrame: {'epoch': 2, 'iteration': 0, 'total': 0.26653721928596497, 'last': 0.21487388014793396, 'assignment_nll': 0.21487388014793396, 'nll_pos': 0.32489335536956787, 'nll_neg': 0.10485437512397766, 'num_matchable': 247.5, 'num_unmatchable': 125.71875, 'confidence': 0.04972054809331894, 'row_norm': 0.184337317943573}
Current iteraiton: 100
Data logged to training DataFrame: {'epoch': 2, 'iteration': 100, 'total': 0.3406094014644623, 'last': 0.28205928206443787, 'assignment_nll': 0.28205928206443787, 'nll_pos': 0.4423277974128723, 'nll_neg': 0.12179076671600342, 'num_matchable': 197.46875, 'num_unmatchable': 159.578125, 'confidence': 0.053955063223838806, 'row_norm': 0.17683307826519012}
Current iteraiton: 200
Data logged to training DataFrame: {'epoch': 2, 'iteration': 200, 'total': 0.2416660636663437, 'last': 0.18586182594299316, 'assignment_nll': 0.18586182594299316, 'nll_pos': 0.28493690490722656, 'nll_neg': 0.08678673207759857, 'num_matchable': 267.5625, 'num_unmatchable': 113.3125, 'confidence': 0.05746693164110184, 'row_norm': 0.1870071291923523}
line 813in train.py doing validation
/homes/tp4618/Documents/bitbucket/miniconda3/envs/redoGlue/lib/python3.12/site-packages/kornia/feature/lightglue.py:44: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
  @torch.cuda.amp.custom_fwd(cast_inputs=torch.float32)
/vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/gluefactory/models/matchers/loadedLightGlue.py:25: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
  @torch.cuda.amp.custom_fwd(cast_inputs=torch.float32)
32)
[08/11/2024 17:39:56 gluefactory INFO] VALIDATION Intermediate DataFrame saved to: /homes/tp4618/Documents/bitbucket/SuperGlueThesis/external/glue-factory/outputs/training/11-08-fullDepth-newScenes/validation_11-08-fullDepth-newScenes_20240811_173956.pkl

[08/11/2024 17:39:56 gluefactory INFO] [Validation] {match_recall 7.375E-01, match_precision 9.240E-01, accuracy 8.149E-01, average_precision 6.851E-01, loss/total 8.836E-01, loss/last 8.836E-01, loss/assignment_nll 8.836E-01, loss/nll_pos 1.376E+00, loss/nll_neg 3.917E-01, loss/num_matchable 1.001E+03, loss/num_unmatchable 4.911E+02, loss/row_norm 7.399E-01}
[08/11/2024 17:40:00 gluefactory.utils.experiments INFO] Saving checkpoint checkpoint_2_2912.tar
[08/11/2024 17:40:02 gluefactory INFO] Starting epoch 3
[08/11/2024 17:40:02 gluefactory INFO] lr changed from 1e-05 to 1e-05
[08/11/2024 17:40:02 gluefactory.datasets.treedepth INFO] Sampling new items for train with seed 45.
Data logged to validation: {'epoch': 2, 'iteration': 242, 'match_recall': 0.7375226079727232, 'match_precision': 0.9239597825009851, 'accuracy': 0.8149410372995501, 'average_precision': 0.6850722721644811, 'loss/total': 0.8836256813358616, 'loss/last': 0.8836256813358616, 'loss/assignment_nll': 0.8836256813358616, 'loss/nll_pos': 1.3755470093612965, 'loss/nll_neg': 0.3917043457620392, 'loss/num_matchable': 1001.488416988417, 'loss/num_unmatchable': 491.1061776061776, 'loss/row_norm': 0.739910566484606}
comparing to see if it hte best experiment 875 train.py
key used was loss/total
 FROM RESULTS [conf.train.best_key] 0.8836256813358616
 best_eval 0.6355505599938764
591 train.py, conf.train.lr_schedule.on_epoch is: True
!!!!!sample_new_items: num_per_scene: 300
did a random shuffle!!!
/homes/tp4618/Documents/bitbucket/miniconda3/envs/redoGlue/lib/python3.12/site-packages/kornia/feature/lightglue.py:44: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
  @torch.cuda.amp.custom_fwd(cast_inputs=torch.float32)
/vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/gluefactory/models/matchers/loadedLightGlue.py:25: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
  @torch.cuda.amp.custom_fwd(cast_inputs=torch.float32)
[08/11/2024 17:42:38 gluefactory INFO] \Training intermediate DataFrame saved to: /homes/tp4618/Documents/bitbucket/SuperGlueThesis/external/glue-factory/outputs/training/11-08-fullDepth-newScenes/train_11-08-fullDepth-newScenes_intermediate_3_0_20240811_174237.pkl

[08/11/2024 17:42:38 gluefactory INFO] [E 3 | it 0] loss {total 3.365E-01, last 2.794E-01, assignment_nll 2.794E-01, nll_pos 4.582E-01, nll_neg 1.006E-01, num_matchable 2.119E+02, num_unmatchable 1.450E+02, confidence 5.713E-02, row_norm 1.741E-01}
neg 3.917E-01, loss/num_matchable 1.002E+03, loss/num_unmatchable 4.911E+02, loss/row_norm 7.399E-01}
[08/11/2024 17:42:12 gluefactory.utils.experiments INFO] Saving checkpoint checkpoint_2_2912.tar
[08/11/2024 17:42:14 gluefactory INFO] Starting epoch 3
[08/11/2024 17:42:14 gluefactory INFO] lr changed from 1e-05 to 1e-05
[08/11/2024 17:42:14 gluefactory.datasets.treedepth INFO] Sampling new items for train with seed 45.
Data logged to validation: {'epoch': 2, 'iteration': 242, 'match_recall': 0.7375234049719733, 'match_precision': 0.9240544970891651, 'accuracy': 0.814952530363812, 'average_precision': 0.6851318877636235, 'loss/total': 0.8836591543838325, 'loss/last': 0.8836591543838325, 'loss/assignment_nll': 0.8836591543838325, 'loss/nll_pos': 1.3756567321688973, 'loss/nll_neg': 0.391661574481537, 'loss/num_matchable': 1001.5061776061776, 'loss/num_unmatchable': 491.1034749034749, 'loss/row_norm': 0.7399344608130142}
comparing to see if it hte best experiment 875 train.py
key used was loss/total
 FROM RESULTS [conf.train.best_key] 0.8836591543838325
 best_eval 0.6355035161419725
591 train.py, conf.train.lr_schedule.on_epoch is: True
!!!!!sample_new_items: num_per_scene: 300
did a random shuffle!!!
/homes/tp4618/Documents/bitbucket/miniconda3/envs/redoGlue/lib/python3.12/site-packages/kornia/feature/lightglue.py:44: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
  @torch.cuda.amp.custom_fwd(cast_inputs=torch.float32)
/vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/gluefactory/models/matchers/loadedLightGlue.py:25: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
  @torch.cuda.amp.custom_fwd(cast_inputs=torch.float32)
[08/11/2024 17:42:37 gluefactory INFO] \Training intermediate DataFrame saved to: /homes/tp4618/Documents/bitbucket/SuperGlueThesis/external/glue-factory/outputs/training/11-08-fullDepth-newScenes/train_11-08-fullDepth-newScenes_intermediate_3_0_20240811_174237.pkl

[08/11/2024 17:42:37 gluefactory INFO] [E 3 | it 0] loss {total 2.949E-01, last 2.375E-01, assignment_nll 2.375E-01, nll_pos 3.671E-01, nll_neg 1.078E-01, num_matchable 2.400E+02, num_unmatchable 1.237E+02, confidence 5.767E-02, row_norm 1.839E-01}
[08/11/2024 17:54:35 gluefactory INFO] \Training intermediate DataFrame saved to: /homes/tp4618/Documents/bitbucket/SuperGlueThesis/external/glue-factory/outputs/training/11-08-fullDepth-newScenes/train_11-08-fullDepth-newScenes_intermediate_3_100_20240811_175435.pkl

[08/11/2024 17:54:35 gluefactory INFO] [E 3 | it 100] loss {total 2.913E-01, last 2.380E-01, assignment_nll 2.380E-01, nll_pos 3.897E-01, nll_neg 8.633E-02, num_matchable 2.391E+02, num_unmatchable 1.404E+02, confidence 5.743E-02, row_norm 1.853E-01}
[08/11/2024 18:06:32 gluefactory INFO] \Training intermediate DataFrame saved to: /homes/tp4618/Documents/bitbucket/SuperGlueThesis/external/glue-factory/outputs/training/11-08-fullDepth-newScenes/train_11-08-fullDepth-newScenes_intermediate_3_200_20240811_180632.pkl

[08/11/2024 18:06:32 gluefactory INFO] [E 3 | it 200] loss {total 2.734E-01, last 2.165E-01, assignment_nll 2.165E-01, nll_pos 3.136E-01, nll_neg 1.195E-01, num_matchable 2.511E+02, num_unmatchable 1.180E+02, confidence 5.906E-02, row_norm 1.786E-01}
Current iteraiton: 0
Data logged to training DataFrame: {'epoch': 3, 'iteration': 0, 'total': 0.3364984393119812, 'last': 0.2793594002723694, 'assignment_nll': 0.2793594002723694, 'nll_pos': 0.4581647217273712, 'nll_neg': 0.10055409371852875, 'num_matchable': 211.90625, 'num_unmatchable': 144.96875, 'confidence': 0.05713178217411041, 'row_norm': 0.17408841848373413}
Current iteraiton: 100
Data logged to training DataFrame: {'epoch': 3, 'iteration': 100, 'total': 0.3622322082519531, 'last': 0.29260823130607605, 'assignment_nll': 0.29260823130607605, 'nll_pos': 0.47104039788246155, 'nll_neg': 0.11417605727910995, 'num_matchable': 192.25, 'num_unmatchable': 156.828125, 'confidence': 0.0708300918340683, 'row_norm': 0.17991343140602112}
Current iteraiton: 200
Data logged to training DataFrame: {'epoch': 3, 'iteration': 200, 'total': 0.31956854462623596, 'last': 0.2691047787666321, 'assignment_nll': 0.2691047787666321, 'nll_pos': 0.40253880620002747, 'nll_neg': 0.1356707513332367, 'num_matchable': 211.5625, 'num_unmatchable': 149.609375, 'confidence': 0.05071090906858444, 'row_norm': 0.17663615942001343}
line 813in train.py doing validation
/homes/tp4618/Documents/bitbucket/miniconda3/envs/redoGlue/lib/python3.12/site-packages/kornia/feature/lightglue.py:44: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
  @torch.cuda.amp.custom_fwd(cast_inputs=torch.float32)
/vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/gluefactory/models/matchers/loadedLightGlue.py:25: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
  @torch.cuda.amp.custom_fwd(cast_inputs=torch.float32)
[08/11/2024 18:17:13 gluefactory INFO] VALIDATION Intermediate DataFrame saved to: /homes/tp4618/Documents/bitbucket/SuperGlueThesis/external/glue-factory/outputs/training/11-08-fullDepth-newScenes/validation_11-08-fullDepth-newScenes_20240811_181712.pkl

[08/11/2024 18:17:12 gluefactory INFO] [Validation] {match_recall 7.527E-01, match_precision 9.181E-01, accuracy 8.226E-01, average_precision 6.948E-01, loss/total 8.937E-01, loss/last 8.937E-01, loss/assignment_nll 8.937E-01, loss/nll_pos 1.310E+00, loss/nll_neg 4.776E-01, loss/num_matchable 1.001E+03, loss/num_unmatchable 4.911E+02, loss/row_norm 7.285E-01}
[08/11/2024 18:17:15 gluefactory.utils.experiments INFO] Saving checkpoint checkpoint_3_3884.tar
[08/11/2024 18:17:17 gluefactory INFO] Starting epoch 4
[08/11/2024 18:17:17 gluefactory INFO] lr changed from 1e-05 to 1e-05
[08/11/2024 18:17:17 gluefactory.datasets.treedepth INFO] Sampling new items for train with seed 46.
Data logged to validation: {'epoch': 3, 'iteration': 242, 'match_recall': 0.7527427351152575, 'match_precision': 0.9180750331363162, 'accuracy': 0.8225865487426405, 'average_precision': 0.6948305652868794, 'loss/total': 0.8937462646528561, 'loss/last': 0.8937462646528561, 'loss/assignment_nll': 0.8937462646528561, 'loss/nll_pos': 1.309904431008004, 'loss/nll_neg': 0.47758810041493893, 'loss/num_matchable': 1001.4895752895753, 'loss/num_unmatchable': 491.1065637065637, 'loss/row_norm': 0.7284688209474777}
comparing to see if it hte best experiment 875 train.py
key used was loss/total
 FROM RESULTS [conf.train.best_key] 0.8937462646528561
 best_eval 0.6355512776430049
591 train.py, conf.train.lr_schedule.on_epoch is: True
!!!!!sample_new_items: num_per_scene: 300
did a random shuffle!!!
/homes/tp4618/Documents/bitbucket/miniconda3/envs/redoGlue/lib/python3.12/site-packages/kornia/feature/lightglue.py:44: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
  @torch.cuda.amp.custom_fwd(cast_inputs=torch.float32)
/vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/gluefactory/models/matchers/loadedLightGlue.py:25: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
  @torch.cuda.amp.custom_fwd(cast_inputs=torch.float32)
[08/11/2024 18:19:44 gluefactory INFO] \Training intermediate DataFrame saved to: /homes/tp4618/Documents/bitbucket/SuperGlueThesis/external/glue-factory/outputs/training/11-08-fullDepth-newScenes/train_11-08-fullDepth-newScenes_intermediate_4_0_20240811_181944.pkl

[08/11/2024 18:19:44 gluefactory INFO] [E 4 | it 0] loss {total 2.789E-01, last 2.296E-01, assignment_nll 2.296E-01, nll_pos 3.476E-01, nll_neg 1.116E-01, num_matchable 2.178E+02, num_unmatchable 1.495E+02, confidence 4.640E-02, row_norm 1.869E-01}
neg 4.776E-01, loss/num_matchable 1.002E+03, loss/num_unmatchable 4.911E+02, loss/row_norm 7.285E-01}
[08/11/2024 18:19:20 gluefactory.utils.experiments INFO] Saving checkpoint checkpoint_3_3884.tar
[08/11/2024 18:19:21 gluefactory INFO] Starting epoch 4
[08/11/2024 18:19:21 gluefactory INFO] lr changed from 1e-05 to 1e-05
[08/11/2024 18:19:21 gluefactory.datasets.treedepth INFO] Sampling new items for train with seed 46.
Data logged to validation: {'epoch': 3, 'iteration': 242, 'match_recall': 0.7527584149570539, 'match_precision': 0.9181237920370802, 'accuracy': 0.8225845311139081, 'average_precision': 0.6948869007434624, 'loss/total': 0.8937753309154143, 'loss/last': 0.8937753309154143, 'loss/assignment_nll': 0.8937753309154143, 'loss/nll_pos': 1.309910248973655, 'loss/nll_neg': 0.4776404163552067, 'loss/num_matchable': 1001.5061776061776, 'loss/num_unmatchable': 491.1034749034749, 'loss/row_norm': 0.7284997511094141}
comparing to see if it hte best experiment 875 train.py
key used was loss/total
 FROM RESULTS [conf.train.best_key] 0.8937753309154143
 best_eval 0.6355035161419725
591 train.py, conf.train.lr_schedule.on_epoch is: True
!!!!!sample_new_items: num_per_scene: 300
did a random shuffle!!!
/homes/tp4618/Documents/bitbucket/miniconda3/envs/redoGlue/lib/python3.12/site-packages/kornia/feature/lightglue.py:44: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
  @torch.cuda.amp.custom_fwd(cast_inputs=torch.float32)
/vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/gluefactory/models/matchers/loadedLightGlue.py:25: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
  @torch.cuda.amp.custom_fwd(cast_inputs=torch.float32)
[08/11/2024 18:19:44 gluefactory INFO] \Training intermediate DataFrame saved to: /homes/tp4618/Documents/bitbucket/SuperGlueThesis/external/glue-factory/outputs/training/11-08-fullDepth-newScenes/train_11-08-fullDepth-newScenes_intermediate_4_0_20240811_181944.pkl

[08/11/2024 18:19:44 gluefactory INFO] [E 4 | it 0] loss {total 4.501E-01, last 3.933E-01, assignment_nll 3.933E-01, nll_pos 6.430E-01, nll_neg 1.435E-01, num_matchable 1.826E+02, num_unmatchable 1.599E+02, confidence 4.985E-02, row_norm 1.678E-01}
[08/11/2024 18:31:40 gluefactory INFO] \Training intermediate DataFrame saved to: /homes/tp4618/Documents/bitbucket/SuperGlueThesis/external/glue-factory/outputs/training/11-08-fullDepth-newScenes/train_11-08-fullDepth-newScenes_intermediate_4_100_20240811_183140.pkl

[08/11/2024 18:31:40 gluefactory INFO] [E 4 | it 100] loss {total 2.966E-01, last 2.475E-01, assignment_nll 2.475E-01, nll_pos 3.918E-01, nll_neg 1.032E-01, num_matchable 2.321E+02, num_unmatchable 1.331E+02, confidence 5.120E-02, row_norm 1.812E-01}
[08/11/2024 18:43:37 gluefactory INFO] \Training intermediate DataFrame saved to: /homes/tp4618/Documents/bitbucket/SuperGlueThesis/external/glue-factory/outputs/training/11-08-fullDepth-newScenes/train_11-08-fullDepth-newScenes_intermediate_4_200_20240811_184337.pkl

[08/11/2024 18:43:37 gluefactory INFO] [E 4 | it 200] loss {total 3.778E-01, last 3.269E-01, assignment_nll 3.269E-01, nll_pos 5.186E-01, nll_neg 1.353E-01, num_matchable 2.115E+02, num_unmatchable 1.471E+02, confidence 5.385E-02, row_norm 1.640E-01}
Current iteraiton: 0
Data logged to training DataFrame: {'epoch': 4, 'iteration': 0, 'total': 0.4501326084136963, 'last': 0.393268346786499, 'assignment_nll': 0.393268346786499, 'nll_pos': 0.6430438160896301, 'nll_neg': 0.14349284768104553, 'num_matchable': 182.5625, 'num_unmatchable': 159.890625, 'confidence': 0.0498487651348114, 'row_norm': 0.16781558096408844}
Current iteraiton: 100
Data logged to training DataFrame: {'epoch': 4, 'iteration': 100, 'total': 0.2965772747993469, 'last': 0.24751943349838257, 'assignment_nll': 0.24751943349838257, 'nll_pos': 0.3918102979660034, 'nll_neg': 0.10322858393192291, 'num_matchable': 232.125, 'num_unmatchable': 133.140625, 'confidence': 0.0511983186006546, 'row_norm': 0.18117058277130127}
Current iteraiton: 200
Data logged to training DataFrame: {'epoch': 4, 'iteration': 200, 'total': 0.37778374552726746, 'last': 0.3269401788711548, 'assignment_nll': 0.3269401788711548, 'nll_pos': 0.5186115503311157, 'nll_neg': 0.13526885211467743, 'num_matchable': 211.53125, 'num_unmatchable': 147.140625, 'confidence': 0.053847622126340866, 'row_norm': 0.16398011147975922}
line 813in train.py doing validation
/homes/tp4618/Documents/bitbucket/miniconda3/envs/redoGlue/lib/python3.12/site-packages/kornia/feature/lightglue.py:44: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
  @torch.cuda.amp.custom_fwd(cast_inputs=torch.float3/vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/gluefactory/models/matchers/loadedLightGlue.py:25: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
  @torch.cuda.amp.custom_fwd(cast_inputs=torch.float32)
[08/11/2024 18:54:09 gluefactory INFO] VALIDATION Intermediate DataFrame saved to: /homes/tp4618/Documents/bitbucket/SuperGlueThesis/external/glue-factory/outputs/training/11-08-fullDepth-newScenes/validation_11-08-fullDepth-newScenes_20240811_185409.pkl

[08/11/2024 18:54:09 gluefactory INFO] [Validation] {match_recall 6.295E-01, match_precision 9.384E-01, accuracy 7.479E-01, average_precision 5.931E-01, loss/total 1.145E+00, loss/last 1.145E+00, loss/assignment_nll 1.145E+00, loss/nll_pos 1.803E+00, loss/nll_neg 4.874E-01, loss/num_matchable 1.001E+03, loss/num_unmatchable 4.911E+02, loss/row_norm 6.775E-01}
[08/11/2024 18:54:12 gluefactory.utils.experiments INFO] Saving checkpoint checkpoint_4_4856.tar
[08/11/2024 18:54:13 gluefactory INFO] Starting epoch 5
[08/11/2024 18:54:14 gluefactory INFO] lr changed from 1e-05 to 1e-05
[08/11/2024 18:54:14 gluefactory.datasets.treedepth INFO] Sampling new items for train with seed 47.
Data logged to validation: {'epoch': 4, 'iteration': 242, 'match_recall': 0.6294999500959536, 'match_precision': 0.9384280641106565, 'accuracy': 0.7479397694576662, 'average_precision': 0.5931040387356143, 'loss/total': 1.145027105044214, 'loss/last': 1.145027105044214, 'loss/assignment_nll': 1.145027105044214, 'loss/nll_pos': 1.8026185643258703, 'loss/nll_neg': 0.487435643093006, 'loss/num_matchable': 1001.4895752895753, 'loss/num_unmatchable': 491.1065637065637, 'loss/row_norm': 0.6774792971298041}
comparing to see if it hte best experiment 875 train.py
key used was loss/total
 FROM RESULTS [conf.train.best_key] 1.145027105044214
 best_eval 0.6355512776430049
591 train.py, conf.train.lr_schedule.on_epoch is: True
!!!!!sample_new_items: num_per_scene: 300
did a random shuffle!!!
/homes/tp4618/Documents/bitbucket/miniconda3/envs/redoGlue/lib/python3.12/site-packages/kornia/feature/lightglue.py:44: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
  @torch.cuda.amp.custom_fwd(cast_inputs=torch.float32)
/vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/gluefactory/models/matchers/loadedLightGlue.py:25: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
  @torch.cuda.amp.custom_fwd(cast_inputs=torch.float32)
[08/11/2024 18:56:53 gluefactory INFO] \Training intermediate DataFrame saved to: /homes/tp4618/Documents/bitbucket/SuperGlueThesis/external/glue-factory/outputs/training/11-08-fullDepth-newScenes/train_11-08-fullDepth-newScenes_intermediate_5_0_20240811_185653.pkl

[08/11/2024 18:56:53 gluefactory INFO] [E 5 | it 0] loss {total 4.544E-01, last 3.813E-01, assignment_nll 3.813E-01, nll_pos 6.225E-01, nll_neg 1.400E-01, num_matchable 2.084E+02, num_unmatchable 1.374E+02, confidence 7.498E-02, row_norm 1.579E-01}
neg 4.875E-01, loss/num_matchable 1.002E+03, loss/num_unmatchable 4.911E+02, loss/row_norm 6.774E-01}
[08/11/2024 18:56:30 gluefactory.utils.experiments INFO] Saving checkpoint checkpoint_4_4856.tar
[08/11/2024 18:56:31 gluefactory INFO] Starting epoch 5
[08/11/2024 18:56:31 gluefactory INFO] lr changed from 1e-05 to 1e-05
[08/11/2024 18:56:31 gluefactory.datasets.treedepth INFO] Sampling new items for train with seed 47.
Data logged to validation: {'epoch': 4, 'iteration': 242, 'match_recall': 0.6294121590360251, 'match_precision': 0.9384171364390251, 'accuracy': 0.7478739359203913, 'average_precision': 0.5930248134384745, 'loss/total': 1.1450377107126832, 'loss/last': 1.1450377107126832, 'loss/assignment_nll': 1.1450377107126832, 'loss/nll_pos': 1.8026185057798407, 'loss/nll_neg': 0.4874569180389169, 'loss/num_matchable': 1001.5061776061776, 'loss/num_unmatchable': 491.1034749034749, 'loss/row_norm': 0.6774102400628756}
comparing to see if it hte best experiment 875 train.py
key used was loss/total
 FROM RESULTS [conf.train.best_key] 1.1450377107126832
 best_eval 0.6355035161419725
591 train.py, conf.train.lr_schedule.on_epoch is: True
!!!!!sample_new_items: num_per_scene: 300
did a random shuffle!!!
/homes/tp4618/Documents/bitbucket/miniconda3/envs/redoGlue/lib/python3.12/site-packages/kornia/feature/lightglue.py:44: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
  @torch.cuda.amp.custom_fwd(cast_inputs=torch.float32)
/vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/gluefactory/models/matchers/loadedLightGlue.py:25: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
  @torch.cuda.amp.custom_fwd(cast_inputs=torch.float32)
[08/11/2024 18:56:53 gluefactory INFO] \Training intermediate DataFrame saved to: /homes/tp4618/Documents/bitbucket/SuperGlueThesis/external/glue-factory/outputs/training/11-08-fullDepth-newScenes/train_11-08-fullDepth-newScenes_intermediate_5_0_20240811_185653.pkl

[08/11/2024 18:56:53 gluefactory INFO] [E 5 | it 0] loss {total 3.213E-01, last 2.535E-01, assignment_nll 2.535E-01, nll_pos 3.903E-01, nll_neg 1.166E-01, num_matchable 2.734E+02, num_unmatchable 1.108E+02, confidence 7.013E-02, row_norm 1.735E-01}
[08/11/2024 19:01:02 gluefactory.utils.experiments INFO] Saving checkpoint checkpoint_5_5000.tar
[08/11/2024 19:08:51 gluefactory INFO] \Training intermediate DataFrame saved to: /homes/tp4618/Documents/bitbucket/SuperGlueThesis/external/glue-factory/outputs/training/11-08-fullDepth-newScenes/train_11-08-fullDepth-newScenes_intermediate_5_100_20240811_190851.pkl

[08/11/2024 19:08:51 gluefactory INFO] [E 5 | it 100] loss {total 4.116E-01, last 3.532E-01, assignment_nll 3.532E-01, nll_pos 5.353E-01, nll_neg 1.711E-01, num_matchable 2.170E+02, num_unmatchable 1.390E+02, confidence 5.806E-02, row_norm 1.613E-01}
[08/11/2024 19:20:47 gluefactory INFO] \Training intermediate DataFrame saved to: /homes/tp4618/Documents/bitbucket/SuperGlueThesis/external/glue-factory/outputs/training/11-08-fullDepth-newScenes/train_11-08-fullDepth-newScenes_intermediate_5_200_20240811_192047.pkl

[08/11/2024 19:20:47 gluefactory INFO] [E 5 | it 200] loss {total 3.114E-01, last 2.565E-01, assignment_nll 2.565E-01, nll_pos 3.539E-01, nll_neg 1.591E-01, num_matchable 2.432E+02, num_unmatchable 1.283E+02, confidence 5.850E-02, row_norm 1.688E-01}
Current iteraiton: 0
Data logged to training DataFrame: {'epoch': 5, 'iteration': 0, 'total': 0.3989049196243286, 'last': 0.335681289434433, 'assignment_nll': 0.335681289434433, 'nll_pos': 0.5423226356506348, 'nll_neg': 0.1290399432182312, 'num_matchable': 221.6875, 'num_unmatchable': 138.359375, 'confidence': 0.06550313532352448, 'row_norm': 0.16320830583572388}
Current iteraiton: 100
Data logged to training DataFrame: {'epoch': 5, 'iteration': 100, 'total': 0.40180736780166626, 'last': 0.34274351596832275, 'assignment_nll': 0.34274351596832275, 'nll_pos': 0.5096281170845032, 'nll_neg': 0.17585888504981995, 'num_matchable': 204.03125, 'num_unmatchable': 150.609375, 'confidence': 0.060364075005054474, 'row_norm': 0.154571071267128}
Current iteraiton: 200
Data logged to training DataFrame: {'epoch': 5, 'iteration': 200, 'total': 0.3670777380466461, 'last': 0.31518226861953735, 'assignment_nll': 0.31518226861953735, 'nll_pos': 0.468600869178772, 'nll_neg': 0.16176368296146393, 'num_matchable': 222.78125, 'num_unmatchable': 134.203125, 'confidence': 0.05147571861743927, 'row_norm': 0.1599082350730896}
line 813in train.py doing validation
/homes/tp4618/Documents/bitbucket/miniconda3/envs/redoGlue/lib/python3.12/site-packages/kornia/feature/lightglue.py:44: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
  @torch.cuda.amp.custom_fwd(cast_inputs=torch.float32)
/vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/gluefactory/models/matchers/loadedLightGlue.py:25: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
  @torch.cuda.amp.custom_fwd(cast_inputs=torch.float32)
[08/11/2024 19:31:26 gluefactory INFO] VALIDATION Intermediate DataFrame saved to: /homes/tp4618/Documents/bitbucket/SuperGlueThesis/external/glue-factory/outputs/training/11-08-fullDepth-newScenes/validation_11-08-fullDepth-newScenes_20240811_193114.pkl

[08/11/2024 19:31:14 gluefactory INFO] [Validation] {match_recall 6.510E-01, match_precision 9.338E-01, accuracy 7.604E-01, average_precision 6.106E-01, loss/total 1.106E+00, loss/last 1.106E+00, loss/assignment_nll 1.106E+00, loss/nll_pos 1.761E+00, loss/nll_neg 4.509E-01, loss/num_matchable 1.001E+03, loss/num_unmatchable 4.911E+02, loss/row_norm 6.986E-01}
[08/11/2024 19:31:18 gluefactory.utils.experiments INFO] Saving checkpoint checkpoint_5_5828.tar
[08/11/2024 19:31:19 gluefactory INFO] Starting epoch 6
[08/11/2024 19:31:19 gluefactory INFO] lr changed from 1e-05 to 1e-05
[08/11/2024 19:31:19 gluefactory.datasets.treedepth INFO] Sampling new items for train with seed 48.
Data logged to validation: {'epoch': 5, 'iteration': 242, 'match_recall': 0.6510273393056567, 'match_precision': 0.933835287351866, 'accuracy': 0.7603858759964756, 'average_precision': 0.610620944564407, 'loss/total': 1.1058460714273932, 'loss/last': 1.1058460714273932, 'loss/assignment_nll': 1.1058460714273932, 'loss/nll_pos': 1.7607882694848256, 'loss/nll_neg': 0.45090388441638135, 'loss/num_matchable': 1001.488416988417, 'loss/num_unmatchable': 491.1061776061776, 'loss/row_norm': 0.6986132119152997}
comparing to see if it hte best experiment 875 train.py
key used was loss/total
 FROM RESULTS [conf.train.best_key] 1.1058460714273932
 best_eval 0.6355505599938764
591 train.py, conf.train.lr_schedule.on_epoch is: True
!!!!!sample_new_items: num_per_scene: 300
did a random shuffle!!!
/homes/tp4618/Documents/bitbucket/miniconda3/envs/redoGlue/lib/python3.12/site-packages/kornia/feature/lightglue.py:44: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
  @torch.cuda.amp.custom_fwd(cast_inputs=torch.float32)
/vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/gluefactory/models/matchers/loadedLightGlue.py:25: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
  @torch.cuda.amp.custom_fwd(cast_inputs=torch.float32)
[08/11/2024 19:33:59 gluefactory INFO] \Training intermediate DataFrame saved to: /homes/tp4618/Documents/bitbucket/SuperGlueThesis/external/glue-factory/outputs/training/11-08-fullDepth-newScenes/train_11-08-fullDepth-newScenes_intermediate_6_0_20240811_193359.pkl

[08/11/2024 19:33:59 gluefactory INFO] [E 6 | it 0] loss {total 1.621E+00, last 1.416E+00, assignment_nll 1.416E+00, nll_pos 2.372E+00, nll_neg 4.596E-01, num_matchable 8.642E+02, num_unmatchable 5.778E+02, confidence 2.221E-01, row_norm 6.924E-01}
neg 4.509E-01, loss/num_matchable 1.002E+03, loss/num_unmatchable 4.911E+02, loss/row_norm 6.986E-01}
[08/11/2024 19:33:34 gluefactory.utils.experiments INFO] Saving checkpoint checkpoint_5_5828.tar
[08/11/2024 19:33:35 gluefactory INFO] Starting epoch 6
[08/11/2024 19:33:35 gluefactory INFO] lr changed from 1e-05 to 1e-05
[08/11/2024 19:33:35 gluefactory.datasets.treedepth INFO] Sampling new items for train with seed 48.
Data logged to validation: {'epoch': 5, 'iteration': 242, 'match_recall': 0.6510937604204569, 'match_precision': 0.9338410576338013, 'accuracy': 0.7604223577212182, 'average_precision': 0.610695491150079, 'loss/total': 1.105767483508725, 'loss/last': 1.105767483508725, 'loss/assignment_nll': 1.105767483508725, 'loss/nll_pos': 1.7605925672302836, 'loss/nll_neg': 0.4509423943560096, 'loss/num_matchable': 1001.5061776061776, 'loss/num_unmatchable': 491.1034749034749, 'loss/row_norm': 0.6985640384055473}
comparing to see if it hte best experiment 875 train.py
key used was loss/total
 FROM RESULTS [conf.train.best_key] 1.105767483508725
 best_eval 0.6355035161419725
591 train.py, conf.train.lr_schedule.on_epoch is: True
!!!!!sample_new_items: num_per_scene: 300
did a random shuffle!!!
/homes/tp4618/Documents/bitbucket/miniconda3/envs/redoGlue/lib/python3.12/site-packages/kornia/feature/lightglue.py:44: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
  @torch.cuda.amp.custom_fwd(cast_inputs=torch.float32)
/vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/gluefactory/models/matchers/loadedLightGlue.py:25: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
  @torch.cuda.amp.custom_fwd(cast_inputs=torch.float32)
[08/11/2024 19:33:59 gluefactory INFO] \Training intermediate DataFrame saved to: /homes/tp4618/Documents/bitbucket/SuperGlueThesis/external/glue-factory/outputs/training/11-08-fullDepth-newScenes/train_11-08-fullDepth-newScenes_intermediate_6_0_20240811_193359.pkl

[08/11/2024 19:33:59 gluefactory INFO] [E 6 | it 0] loss {total 3.993E-01, last 3.444E-01, assignment_nll 3.444E-01, nll_pos 5.772E-01, nll_neg 1.117E-01, num_matchable 2.191E+02, num_unmatchable 1.418E+02, confidence 5.769E-02, row_norm 1.720E-01}
[08/11/2024 19:45:55 gluefactory INFO] \Training intermediate DataFrame saved to: /homes/tp4618/Documents/bitbucket/SuperGlueThesis/external/glue-factory/outputs/training/11-08-fullDepth-newScenes/train_11-08-fullDepth-newScenes_intermediate_6_100_20240811_194555.pkl

[08/11/2024 19:45:55 gluefactory INFO] [E 6 | it 100] loss {total 3.467E-01, last 2.991E-01, assignment_nll 2.991E-01, nll_pos 4.606E-01, nll_neg 1.376E-01, num_matchable 2.159E+02, num_unmatchable 1.470E+02, confidence 4.977E-02, row_norm 1.741E-01}
[08/11/2024 19:57:51 gluefactory INFO] \Training intermediate DataFrame saved to: /homes/tp4618/Documents/bitbucket/SuperGlueThesis/external/glue-factory/outputs/training/11-08-fullDepth-newScenes/train_11-08-fullDepth-newScenes_intermediate_6_200_20240811_195751.pkl

[08/11/2024 19:57:51 gluefactory INFO] [E 6 | it 200] loss {total 3.427E-01, last 2.968E-01, assignment_nll 2.968E-01, nll_pos 4.097E-01, nll_neg 1.840E-01, num_matchable 2.178E+02, num_unmatchable 1.442E+02, confidence 4.978E-02, row_norm 1.622E-01}
Current iteraiton: 0
Data logged to training DataFrame: {'epoch': 6, 'iteration': 0, 'total': 0.3454236686229706, 'last': 0.29113370180130005, 'assignment_nll': 0.29113370180130005, 'nll_pos': 0.46782687306404114, 'nll_neg': 0.11444051563739777, 'num_matchable': 240.15625, 'num_unmatchable': 126.328125, 'confidence': 0.057585667818784714, 'row_norm': 0.1760004460811615}
Current iteraiton: 100
Data logged to training DataFrame: {'epoch': 6, 'iteration': 100, 'total': 0.36252889037132263, 'last': 0.3100934624671936, 'assignment_nll': 0.3100934624671936, 'nll_pos': 0.4768229126930237, 'nll_neg': 0.14336401224136353, 'num_matchable': 232.6875, 'num_unmatchable': 131.21875, 'confidence': 0.05105161666870117, 'row_norm': 0.16545309126377106}
Current iteraiton: 200
Data logged to training DataFrame: {'epoch': 6, 'iteration': 200, 'total': 0.3929760456085205, 'last': 0.3525015711784363, 'assignment_nll': 0.3525015711784363, 'nll_pos': 0.5220605134963989, 'nll_neg': 0.18294262886047363, 'num_matchable': 219.34375, 'num_unmatchable': 149.953125, 'confidence': 0.04904123395681381, 'row_norm': 0.1610935628414154}
line 813in train.py doing valid/ho/homes/tp4618/Documents/bitbucket/miniconda3/envs/redoGlue/lib/python3.12/site-packages/kornia/feature/lightglue.py:44: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
  @torch.cuda.amp.custom_fwd(cast_inputs=torch.float32)
2)
/vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/gluefactory/models/matchers/loadedLightGlue.py:25: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
  @torch.cuda.amp.custom_fwd(cast_inputs=torch.float32)
[08/11/2024 20:08:30 gluefactory INFO] VALIDATION Intermediate DataFrame saved to: /homes/tp4618/Documents/bitbucket/SuperGlueThesis/external/glue-factory/outputs/training/11-08-fullDepth-newScenes/validation_11-08-fullDepth-newScenes_20240811_200823.pkl

[08/11/2024 20:08:23 gluefactory INFO] [Validation] {match_recall 7.129E-01, match_precision 9.207E-01, accuracy 7.972E-01, average_precision 6.599E-01, loss/total 1.041E+00, loss/last 1.041E+00, loss/assignment_nll 1.041E+00, loss/nll_pos 1.517E+00, loss/nll_neg 5.639E-01, loss/num_matchable 1.001E+03, loss/num_unmatchable 4.911E+02, loss/row_norm 6.929E-01}
[08/11/2024 20:08:27 gluefactory.utils.experiments INFO] Saving checkpoint checkpoint_6_6800.tar
[08/11/2024 20:08:28 gluefactory INFO] Starting epoch 7
[08/11/2024 20:08:28 gluefactory INFO] lr changed from 1e-05 to 1e-05
[08/11/2024 20:08:28 gluefactory.datasets.treedepth INFO] Sampling new items for train with seed 49.
Data logged to validation: {'epoch': 6, 'iteration': 242, 'match_recall': 0.7129163267069342, 'match_precision': 0.9206568920474255, 'accuracy': 0.7972284009557893, 'average_precision': 0.6599354795507483, 'loss/total': 1.0405073705327097, 'loss/last': 1.0405073705327097, 'loss/assignment_nll': 1.0405073705327097, 'loss/nll_pos': 1.5170909868704305, 'loss/nll_neg': 0.5639237616513226, 'loss/num_matchable': 1001.488416988417, 'loss/num_unmatchable': 491.1061776061776, 'loss/row_norm': 0.6929279141444498}
comparing to see if it hte best experiment 875 train.py
key used was loss/total
 FROM RESULTS [conf.train.best_key] 1.0405073705327097
 best_eval 0.6355505599938764
591 train.py, conf.train.lr_schedule.on_epoch is: True
!!!!!sample_new_items: num_per_scene: 300
did a random shuffle!!!
/homes/tp4618/Documents/bitbucket/miniconda3/envs/redoGlue/lib/python3.12/site-packages/kornia/feature/lightglue.py:44: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
  @torch.cuda.amp.custom_fwd(cast_inputs=torch.float32)
/vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/gluefactory/models/matchers/loadedLightGlue.py:25: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
  @torch.cuda.amp.custom_fwd(cast_inputs=torch.float32)
[08/11/2024 20:11:07 gluefactory INFO] \Training intermediate DataFrame saved to: /homes/tp4618/Documents/bitbucket/SuperGlueThesis/external/glue-factory/outputs/training/11-08-fullDepth-newScenes/train_11-08-fullDepth-newScenes_intermediate_7_0_20240811_201107.pkl

[08/11/2024 20:11:07 gluefactory INFO] [E 7 | it 0] loss {total 2.594E-01, last 2.123E-01, assignment_nll 2.123E-01, nll_pos 2.773E-01, nll_neg 1.472E-01, num_matchable 2.588E+02, num_unmatchable 1.256E+02, confidence 4.992E-02, row_norm 1.745E-01}
neg 5.638E-01, loss/num_matchable 1.002E+03, loss/num_unmatchable 4.911E+02, loss/row_norm 6.929E-01}
[08/11/2024 20:10:43 gluefactory.utils.experiments INFO] Saving checkpoint checkpoint_6_6800.tar
[08/11/2024 20:10:45 gluefactory INFO] Starting epoch 7
[08/11/2024 20:10:45 gluefactory INFO] lr changed from 1e-05 to 1e-05
[08/11/2024 20:10:45 gluefactory.datasets.treedepth INFO] Sampling new items for train with seed 49.
Data logged to validation: {'epoch': 6, 'iteration': 242, 'match_recall': 0.7128930530032596, 'match_precision': 0.9206130390461807, 'accuracy': 0.7972154994728943, 'average_precision': 0.6598931069539781, 'loss/total': 1.0405432574077003, 'loss/last': 1.0405432574077003, 'loss/assignment_nll': 1.0405432574077003, 'loss/nll_pos': 1.51732216599365, 'loss/nll_neg': 0.563764341089256, 'loss/num_matchable': 1001.5061776061776, 'loss/num_unmatchable': 491.1034749034749, 'loss/row_norm': 0.6929195358025981}
comparing to see if it hte best experiment 875 train.py
key used was loss/total
 FROM RESULTS [conf.train.best_key] 1.0405432574077003
 best_eval 0.6355035161419725
591 train.py, conf.train.lr_schedule.on_epoch is: True
!!!!!sample_new_items: num_per_scene: 300
did a random shuffle!!!
/homes/tp4618/Documents/bitbucket/miniconda3/envs/redoGlue/lib/python3.12/site-packages/kornia/feature/lightglue.py:44: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
  @torch.cuda.amp.custom_fwd(cast_inputs=torch.float32)
/vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/gluefactory/models/matchers/loadedLightGlue.py:25: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
  @torch.cuda.amp.custom_fwd(cast_inputs=torch.float32)
[08/11/2024 20:11:07 gluefactory INFO] \Training intermediate DataFrame saved to: /homes/tp4618/Documents/bitbucket/SuperGlueThesis/external/glue-factory/outputs/training/11-08-fullDepth-newScenes/train_11-08-fullDepth-newScenes_intermediate_7_0_20240811_201107.pkl

[08/11/2024 20:11:07 gluefactory INFO] [E 7 | it 0] loss {total 3.360E-01, last 2.901E-01, assignment_nll 2.901E-01, nll_pos 4.321E-01, nll_neg 1.481E-01, num_matchable 2.297E+02, num_unmatchable 1.402E+02, confidence 4.728E-02, row_norm 1.707E-01}
[08/11/2024 20:23:03 gluefactory INFO] \Training intermediate DataFrame saved to: /homes/tp4618/Documents/bitbucket/SuperGlueThesis/external/glue-factory/outputs/training/11-08-fullDepth-newScenes/train_11-08-fullDepth-newScenes_intermediate_7_100_20240811_202303.pkl

[08/11/2024 20:23:03 gluefactory INFO] [E 7 | it 100] loss {total 3.487E-01, last 3.014E-01, assignment_nll 3.014E-01, nll_pos 4.667E-01, nll_neg 1.360E-01, num_matchable 2.530E+02, num_unmatchable 1.181E+02, confidence 4.611E-02, row_norm 1.704E-01}
[08/11/2024 20:35:43 gluefactory INFO] \Training intermediate DataFrame saved to: /homes/tp4618/Documents/bitbucket/SuperGlueThesis/external/glue-factory/outputs/training/11-08-fullDepth-newScenes/train_11-08-fullDepth-newScenes_intermediate_7_200_20240811_203543.pkl

[08/11/2024 20:35:43 gluefactory INFO] [E 7 | it 200] loss {total 3.037E-01, last 2.588E-01, assignment_nll 2.588E-01, nll_pos 3.963E-01, nll_neg 1.214E-01, num_matchable 2.184E+02, num_unmatchable 1.427E+02, confidence 5.030E-02, row_norm 1.782E-01}
Current iteraiton: 0
Data logged to training DataFrame: {'epoch': 7, 'iteration': 0, 'total': 0.33595389127731323, 'last': 0.2901207208633423, 'assignment_nll': 0.2901207208633423, 'nll_pos': 0.4321327209472656, 'nll_neg': 0.14810875058174133, 'num_matchable': 229.65625, 'num_unmatchable': 140.25, 'confidence': 0.04728458821773529, 'row_norm': 0.17068979144096375}
Current iteraiton: 100
Data logged to training DataFrame: {'epoch': 7, 'iteration': 100, 'total': 0.30874747037887573, 'last': 0.25860071182250977, 'assignment_nll': 0.25860071182250977, 'nll_pos': 0.36812955141067505, 'nll_neg': 0.14907191693782806, 'num_matchable': 239.8125, 'num_unmatchable': 125.8125, 'confidence': 0.05068158730864525, 'row_norm': 0.1694457232952118}
Current iteraiton: 200
Data logged to training DataFrame: {'epoch': 7, 'iteration': 200, 'total': 0.4333861470222473, 'last': 0.39086979627609253, 'assignment_nll': 0.39086979627609253, 'nll_pos': 0.6341497898101807, 'nll_neg': 0.14758983254432678, 'num_matchable': 179.6875, 'num_unmatchable': 161.046875, 'confidence': 0.044713765382766724, 'row_norm': 0.16544118523597717}
line 813in train.py doing validation
/homes/tp4618/Documents/bitbucket/miniconda3/envs/redoGlue/lib/python3.12/site-packages/kornia/feature/lightglue.py:44: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
  @torch.cuda.amp.custom_fwd(cast_inputs=torch.float32)
/vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/gluefactory/models/matchers/loadedLightGlue.py:25: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
  @torch.cuda.amp.custom_fwd(cast_inputs=torch.float32)
[08/11/2024 20:46:21 gluefactory INFO] VALIDATION Intermediate DataFrame saved to: /homes/tp4618/Documents/bitbucket/SuperGlueThesis/external/glue-factory/outputs/training/11-08-fullDepth-newScenes/validation_11-08-fullDepth-newScenes_20240811_204621.pkl

[08/11/2024 20:46:21 gluefactory INFO] [Validation] {match_recall 7.271E-01, match_precision 9.160E-01, accuracy 8.044E-01, average_precision 6.698E-01, loss/total 1.046E+00, loss/last 1.046E+00, loss/assignment_nll 1.046E+00, loss/nll_pos 1.433E+00, loss/nll_neg 6.596E-01, loss/num_matchable 1.001E+03, loss/num_unmatchable 4.911E+02, loss/row_norm 6.724E-01}
[08/11/2024 20:46:25 gluefactory.utils.experiments INFO] Saving checkpoint checkpoint_7_7772.tar
[08/11/2024 20:46:26 gluefactory INFO] Starting epoch 8
[08/11/2024 20:46:26 gluefactory INFO] lr changed from 1e-05 to 1e-05
[08/11/2024 20:46:26 gluefactory.datasets.treedepth INFO] Sampling new items for train with seed 50.
Data logged to validation: {'epoch': 7, 'iteration': 242, 'match_recall': 0.7270683916379126, 'match_precision': 0.9160386641513427, 'accuracy': 0.8044230238351122, 'average_precision': 0.6697980136944981, 'loss/total': 1.0463088186551246, 'loss/last': 1.0463088186551246, 'loss/assignment_nll': 1.0463088186551246, 'loss/nll_pos': 1.4329792129487144, 'loss/nll_neg': 0.6596384252820696, 'loss/num_matchable': 1001.488416988417, 'loss/num_unmatchable': 491.1061776061776, 'loss/row_norm': 0.6723851507695024}
comparing to see if it hte best experiment 875 train.py
key used was loss/total
 FROM RESULTS [conf.train.best_key] 1.0463088186551246
 best_eval 0.6355505599938764
591 train.py, conf.train.lr_schedule.on_epoch is: True
!!!!!sample_new_items: num_per_scene: 300
did a random shuffle!!!
/homes/tp4618/Documents/bitbucket/miniconda3/envs/redoGlue/lib/python3.12/site-packages/kornia/feature/lightglue.py:44: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
  @torch.cuda.amp.custom_fwd(cast_inputs=torch.float32)
/vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/gluefactory/models/matchers/loadedLightGlue.py:25: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
  @torch.cuda.amp.custom_fwd(cast_inputs=torch.float32)
[08/11/2024 20:48:49 gluefactory INFO] \Training intermediate DataFrame saved to: /homes/tp4618/Documents/bitbucket/SuperGlueThesis/external/glue-factory/outputs/training/11-08-fullDepth-newScenes/train_11-08-fullDepth-newScenes_intermediate_8_0_20240811_204849.pkl

[08/11/2024 20:48:49 gluefactory INFO] [E 8 | it 0] loss {total 2.981E-01, last 2.561E-01, assignment_nll 2.561E-01, nll_pos 3.477E-01, nll_neg 1.646E-01, num_matchable 2.357E+02, num_unmatchable 1.390E+02, confidence 4.858E-02, row_norm 1.668E-01}
neg 6.594E-01, loss/num_matchable 1.002E+03, loss/num_unmatchable 4.911E+02, loss/row_norm 6.724E-01}
[08/11/2024 20:48:28 gluefactory.utils.experiments INFO] Saving checkpoint checkpoint_7_7772.tar
[08/11/2024 20:48:29 gluefactory INFO] Starting epoch 8
[08/11/2024 20:48:29 gluefactory INFO] lr changed from 1e-05 to 1e-05
[08/11/2024 20:48:29 gluefactory.datasets.treedepth INFO] Sampling new items for train with seed 50.
Data logged to validation: {'epoch': 7, 'iteration': 242, 'match_recall': 0.7269626306290792, 'match_precision': 0.9161284536928744, 'accuracy': 0.8043589186944556, 'average_precision': 0.6697658052775851, 'loss/total': 1.046272745647946, 'loss/last': 1.046272745647946, 'loss/assignment_nll': 1.046272745647946, 'loss/nll_pos': 1.4331409314424375, 'loss/nll_neg': 0.659404553225602, 'loss/num_matchable': 1001.5061776061776, 'loss/num_unmatchable': 491.1034749034749, 'loss/row_norm': 0.6724159244404796}
comparing to see if it hte best experiment 875 train.py
key used was loss/total
 FROM RESULTS [conf.train.best_key] 1.046272745647946
 best_eval 0.6355035161419725
591 train.py, conf.train.lr_schedule.on_epoch is: True
!!!!!sample_new_items: num_per_scene: 300
did a random shuffle!!!
/homes/tp4618/Documents/bitbucket/miniconda3/envs/redoGlue/lib/python3.12/site-packages/kornia/feature/lightglue.py:44: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
  @torch.cuda.amp.custom_fwd(cast_inputs=torch.float32)
/vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/gluefactory/models/matchers/loadedLightGlue.py:25: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
  @torch.cuda.amp.custom_fwd(cast_inputs=torch.float32)
[08/11/2024 20:48:49 gluefactory INFO] \Training intermediate DataFrame saved to: /homes/tp4618/Documents/bitbucket/SuperGlueThesis/external/glue-factory/outputs/training/11-08-fullDepth-newScenes/train_11-08-fullDepth-newScenes_intermediate_8_0_20240811_204849.pkl

[08/11/2024 20:48:49 gluefactory INFO] [E 8 | it 0] loss {total 3.504E-01, last 3.094E-01, assignment_nll 3.094E-01, nll_pos 4.478E-01, nll_neg 1.710E-01, num_matchable 2.222E+02, num_unmatchable 1.409E+02, confidence 4.601E-02, row_norm 1.593E-01}
[08/11/2024 21:00:45 gluefactory INFO] \Training intermediate DataFrame saved to: /homes/tp4618/Documents/bitbucket/SuperGlueThesis/external/glue-factory/outputs/training/11-08-fullDepth-newScenes/train_11-08-fullDepth-newScenes_intermediate_8_100_20240811_210045.pkl

[08/11/2024 21:00:45 gluefactory INFO] [E 8 | it 100] loss {total 3.246E-01, last 2.719E-01, assignment_nll 2.719E-01, nll_pos 4.287E-01, nll_neg 1.152E-01, num_matchable 2.326E+02, num_unmatchable 1.365E+02, confidence 5.572E-02, row_norm 1.759E-01}
[08/11/2024 21:12:41 gluefactory INFO] \Training intermediate DataFrame saved to: /homes/tp4618/Documents/bitbucket/SuperGlueThesis/external/glue-factory/outputs/training/11-08-fullDepth-newScenes/train_11-08-fullDepth-newScenes_intermediate_8_200_20240811_211241.pkl

[08/11/2024 21:12:41 gluefactory INFO] [E 8 | it 200] loss {total 3.629E-01, last 3.216E-01, assignment_nll 3.216E-01, nll_pos 4.993E-01, nll_neg 1.440E-01, num_matchable 2.065E+02, num_unmatchable 1.496E+02, confidence 4.679E-02, row_norm 1.697E-01}
Current iteraiton: 0
Data logged to training DataFrame: {'epoch': 8, 'iteration': 0, 'total': 0.35037779808044434, 'last': 0.30942583084106445, 'assignment_nll': 0.30942583084106445, 'nll_pos': 0.4478268623352051, 'nll_neg': 0.17102482914924622, 'num_matchable': 222.21875, 'num_unmatchable': 140.875, 'confidence': 0.046005457639694214, 'row_norm': 0.15928736329078674}
Current iteraiton: 100
Data logged to training DataFrame: {'epoch': 8, 'iteration': 100, 'total': 0.36004096269607544, 'last': 0.31373608112335205, 'assignment_nll': 0.31373608112335205, 'nll_pos': 0.49611833691596985, 'nll_neg': 0.13135379552841187, 'num_matchable': 234.125, 'num_unmatchable': 130.453125, 'confidence': 0.05114525556564331, 'row_norm': 0.16639277338981628}
Current iteraiton: 200
Data logged to training DataFrame: {'epoch': 8, 'iteration': 200, 'total': 0.31268179416656494, 'last': 0.27465128898620605, 'assignment_nll': 0.27465128898620605, 'nll_pos': 0.4105897545814514, 'nll_neg': 0.1387127935886383, 'num_matchable': 236.46875, 'num_unmatchable': 132.5625, 'confidence': 0.043664149940013885, 'row_norm': 0.16812878847122192}
line 813in train.py doing validation
/homes/tp4618/Documents/bitbucket/miniconda3/envs/redoGlue/lib/python3.12/site-packages/kornia/feature/lightglue.py:44: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
  @torch.cuda.amp.custom_fwd(cast_inputs=torch.float32)
//vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/gluefactory/models/matchers/loadedLightGlue.py:25: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
  @torch.cuda.amp.custom_fwd(cast_inputs=torch.float32)
[08/11/2024 21:23:15 gluefactory INFO] VALIDATION Intermediate DataFrame saved to: /homes/tp4618/Documents/bitbucket/SuperGlueThesis/external/glue-factory/outputs/training/11-08-fullDepth-newScenes/validation_11-08-fullDepth-newScenes_20240811_212315.pkl

[08/11/2024 21:23:15 gluefactory INFO] [Validation] {match_recall 7.019E-01, match_precision 9.263E-01, accuracy 7.917E-01, average_precision 6.534E-01, loss/total 1.024E+00, loss/last 1.024E+00, loss/assignment_nll 1.024E+00, loss/nll_pos 1.552E+00, loss/nll_neg 4.960E-01, loss/num_matchable 1.001E+03, loss/num_unmatchable 4.911E+02, loss/row_norm 6.992E-01}
[08/11/2024 21:23:16 gluefactory.utils.experiments INFO] Saving checkpoint checkpoint_8_8744.tar
[08/11/2024 21:23:18 gluefactory INFO] Starting epoch 9
[08/11/2024 21:23:18 gluefactory INFO] lr changed from 1e-05 to 1e-05
[08/11/2024 21:23:18 gluefactory.datasets.treedepth INFO] Sampling new items for train with seed 51.
Data logged to validation: {'epoch': 8, 'iteration': 242, 'match_recall': 0.7019133567810059, 'match_precision': 0.9263259149426198, 'accuracy': 0.791730430227449, 'average_precision': 0.6534333242872967, 'loss/total': 1.0238706588745117, 'loss/last': 1.0238706588745117, 'loss/assignment_nll': 1.0238706588745117, 'loss/nll_pos': 1.551744339364836, 'loss/nll_neg': 0.4959969865769493, 'loss/num_matchable': 1001.488416988417, 'loss/num_unmatchable': 491.1061776061776, 'loss/row_norm': 0.6991939329272532}
comparing to see if it hte best experiment 875 train.py
key used was loss/total
 FROM RESULTS [conf.train.best_key] 1.0238706588745117
 best_eval 0.6355505599938764
591 train.py, conf.train.lr_schedule.on_epoch is: True
!!!!!sample_new_items: num_per_scene: 300
did a random shuffle!!!
/homes/tp4618/Documents/bitbucket/miniconda3/envs/redoGlue/lib/python3.12/site-packages/kornia/feature/lightglue.py:44: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
  @torch.cuda.amp.custom_fwd(cast_inputs=torch.float32)
/vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/gluefactory/models/matchers/loadedLightGlue.py:25: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
  @torch.cuda.amp.custom_fwd(cast_inputs=torch.float32)
[08/11/2024 21:25:54 gluefactory INFO] \Training intermediate DataFrame saved to: /homes/tp4618/Documents/bitbucket/SuperGlueThesis/external/glue-factory/outputs/training/11-08-fullDepth-newScenes/train_11-08-fullDepth-newScenes_intermediate_9_0_20240811_212554.pkl

[08/11/2024 21:25:54 gluefactory INFO] [E 9 | it 0] loss {total 2.878E-01, last 2.449E-01, assignment_nll 2.449E-01, nll_pos 3.678E-01, nll_neg 1.219E-01, num_matchable 2.409E+02, num_unmatchable 1.362E+02, confidence 4.943E-02, row_norm 1.746E-01}
neg 4.959E-01, loss/num_matchable 1.002E+03, loss/num_unmatchable 4.911E+02, loss/row_norm 6.992E-01}
[08/11/2024 21:25:31 gluefactory.utils.experiments INFO] Saving checkpoint checkpoint_8_8744.tar
[08/11/2024 21:25:33 gluefactory INFO] Starting epoch 9
[08/11/2024 21:25:33 gluefactory INFO] lr changed from 1e-05 to 1e-05
[08/11/2024 21:25:33 gluefactory.datasets.treedepth INFO] Sampling new items for train with seed 51.
Data logged to validation: {'epoch': 8, 'iteration': 242, 'match_recall': 0.7019278418603551, 'match_precision': 0.9264152742260672, 'accuracy': 0.79174147002025, 'average_precision': 0.6534988836892324, 'loss/total': 1.0238853762048552, 'loss/last': 1.0238853762048552, 'loss/assignment_nll': 1.0238853762048552, 'loss/nll_pos': 1.5518525558088736, 'loss/nll_neg': 0.4959181969690507, 'loss/num_matchable': 1001.5061776061776, 'loss/num_unmatchable': 491.1034749034749, 'loss/row_norm': 0.6991911840254735}
comparing to see if it hte best experiment 875 train.py
key used was loss/total
 FROM RESULTS [conf.train.best_key] 1.0238853762048552
 best_eval 0.6355035161419725
591 train.py, conf.train.lr_schedule.on_epoch is: True
!!!!!sample_new_items: num_per_scene: 300
did a random shuffle!!!
/homes/tp4618/Documents/bitbucket/miniconda3/envs/redoGlue/lib/python3.12/site-packages/kornia/feature/lightglue.py:44: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
  @torch.cuda.amp.custom_fwd(cast_inputs=torch.float32)
/vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/gluefactory/models/matchers/loadedLightGlue.py:25: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
  @torch.cuda.amp.custom_fwd(cast_inputs=torch.float32)
[08/11/2024 21:25:54 gluefactory INFO] \Training intermediate DataFrame saved to: /homes/tp4618/Documents/bitbucket/SuperGlueThesis/external/glue-factory/outputs/training/11-08-fullDepth-newScenes/train_11-08-fullDepth-newScenes_intermediate_9_0_20240811_212554.pkl

[08/11/2024 21:25:54 gluefactory INFO] [E 9 | it 0] loss {total 3.492E-01, last 3.099E-01, assignment_nll 3.099E-01, nll_pos 4.811E-01, nll_neg 1.388E-01, num_matchable 2.105E+02, num_unmatchable 1.410E+02, confidence 4.591E-02, row_norm 1.686E-01}
[08/11/2024 21:37:50 gluefactory INFO] \Training intermediate DataFrame saved to: /homes/tp4618/Documents/bitbucket/SuperGlueThesis/external/glue-factory/outputs/training/11-08-fullDepth-newScenes/train_11-08-fullDepth-newScenes_intermediate_9_100_20240811_213750.pkl

[08/11/2024 21:37:50 gluefactory INFO] [E 9 | it 100] loss {total 3.160E-01, last 2.740E-01, assignment_nll 2.740E-01, nll_pos 4.209E-01, nll_neg 1.272E-01, num_matchable 2.257E+02, num_unmatchable 1.453E+02, confidence 4.738E-02, row_norm 1.691E-01}
[08/11/2024 21:49:45 gluefactory INFO] \Training intermediate DataFrame saved to: /homes/tp4618/Documents/bitbucket/SuperGlueThesis/external/glue-factory/outputs/training/11-08-fullDepth-newScenes/train_11-08-fullDepth-newScenes_intermediate_9_200_20240811_214945.pkl

[08/11/2024 21:49:45 gluefactory INFO] [E 9 | it 200] loss {total 2.541E-01, last 2.100E-01, assignment_nll 2.100E-01, nll_pos 3.146E-01, nll_neg 1.055E-01, num_matchable 2.772E+02, num_unmatchable 1.129E+02, confidence 4.980E-02, row_norm 1.790E-01}
Current iteraiton: 0
Data logged to training DataFrame: {'epoch': 9, 'iteration': 0, 'total': 0.2878127694129944, 'last': 0.2448509931564331, 'assignment_nll': 0.2448509931564331, 'nll_pos': 0.3678348958492279, 'nll_neg': 0.12186707556247711, 'num_matchable': 240.9375, 'num_unmatchable': 136.234375, 'confidence': 0.04942839592695236, 'row_norm': 0.1746159940958023}
Current iteraiton: 100
Data logged to training DataFrame: {'epoch': 9, 'iteration': 100, 'total': 0.4341144561767578, 'last': 0.38587290048599243, 'assignment_nll': 0.38587290048599243, 'nll_pos': 0.6140919923782349, 'nll_neg': 0.15765380859375, 'num_matchable': 191.0, 'num_unmatchable': 142.453125, 'confidence': 0.050241969525814056, 'row_norm': 0.1637118011713028}
Current iteraiton: 200
Data logged to training DataFrame: {'epoch': 9, 'iteration': 200, 'total': 0.3494813144207001, 'last': 0.3012267053127289, 'assignment_nll': 0.3012267053127289, 'nll_pos': 0.49265098571777344, 'nll_neg': 0.10980237275362015, 'num_matchable': 228.3125, 'num_unmatchable': 136.515625, 'confidence': 0.052088093012571335, 'row_norm': 0.1757831871509552}
line 813in train.py doing validation
/homes/tp4618/Documents/bitbucket/miniconda3/envs/redoGlue/lib/python3.12/site-packages/kornia/feature/lightglue.py:44: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
  @torch.cuda.amp.custom_fwd(cast_inputs=torch.float32)
/vol/bi/vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/gluefactory/models/matchers/loadedLightGlue.py:25: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
  @torch.cuda.amp.custom_fwd(cast_inputs=torch.float32)
[08/11/2024 22:00:29 gluefactory INFO] VALIDATION Intermediate DataFrame saved to: /homes/tp4618/Documents/bitbucket/SuperGlueThesis/external/glue-factory/outputs/training/11-08-fullDepth-newScenes/validation_11-08-fullDepth-newScenes_20240811_220029.pkl

[08/11/2024 22:00:29 gluefactory INFO] [Validation] {match_recall 7.007E-01, match_precision 9.283E-01, accuracy 7.918E-01, average_precision 6.537E-01, loss/total 1.012E+00, loss/last 1.012E+00, loss/assignment_nll 1.012E+00, loss/nll_pos 1.564E+00, loss/nll_neg 4.611E-01, loss/num_matchable 1.001E+03, loss/num_unmatchable 4.911E+02, loss/row_norm 7.080E-01}
[08/11/2024 22:00:32 gluefactory.utils.experiments INFO] Saving checkpoint checkpoint_9_9716.tar
[08/11/2024 22:00:34 gluefactory INFO] Starting epoch 10
[08/11/2024 22:00:34 gluefactory INFO] lr changed from 1e-05 to 9.501185073181438e-06
[08/11/2024 22:00:34 gluefactory.datasets.treedepth INFO] Sampling new items for train with seed 52.
Data logged to validation: {'epoch': 9, 'iteration': 242, 'match_recall': 0.700713265908731, 'match_precision': 0.9283032295786736, 'accuracy': 0.7917613624145625, 'average_precision': 0.6536555254321301, 'loss/total': 1.0124023212889446, 'loss/last': 1.0124023212889446, 'loss/assignment_nll': 1.0124023212889446, 'loss/nll_pos': 1.563705149028292, 'loss/nll_neg': 0.46109949548272094, 'loss/num_matchable': 1001.488416988417, 'loss/num_unmatchable': 491.1061776061776, 'loss/row_norm': 0.7080323421817029}
comparing to see if it hte best experiment 875 train.py
key used was loss/total
 FROM RESULTS [conf.train.best_key] 1.0124023212889446
 best_eval 0.6355505599938764
591 train.py, conf.train.lr_schedule.on_epoch is: True
!!!!!sample_new_items: num_per_scene: 300
did a random shuffle!!!
/homes/tp4618/Documents/bitbucket/miniconda3/envs/redoGlue/lib/python3.12/site-packages/kornia/feature/lightglue.py:44: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
  @torch.cuda.amp.custom_fwd(cast_inputs=torch.float32)
/vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/gluefactory/models/matchers/loadedLightGlue.py:25: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
  @torch.cuda.amp.custom_fwd(cast_inputs=torch.float32)
[08/11/2024 22:02:55 gluefactory INFO] \Training intermediate DataFrame saved to: /homes/tp4618/Documents/bitbucket/SuperGlueThesis/external/glue-factory/outputs/training/11-08-fullDepth-newScenes/train_11-08-fullDepth-newScenes_intermediate_10_0_20240811_220255.pkl

[08/11/2024 22:02:55 gluefactory INFO] [E 10 | it 0] loss {total 1.349E+00, last 1.181E+00, assignment_nll 1.181E+00, nll_pos 1.868E+00, nll_neg 4.931E-01, num_matchable 8.846E+02, num_unmatchable 5.531E+02, confidence 1.978E-01, row_norm 6.936E-01}
g 4.611E-01, loss/num_matchable 1.002E+03, loss/num_unmatchable 4.911E+02, loss/row_norm 7.079E-01}
[08/11/2024 22:02:32 gluefactory.utils.experiments INFO] Saving checkpoint checkpoint_9_9716.tar
[08/11/2024 22:02:34 gluefactory INFO] Starting epoch 10
[08/11/2024 22:02:34 gluefactory INFO] lr changed from 1e-05 to 9.501185073181438e-06
[08/11/2024 22:02:34 gluefactory.datasets.treedepth INFO] Sampling new items for train with seed 52.
Data logged to validation: {'epoch': 9, 'iteration': 242, 'match_recall': 0.7007667502841434, 'match_precision': 0.928356460254625, 'accuracy': 0.7918024602543894, 'average_precision': 0.65372114245956, 'loss/total': 1.0123538799727745, 'loss/last': 1.0123538799727745, 'loss/assignment_nll': 1.0123538799727745, 'loss/nll_pos': 1.5636004720415388, 'loss/nll_neg': 0.46110728735168927, 'loss/num_matchable': 1001.5061776061776, 'loss/num_unmatchable': 491.1034749034749, 'loss/row_norm': 0.7079493695704633}
comparing to see if it hte best experiment 875 train.py
key used was loss/total
 FROM RESULTS [conf.train.best_key] 1.0123538799727745
 best_eval 0.6355035161419725
591 train.py, conf.train.lr_schedule.on_epoch is: True
!!!!!sample_new_items: num_per_scene: 300
did a random shuffle!!!
/homes/tp4618/Documents/bitbucket/miniconda3/envs/redoGlue/lib/python3.12/site-packages/kornia/feature/lightglue.py:44: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
  @torch.cuda.amp.custom_fwd(cast_inputs=torch.float32)
/vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/gluefactory/models/matchers/loadedLightGlue.py:25: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
  @torch.cuda.amp.custom_fwd(cast_inputs=torch.float32)
[08/11/2024 22:02:55 gluefactory INFO] \Training intermediate DataFrame saved to: /homes/tp4618/Documents/bitbucket/SuperGlueThesis/external/glue-factory/outputs/training/11-08-fullDepth-newScenes/train_11-08-fullDepth-newScenes_intermediate_10_0_20240811_220255.pkl

[08/11/2024 22:02:55 gluefactory INFO] [E 10 | it 0] loss {total 4.006E-01, last 3.609E-01, assignment_nll 3.609E-01, nll_pos 5.968E-01, nll_neg 1.250E-01, num_matchable 2.112E+02, num_unmatchable 1.374E+02, confidence 4.628E-02, row_norm 1.698E-01}
[08/11/2024 22:11:14 gluefactory.utils.experiments INFO] Saving checkpoint checkpoint_10_10000.tar
[08/11/2024 22:14:52 gluefactory INFO] \Training intermediate DataFrame saved to: /homes/tp4618/Documents/bitbucket/SuperGlueThesis/external/glue-factory/outputs/training/11-08-fullDepth-newScenes/train_11-08-fullDepth-newScenes_intermediate_10_100_20240811_221452.pkl

[08/11/2024 22:14:52 gluefactory INFO] [E 10 | it 100] loss {total 2.886E-01, last 2.510E-01, assignment_nll 2.510E-01, nll_pos 3.678E-01, nll_neg 1.342E-01, num_matchable 2.472E+02, num_unmatchable 1.319E+02, confidence 4.123E-02, row_norm 1.725E-01}
[08/11/2024 22:26:47 gluefactory INFO] \Training intermediate DataFrame saved to: /homes/tp4618/Documents/bitbucket/SuperGlueThesis/external/glue-factory/outputs/training/11-08-fullDepth-newScenes/train_11-08-fullDepth-newScenes_intermediate_10_200_20240811_222647.pkl

[08/11/2024 22:26:47 gluefactory INFO] [E 10 | it 200] loss {total 3.896E-01, last 3.463E-01, assignment_nll 3.463E-01, nll_pos 5.682E-01, nll_neg 1.245E-01, num_matchable 2.082E+02, num_unmatchable 1.465E+02, confidence 4.488E-02, row_norm 1.673E-01}
Current iteraiton: 0
Data logged to training DataFrame: {'epoch': 10, 'iteration': 0, 'total': 0.3665929436683655, 'last': 0.3307303488254547, 'assignment_nll': 0.3307303488254547, 'nll_pos': 0.536234974861145, 'nll_neg': 0.1252257227897644, 'num_matchable': 201.59375, 'num_unmatchable': 157.59375, 'confidence': 0.04594730585813522, 'row_norm': 0.1731368601322174}
Current iteraiton: 100
Data logged to training DataFrame: {'epoch': 10, 'iteration': 100, 'total': 0.3978500962257385, 'last': 0.357438862323761, 'assignment_nll': 0.357438862323761, 'nll_pos': 0.5504551529884338, 'nll_neg': 0.16442254185676575, 'num_matchable': 200.46875, 'num_unmatchable': 152.828125, 'confidence': 0.04303859919309616, 'row_norm': 0.16147387027740479}
Current iteraiton: 200
Data logged to training DataFrame: {'epoch': 10, 'iteration': 200, 'total': 0.38956570625305176, 'last': 0.34631282091140747, 'assignment_nll': 0.34631282091140747, 'nll_pos': 0.5681623220443726, 'nll_neg': 0.12446334958076477, 'num_matchable': 208.15625, 'num_unmatchable': 146.453125, 'confidence': 0.04487738385796547, 'row_norm': 0.1673300862312317}
line 813in train.py doing validation
on//homes/tp4618/Documents/bitbucket/miniconda3/envs/redoGlue/lib/python3.12/site-packages/kornia/feature/lightglue.py:44: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
  @torch.cuda.amp.custom_fwd(cast_inputs=torch.float32)/vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/gluefactory/models/matchers/loadedLightGlue.py:25: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
  @torch.cuda.amp.custom_fwd(cast_inputs=torch.float32)

[08/11/2024 22:37:26 gluefactory INFO] VALIDATION Intermediate DataFrame saved to: /homes/tp4618/Documents/bitbucket/SuperGlueThesis/external/glue-factory/outputs/training/11-08-fullDepth-newScenes/validation_11-08-fullDepth-newScenes_20240811_223726.pkl

[08/11/2024 22:37:26 gluefactory INFO] [Validation] {match_recall 7.314E-01, match_precision 9.180E-01, accuracy 8.081E-01, average_precision 6.752E-01, loss/total 1.007E+00, loss/last 1.007E+00, loss/assignment_nll 1.007E+00, loss/nll_pos 1.419E+00, loss/nll_neg 5.952E-01, loss/num_matchable 1.001E+03, loss/num_unmatchable 4.911E+02, loss/row_norm 6.890E-01}
[08/11/2024 22:37:29 gluefactory.utils.experiments INFO] Saving checkpoint checkpoint_10_10688.tar
[08/11/2024 22:37:31 gluefactory INFO] Starting epoch 11
[08/11/2024 22:37:31 gluefactory INFO] lr changed from 9.501185073181438e-06 to 9.027251779484575e-06
[08/11/2024 22:37:31 gluefactory.datasets.treedepth INFO] Sampling new items for train with seed 53.
Data logged to validation: {'epoch': 10, 'iteration': 242, 'match_recall': 0.731353084092895, 'match_precision': 0.9179923155132869, 'accuracy': 0.8080576432717813, 'average_precision': 0.6751842586230127, 'loss/total': 1.006938041591276, 'loss/last': 1.006938041591276, 'loss/assignment_nll': 1.006938041591276, 'loss/nll_pos': 1.4186670498498157, 'loss/nll_neg': 0.5952090402367493, 'loss/num_matchable': 1001.488416988417, 'loss/num_unmatchable': 491.1061776061776, 'loss/row_norm': 0.6890127682777906}
comparing to see if it hte best experiment 875 train.py
key used was loss/total
 FROM RESULTS [conf.train.best_key] 1.006938041591276
 best_eval 0.6355505599938764
591 train.py, conf.train.lr_schedule.on_epoch is: True
!!!!!sample_new_items: num_per_scene: 300
did a random shuffle!!!
/homes/tp4618/Documents/bitbucket/miniconda3/envs/redoGlue/lib/python3.12/site-packages/kornia/feature/lightglue.py:44: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
  @torch.cuda.amp.custom_fwd(cast_inputs=torch.float32)
/vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/gluefactory/models/matchers/loadedLightGlue.py:25: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
  @torch.cuda.amp.custom_fwd(cast_inputs=torch.float32)
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/homes/tp4618/Documents/bitbucket/miniconda3/envs/redoGlue/lib/python3.12/multiprocessing/spawn.py", line 122, in spawn_main
    exitcode = _main(fd, parent_sentinel)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/homes/tp4618/Documents/bitbucket/miniconda3/envs/redoGlue/lib/python3.12/multiprocessing/spawn.py", line 132, in _main
    self = reduction.pickle.load(from_parent)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/gluefactory/datasets/treedepth.py", line 1482
    scene1 = #
             ^^
SyntaxError: invalid syntax
yntax
[08/11/2024 22:39:34 gluefactory INFO] VALIDATION Intermediate DataFrame saved to: /homes/tp4618/Documents/bitbucket/SuperGlueThesis/external/glue-factory/outputs/training/11-08-fullDepth-newScenes/validation_11-08-fullDepth-newScenes_20240811_223934.pkl

[08/11/2024 22:39:34 gluefactory INFO] [Validation] {match_recall 7.313E-01, match_precision 9.181E-01, accuracy 8.080E-01, average_precision 6.752E-01, loss/total 1.007E+00, loss/last 1.007E+00, loss/assignment_nll 1.007E+00, loss/nll_pos 1.419E+00, loss/nll_neg 5.952E-01, loss/num_matchable 1.002E+03, loss/num_unmatchable 4.911E+02, loss/row_norm 6.890E-01}
[08/11/2024 22:39:37 gluefactory.utils.experiments INFO] Saving checkpoint checkpoint_10_10688.tar
[08/11/2024 22:39:39 gluefactory INFO] Starting epoch 11
[08/11/2024 22:39:39 gluefactory INFO] lr changed from 9.501185073181438e-06 to 9.027251779484575e-06
[08/11/2024 22:39:39 gluefactory.datasets.treedepth INFO] Sampling new items for train with seed 53.
Data logged to validation: {'epoch': 10, 'iteration': 242, 'match_recall': 0.7313177736569556, 'match_precision': 0.9180549146585943, 'accuracy': 0.8080205666972864, 'average_precision': 0.6751684824011961, 'loss/total': 1.0069626664563036, 'loss/last': 1.0069626664563036, 'loss/assignment_nll': 1.0069626664563036, 'loss/nll_pos': 1.4187418980027717, 'loss/nll_neg': 0.5951834249680567, 'loss/num_matchable': 1001.5061776061776, 'loss/num_unmatchable': 491.1034749034749, 'loss/row_norm': 0.6890338549742827}
comparing to see if it hte best experiment 875 train.py
key used was loss/total
 FROM RESULTS [conf.train.best_key] 1.0069626664563036
 best_eval 0.6355035161419725
591 train.py, conf.train.lr_schedule.on_epoch is: True
!!!!!sample_new_items: num_per_scene: 300
did a random shuffle!!!
/homes/tp4618/Documents/bitbucket/miniconda3/envs/redoGlue/lib/python3.12/site-packages/kornia/feature/lightglue.py:44: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
  @torch.cuda.amp.custom_fwd(cast_inputs=torch.float32)
/vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/gluefactory/models/matchers/loadedLightGlue.py:25: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
  @torch.cuda.amp.custom_fwd(cast_inputs=torch.float32)
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/homes/tp4618/Documents/bitbucket/miniconda3/envs/redoGlue/lib/python3.12/multiprocessing/spawn.py", line 122, in spawn_main
    exitcode = _main(fd, parent_sentinel)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/homes/tp4618/Documents/bitbucket/miniconda3/envs/redoGlue/lib/python3.12/multiprocessing/spawn.py", line 132, in _main
    self = reduction.pickle.load(from_parent)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/gluefactory/datasets/treedepth.py", line 1482
    scene1 = #
             ^^
SyntaxError: invalid syntax