Aborted error when attempting to use ctc-decoding

[GarlandZhang]

I receive an error trying to decode using ctc-decoding. I am using the master branch without code changes but using a model trained for several epochs.

command

gdb --args python ./conformer_ctc/decode.py --avg 1 --epoch 5 --method ctc-decoding --exp-dir exp_dir/ --lang-dir data/lang_bpe_5000 --max-duration 300

result

[...]

2021-11-02 10:06:55,671 INFO [decode.py:476] batch 0/?, cuts processed until now is 6
[F] /usr/share/miniconda/envs/k2/conda-bld/k2_1631619831677/work/k2/csrc/array.h:501:void k2::Array1<T>::Init(k2::ContextPtr, int32_t, k2::Dtype) [with T = char; k2::ContextPtr = std::shared_ptr<k2::Context>; int32_t = int] Check failed: size >= 0 (-1383015021 vs. 0) Array size MUST be greater than or equal to 0, given :-1383015021


[ Stack-Trace: ]
/anaconda3/envs/selfsl/lib/python3.8/site-packages/libk2_log.so(k2::internal::GetStackTrace()+0x4c) [0x7fff5b01021c]
/anaconda3/envs/selfsl/lib/python3.8/site-packages/libk2context.so(k2::internal::Logger::~Logger()+0x5a) [0x7fff5b33535a]
/anaconda3/envs/selfsl/lib/python3.8/site-packages/libk2context.so(k2::Array1<char>::Init(std::shared_ptr<k2::Context>, int, k2::Dtype)+0x1cd) [0x7fff5b3630dd]
/anaconda3/envs/selfsl/lib/python3.8/site-packages/libk2context.so(k2::Renumbering::Init(std::shared_ptr<k2::Context>, int, bool)+0xa7) [0x7fff5b3697a7]
/anaconda3/envs/selfsl/lib/python3.8/site-packages/libk2context.so(k2::Renumbering::Renumbering(std::shared_ptr<k2::Context>, int, bool)+0xbc) [0x7fff5b36ac5c]
/anaconda3/envs/selfsl/lib/python3.8/site-packages/libk2context.so(k2::MultiGraphDenseIntersectPruned::PruneTimeRange(int, int)+0x37c) [0x7fff5b4c945c]
/anaconda3/envs/selfsl/lib/python3.8/site-packages/libk2context.so(std::_Function_handler<void (), k2::MultiGraphDenseIntersectPruned::Intersect()::{lambda()#1}>::_M_invoke(std::_Any_data const&)+0x203) [0x7fff5b4cc763]
/anaconda3/envs/selfsl/lib/python3.8/site-packages/libk2context.so(k2::ThreadPool::ProcessTasks()+0x164) [0x7fff5b610b94]
/anaconda3/envs/selfsl/lib/python3.8/site-packages/torch/lib/../../../.././libstdc++.so.6(+0xc8421) [0x7fffd25b5421]
/lib64/libpthread.so.0(+0x7ea5) [0x7ffff7bc6ea5]
/lib64/libc.so.6(clone+0x6d) [0x7ffff78ef9fd]

terminate called after throwing an instance of 'std::runtime_error'
  what():
      Some bad things happened. Please read the above error messages and stack
      trace. If you are using Python, the following command may be helpful:

            gdb --args python /path/to/your/code.py

                (You can use `gdb` to debug the code. Please consider compiling
                    a debug version of k2.).

                        If you are unable to fix it, please open an issue at:

                              https://github.com/k2-fsa/k2/issues/new


                              Program received signal SIGABRT, Aborted.
                              [Switching to Thread 0x7fff3f7ee700 (LWP 24121)]
                              0x00007ffff7827387 in raise () from /lib64/libc.so.6
                              Missing separate debuginfos, use: debuginfo-install glibc-2.17-324.el7_9.x86_64

As part of debugging (this is sort of separate issue) but I tried using pretrained model for decode script to ensure its not model related but I received the error

_pickle.UnpicklingError: invalid load key, 'v'

when running the line:

checkpoint = torch.load(filename, map_location="cpu")

[csukuangfj]

I receive an error trying to decode using ctc-decoding

ctc-decoding is tested with a model using vocab size 500. To fix

[F] /usr/share/miniconda/envs/k2/conda-bld/k2_1631619831677/work/k2/csrc/array.h:501:void k2::Array1<T>::Init(k2::ContextPtr, int32_t, k2::Dtype) [with T = char; k2::ContextPtr = std::shared_ptr<k2::Context>; int32_t = int] Check failed: size >= 0 (-1383015021 vs. 0) Array size MUST be greater than or equal to 0, given :-1383015021

please change

icefall/egs/librispeech/ASR/conformer_ctc/decode.py

Line 566 in 42b437b

modified=False,

to

 modified=True, 

if you are using a model with vocab size 5000.

Also, you may need to reduce the value of --max-duration.
(Currently, you are using 300. It is OK for a model with vocab size 500).

Sorry that this is not documented. There is an ongoing PR to fix it (https://github.com/k2-fsa/icefall/pull/91/files)
and there are discussions about it (#70 (comment))

---

As for the second error,

_pickle.UnpicklingError: invalid load key, 'v'

I think you skipped an essential step listed in https://icefall.readthedocs.io/en/latest/recipes/librispeech/conformer_ctc.html#download-the-pre-trained-model

git lfs install

To fix that, please first install git-lfs (using sudo apt-get install git-lfs if you are using Ubuntu),
go to the pre-trained model repo and run

cd icefall_asr_librispeech_conformer_ctc
git lfs pull

After that, everything should work.

BTW, please use the pre-trained model in https://github.com/csukuangfj/icefall-asr-conformer-ctc-bpe-500
to test ctc-decoding if you don't modify any code.

[GarlandZhang]

Thank you this worked!