implement ctc loss function

[wcshds]

I need ctc loss function in CRNN model. I tried to implement it based on PyTorch implementation, but the results obtained after calling forward() are somewhat different from PyTorch's.

I don't know what went wrong, I'd appreciate it if someone could tell me.

Reference

• PyTorch implementation
• Connectionist Temporal Classification: Labelling Unsegmented Sequence Data with Recurrent Neural Networks

Checklist

• Confirmed that run-checks all script has been executed.

[louisfd]

Hi,
I can take a look at it later today

[wcshds]

I believe the result is now the same as PyTorch. But the performance of this implementation seems to be less than ideal.

[wcshds]

The implementation doesn't work on the NdArray backend because of #1053. It also doesn't work on the LibTorch backend because of #1055.

I believe the current performance bottleneck lies in creating the one-hot. This is because the repeat() method is very slow on the Wgpu backend.

burn/burn-core/src/nn/loss/ctc.rs

Lines 330 to 341 in 7121852

	fn one_hot<B: Backend>(tensor: Tensor<B, 2, Int>, num_classes: usize) -> Tensor<B, 3> {
	let device = tensor.device();
	let shape = tensor.dims();
	let labels: Tensor<B, 3, Int> = tensor.unsqueeze_dim(2).repeat(2, num_classes);
	let indices = Tensor::<B, 1, Int>::arange_device(0..num_classes, &device)
	.reshape([1, 1, num_classes])
	.repeat(1, shape[1])
	.repeat(0, shape[0]);
	labels.equal(indices).float()
	}

[louisfd]

Hi @wcshds
I haven't had time like I thought last week and then I was abroad for several days. I'm sorry I said I was gonna look at it last week, but I certainly haven't forgotten you! Glad to see you continued working on it since then. I will definitely take a look real soon

[wcshds]

@louisfd Thank you! The current implementation still significantly consumes graphics memory. I believe that separately calculating the alpha values for blanks and letters can significantly reduce the graphics memory usage, but I don't know how to implement it.

[louisfd]

@wcshds
I took your word that repeat was the bottleneck in wgpu. This made a lot of sense because we relied on the default implementation which launches as many slice_assign kernels as there are repetitions. For large times argument this is awful.
I wrote a repeat kernel so that only one kernel is launched instead of times: #1068

Please tell me if this is better now

[wcshds]

@louisfd Thank you! Now repeat() is much faster.

[wcshds]

I tried to use this implementation of ctc loss in the CRNN model, but after the first iteration loss became NaN. I don't know what went wrong. wcshds/crnn-cjk

[antimora]

Just noticed 1-e15 magic number. Please refactor to a constant and explain how this number is derived. It would also be preferable if float number precision independent (we use half and full precisions)

[wcshds]

@antimora I just need a small value to prevent log(0), so now I think it may not be necessary to use 1e-15; 1e-5 should be small enough.
However, I think CTC Loss may not be suitable for half precision because I previously attempted to use mixed precision training in PyTorch, but PyTorch's CTC Loss does not support fp16. [CTC Loss] CTC Loss not support float16? Perhaps I need to explore the use of half precision training in future practices to see if CTC Loss can work with it.

[antimora]

Did you mean to have this number as negative? The literal number you has is positive. In an unlikely event, (l1 - m.clone()).exp() + (l2 - m.clone()).exp() expression could be equal to abs(DELTA) which would still lead to log(0) situation.

Additionally, I suggest we use [https://doc.rust-lang.org/std/primitive.f32.html#associatedconstant.EPSILON](f32's EPSILON) or [f16's EPSILON]https://docs.rs/tract-core/latest/tract_core/prelude/struct.f16.html#associatedconstant.EPSILON constants depending what on Backend's precision settings. @nathanielsimard or @louisfd can suggest on how we can extract this. -1e-5 seems a rather big number for f16 or f32. (probably it may not work for f16 because its epsilon is 4.88e-04. we need to double check it)

[wcshds]

I'm sorry, it's a typo. DELTA should be positive.

1e-5 can ensure that the results of the loss are accurate to three decimal places, but 4.88e-4 is a bit large. Perhaps CTC Loss is indeed not suitable for the use of half-precision training.

[antimora]

Closing this ticket and linking to an issue ticket so someone else can pick up: #1536