training the CRNNImage2Text model

[kghezelbash]

Hi, I am trying to finetune the 'hezarai/crnn-fa-printed-96-long' model. But I am getting this error while running trainer.train:
TypeError: CRNNImage2Text.forward() missing 1 required positional argument: 'pixel_values'

Also my dataset have 'image_path' and 'text' fields. Where I should give the trainer 'pixel_values'?

Thank you

[arxyzan]

Hi @kghezelbash, can you share your training script? or any other details about how you're constructing the dataset class?
The general procedure to finetune this model is to first construct your dataset. The dataset must be a hezar.data.Dataset or torch.utils.data.Dataset subclass and has to implement some methods. You can check the issue #134 to find out more. There's also a ready to use example at https://github.com/hezarai/hezar/blob/main/examples/train/train_ocr.py

[kghezelbash]

Of course.
this is my training process:
train_config = TrainerConfig(
output_dir="crnn-fa-printed-96-long-better",
task="image2text",
device="cpu",
init_weights_from=BASE_MODEL_PATH,
batch_size=8,
num_epochs=2,
metrics=["cer"],
num_dataloader_workers=0,
seed=42,
optimizer="adamw",
learning_rate=2e-5,
weight_decay=.0,
save_freq=1
)

from transformers import default_data_collator

trainer = Trainer(
config=train_config,
model=model,
train_dataset=train_dataset,
eval_dataset=eval_dataset,
data_collator=default_data_collator
)

trainer.train()

And this is how I make my dataset:
from datasets import Dataset, DatasetDict
import random

keys = list(data_dict.keys())
random.shuffle(keys)
train_keys = keys[:int(0.9 * len(keys))] # 90% for training
test_keys = keys[int(0.9 * len(keys)):] # 10% for testing

train_data = {"image_path": [], "text": []}
for key in train_keys:
train_data["image_path"].append(key)
train_data["text"].append(data_dict[key])

test_data = {"image_path": [], "text": []}
for key in test_keys:
test_data["image_path"].append(key)
test_data["text"].append(data_dict[key])

Create Dataset objects

train_dataset = Dataset.from_dict(train_data)
test_dataset = Dataset.from_dict(test_data)

Create DatasetDict

dataset_dict = DatasetDict({"train": train_dataset, "test": test_dataset})

[arxyzan]

@kghezelbash Got it, you're only creating a dataset from raw inputs. But the correct way to do it is to firstly, transform the images in the dataset which includes resizing, normalization, scaling, etc and converting to torch.tensor and that is your pixel_values. Secondly, you have to also construct your label tensors which would be your labels. You can take a look at the code starting from below:

hezar/hezar/data/datasets/ocr_dataset.py

Line 190 in 39763bd

path, text = self.data[index].values()

[kghezelbash]

So my dataset should have these:
"pixel_values": pixel_values,
"labels": labels

Instead of image_path and text?

[kghezelbash]

Now I got it; thank you for your answer.

[arxyzan]

@kghezelbash Exactly!