while extracting japanese text from pdf to text file, it is showing unwanted noise with japanese text.

[rohit-humne]

This is my code:-

import pymupdf

doc = pymupdf.open("output_S19990401A10010001001.pdf") # open a document
out = open("output5_S19990401A10010001001.txt", "wb") # create a text output
for page in doc: # iterate the document pages
    text = page.get_text().encode("utf16") # get plain text (is in UTF-8)
    out.write(text) # write text of page
    out.write(bytes((12,))) # write page delimiter (form feed 0x0C)
out.close()

output:-

i hope that i can find a solution.

Thanks in advance!

[JorjMcKie]

You did not share the file.
But anyway, did you check whether this also happens without your encoding change?

[rohit-humne]

@JorjMcKie
Thanks for your reply!

this is a pdf file
S19990401A10010001001.pdf

when i tried with other code instead of red marks as you can seen in ouput image, it is showing (cid:2) kind of characters.

[rohit-humne]

@JorjMcKie
Thank you!

if in case, i want to ignore this glyph characters, how can i do it?

[JorjMcKie]

Use a text extraction flags that does not try to find replacements for the so-called Invalid Unicode 0xFFFD (�), e.g. flags = pymupdf.TEXTFLAGS_TEXT & ~pymupdf.TEXT_CID_FOR_UNKNOWN_UNICODE.
Then text = page.get_text(flags=flags).
Finally text = text.replace(chr(0xfffd), "").

[rohit-humne]

Thank you! @JorjMcKie,

i am very new to programming. I have just manage to extract data from pdf into text file and then remove cid:x characters from the text file and save it into another textfile.

This is the code:-

from pdfminer.converter import PDFPageAggregator
from pdfminer.layout import LAParams, LTContainer, LTTextBox, LTTextLine, LTChar
from pdfminer.pdfinterp import PDFPageInterpreter, PDFResourceManager
from pdfminer.pdfpage import PDFPage
import re

def pdfminer_config(line_overlap, word_margin, char_margin,line_margin, detect_vertical):
    laparams = LAParams(line_overlap=line_overlap,
                        word_margin=word_margin,
                        char_margin=char_margin,
                        line_margin=line_margin,
                        detect_vertical=detect_vertical)
    resource_manager = PDFResourceManager()
    device = PDFPageAggregator(resource_manager, laparams=laparams)
    interpreter = PDFPageInterpreter(resource_manager, device)
    return (interpreter, device)

def find_textboxes(layout_obj):
    if isinstance(layout_obj, LTTextBox):
        return [layout_obj]
    if isinstance(layout_obj, LTContainer):
        boxes = []
        for child in layout_obj:
            boxes.extend(find_textboxes(child))
        return boxes
    return []

def find_textlines(layout_obj):
    if isinstance(layout_obj, LTTextLine):
        return [layout_obj]
    if isinstance(layout_obj, LTTextBox):
        lines = []
        for child in layout_obj:
            lines.extend(find_textlines(child))
        return lines
    return []

def find_characters(layout_obj):
    if isinstance(layout_obj, LTChar):
        return [layout_obj]
    if isinstance(layout_obj, LTTextLine):
        characters = []
        for child in layout_obj:
            characters.extend(find_characters(child))
        return characters
    return []

def write_text(text_file, text):
    text_file.write(text)

text_file = open('output11.txt', 'w')
with open("./S19990401A100100010011.pdf", 'rb') as f:
    interpreter, device = pdfminer_config(line_overlap=0.5, word_margin=0.1, char_margin=2, line_margin=0.5, detect_vertical=True)
    for page in PDFPage.get_pages(f):
        interpreter.process_page(page)  # ページを処理する。
        layout = device.get_result()  # LTPageオブジェクトを取得。
        boxes = find_textboxes(layout)
        for box in boxes:
            write_text(text_file, box.get_text().strip())
text_str = '(cid:2)'
if 'cid' in text_str.lower():
	text_str = text_str.strip('(')
	text_str = text_str.strip(')')
	ascii_num = text_str.split(':')[-1]
	ascii_num = int(ascii_num)
	text_val = chr(ascii_num)  # 66 = 'B' in ascii            
        
text_file.close()

with open("output11.txt", "r") as infile, open("output11_1.txt", "w") as outfile:
    content = infile.read()
    cleaned_content = re.sub(r"\(cid:\w+\)", "", content)
    outfile.write(cleaned_content)
re.sub(r"\(cid:\w+\)", "", content) #this will remove that pattern

output11_1.txt
output11.txt

if you see output11_1.text file still some noise is present....
as well as i want to extract text from this kind of pdfs from the folder and save into the same folder as same name, how can i imporve this code?

[JorjMcKie]

This script does not use PyMuPDF but pdfminer to access PDFs.
We do not give advice for this package.

[rohit-humne]

Thank you! @JorjMcKie

[rohit-humne]

@JorjMcKie Hi,
some how i managed to remove the cid characters, but there is some issue from the pdf file some text is getting extracted repeatedly.
how can i modify the following code?

from io import StringIO
from pdfminer.high_level import extract_text_to_fp
import re


output_string = StringIO()
with open('/Users/humnerohit/python_pdf_to_text/S19990401A10010001001.pdf', 'rb') as fin:
    extract_text_to_fp(fin, output_string)
    
with open("outputx.txt","w") as infile, open("outputx.txt", "r") as outfile :
    infile.write(output_string.getvalue().strip()) 



with open("outputx.txt", "r") as infile, open("outputx_1.txt", "w") as outfile:
    content = infile.read()
    cleaned_content = re.sub(r"\(cid:\w+\)", "", content)
    outfile.write(cleaned_content)
re.sub(r"\(cid:\w+\)", "", content) #this will remove that pattern      
print(output_string.getvalue().strip())

output:-

[rohit-humne]

this repeated text is present in pdf file, which kind of text is this?
why this text only is repeating?

[rohit-humne]

this text is getting repeated.

[JorjMcKie]

This is the consequence of the PDF creator's decision to show us text effects like those gray shadows.
PDF has no way of supporting this directly like e.g. Word does that.
The gray shadows represent separate characters written in a way that looks like shadowing.
Therefore we get multiple copies of the same text upon text extraction.

[rohit-humne]

Thank you!