Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Problems with pygtail running on Windows with reading log file generated on a Linux box #68

Open
Jacek-ghub opened this issue Jan 26, 2023 · 0 comments

Comments

@Jacek-ghub
Copy link

I have a third-party process (don't have control of its behavior) running on Linux / CentOS that writes to a log file on the local system. The folder with that file is samba exported, so other boxes can see it. I mapped that folder on a Win box and am running Python3 (v3.11.1) code to read updates to that file.

Somehow the minute i loop on pygtail, Python is not seeing file being increased anymore. I have also created a test file, just to read that file size, and it works OK. However, when that file test file is running, and I start the pygtail program, the test program also gets stuck. Once the pygtail program is killed, the test program again sees increasing file sizes. Also, when I comment out the for loop on pygtail results (but leave the constructor), the code sees increasing file sizes.

I also have that code running on Linux that samba mapped that folder, and it works like a charm.

Any help would be greatly appreciated.

Here are those programs:

test.py:

from time import sleep
import os
from os import fstat, stat

filename = 'z:\debug.log'
while True:
  print ('stat(filename).st_size', stat(filename).st_size)

  fh = open (filename, "r", 1)
  fh.seek (0, os.SEEK_END)
  fh.close ()

  sleep (5)

pyg.py

from pygtail import Pygtail
from time import sleep
import os
from os import fstat, stat

filename = 'z:\debug.log'

while True:
  print ('stat(filename).st_size', stat(filename).st_size)

  py = Pygtail (filename, read_from_end = True, offset_file = 'some_offset.txt')
  print ("pygtail pygtail:", py)
  for log_line in py:
    print ("in the pygtail for loop")

  sleep (5)

My test procedures:

  1. Running test.py
    1.1. program sees increasing file sizes
  2. Running in parallel pyg.py
    2.1. both programs stall at the first size that pygtail grabs
  3. Killing pyg.py
    3.1. test.py start seeing size increases
  4. Running in parallel pyg.py with "for log_line ... / print ('in the ..." lines commented out
    4.1. both programs see file size increases

Of course, there is no need to run test.py to have pyg.py stall. I added that test program to try to see whether this is a general Python or Windows problem, and better illustrate the process.

I really do not understand how pyg.py can influence test.py other than somehow keeping a lock or something like that on that log file.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant