intermittent problem building bom.csv

[kroesche]

Thank you so much for putting this together. I used it to (almost) completely automate my tasks for JLCPCB board production. This is not so much a request for you to fix something as a question about how I can troubleshoot something going wrong with my BOM generation.

Here is my problem with the make bom target:

INFO:__main__:Run generate
INFO:util.ui_automation:Waiting for plot window...
INFO:util.ui_automation:Found plot window
DEBUG:util.ui_automation:Window id: 
INFO:__main__:Select commandline window
INFO:__main__:Paste xslt command
INFO:__main__:Waiting for bom file
DEBUG:util.file_util:Waiting for process to create file
DEBUG:util.file_util:Waiting for process to create file
DEBUG:util.file_util:Waiting for process to create file

(snip many repeats)

DEBUG:util.file_util:Waiting for process to create file
DEBUG:util.file_util:Waiting for process to create file
   
Done.
Written 1099251 bytes
(1099251 of which were video data and 0 audio data)

Shutting down......
Goodbye!
XIO:  fatal IO error 11 (Resource temporarily unavailable) on X server ":303972457"
      after 11 requests (8 known processed) with 0 events remaining.
Traceback (most recent call last):
  File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/usr/lib/python2.7/dist-packages/kicad-automation/eeschema/export_bom.py", line 199, in <module>
    export_bom(args.schematic, output_dir, args.screencast_dir)
  File "/usr/lib/python2.7/dist-packages/kicad-automation/eeschema/export_bom.py", line 164, in export_bom
    eeschema_export_bom(output_dir, eeschema_proc)
  File "/usr/lib/python2.7/dist-packages/kicad-automation/eeschema/export_bom.py", line 143, in eeschema_export_bom
    file_util.wait_for_file_created_by_process(eeschema_proc.pid, bom_file)
  File "/usr/lib/python2.7/dist-packages/kicad-automation/util/file_util.py", line 37, in wait_for_file_created_by_process
    raise RuntimeError('Timed out waiting for creation of %s' % file)
RuntimeError: Timed out waiting for creation of /output/bom/bom.csv

This problem is intermittent but is happening a lot on my current project. I thought it might be something in my project, but then it does work some of the time. Then I thought maybe the timeout wasn't long enough so I lengthen that (in upstream/kicad-automation-scripts/util/file_util.py). The first time I increased the timeout, it started working again so I thought that was the problem, but later it started producing the error again.

I also modified rules.mk so that it would generate a screencast. When I reviewed the screen capture it looked normal for producing the BOM and there was no error indication of any kind (and no indication the file was produced either).

I also did the exact same steps through the GUI on my non-docker Kicad and it always produces the bom.csv file without any problem.

The host machine is a Mac and I am using docker for the kicad-tools and building the docker container using your Dockerfile.

Do you have any idea what might cause this, or suggestions where I should look next?

Thank you.

[obra]

This is weird, but unsurprising, given how brittle our eeschema automation is. I'd love having a PR to make that screencast an optional choice. I'd also love a PR to add explicit generation steps for jlcpcb (and any other pcb house), maybe integrating https://github.com/matthewlai/JLCKicadTools for JLC.

On Sat, Sep 18, 2021, at 9:58 AM, kroesche wrote: Thank you so much for putting this together. I used it to (almost) completely automate my tasks for JLCPCB board production. This is not so much a request for you to fix something as a question about how I can troubleshoot something going wrong with my BOM generation. Here is my problem with the make bom target: `INFO:__main__:Run generate INFO:util.ui_automation:Waiting for plot window... INFO:util.ui_automation:Found plot window DEBUG:util.ui_automation:Window id: INFO:__main__:Select commandline window INFO:__main__:Paste xslt command INFO:__main__:Waiting for bom file DEBUG:util.file_util:Waiting for process to create file DEBUG:util.file_util:Waiting for process to create file DEBUG:util.file_util:Waiting for process to create file (snip many repeats) DEBUG:util.file_util:Waiting for process to create file DEBUG:util.file_util:Waiting for process to create file Done. Written 1099251 bytes (1099251 of which were video data and 0 audio data) Shutting down...... Goodbye! XIO: fatal IO error 11 (Resource temporarily unavailable) on X server ":303972457" after 11 requests (8 known processed) with 0 events remaining. Traceback (most recent call last): File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main "__main__", fname, loader, pkg_name) File "/usr/lib/python2.7/runpy.py", line 72, in _run_code exec code in run_globals File "/usr/lib/python2.7/dist-packages/kicad-automation/eeschema/export_bom.py", line 199, in <module> export_bom(args.schematic, output_dir, args.screencast_dir) File "/usr/lib/python2.7/dist-packages/kicad-automation/eeschema/export_bom.py", line 164, in export_bom eeschema_export_bom(output_dir, eeschema_proc) File "/usr/lib/python2.7/dist-packages/kicad-automation/eeschema/export_bom.py", line 143, in eeschema_export_bom file_util.wait_for_file_created_by_process(eeschema_proc.pid, bom_file) File "/usr/lib/python2.7/dist-packages/kicad-automation/util/file_util.py", line 37, in wait_for_file_created_by_process raise RuntimeError('Timed out waiting for creation of %s' % file) RuntimeError: Timed out waiting for creation of /output/bom/bom.csv

`

…

This problem is intermittent but is happening a lot on my current project. I thought it might be something in my project, but then it does work some of the time. Then I thought maybe the timeout wasn't long enough so I lengthen that (`in upstream/kicad-automation-scripts/util/file_util.py`). The first time I increased the timeout, it started working again so I thought that was the problem, but later it started producing the error again. I also modified rules.mk so that it would generate a screencast. When I reviewed the screen capture it looked normal for producing the BOM and there was no error indication of any kind (and no indication the file was produced either). I also did the exact same steps through the GUI on my non-docker Kicad and it always produces the bom.csv file without any problem. The host machine is a Mac and I am using docker for the kicad-tools and building the docker container using your Dockerfile. Do you have any idea what might cause this, or suggestions where I should look next? Thank you. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#27>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAALC2A7UD3PPHGYRBIJESTUCTAKPANCNFSM5EJMAAVA>.

[kroesche]

I'll take a look at creating the PRs for you.

[kroesche]

To be clear about this ...

"I'd also love a PR to add explicit generation steps for jlcpcb (and any other pcb house), maybe integrating https://github.com/matthewlai/JLCKicadTools for JLC."

Do you mean that that python package should be added to the Docker container and then the JLC bom build steps done using the Docker container? Or did you mean that the JLCKicadTools was integrated separately from the Docker container.

In my view it makes sense to add it to the Docker container and then it is one less tool the developer needs to install. For example, I am using JLCKicadTools in my project and so I have to install it separately from kicad-tools right now.

[obra]

I think that in my ideal world, everything would go into the docker container. That's part of the point of having Docker there at all - getting a known-good environment that should work cross-platform.

What I'm envisioning is a new makefile target of something like

make fabrication-output-jlcpcb that would create the zipfile, the BOM and the placement file.
(I'd love folks to do this for other vendors, too).

From there, for extra credit, a make send-to-jlcpcb target that actually uploads would be mind-blowing. But also not a blocker to merge ;)

[kroesche]

I submitted a PR in the kicad-automation-scripts repo which I believe resolves my original problem reported in this issue.

obra/kicad-automation-scripts#2

BTW are you aware of this fork of your project? https://github.com/INTI-CMNB/KiAuto

I didn't try it but it looks like he did some work to try and resolve some of the funny x server error messages.

[obra]

I wasn't aware of that fork but...it looks like a nice improvement and I'd be thrilled for us to switch the docker image and makefiles to use it under the covers. It looks like it's getting actual dev effort.

…

On Thu, Sep 23, 2021, at 12:17 PM, kroesche wrote: I submitted a PR in the kicad-automation-scripts repo which I believe resolves my original problem reported in this issue. obra/kicad-automation-scripts#2 <obra/kicad-automation-scripts#2> BTW are you aware of this fork of your project? https://github.com/INTI-CMNB/KiAuto I didn't try it but it looks like he did some work to try and resolve some of the funny x server error messages. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#27 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAALC2AF4CJHUTF4UOD64ZTUDN4MJANCNFSM5EJMAAVA>.

[kroesche]

I am sorry @obra but I discovered a new problem. I dont know if this existed before I made the timing change. If the bom.csv file already exists in the output directory, then the make bom script hangs because when wait_for_file...() is called the bom.csv already exists and wait_for_file...() returns immediately before the actual BOM exporting process finishes and then there is some kind of conflict between the process still running and the eeschema_export_bom() function which thinks it is finished and trying to shut it down. This causes the process to hang.

The reason I didn't notice this before because I am using a higher level Makefile that always just cleaned or moved the outputs before the bom export step was ever kicked off and so I never encountered this situation.

A hacky fix is to modify the rules.mk in kicad-tools to rm the bom.csv file before kicking off the eeschema export script. I made this modification locally and it does solve the problem. If this solution is acceptable to you then I will submit the PR.

[obra]

I'd rather `mv bom.csv bom.csv.bak` than an outirght delete, but sure.

…

On Fri, Sep 24, 2021, at 1:06 PM, kroesche wrote: I am sorry @obra <https://github.com/obra> but I discovered a new problem. I dont know if this existed before I made the timing change. If the bom.csv file already exists in the output directory, then the `make bom` script hangs because when `wait_for_file...()` is called the bom.csv already exists and `wait_for_file...()` returns immediately before the actual BOM exporting process finishes and then there is some kind of conflict between the process still running and the `eeschema_export_bom()` function which thinks it is finished and trying to shut it down. This causes the process to hang. The reason I didn't notice this before because I am using a higher level Makefile that always just cleaned or moved the outputs before the bom export step was ever kicked off and so I never encountered this situation. A hacky fix is to modify the rules.mk in kicad-tools to `rm` the bom.csv file before kicking off the eeschema export script. I made this modification locally and it does solve the problem. If this solution is acceptable to you then I will submit the PR. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#27 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAALC2BNXJ46QREP3ZWHXC3UDTK2XANCNFSM5EJMAAVA>.

[kroesche]

I think this can be closed.