LiTeX should work with Xpdf utils as well as Poppler utils

[flerro]

Summary by @rschroll: The Xpdf version of pdftoppm doesn't support the -png flag, but Xpdf does provide a pdftopng command that we could use instead. It currently has problems on Windows, but we should support it as soon as those problems are fixed.

---

PDF content is not rendered.

Light Table version: 0.6.5 (on Windows 7 x64)
Binary version: 0.8.4
LiTeX version: 0,0.7
pdftoppm version 3.04

[rschroll]

I can't claim to be too surprised at this -- I haven't done any testing
on Windows.

To make sure that I understand properly: Compiling the latex file
works, resulting in an up-to-date PDF file. The proper number of pages
are created in the viewer, but they are left blank. Is that correct?

The first thing to check is if there are any messages in the Light
Table console (View > Console). Litex will spit out error messages
there if it detects something's gone wrong.

Then, let's check that pdftoppm is working. On the command line, in
the same directory as your PDF file, run
pdftoppm -f 1 -l 1 -r 72 -png filename.pdf > test.png
replacing filename.pdf with the appropriate filename. (I'm assuming
Windows uses '>' for output redirection; if it doesn't, replace that
character with the appropriate one.) Check that test.png is indeed a
PNG file with an image of the first page.

If any of this fails, let me know. Otherwise, we'll move on to
checking some other things.

[flerro]

PDF generation is OK. Blank pages are shown in place of the actual content (total number of pages is correct).

Looks like the pdftoppm version I installed (from foolabs.com/xpdf) does not support the -png option:

C:\...\>pdftoppm -f 1 -l 1 -r 72 -png sample.pdf 
Config Error: No display font for 'Symbol'
Config Error: No display font for 'ZapfDingbats'
I/O Error: Couldn't open file '-png'

Without the -png option, pdftoppm complains about a missing <PPM-root> compulsory parameter:

C:\...\>pdftoppm -f 1 -l 1 -r 72 sample.pdf
pdftoppm version 3.04
Copyright 1996-2014 Glyph & Cog, LLC
Usage: pdftoppm [options] <PDF-file> <PPM-root>
  -f <int>          : first page to print
  -l <int>          : last page to print
  -r <int>          : resolution, in DPI (default is 150)
  -mono             : generate a monochrome PBM file
  -gray             : generate a grayscale PGM file
  -freetype <string>: enable FreeType font rasterizer: yes, no
  -aa <string>      : enable font anti-aliasing: yes, no
  -aaVector <string>: enable vector anti-aliasing: yes, no
  -opw <string>     : owner password (for encrypted files)
  -upw <string>     : user password (for encrypted files)
  -q                : don't print any messages or errors
  -cfg <string>     : configuration file to use in place of .xpdfrc
  -v                : print copyright and version info
  -h                : print usage information
  -help             : print usage information
  --help            : print usage information
  -?                : print usage information

If you need more info, let me know

[rschroll]

I had been expecting the options to be the same between the Xpdf and Poppler versions of pdftoppm. How silly of me.

The bad news is that the Xpdf version of pdftoppm just won't work for us. The good news is that the new version of Xpdf, which you have (and which was quite literally released yesterday!) has a pdftopng command that will work. If you don't mind digging into some clojurescript, try changing lines 215 and 216 of litex-viewer.cjls (in the src/lt/plugins/ folder of the litex package) from

(lt.plugins.litex/run-commands [(str "pdftoppm -f " pagenum " -l " pagenum " -r " (* 72 zoom)
                                     " -png \"" (:pdfname @this) "\"")]

to

(lt.plugins.litex/run-commands [(str "pdftopng -f " pagenum " -l " pagenum " -r " (* 72 zoom)
                                     "\"" (:pdfname @this) "\" -")]

After making the change, you'll need to recompile the Javascript. Saving the file should trigger this, and restarting Light Table should load the new version. Alternatively, you can connect to the "Light Table UI" and then execute the changed bit of code with Ctrl-Enter to alter the currently loaded plugin. With a bit of luck, that will work. That said, I wouldn't be surprised to have something else trip us up.

Let me know how it goes.

[flerro]

Thanks for you very quick answer.

To make the generation work I had to include the output file base name as a parameter the pdftopng invocation command line:

 (lt.plugins.litex/run-commands [(str "pdftopng -f " pagenum " -l " pagenum " -r " (* 72 zoom)
       " \"" (:pdfname @this) "\" \"" (:pdfname @this) "\"")]

We are a step closer to the solution. A bunch of files are generated, eg:

mypdffile.pdf-000001.png
mypdffile.pdf-000002.png
mypdffile.pdf-000002.png

Each image contains the related PDF page rendered, but images do not get displayed in LightTable (I bet the plugin expects different names for generated PNGs).

[rschroll]

Very good to hear. I was worried that I'd have problems with path
separators on Windows, but if you're getting PNGs produced, I guess not!

Litex isn't looking for the PNGs in any files; it's expecting them to
be written to stdout. pdftopng should do that if the output base name
is '-'. Can you double check whether this works in Windows? (I know
it does in Linux, but this is something that may be different across
OSs.)

Previously, I had been writing the PNGs to files and then reading the
back in. We can switch back to that if we can't get this to work.

[flerro]

The command

pdftopng -f 1 -l 1 -r 72 sample.pdf -

Dumps binary content in the terminal

I tried to change the plugin in the following way

(lt.plugins.litex/run-commands [(str "pdftopng -f " pagenum " -l " pagenum " -r " (* 72 zoom)
       " \"" (:pdfname @this) "\" - ")]

with no luck.

[rschroll]

On Fri, May 30, 2014 at 8:51 AM, Francesco [email protected]
wrote:

The command

pdftopng -f 1 -l 1 -r 72 sample.pdf -
Dumps binary content in the terminal

That's what it's supposed to do. Then, Light Table should read that
content, base64-encode it, and pass it to litex, which puts it in a
data URL for the image. I'm not sure why this last bit isn't working.

I'll boot over into a Windows partition later today and see if I can
figure out what's going on.

[rschroll]

I think I know what's going wrong. Windows expects everything sent to
stdout to be text and all text to have \r\n instead of bare \n's.
Therefore, it helpfully converts all 0x0A bytes to 0x0D 0x0A. This
happens before node.js has a chance to base64-encode everything. The
result is that we end up with a corrupted PNG file. Thanks, Windows!

Apparently, you can convince Windows to treat stdout as a binary
stream, so it doesn't munge your data. For instance, a python script
will display this problem with the standard interpreter, but if the
interpreter is run with the '-u' flag, the switch is flipped and you
get un-munged data. I don't know if we can flip this switch from
within node or not. I'll ask about it on the mailing list.

Arguably, the fault lies in pdftopng, which dumps binary data to stdout
without checking that stdout is ready to receive binary data. But I
can't find a bug tracker for Xpdf to report this.

[rschroll]

I've heard from the Xpdf author that this is indeed a bug in pdftopng. I'll give him a few days to try to fix it before we start considering work-arounds.

[rschroll]

More differences between Poppler and Xpdf: Xpdf doesn't take non-integer resolutions; Poppler does. Xpdf rounds to the nearest pixel when working out the bitmap size; Poppler always rounds up.

Fun.

[rschroll]

While we're waiting for Xpdf to patch itself, you might be interested
in this: I've found two sources for Poppler on Windows:
http://blog.alivate.com.au/poppler-windows/
http://manifestwebdesign.com/2013/01/09/xpdf-and-poppler-utils-on-windows/

I can't vouch for either of them, but you might be able to get pdftoppm
working from there.

[rschroll]

Didn't mean to close this.

[rschroll]

No news from Xpdf. I did get a chance to try the poppler utils from alivate.com.au, and they seemed to work correctly. (I don't have a TeX install on my Windows, so I couldn't test the full stack.) If you feel like giving that a try, I'd love to know whether it works.

[flerro]

It works!

Using: poppler.0.26.1_x86.7z

[rschroll]

Fantastic!

How did you install the program from the 7z file? I was able to test
it locally, but I didn't know enough about Windows to know how to get
them to work from anywhere. If you can provide a brief description,
I'll add it to the README.

I'm going to leave this bug open, but re-title it to be about getting
litex to work with the Xpdf utils.

[n2o]

This also solves the same issues occuring with OS X 10.10

Use poppler and the pdf file compiles and shows up as expected.

$ brew install poppler