support for .lines file, version=6 (rM v3.0)

[RedTartan04]

It seems reMarkable have changed the .lines file format with their recent v3.0 software.
It now says version=6 and even if remy's version check is bypassed there's errors (of course),
example from lines.py:
readStruct
return fmt.unpack(buff)
struct.error: unpack requires a buffer of 24 bytes

Any idea how to fix this?

[bordaigorl]

First off: I am not on beta and I haven't had the chance to try v3.0.
So I have no first-hand information about this.

With the addition of text input, they seem to have radically redesigned the (undocumented) internal file format.
Currently, to the best of my knowledge, there's no complete reverse-engineered reimplementation of the file format.
Until this is figured out, there's no chance Remy will support the new file format.

I know this sucks; I spent all this time writing Remy and now it will become obsolete.
If somebody figures out the file format I will look into integrating it in Remy, but I cannot promise anything.
I'm open to reviewing PRs though.

Edit:
I only know of this partial attempt at parsing the v6 file format:
https://github.com/ddvk/reader

[RedTartan04]

Thank you! "radically redesigned" - not good :-( I was hoping for a quick solution to render files without using new features. Promising though ddvk is addressing this.
Are the rev eng'd file formats (including v5) documented anywhere?

[bordaigorl]

Are the rev eng'd file formats (including v5) documented anywhere?

The closest to a documentation is https://remarkablewiki.com/tech/filesystem#lines_file_format and the linked pages.
There's no official documentation/API.

[RedTartan04]

Ah, totally forgot about the wiki 🫢 Maybe I'll briefly look into it. Learning about the python struct lib can't be wrong :)

[sgoerres]

Just found this. Not sure how functional this is but maybe it helps? https://github.com/ricklupton/rmscene

[bordaigorl]

One main problem for me is that upgrading to v3 would completely disrupt my current Remy-based workflow;
so I don't want to do that until Remy can handle the new format.
But I could look into supporting it (time permitting) if I was given some example files in the new format.

Ideally I'd need:

1. a notebook (metadata+data) with multiple pages, multiple layers, and every tool used (with as many combinations of options as possible)
2. a PDF with highlighting, annotations, couple of layers on top
3. their renderings (ideally PNG, PDF and SVG) as exported from the official app

Any volunteers?

[sgoerres]

Just send a sample Notebook by mail. Will try create a pdf later.

[sgoerres]

Maybe this helps: https://github.com/chemag/maxio/blob/master/version6.md

[RedTartan04]

Maybe this helps: https://github.com/chemag/maxio/blob/master/version6.md

Pretty much in an early stage. I've emailed him, pointing to this discussion here :)

[RedTartan04]

Just found this. Not sure how functional this is but maybe it helps? https://github.com/ricklupton/rmscene

Looks like a well-structured python version from the ddvk go source plus more.
I commented about what it seems is the underlying data structure of the new format ("LWW-CRDT") ricklupton/rmscene#1

[RedTartan04]

On a side note: also the .content file has changed. At first I thought "um, lots of timestamps in there". Now with that LWW-CRDT in mind, this seems to make sense, to sync not only the actual lines file but also some of the notebook's metadata. Excerpt from a .content with significant changes to the page list:

{
    "cPages": {
        "lastOpened": {
            "timestamp": "0:0",
            "value": ""
        },
        "original": {
            "timestamp": "0:0",
            "value": -1
        },
        "pages": [
            {
                "id": "00e1f966-f2aa-48b6-b2d7-c763081a3b60",
                "idx": {
                    "timestamp": "1:1",
                    "value": "ba"
                },
                "scrollTime": {
                    "timestamp": "1:3",
                    "value": "2022-12-20T17:38:48Z"
                },
                "template": {
                    "timestamp": "1:1",
                    "value": "Blank"
                },
                "verticalScroll": {
                    "timestamp": "1:3",
                    "value": 1872
                }
            }
        ],
        "uuids": [
            {
                "first": "188a921f-7854-54ce-86c8-167c1bf619a8",
                "second": 1
            }
        ]
    },
    "coverPageNumber": -1,
    "customZoomCenterX": 0,
    "customZoomCenterY": 936,
    "customZoomOrientation": "portrait",
    "customZoomPageHeight": 1872,
    "customZoomPageWidth": 1404,
    "customZoomScale": 1,
    "documentMetadata": {
    },
    "dummyDocument": false,
    "extraMetadata": {
        "LastBallpointv2Color": "Black",
...
        "LastTool": "Finelinerv2"
    },
    "fileType": "notebook",
    "fontName": "",
    "formatVersion": 2,
    "lineHeight": -1,
    "margins": 125,
    "orientation": "portrait",
    "pageCount": 1,
    "pageTags": [
        {
            "name": "my tag",
            "pageId": "00e1f966-f2aa-48b6-b2d7-c763081a3b60",
            "timestamp": 1671556514488
        }
    ],
    "sizeInBytes": "5474",
    "tags": [
        {
            "name": "my doc tag",
            "timestamp": 1671556761064
        }
    ],
    "textAlignment": "justify",
    "textScale": 1,
    "zoomMode": "bestFit"
}

[ricklupton]

Just found this. Not sure how functional this is but maybe it helps? https://github.com/ricklupton/rmscene

Looks like a well-structured python version from the ddvk go source plus more. I commented about what it seems is the underlying data structure of the new format ("LWW-CRDT") ricklupton/rmscene#1

Maybe you can use this as it is to read the files to avoid having to re-implement all the low-level reading? And then feed the stroke data into the same rendering you do now. See also ricklupton/rmscene#2

The rmscene code is fairly complete for reading the "blocks" of data. I was mostly interest in accessing the text data, so haven't looked at the stroke data much, but I think it's fairly straightforward to extract the same kind of data as in the old format.

[sgoerres]

Just saw this: https://www.reddit.com/r/RemarkableTablet/comments/10hxe3j/updates_regarding_reverse_engineering_remarkable/

[chemag]

[I forgot to mention it here]

I mixed rmscene's parser with maxio's SVG renderer, and now I can convert version=6 lines (.rm) files into PDF.

From https://www.reddit.com/r/RemarkableTablet/comments/10bbteg/support_for_remarkable_lines_version6_file_format/

Not sure if it helps, but found this recently : https://github.com/ricklupton/rmscene

Wow, this is a full version=6 parser :). This was great, and saved me lots of effort.

I added an rm2svg (and rm2pdf) converter to rmscene (see pull request here), and then added glue for maxio (the package I use for conversion) to resort to rmscene for version=6 files (see pull request here). The setup is cumbersome (you need to download both repos in the same root directory, as maxio will look for ../rmscene/), but I'm not sure whether the upstream repo owners are interested in any of this work, or whether they're willing to unify their repos (which IMO is the right solution).

tl;dr: Now I get PDFs of all the .rm files, either version=5 or version=6.

This works for me:

(1) setup

$ mkdir ~/remarkable
$ git clone https://github.com/chemag/maxio
$ git clone https://github.com/chemag/rmscene

(2) sync all your files to ~/raw/

$ rsync -avut --progress --rsync-path=/opt/bin/rsync remarkable:/home/root/.local/share/remarkable/xochitl/ ~/raw/

(3) convert the raw files to pdf.

$ ./remarkable/maxio/rm_tools/rmtool.py convert-all --root ~/raw/ --outdir ~/pdf -dd

Some caveats:

• only tested in Linux.
• the implementation of the converter is very simple: It loops through all the blocks, and converts to SVG (and later PDF) 2x of them (the ones containing strokes and the ones containing text). There seem to be some hierarchical relationship between blocks: I am not sure whether this matter for SVG/PDF conversion, or for any other reason.
• I haven't figured out completely the positioning mechanism. While you keep strokes and text separately, the positioning works well. If you mix them, then all the positioning is broken. This needs to be fixed.
• text with multiple lines renders all of them in the same place. This needs to be fixed.

[lonvia]

I've hacked together a quick proof-of-concept support for the version 6 file here: https://github.com/lonvia/remy/tree/support-for-version-6. It is using https://github.com/ricklupton/rmscene for reading the lines files. Simply 'pip install rmscene' to make it work.

This is very rudimentary. Only strokes are supported, not the new text annotations. It can read the page info from the new cPages structure in the .content file to get the page UUIDs and templates. There is other interesting information there. At least the VerticalScroll should be used to determine the actual page size. It currently segfaults on loading PDFs.

So still a bit immature for a PR but maybe still a useful starting point for others.

[bordaigorl]

@lonvia thanks so much for sharing your efforts! Much appreciated.
I was hoping to give some attention to this issue (I am still using version 2.15 so to preserve my workflow but this is not going to be sustainable).
I cannot promise a timeline, but the project is not abandoned and your edits can help me make progress.

[lonvia]

It's good to hear the project is not dead. Remy is just the right piece of software for me and I discovered too late that updating my Remarkable would break it. So I'm just glad I have it working again for what I need.

If you are still active and interested, I'm willing to contribute a few PRs towards Remarkable version 3. However, it would have to be in smaller steps with gradually growing support for the new format because I'm pretty much in the same situation as you (very limited time and mostly doing this to support my own workflows). As far as I can see, it can be done without breaking 2.15 (although I can't really test that for obvious reasons). It might give you a version of remy that gets you out of the current deadlock, i.e. a version that allows you to update your remarkable without crippling your workflow for weeks while you readd the basic support to remy.

[benneti]

If you don't mind me asking (and somewhat hijacking this thread), I added support to the newer versions to https://github.com/benneti/rmrl , however I did not figure out how I could transform the annotations to correctly align with a pdf.

Do you have more information or ideas about how to use the VerticalScroll for this?

[gimmy]

I tested the work of @lonvia after updating to v3.11 from v2.15. The software is good enough to keep the workflow for me, in particular:

👍 Strokes in new and previous notebook work
👍 Straight lines work
👎 typed text is not supported, as expected

⚠️ Inserting typed-text (or converted text) in a former notebook would mess up the page on remy. This seems to be caused by the extra Text layer (which cannot be removed, even after the typed-text has been erased). Anyway, it is possible to copy the strokes in a new page in the same notebook: this won't have a Text layer and remy will read it correctly.
⚠️ longer pages (scrolling down in a notebook) will be cut on remy to a standard length.

In conclusion, provided that you use the reMarkable with remy in the same way as v2.15 (no typed-text, no down scrolling) everything should work. And even if you mess something up, there is a simple way to fix it.