Word tokenization in puzzle 02

[romanows]

Thanks for releasing this, I'm excited to continue tackling the novel!

The tokenization rules in puzzle 02 are confusing me. The relevant text currently reads "... consider a word to be whatever characters in a sentence fall between two spaces, or between a space and the start or end of the sentence." To me, this means I would tokenize the string "foo bar ." (i.e, "foo<space>bar<space><period>") like ["foo", "bar"], since "bar" is between two spaces, and there is nothing between the last space and the end of the sentence.

However, from the example in the decoded text from puzzle 01, I'm getting the sense that you're expecting us to tokenize this like: ["foo", "bar", " "]? But that doesn't seem to be correct given the definition of word tokens.

Similarly, I would think that "foo bar \n" (i.e, "foo\<space>bar<space><space><space><newline>") would be tokenized like ["foo", "bar", " ", "\n"] since "bar" is delimited by spaces, then the next word is a space that is delimited by spaces, then the last word is the newline, since it's between a space and the end of the sentence.

Edits: fixed markdown-related bugs

[larryboyko]

Thanks for the feedback! You make an excellent point.

The catch is that (as this is a transposition cipher) no characters can be thrown away. The text points to this in two ways: The hint ("If you're doing it right, deciphering and enciphering should be the same process: if you re-encipher an enciphered text, you should end up with the plaintext you started with"), and the waiver itself, which has some extra spaces acting as indentations when deciphered.

So, the tokenizations you suggest (["foo", "bar"] and ["foo", "bar", " ", "\n"]) are destructive, and not reversible. But they DO follow the letter of the instructions! So I think the instructions need improving.

I could explicitly state, "consider a word to be 1) a space, or 2) whatever characters in a sentence fall between two spaces, or between a space and the start or end of the sentence", but that feels a clunky -- calling a space a word.

Alternatively, I could extend the definition of a word to be "whatever falls between two spaces, or between a space and the start of end of the sentence -- even if it is zero characters long". Then you'd get ["foo", "bar", ""] in the first case, and ["foo", "bar", "", "", "\n"] in the second. And nothing is lost, as these can just be rejoined with single spaces.

But I'm leaning towards this edit:

The rest of the story's text has been enciphered in the same way as the waiver above. (For the purposes of deciphering, consider a sentence to be whatever characters fall between two periods, or between a period and the start or end of the ciphertext; and consider a word to be whatever characters in a sentence fall between two spaces, or between a space and the start or end of the sentence. But the spaces themselves should be preserved!)

(Hint: If you're doing it right, deciphering and enciphering should be the same process: if you re-encipher an enciphered text, you should end up with the plaintext you started with. You can test your method by unscrambling, and rescrambling, and unscrambling, the waiver above.)

[romanows]

I haven't gotten back to fully implement the solution for this, but I had some thoughts. The general solution is pretty clear, it's just how exactly to deal with spaces and newlines that wasn't clear to me. (Dealing with whitespace isn't the satisfying part of the puzzle, hence my whining :) ). If I knew that the example ciphertext decoded into plaintext where there were N lines, and some of which were indented, I think it would be much easier to figure out the correct algorithm.

I think you could make this happen simply by including text like: "Hint: the waiver above has text on three lines, two of which are indented by two spaces". Or you could shorten the waiver text (to make it easier to manually manipulate) and include a hash that readers could check to make sure that they had basically recovered the test case, before implementing and debugging their algorithm. Or you could have Dimity be required to write her acceptance text, and sign it with her name, indented? Like:

"Oh, I get it!" said Dimity; and after a minute of deciphering, she wrote on the bottom of the page:

I hereby understand and solemnly agree to the terms of this contract.
  - Dimity Jones

I'm not sure I'd attempt to hint how to handle words of whitespace even, since that's confusing. Note that your proposed definition could be interpreted like "foo<space>bar<space><space><space><newline>" -> "bar<space>foo<space><space><space><newline>". The spaces are preserved and the cipher process is reversible. But it doesn't give you the output that you are looking for.

Anyway, maybe I'm trying to make this too comfortable for me to solve, and I should be more willing to write a few candidate implementations until I get one where the decrypted hashes match. I'm learning a new programming language while reading the novel, which I think maybe is where some of the friction is coming from.

[larryboyko]

I'm not sure I'd attempt to hint how to handle words of whitespace even, since that's confusing. Note that your proposed definition could be interpreted like "foo<space>bar<space><space><space><newline>" -> "bar<space>foo<space><space><space><newline>". The spaces are preserved and the cipher process is reversible. But it doesn't give you the output that you are looking for.

This is the crux of the problem, I think. Are the spaces like unmoving rocks in a riverbed around which the water-words get rearranged, or do they, too, go with the flow? I'd been hoping that a glance at the waiver would indicate that it is the latter that's happening: liability of Limitation.risk own my [...] If the spaces didn't also move, there would be a space after the period, as is usual with prose.

Perhaps I'll allude to the indentations as you suggest. And perhaps, instead of merely saying the spaces should be preserved, I'll find a way of hinting that they should also be included in the reversal of elements.

[frogpelt]

I'm also having an issue with puzzle 2. The quote at the very end and the period in front of ".time!" is really confusing when trying to figure out how to approach this. Does the period go with sentence after it?

[larryboyko]

mild spoiler

Just think of the periods as stationary delimiters. The period before "time!" will end up adhering to the previous sentence.

Chunk the sentence into 1. spaces, and 2. "words" (groups of not-spaces), then reverse those chunks. Thus the final "sentence" of the waiver will start with "\n, <space>, <space>, <space>, Finally, <space>, etc., and end with time!

[frogpelt]

Ok thanks for that. I think I'm really close. I was able to decipher and re-encipher the waiver using the same code and end up with what I started with.

However, when I perform it on the next section of the book I get this: "Well done! But before I return you to the story, I'll get you to perform another decipherment, as described in the following lightly enciphered roF
"""
And the wrong checksum....

[frogpelt]

Ok now I figured that part out. The phrase:

"roF
"""
paragraph:"

has new lines but no spaces which was throwing me off.

I now have the correct start to the next section. But my checksum still doesn't match!!

I'm guessing it's not gonna work to have you walk me through the entire book.

It might defeat the purpose of this project but maybe it would be helpful to create a walk-through/hint book for when someone gets stuck.

[larryboyko]

I'm guessing it's not gonna work to have you walk me through the entire book.

I'm happy to help where I can, as it helps me, too, to find potential bugs or see if instructions are ambiguous.

Do you want to email me your code, and output?

[frogpelt]

Thanks I will do that. I got stuck and lost interest for a couple of weeks (story of my life!)