Explore and documentation what Backtrackable means

[MangelMaxime]

• Describe what Backtrackable does
• Create examples / matrice of what is the result of the different composition functions in regards of Backtrackable state
  • oneOf
  • map
  • map2
  • andThen
  • etc.

[MangelMaxime]

Parser.token seems to be a good place to understand what Backtrackable means.

However, I would say there is a conflict in the naming between progress and Backtrackable. Also, why is the decision based on the input str and not on result of findSubString.

Is it to protect user code from empty tokens ?

    let token (Token(str, expecting)) : Parser<'Context, 'Problem, unit> =
        ParserFunc
        <| fun state ->
            let progress = not (String.isEmpty str)

            let result = findSubString str state.Offset state.Row state.Column state.Source

            match result with
            | SubStringResult.NoMatch _ ->
                ParserStep.Failed
                    {
                        Backtrackable = false
                        Bag = fromState state expecting
                    }

            | SubStringResult.Match result ->
                ParserStep.Success
                    {
                        Backtrackable = progress
                        Value = ()
                        State =
                            { state with
                                Offset = result.Offset
                                Row = result.Row
                                Column = result.Column
                            }
                    }