Clap derive should parse markdown doc comment into normal text

[ducaale]

Maintainer's notes:

• Workaround: verbatim_doc_comment attribute

Blocked on finalizing details of StyledStr, including

• Allow good/warning/error/hint color to be customized #3234
• Allow styled text in template #1433
• The header for my examples are not colored like the others #3108

---

Please complete the following tasks

• I have searched the discussions
• I have searched the existing issues

Rust Version

rustc 1.48.0 (7eac88abb 2020-11-16)

Clap Version

3.0.0-beta.2

Minimal reproducible code

use clap::Clap;

#[derive(Clap, Debug)]
#[clap(name = "xh")]
struct Cli {
    /// Optional key-value pairs to be included in the request.
    ///
    /// - key:=value to add a complex JSON value (e.g. `numbers:=[1,2,3]`)
    /// - key@filename to upload a file from filename (with --form)
    /// - header:value to add a header
    /// - header: to unset a header
    /// - header; to add a header with an empty value
    ///
    /// A backslash can be used to escape special characters (e.g. weird\:key=value).
    #[clap(value_name = "REQUEST_ITEM")]
    raw_rest_args: Vec<String>
}

fn main() {
      let _ = Cli::parse();
}

Steps to reproduce the bug with the above code

cargo run -- --help

Actual Behaviour

ARGS:
    <REQUEST_ITEM>...
            Optional key-value pairs to be included in the request.

            * key==value to add a parameter to the URL \n * key=value to add a JSON field (--json)
            or form field (--form) * key:=value to add a complex JSON value (e.g.
            `numbers:=[1,2,3]`) * key@filename to upload a file from filename (with --form) *
            header:value to add a header * header: to unset a header * header; to add a header with
            an empty value

            A backslash can be used to escape special characters (e.g. weird\:key=value).

Expected Behaviour

ARGS:
    <REQUEST_ITEM>...
            Optional key-value pairs to be included in the request.

            * key==value to add a parameter to the URL
            * key=value to add a JSON field (--json) or form field (--form)
            * key:=value to add a complex JSON value (e.g. `numbers:=[1,2,3]`)
            * key@filename to upload a file from filename (with --form)
            * header:value to add a header
            * header: to unset a header
            * header; to add a header with an empty value

            A backslash can be used to escape special characters (e.g. weird\:key=value).

Additional Context

In StructOpt, it possible to use the long_help attribute to preserve newlines but I couldn't find something similar in clap-derive.

Just found that it is called long_about in clap-derive.

Debug Output

Will add if requested

[pksunkara]

There is quite a bit of discussion in #2401

[epage]

One of the challenges with this is we'd basically need to pull in pulldown-cmark which is quite a large dependency. This would need to be put behind a feature flag.

[ducaale]

My main goal was to preserve newlines in the doc comment and that should be achievable with the verbatim_doc_comment attribute.

Example

// clap = { version = "3.0.0-rc.4", features = ["derive"] }

use clap::Parser;

#[derive(Parser, Debug)]
#[clap(name = "xh")]
struct Cli {
    /// Optional key-value pairs to be included in the request.
    ///
    ///   • key:=value to add a complex JSON value (e.g. `numbers:=[1,2,3]`)
    ///   • key@filename to upload a file from filename (with --form)
    ///   • header:value to add a header
    ///   • header: to unset a header
    ///   • header; to add a header with an empty value
    ///
    /// A backslash can be used to escape special characters (e.g. weird\:key=value).
    #[clap(value_name = "REQUEST_ITEM", verbatim_doc_comment)]
    raw_rest_args: Vec<String>,
}

fn main() {
    let _ = Cli::parse();
}

Output

USAGE:
    temp-clap [REQUEST_ITEM]...

ARGS:
    <REQUEST_ITEM>...
            Optional key-value pairs to be included in the request.

              • key:=value to add a complex JSON value (e.g. `numbers:=[1,2,3]`)
              • key@filename to upload a file from filename (with --form)
              • header:value to add a header
              • header: to unset a header
              • header; to add a header with an empty value

            A backslash can be used to escape special characters (e.g. weird\:key=value).

OPTIONS:
    -h, --help
            Print help information

I wouldn't mind if this issue was to be closed.

[epage]

I think its still an interesting topic for us to weigh out. I did call out the workaround in the issue so its more discoverable.

[I60R]

We may need only a subset of markdown features in CLI parser:

• newline escaping (implemented in Allow to escape newline with \ like in markdown #3228)
• lists as suggested here (should be relatively easy to implement, even with nesting)
• bold, italic, bold-italic, inline code - if colors enabled (not sure whether it's worth effort, but why not?)
• fenced code blocks (without syntax highlighting, of course)

There also could be some mechanism to cut off help message from doc comment as @epage suggested here — IMO markdown headers are a perfect mechanism for that e.g.:

• without any header everything goes to -h --help output
  • that's the same as if everything was placed under # Help header
• if # Long help is specified that section will be appended to --help output (only)
• we may also have # Manpage section (if clap will provide some mechanism for manpage generation)
• text under all other headers will be ignored by clap

---

I think this subset should satisfy 99.9% of all use cases of clap, everyone should like it, and it doesn't seem to be so complicated to require pulldown-cmark or any other heavy dependency.

[epage]

My experience writing one off parsers has made me a bit cautious of doing so. I threw together some md parser benchmarks to see what might be small enough for being an optional dependency. minimad looks tempting.

[epage]

Another thought on markdown support, should we support it at runtime, compile time, or both?

The core of this Issue is our line breaks which is a problem exclusive to the derive API. For that, we could build markdown parsing directly into clap_derive.

Another step up is markdown formatting. For that, we'd want to expose it to both APIs. Say we generalized our Colorizer struct and accepted it for all our user-facing inputs (for #1790, #1433), users can do whatever styling they want, though it might be a bit arduous. We then provided a markdown-parsing macro that would code-generate Colorizer. clap_derive would then have this built-in for all doc comments.

The main benefit of this is that this would remove the binary-size overhead of the markdown parser. We'd still need to compile it (though it'd be optional). Though this would mean we wouldn't regress in --help and error performance by introducing markdown, the performance would have to be pretty bad for us to care.

This has the added benefit that we wouldn't need yet-another Setting in the API for doing controlling this at runtime.

[epage]

Another question for us to answer in this issue is how disruptive would markdown parsing be. While --helps output is not considered "stable", we shouldn't dramatically alter users crafting of their help output. To what degree of markdown parsing can we add in a patch, a feature release, or a breaking release?

[epage]

One design problem I've been running into is how to render markdown at compile time when the colors aren't known until runtime.

I think the way to do this is to allow a fn(fmt, palette) -> fmt::Result to be passed into StyledStr and the derive would generate this, parsing the markdown at compile time but rendering the styles at runtime.

[pronebird]

I'd only ask for italic and bold for basic formatting.

[nrdxp]

In my case I just want to add a space in a list without having to add a newline between the bullets and without having to use verbatim doc comments (so wrap_help is respected properly)

[ModProg]

Is there some specification on how this should work? Should we fully parse markdown, or should we just implement the parts that make sense, i.e., lists, bold, italic? I'm currently quite annoyed by lists not working so I'd be willing to invest some time into implementing some solution here.

[alerque]

How established is CommonMark in Rust docs contexts? I would suggest pulldown-cmark or, if another Markdown parser is established in Rust context whatever that is, as a full flow Markdown parser makes more sense than a "whatever makes sense/minimalist" approach. Markdown is notoriously hard to actually get right even if small things like emphasis or list parsing.

[epage]

Subsets of markdown were covered earlier in the thread
#2389 (comment)

[ModProg]

#2389 (comment) you also cautioned against writing a custom parser should we just use pulldown-cmark and ignore the ones we don't support?

Also should this be configurable, for some the mapping to ANSI is pretty clear e.g. bold or italic while for others it's unclear e.g. code_block should that set a backgroun? or keep the quotes ` ?

Another thing to consider would be should support be implemented for html tags like in color_print: https://docs.rs/color-print/latest/color_print/index.html#list-of-accepted-tags

[epage]

#2389 (comment) you also cautioned against writing a custom parser should we just use pulldown-cmark and ignore the ones we don't support?

Looking at md parser benchmarks, I would be curious about how well we could get by with minimad.

Details like code_lock would need to be figured out. In a way, it is another form of literal.

It would be good to have a way for people to set colors but I would be worried about full HTML support.

[ModProg]

It would be good to have a way for people to set colors but I would be worried about full HTML support.

The only reason I mentioned color_print is that it is referenced in the clap docs, and it could be nice to also be able to use it inside of markdown.

I would agree that any block elements would be annoying to deal with, so only inline ones could work. Maybe adding some tags for all the styles definable could be useful, i.e., <header>, <error>, usage, ...

Details like code_block would need to be figured out. In a way, it is another form of literal.

Makes sense, though one could also add the different markdown elements to Styles, e.g., if someone wants to color their bold text red.

The other question is should mark down parsing be enabled at runtime as well? If not, the compile time parsed markdown needs to be converted like you suggested here:

One design problem I've been running into is how to render markdown at compile time when the colors aren't known until runtime.
I think the way to do this is to allow a fn(fmt, palette) -> fmt::Result to be passed into StyledStr and the derive would generate this, parsing the markdown at compile time but rendering the styles at runtime.

Even if we support markdown parsing at runtime, one might like to move that effort to compilation where possible, but we could also just implement it for runtime first and then check if there is a relevant performance penalty end disable it if so.

The whole markdown feature should probably be behind a feature flag anyway.

Would you be in favor of building a PoC for markdown parsing to test it out?

[epage]

I suspect we should do this all at compile time though either direction comes with its problems.

One combination of challenges

• I intentionally switched StyledStr to only hold ANSI escape codes
• If we want to provide style classes like "header", the conversion of that to ANSI is only really available at runtime

iirc there were more. Feel free to play around and come back with more ideas. I feel like I'd want things settled a little more before adding an unstable feature for this.