9.10b5 General Purpose Block Feedback

[facelessuser]

Beta 5

Documentation: https://facelessuser.github.io/pymdown-extensions/extensions/blocks/

The purpose of this discussion is to allow feedback for the latest iteration of General Purpose Blocks 9.10.b1. It should be noted that any and all feedback is welcome.

Changes in Beta 1:

• NEW: HTML General block now accepts Emmet style attribute: /// html | div.class#id[name=value].
• NEW: Block attribute specifier is renamed from attributes to attrs.
• NEW: Remove colon_syntax option and cement that we are using /// format moving forward.
• NEW: Revise available validators for Block options. Remove unnecessary validators and replace some with new ones.
• NEW: Simplify argument API.
• NEW: Block extensions can now be registered directly as normal Python Markdown extensions.

Feedback Needed

General feedback is desired. Are there any bugs we missed, or some functionality we overlooked? The next release plans to make this public as a set of experimental extensions. Catching any big issues now is the goal.

[waylan]

This looks great! 👍

I see you have not yet updated the example code which demonstrates how to define config options for the Admonitions and Details extensions. Also, the "Per Block Options" table for each block type still needs to be updated for the $ => attrs change. But my understanding is that those are just documentation updates and the code has already been changed.

The changes to the extension API look great. I think the change to make each block its own standalone extension was the right move. And you implemented the API for that just as I expected.

I may have some time over the next few days to play with an idea I have for a block type. I'll report back if/when I get a chance to do so.

[facelessuser]

I must have missed it.

[facelessuser]

Maybe you are hitting an outdated cache? I'll try to find the $ case in the docs, but I'm pretty sure I replaced them all...

[facelessuser]

Never mind, I'll fix the docs, I found the issues.

[facelessuser]

I also have some extension registration that is outdated 😞

[facelessuser]

Docs fixed

[waylan]

You can find my first attempt at building an extension with this in this Gist. It sort-of works in that it only recognizes some of my test blocks but not others and I don't know why. I eventually gave up. I was making notes as I worked on it and all of those notes are preserved in a separate file in the gist. Some issues I noted I later resolved as I note later on. However, I left the entire set of notes intact to properly convey my entire experience. There seems to be a lot of holes and inconsistencies in the documentation. Eventually I found myself using the source code as a reference and mostly ignoring the docs. I started with the included block types as examples, but eventually went to the underlying parent Block class itself to understand what was expected.

My general observations of the project (ignoring the documentation issues) is that it is great when it works. It has never been so easy to create support for a new block type in Markdown. However, it appears that something is causing the parser to be way to strict and I can't seem to get consistent results. Work those kinks out and this will be a great tool and asset to the community.

[facelessuser]

All relevant changes based on the feedback:

• Better describe on_create and on_add
• Show function signatures of validators and describe some validators
  a bit better: type_string_in and type_html_attribute_dict.
• Cover attrs in api page and also correct some misleading info about
  attrs on main page.
• type_string_in no longer requires a callback, but instead exposes
  an option to control case insensitive comparison.
• Ensure that when attrs is applied to an element, it accepts the
  classes on the element as manually added by the developer.
• type_html_attribute_dict runs type_html_indentifier on id.

I have not pushed new docs as there are some functional changes reflected in the docs that are not in Beta 1. If desired, I can push a Beta 2.

[facelessuser]

A small correction in the earlier statement, attrs does not join the classes in the final result, that only occurs in order to combine them with existing class values on the wrapping element. They are accessible to the user as a list of strings and are the only attribute that is available as a list of strings. This is noted in the documentation now.

[waylan]

Thanks. I'm a little embarrassed that the issue was some silly typos. I have tried all three possible values for ARGUMENT, but they didn't make a difference because of the = instead of :. Regardless, up until the end, I was getting errors as I worked through all the other issues. Then when I finally eliminated all the errors and it still wasn't working I just gave up in frustration. In any event, it sounds like the feedback has helped make things better.

[facelessuser]

No worries. It's a lot of stuff to go through and absorb all at once. I'm glad to see that it is fairly easy for others to get up and running. I also appreciate the critical eye on some of the finer details. You spend too much time on something, you find yourself to close or to tired to see some minor improvements that can be made.

[waylan]

I have not pushed new docs as there are some functional changes reflected in the docs that are not in Beta 1. If desired, I can push a Beta 2.

If you could when it is ready, that would be appreciated. I don't intend to keep working against this version but will be willing to try again with Beta 2.

[facelessuser]

Functional changes for the next release. When processing attrs, we will not run type_html_classes on the wrapping element's classes. Whatever the developer has manually added will be untouched. type_html_classes will only be run on class entry in attrs. Documentation will note how attrs is processed.

[facelessuser]

type_string_in will now compare strings case-insensitively unless insensitive is set to #!py3 False. No callback is needed anymore.

[facelessuser]

type_html_attribute_dict now runs type_html_identifier on id as well.

[facelessuser]

I released Beta 2

[waylan]

Just ran into an interesting hiccup with my experimental Bootstrap Alert block. I added the on_markdown event and set it to inline. That resulted in the body being set as the text if the wrapping div, which better follows Bootstrap's spec. However, in alerts that had a title set, that meant that the title came after the body because it was set as the text of the container element. Of course, the way etree works, the content would need to be assigned as the tail of the title instead. Not sure how the general purpose extension should handle that. As I understand it, on_add won't help here. because that doesn't address the tail.

In my case, I just added a check for a title in on_markdown and always return block if a title is set. That mostly works for my specific use-case, but won't for everyone. I say mostly, because, ideally, with a one paragraph body, the content should ideally be set to the tail of the title. It's a small inconvenience for a simple solution.

I would have actually preferred for the auto option to result in inline behavior if the body consisted of only one paragraph and block if there was more than one paragraph. But I suspect that would be more complex than either of us would want to implement. I can live without it but would not be surprised of other users expect similar behavior. While one could certainly include more sophisticated logic in the on_markdown method, it's not clear how one would get access to the information one needs about the content from within that method.

I suppose one could use on_end for this. Have in run with block and then check the content and rearrange it in on_end if there is only on child p element. But that feels more like a hack to me that the official solution. What are your thoughts about this @facelessuser.

[waylan]

Regarding auto, I think maybe the issue is a lack of understanding on my part about how it decides. My thinking was that since the content only consists of a single paragraph, then it should recognize that and only insert the inline content of the paragraph. However, I suppose the only way that it could know that the content consists of a single paragraph is if the content is first parsed as Markdown.

Since the content is returned as a p element, then it correctly identifies that as inline content. But what I don't understand is under what conditions would it ever receive inline content only and cause auto to use inline? Because anytime Markdown processing happens it will always wrap any content in a block-level element.

[facelessuser]

So, when block is used, all content is parsed as block items, which means sent through Python Markdown blockprocessors.

When inline is used, all content is attached to the parent element and skips the block parser. That means no more block processors will be applied to the block.

The content is just redirected. Basically, the same kind of logic used in md_in_html and such to identify "block" content vs "inline" when auto is enabled.

It's probably not as intelligent as you may have thought it was. What this means is that inline tags will not parse block Markdown stuff, just inline parsing. So it doesn't just control wrapping elements in <p> or not, it dictates how the Markdown parser will see that content. Will it go through block parsers, or head straight through inline.

It is impossible for us to have full understanding and context of the input text stream. Do to the way lists and such work, we can never fully look ahead to see all the content that is coming. We don't know all the plugins that are enabled and how they work, so it makes it hard for us to count blocks while the data is coming through. Only during on_end do we truly know what we are dealing with.

[waylan]

I just looked at the code for this and now I get it. auto relies on the wrapping element (returned by on_create and/or on_add) , not the content as I understood it. So if the wrapping element is either an inline element or cannot contain block elements, then auto results in inline behavior. In other words auto will prevent a p from being inserted in a p.

[facelessuser]

Yeah, it prevents it from being inserted in a p, but it also prevents the content from being touched by any blockprocessors as well.

[facelessuser]

Better put, if you only want to control what gets put into a p, then maybe you want on_end. You can parse all the content and sort it out at the end.

If you want to treat the element as an inline element, and don't care about processing block Markdown content (blockquote, block code, multiple paragraphs, etc.) then inline is fine.

[facelessuser]

Released a Beta 3

[waylan]

I am not sure how to proceed with a new block I am playing with. I am looking to create a block for Bootstrap's Carousel. At first I created a parent block with the idea that child blocks would be the various slides of the carousel. After all, most options are specific to the carousel not the individual slides. In fact, I realized that each slide is nothing more than an image with a few specific classes assigned to it.

Therefore, I initially planned on defining the slides as nothing more than Markdown images (![alt](path.jpg)). The body of a carousel would simply be a number of images, I attempted to iterate over the img tags in on_end (as you can see here). Then I realized that inline parsing has not happened yet so that won't work. I came up with three possible work-arounds, but I don't particularly care for any of them.

1. I could create some companion treeprocessor which runs after inline parsing to deal with the img elements.
2. I could create a slide block which must be a child of the parent carousel block. However, in most cases a slide only needs to be an image with alt text, so the nested block seems unnecessary and would likely be a source of complaint from users. That said, Bootstrap does include the option to define captions which get overlaid on the slides. The text for a caption could easily be the body of the child block and would be an easy way to justify the nested block.
3. I could create a slide block which does not need a parent. The first one would create the parent wrapper and any subsequent slide blocks would simply insert themselves in the existing parent (not unlike the tab blocks work now). The problem with this is that the options are all specific to the parent wrapper so what do I do if they are defined on a slide other than the first one?

Obviously I can make a choice myself. I'm just curious what you would do here?

Regardless, there are a few potential deficiencies in the API for each of the above approaches as outlined below:

1. It would be nice if the API included a hook which ran after inline parsing. Perhaps the wrapping code would include a treeprocessor which would go through each block instance and call an event method, if defined, Then I could simply move my logic from on_end to whatever this new method would be called.

2. How would I enforce requiring the child slide to be nested in a parent? The on_parse event can return False to force a block to not be recognized/parsed as one. However, the parent element is not available there as it is in on_create. So how would I do that?

3. I looked at the source code for tabs and noticed a few things that weren't clear at first (like the use of the tracker attribute of the class instance). Then I noticed that there are various attributes to the Block class that are not mentioned in the documentation at all. And the comments in the code seem to be out of sync with the code itself. Specifically, store is documented in the code comments but doesn't exist and config exists but isn't documented (or used anywhere AFAICT).

   pymdown-extensions/pymdownx/blocks/block.py

   Lines 190 to 201 in 5a40bc0

   	def __init__(self, length, tracker, md, config):
   	"""
   	Initialize.
   	- `store` allows us to store content until all content is found.
   	- `length` specifies the length (number of colons) that the header used
   	- `tracker` is a persistent storage for the life of the current Markdown page.
   	It is a dictionary where we can keep references until the parent extension is reset.
   	- `md` is the Markdown object just in case access is needed to something we
   	didn't think about.
   	"""

   I'm assuming I would want to use tracker here. But I'm not clear on how that works. And the fact that it is not mentioned in the documentation makes me think maybe it is not stable and isn't something I should rely on.

[facelessuser]

I guess if we're going to do a Treeprocessor, we might as well run it on all blocks. I think what we could do is simply queue up completed blocks for the on_inline_end event. The treeprocessor would just work through the queue sending on_inline_end the parent node. I think that should be reasonable.

[facelessuser]

Yeah, adding an entire treeprocessor to process a block after inline parsing is certainly excessive, so there is now a new on_inline_end event. It'll be in the next Beta 4: 995ecae.

If you have a specific, additional action item that you'd like to request, please do make the request. I'm hoping Beta 4 will be the last Beta, but we will of course roll out more if needed.

[waylan]

Awesome. I encountered a problem the current API couldn't address and brought it up without any specific expectations and you provided a solution. Thank you, Now I will go dig in and see if I can use it like I hope to.

[waylan]

Well that worked exactly as expected. 👍

[facelessuser]

Hopefully, that address the biggest sticking points and we have a decent foundation for getting plugins done 🤞🏻. Thanks for taking the time to exercise the API, I think a lot of good things have been caught. Let me know if there are any other major issues, if not, we might be able to get this out publically.

[facelessuser]

Beta 4 just released. Will probably be up on PyPi in a few minutes.

[waylan]

I just encountered a situation where I needed to have a validator accept either a boolean or a known string (for example, one of True, False, or 'auto'). There could be other occasions where one might need to chose from a list of strings or None. Therefore, I'm wondering if we should alter the type_string_in validator to also accept boolean and/or None as valid values. I'm thinking we could add 2 more flags to the validator:

1. bool, which would validate True and False values when set to True (default False), and
2. none, which would validate a None value when set to True (default False)

Essentially this would combine the functionality of the existing type_string_in with either type_boolean or type_ternary depending on which options are chosen.

Obviously, I can (and have) create my own validator to do this. But I think this might be common enough to add to the API proper.

[facelessuser]

Whatever we end up doing, I think it can happen in the final unless we have a reason to release a new beta.

[facelessuser]

I think what I've decided is to add a generic type_multi that can be used to pair any type:

type_multi(type_boolean, type_string_in(['this', 'that']))

It's a bit more versatile and basically gets you what you want.

[facelessuser]

New validator added: f17d03d

[waylan]

Looks good. That is more versatile than my suggestion. 👍

[facelessuser]

I'm going to hold on to the change, for now, to see if any other issues need addressing. Depending on the type of feedback, the next release may be the official or another beta.

[waylan]

I was playing with the options on one of my blocks and checking the output. I was commenting out some of the options (which is valid YAML) to confirm the default when an the option is not defined for a block. However, I inadvertently commented out all options and that caused it to not be recognized as a block. Presumably, by commenting out the entire YAML section it was considered to not be valid YAML. I'm wondering if that should be considered a bug as technically, it is still valid YAML, albeit, not particularly useful. In the end, not a big deal either way, but I thought I would mention it.

I checked and YAML returns None if all lines are commented out

>>> import yaml
>>> yaml.safe_load('foo: bar')
{'foo': 'bar'}
>>> yaml.safe_load('#foo: bar')
>>>

Looking at your code...

pymdown-extensions/pymdownx/blocks/__init__.py

Lines 60 to 67 in 5a40bc0

	try:
	frontmatter = yaml.safe_load(string)
	if not isinstance(frontmatter, dict):
	frontmatter = None
	except Exception:
	pass
	return frontmatter

It would seem that returning None would not cause an issue here. So you must be checking something somewhere else which is causing it to fail. Maybe in your regex which collects the YAML section. Although that seems unlikely because it works fine as long as any single random line in the YAML is uncommented.

This fails (it is not recognized as a valid block):

/// carousel
    #attrs: {id: 'random'}
    #indicators: true
    #fade: true
    #autoplay: carousel

![img 1](img1.jpg)
![img 2](img2.jpg)
///

However, this works fine (as does any other random line being uncommented):

/// carousel
    #attrs: {id: 'random'}
    indicators: true
    #fade: true
    #autoplay: carousel

![img 1](img1.jpg)
![img 2](img2.jpg)
///

Completely removing the YAML also works fine (which I would have expected to match the behavior of commenting all lines out)

/// carousel

![img 1](img1.jpg)
![img 2](img2.jpg)
///

BTW, the fact that I am now reporting minor details like this suggests to me that you are getting very close to a release.

[facelessuser]

As YAML can return various objects, we are specifically checking whether we are getting a dictionary signifying that we are getting an expected YAML return. Anything that triggers the YAML config will expect the end result to be a dictionary of options

I guess, we could make a change to allow null (or Python None) to get returned, but if we get anything else, str, list, etc. we will have to invalidate the result.

[facelessuser]

Frontmatter will now allow a YAML null return: da94db7

[waylan]

I guess, we could make a change to allow null (or Python None) to get returned, but if we get anything else, str, list, etc. we will have to invalidate the result.

I understand that, but that's why I am confused. As we can see in your code which I reference above (blocks/__init__.py#L60-L67), if anything other than a dict is returned by YAML, you are returning None. So where is the error being raised for a list', str` etc.?

[waylan]

Wow that was fast. Thanks.

[facelessuser]

I understand that, but that's why I am confused. As we can see in your code which I reference above (blocks/init.py#L60-L67), if anything other than a dict is returned by YAML, you are returning None. So where is the error being raised for a list', str` etc.?

None signified we couldn't acquire YAML; therefore, the config was invalid. Now, if we don't get an error, and None is returned, we'll change it to an empty dict which will signify YAML parsed okay.

[waylan]

This one is a long-shot and I'm not even sure it is a good idea. I have a block type which would likely contain only an argument in most cases. Like this:

/// name | argument
///

That works fine. However, As there is no need for any additional lines, I am wondering if it would make sense to allow the entire block to be defined on one line. Like this:

/// name | argument ///

I think I find that easier to read, especially if multiple of them occur one right after the other in a document. Obviously, if anything else needed to be defined (options and/or body content) then the existing syntax would still be required. This would only apply to a block which only has an argument defined.

Any thoughts?

[facelessuser]

This one is a long-shot and I'm not even sure it is a good idea.

Yeah, I did think about this early on, but I don't think it is a good idea...or maybe I just don't like it? I'm not really sure.

"Argument" has no restricted syntax, you can put whatever you want in it. It is up to the plugin to decide what is valid and invalid, so anything after | is valid. To allow this, we need to restrict the input with additional rules.

Could we add rules to allow this? Sure, but my current feeling is to not bother. I guess if there were enough people calling for something like this, we could always add it in the future. Since ending and start must match in count and an inline ending would have to be at the very end of the line, you could probably still specify things like below:

/// name | Some weird title that ends with /// ///

/// name | Some weird title that ends with unmatching //////
///

I just don't like it.

[facelessuser]

I guess this may be more sensitive to whitespace at the end, unless you accept <matching ///>[ ]*$. Regardless, I understand the initial draw, I'm just hesitant.

[waylan]

Hmm, hadn't thought of that. Although, one could us backslash escaping:

/// name | Some weird title that ends with \/// ///

Of course, that assumes the argument value gets inserted into HTML as-is and will get Markdown's inline parsing. I suppose it is possible that won't happen in all cases, which means the backslash escape won't always make sense.

This could always be added later if it becomes highly requested. I can work with whatever decision you make here.

[facelessuser]

Escape wouldn't be necessary in this case as the last slash group is anchored at the end, and those would be used

/// name | Some weird title that ends with /// ///

Where escaping would be more required is in a situation like this:

/// name | Some weird title that ends with slashes, but we don't intend them to terminate the block ///
///

But again, I'm certain we could work around it, but I prefer to not have to work around it. We'll see if this ends up being a requested thing.

For now, I'm going to pass on this, but leave it open as a possibility. I'm not necessarily promising to implement it if X% of people request it, but I'm not closing the door either.

[waylan]

I asked this question previously but it was buried in a bigger issue, so I'll ask it separately here.

Suppose I have a block type that is only valid as a direct child of a parent block type. How can I validate that from the child block? Of course, the on_parse event can return False to force a block to not be recognized/parsed as one, which is the behavior I want if my check fails. However, the parent element is not available there like it is in on_create. So I would presumably need to make this check from on_create. But how would I force a failure from on_create? Or is there some other way to accomplish this?

[facelessuser]

Ah, I think I misunderstood the question earlier. I don't think there is currently a way, but we could certainly make one. Maybe on_parse is a bad name. What if we call it on_validate and pass in the parent?

[waylan]

After I posted this question it occurred to me that I might be able to use self.tracker for this. When the parent is first created I could set a value in self.tracker which indicates that a parent is currently open and accepting children. Then in on_end of the parent I could update the status to indicate that the parent is closed and no longer accepting children. In that way, child blocks could check self.tracker and fail if a parent is not currently open. The only issue is that there is no way to confirm that the child is a direct child. It could be nested multiple levels deep. But that may be enough for most use cases.

Maybe on_parse is a bad name. What if we call it on_validate and pass in the parent?

That would work too. Regardless of whether you pass parent in or not, I agree that on_validate makes more sense as the name of the event.

[facelessuser]

I think we will go with on_validate. While your use case specifically requires an element to be a child of a parent you created, it is possible that others may want to use any element as context. And if we pass in the parent, you can tell if you are a direct child or not.

[facelessuser]

Change is here: 3d89440

I imagine that is the last big change that we need to make this reasonably accessible. I think it is a low-risk change, so I'll save it for the final release.

[waylan]

I have a block where body content is optional. However, if it is present, I need to wrap it in a wrapping element which is a child of the block's root element. However, that wrapper should not exist if no body content exists. Therefore, I want to conditionally add the wrapper in on_create and return it from on_add. Is there any access from the Block class to determine if any body content exists for the block?

I suppose I could check the element in on_end and delete it if it is empty, but I would prefer to just not add it at all if it is not needed.

For clarity, here is an example:

/// foo | Title
body content
///

The above becomes:

<div>
Title
<div class='body-wrapper'>
body content
</div>
</div>

However, for this input:

/// foo | Title
///

The output should be:

<div>
Title
</div>

Notice that the <div class='body-wrapper'> is not present as there is no body content.

[facelessuser]

We do mention it here: https://facelessuser.github.io/pymdown-extensions/extensions/blocks/api/#on_markdown-event, but I can make it more explicit and express some of the caveats.

[waylan]

Yeah, I would suggest adding a note about how to get access to that raw content. Of course, the note should mention when and where one could access that content as well (only from on_end or later).

[facelessuser]

I think I'm going to add a small change for raw mode to also dedent the content by tab length automatically (if indented) and suggest indenting content in raw blocks as a way to survive things like Python Markdown's HTML processor. Since it is a preprocessor, and it avoids indented code, it's probably the only reasonable approach to get raw content through.

[facelessuser]

Beta 5 is released.

I went ahead and made the raw change. If a block reports its content as raw, the content should be indented. We'll accept it if it isn't, but to avoid the HTML parser from messing with the content, indentation is recommended. Content will be dedented right before on_end, but no more than the current tab length so that you can specify intentional indentation in a raw block.

[facelessuser]

I think if the HTML processor was ever reworked as a real block processor, this indentation requirement for raw could be retired. I'm fairly certain it could be after exploring fenced blocks in this extension. I'm not quite as familiar with all of the inner workings of that extension though, but I'm fairly certain it could be reworked, though I'm sure no one is wanting to rewrite it again anytime soon 🙃. Who knows, maybe one day I'll sit down and try and prototype one.

[quancore]

@facelessuser Hey, I am trying to use the beta version of HTML block to build bootstrapping card view but I have problems with nested divs

/// html | div.card.text-white.bg-secondary.mb-3[style="max-width: 50rem;"]
/// html | div.card-header
///
/// html | div.card-title 
///
///

This setup in md file produce this:

This is not what I expected, there should be one big div and two children div. Am I missing something?

[facelessuser]

@quancore, this is a general block question, not HTML specific, and is covered in the docs where it talks about the nesting: https://facelessuser.github.io/pymdown-extensions/extensions/blocks/#nesting.

TLDR: Use different numbers of leading / for nested containers.

/// html | div.card.text-white.bg-secondary.mb-3[style="max-width: 50rem;"]
//// html | div.card-header
////
//// html | div.card-title 
////
///

[quancore]

@facelessuser thank you, it is working !

[waylan]

I'm not sure I understand or agree with your reasoning on how raw is handled. Why does it need to be indented? Except for the fact that you dedent it, I don't see any difference in behavior in my limited testing. Or is the indentation to avoid it being stashed by the raw HTML preprocessor? Or is the issue to do with how you are collecting the body content? Apparently you don't collect it all as a single block. While I understand why that might be (based on how the underlying BlockParser works), I haven't tried to work out how you are actually collecting the parts. So, while I understand that you are likely trying to avoid some weird edge cases, I find the API unintuitive in regards to raw.

For example, I was surprised when I used raw on a block which contained unindented raw HTML. I expected the preprocessor to have put my raw HTML in the stash and for the block to contain a placeholder. But block.text actually contained the raw HTML as a string. Are you extracting the content from the stash and swapping that for the placeholder? The concern is that my block-level raw HTML was contained in block.text which resulted in it being HTML-escaped on serialization. While that makes sense when block is a pre or code element, it does not make sense in any other context. The raw text should have stayed in the stash to be handled by Markdown's default postprocessor IMO.

Of course, I realize that my expectations here are based on the fact that I have a very clear understanding of how the Markdown parser works. However, most users do not and the stash is invisible to them. Maybe having the API keep the stash invisible makes more sense. But then those devs who want to pass through some raw HTML will need to manually pull the text from block.text, save it to the stash, and save the placeholder to block.text, So, in the end, the stash is not invisible after all.

To sum up, because you are calling the mode raw, I think it should result in the default behavior for raw content. Namely that it be put in the stash. When I select the raw option, I expect the content of the block to stashed and a placeholder to be inserted in block.text. Of course, if the raw content happens to already be a placeholder (because the preprocessor stashed it), then that existing placeholder can just be passed on. Either way, if the extension dev wants to do some processing of the content, then the dev can use the placeholder to extract the content from the stash.

I suppose maybe one could end up with multiple placeholders in the stash from the preprocessor. To avoid that, your indedent/dedent thing could be useful. However, we can't depend on users to always indent their raw content so I don't see its value.

[waylan]

My primary objection is that the behavior assumes HTML content. I believe that no assumption should be made when the user has expressly indicated that the content is not to be treated as Markdown. So, its not relevant whether its called raw or something else. Yes, that means that you have to do more work when the raw content happens to be HTML, but I'm okay with that. You seem to be resistant because all of your use cases are HTML. I'm pushing back because I think you should not assume that your use cases are the norm.

That's the issue in a nutshell. Either you accept my position that you should not expect HTML content to the norm or you don't. It's your choice.

[facelessuser]

That's the issue in a nutshell. Either you accept my position that you should not expect HTML content to the norm or you don't. It's your choice.

I think I at least understand your stance vs mine now. The decision as to what the content is does not matter until on_end to me. So, as far as the extension is concerned, I guess the content is assumed HTML, regardless of the mode. In on_end, the developer can decide that is not actually the case, but the mechanism to deliver the content to on_end will remain generic and assume HTML to that point.

[facelessuser]

Last question as I prepare the final release. We've determined that the content in raw can only truly be preserved by using indented code blocks.

Should we reject the content in a raw block if it is not indented? I cannot abort at that point. So all we could do is just not return the content and have an empty block. We will have it document the content should be indented, regardless of whether we do this or not. This may be too heavy handed and maybe we should just let it pass through, even if the HTML parser mangles things.

Ignoring the content would at least give a very clear indication to the user that they are doing it wrong, that is probably the only advantage.

[waylan]

I am opposed to data loss. That is not how any other Markdown feature works. The policy is to always produce bad output than to remove the content. I don't see any reason to do any differently here.

[facelessuser]

Sounds good.

[facelessuser]

As I prepare for the final release, there will be some small changes:

• Block objects will have some new API methods is_raw and is_block that will use the same logic that is used internally to determine if a block is raw or block assuming an explicit identifier was not provided.
• self.html_escape(text) is provided as a function to HTML escape basic HTML content and will escape &, <, and >.
• The block HTML extension will store content in the Markdown stash during on_end and will now differentiate between content that should be HTML escaped and not HTML escaped (e.g. script and style). You can force this new unescaped HTML mode in the extension by using markdown: html if needed.

[facelessuser]

Closing this discussing as the official release will be dropped soon.

[facelessuser]

#1973 will be the formal thread for any feedback moving forward.