Popover: check dialog's is modal flag instead of open attribute

The following case currently does not fail

dialog.showModal();
dialog.open = false;
dialog.showPopover();

even though it is a top layer conflict:

1. showModal() pushes to the top layer.
2. open = false then hides the dialog, while keeping it in top layer.
3. showPopover() pushes to the top layer again.

The check for the open attribute is supposed to prevent top layer conflicts, since those conflicts can leave the element in a broken state, notably after hiding. For instance, the previous sequence of actions followed by dialog.hidePopover() will leave the is modal flag set to true despite the dialog being completely removed from the top layer.

Also only remove from top layer in HTMLDialogElement.prototype.close() when the is modal flag is true to prevent that method from removing popovers from the top layer.

For dialog.close() to be symmetric & consistent with dialog.show(), we also stop throwing when show() is called on a dialog in the popover showing state.

Tests: WebKit/WebKit#14454.

Fixes whatwg#9335.

source

@@ -60325,10 +60325,6 @@ interface <dfn interface>HTMLDialogElement</dfn> : <span>HTMLElement</span> {
    <li><p>If <span>this</span> has an <code data-x="attr-dialog-open">open</code> attribute, then
    throw an <span>"<code>InvalidStateError</code>"</span> <code>DOMException</code>.</p></li>
 
-   <li><p>If <span>this</span> is in the <span data-x="popover-showing-state">popover showing
-   state</span>, then throw an <span>"<code>InvalidStateError</code>"</span>
-   <code>DOMException</code>.</p></li>
-
    <li><p>Add an <code data-x="attr-dialog-open">open</code> attribute to <span>this</span>, whose
    value is the empty string.</p></li>
 
@@ -60453,14 +60449,14 @@ interface <dfn interface>HTMLDialogElement</dfn> : <span>HTMLElement</span> {
    <li><p>Remove <var>subject</var>'s <code data-x="attr-dialog-open">open</code>
    attribute.</p></li>
 
+   <li><p>If the <span>is modal</span> flag of <var>subject</var> is true, then <span data-x="list
+   remove">remove</span> it from its <code>Document</code>'s <span>top layer</span>.</p></li>
+
    <li><p>Set the <span>is modal</span> flag of <var>subject</var> to false.</p></li>
 
    <li><p>If <var>result</var> is not null, then set the <code
    data-x="dom-dialog-returnValue">returnValue</code> attribute to <var>result</var>.</p></li>
 
-   <li><p>If <var>subject</var> is in its <code>Document</code>'s <span>top layer</span>, then <span
-   data-x="list remove">remove</span> it.</p></li>
-
    <li>
     <p>If <var>subject</var>'s <span>previously focused element</span> is not null, then:<p>
 
@@ -82957,8 +82953,8 @@ dictionary <dfn dictionary>DragEventInit</dfn> : <span>MouseEventInit</span> {
      <li><p><var>expectedDocument</var> is not null and <var>element</var>'s <span>node
      document</span> is not <var>expectedDocument</var></p></li>
 
-     <li><p><var>element</var> is a <code>dialog</code> element and has an <code
-     data-x="attr-details-open">open</code> attribute</p></li>
+     <li><p><var>element</var> is a <code>dialog</code> element and its <span>is modal</span> flag
+     is set to true.</li>
 
      <li><p><var>element</var>'s <span>fullscreen flag</span> is set</p></li>
     </ul>