Dropping Characters Waiting for Vim's Completion

[orbisvicis]

In either input modes, SkyBison drops characters while waiting for Vim's built-in command completion. I know this is a big requestion and may not be fixable. Its not the lag that is problematic, the fact that key-presses disappear. For example:

h catch -> h ctch

[orbisvicis]

I'm a little surprised this happens in SkyBison as this effect isn't triggered by these examples:

noremap F :call TestChar2()<CR>

function! TestChar1()
    let s = ""
    while 1
        let c = nr2char(getchar())
        echo "Char: " . c
        if c == "\r"
            break
        endif
        let s .= c
        sleep 1000m
    endwhile
    redraw
    echom "Result: " . s
endfunction

function! TestChar2()
    let s = ""
    while 1
        if getchar(1)
            let c = nr2char(getchar())
            echo "Char: " . c
            if c == "\r"
                break
            endif
            let s .= c
            sleep 1000m
        endif
    endwhile
    redraw
    echom "Result: " . s
endfunction

[orbisvicis]

Could this (see comments) be the problem with <Esc> and getchar(1)?

function! TestChar1()
    let s = ""
    while 1
        let c = getchar()
        echo "Char #: " . c
        " getchar() with newline is 13, but here newline is 10?????????
        " :echo getchar()
        if c == 10
            break
        endif
        let c = nr2char(c)
        echo "Char S: " . c
        if c == "\r"
            break
        endif
        let s .= c
        sleep 1000m
    endwhile
    redraw
    echom "Result: " . s
endfunction

[paradigm]

Not sure what order you'll view these in, so I'm going to prefix all of the
recent issues you've made with:

I've got a skybison 2.0 ~90% done with many improvements. There's some
(external) blockers keeping me from finishing and publishing it (and a bunch of
other vim projects of mine...) immediately, but I fully plan to do so as soon
as is reasonably possible.

---

I think the effect is not triggered in your examples because of how sleep is
implemented. You'll want to find a single command (i.e. can't use a loop or
something that strings together multiple commands) that results in heavy
processing. Maybe have it open up and (and syntax highlight) a huge file.
Typing while it's opening that file will probably be dropped.

2.0 completely reworks the input system - no more getchar() vs getchar(1)
However, it still suffers from this issue. This is fundamental to the
single-threaded nature of vim: if one thing is processing, another can't be.

There's a number of work-arounds for this:

• Do some hacky multithread/process stuff. For example, kick off another vim
  that processes the current input and writes the results to a file, which the
  current vim then reads between keystrokes - much less delay. Additionally
  we'd need to do some more hacks faking user input to get it to update a
  potentially long processed item when the user isn't typing. While in theory
  this would "feel" the best, it's also by far the hackiest solution.
• The delay usually happens within the first few keystrokes when there are a
  lot of matches. As the user enters more, there are fewer results, and thus
  less delay. What we could do is delay the match look-up such that it will
  skip over the bad cases. For example, wait until the user stopped typing for
  some short period of time, then do the processing. This again will
  require a hack to fake input to get it to do multiple things between
  keystrokes to do the timing.
• Another route for the previous strategy, skybison could have some
  user-configurable list of slow commands that skybison will recognize and then
  avoid processing until a certain number of keys have been entered, e.g. if
  the command is :h, don't process until the first argument has at least two
  keys.

The last option would be, by far, the cleanest to implement, but it's also
arguably losing the key feature of skybison - it doesn't show completion items
when the user may want it to. If I include defaults, they'll likely result in
people thinking it's a bug, and I expect some people may be upset they have to
add such things themselves, but at least it would provide some relief for the
issue.

I wasn't planning on resolving this issue for the 2.0 release, but I could
reconsider that. There wouldn't be much code for the third option listed.

[orbisvicis]

What about builtin hacky multithread/process stuff? Since I wasn't able to reproduce this effect as a test case, I don't know if this will work:

function! GetChar()
python << EOF
import vim
import sys
vim.command("return '{}'".format(sys.stdin.read(1)))
EOF
endfunction

[orbisvicis]

I think vim disables all input buffering and appends received characters to the 'typeahead' buffer. This explains the difference between c's getchar() and vim's getchar(): the former is line-buffered. (Vim buffers function output, in the above code block a print above sys.stdin.read() won't be displayed until the function returns). So whenever vim is busy, the typeahead buffer isn't written and characters are dropped.

I couldn't duplicate the effect using :call feedkeys(";h c\<Tab>a\<Tab>t\<Tab>c\<Tab>h\<CR>"), perhaps vim uses extra processing.

Unfortunately python and vim share stdin. However if python is a separate thread or process it might be possible to duplicate and buffer python's input file handle, like tee, when the SkyBison is loaded:

fo = os.fdopen(os.dup(sys.stdin.fileno()))

Though probably python internally buffers stdin already.

[orbisvicis]

1. Python doesn't run independently of vim

Then I try reading from vim's stdin (not python's) and find that I unfortunately can't seek. It also turns out:

1. Non-canonical terminal mode is buffered (ie, same problem in gvim)

So I assume this must be a vim bug - it works in NeoVim, albeit slower. Best possible approach would be to get this fixed upstream. Any ideas on creating a simple test-case?

[paradigm]

feedkeys() won't work to automate a test case as it's bypassing the underlying problem.

Here's a simple test that should be easily repeatable on all but extremely fast hardware:

Run this to start vim:

vim -N -u NONE -c 'inoremap <silent> f <c-o>:execute "normal :h e<c-l>"<cr>f'

Enter insert mode then type "fj" relatively quickly. The "j" will be dropped while the processing for "f" is going.

I'd be happy to submit it a bug report upstream myself once my blockers are out of the way. If you want to go for it now (while I'm otherwise unable to), you're more than welcome to use that example to illustrate the issue. I'd prefer you not mention skybison if you can avoid it - I don't want to draw more attention to this project while I'm in the awkward position where I can't update it. Just treat it as a generalized issue if you can.