Question: Suitability for log-like buffers (IRC, etc)?

[alphapapa]

Hi,

This looks very, very interesting! I read your documentation (good job on that, by the way), but don't fully understand how it works, so I'll just ask: would this be suitable for a log-like buffer, like an IRC chat room? For example, I've been working on matrix-client.el for a while, and after I worked on that for a while, I discovered ewoc.el, and I've been thinking about giving it a try for implementing the UI.

Is rem.el suitable for that kind of use? If not, have you thought about it, and would it be possible to add support for it?

For example, in matrix-client.el, each room has a list of message events, newest first. We manually render messages into the room buffer at the appropriate location when they arrive. It would be nice to have a higher-level solution that would simply render the list of message events to the buffer appropriately; that is, it would determine which messages are new and render them in the appropriate position in the message buffer (considering that events may arrive out-of-order), without re-rendering all of the messages every time (which would perform poorly, of course).

Thanks.

[baygeldin]

Hi!

To be honest, I'm not sure if rem.el is the best option for a such use case, because log-like buffers are expected to have a lot of text in them. The approach rem.el takes is more suitable for relatively small amounts of text (like maybe a couple of screens).

I've been developing an Org-mode plugin, but I wasn't sure about what UI exactly do I need. So, I realized that I need a straightforward way to structure the code that would make it easy to refactor and maintain it later (i.e. the performance wasn't the main goal). The idea is that I have a single function that maps my data to a string representation and then I just replace a buffer's contents with it on each action. What rem.el optimizes is the generation of a string representation (although it doesn't have to be a string, but it makes sense for text interfaces). The rendering part (in the sense of replacing buffer contents) is done by rem-bind helper which really just monitors actions calls, erases previous buffer contents and inserts a new string representation. Computers are quite fast nowadays, so it works without glitches :)

But when the string representation is huge and constantly growing, I doubt that the performance would be as nice. Of course, it's possible to narrow the string representation manually and implement the scrolling, but you would lose many benefits of having the whole text in a buffer (e.g. full-text search). The problem is that Emacs buffers don't have some kind of tree representation to which you can apply patches by inserting and removing elements (like in the browser), it's just a wall of text. Maybe we can calculate the difference between string representations and apply patches in a more granular fashion using some kind of diff and patch utilities, but I'm not sure if it will be significantly faster than just replacing buffer contents altogether.

Speaking of "how it works", let's imagine we have a bunch of pure functions and a single function that somehow combines them and maps our data to some representation. Each time we change the data, we just call the top function and it recalculates everything from scratch. Now depending on the complexity of underlying functions it might perform just fine or it might not. But it's a pretty clean structure! We put in the data and get the result. We assume that if we put the same data we get the same result. And we also know that this result is somehow a combination of what underlying functions produced. So, we might say that for the same data we call the same set of functions with the same parameters. If we change data a bit and call the top function again it will produce a (likely) different result using a different set of functions with different parameters. Usually, these sets of function calls will overlap (how much depends on how much we change the data), but since functions are dumb, they don't know it and recalculate overlapping parts again. What rem.el does is simply prevent overlapping parts from unnecessary recalculation and calculates only the difference. In rem.el terms the top function is the view and underlying functions are components.

[alphapapa]

To be honest, I'm not sure if rem.el is the best option for a such use case, because log-like buffers are expected to have a lot of text in them. ...

Thanks, that answers my question.

Usually, these sets of function calls will overlap (how much depends on how much we change the data), but since functions are dumb, they don't know it and recalculate overlapping parts again. What rem.el does is simply prevent overlapping parts from unnecessary recalculation and calculates only the difference.

So rem.el basically memoizes the results of function calls at deeper levels of the hierarchy?

[baygeldin]

So rem.el basically memoizes the results of function calls at deeper levels of the hierarchy?

Yes, but this statement would also be true if components would be just standard functions with memoization. Unlike standard memoization rem.el keeps only relevant results that might be needed in the next calculation.

P.S:

I'm not sure if you've already seen it, but take a look at tui.el. It's still experimental and way more complicated, but looking at the source code it seems that it addresses the problem I've mentioned about Emacs buffers not having a tree representation by actually implementing a full-blown DOM.

Honestly, it's a shame that still there isn't a de-facto solution for building interfaces for Emacs that would suit a wide variety of use cases. I'm starting to think that it might be a good idea to implement interfaces using the embedded WebKit, but allow users to use and configure them via Emacs Lisp.

Please, ping me if at some point you find a good solution. Anyway, good luck with matrix-client.el!

[alphapapa]

Unlike standard memoization rem.el keeps only relevant results that might be needed in the next calculation.

This is the part I don't understand yet. Maybe I should just go stare at the code for a while, but could you explain how it does this at a basic, high level?

I'm not sure if you've already seen it, but take a look at tui.el. It's still experimental and way more complicated, but looking at the source code it seems that it addresses the problem I've mentioned about Emacs buffers not having a tree representation by actually implementing a full-blown DOM.

Yeah, I saw that a while back. It's an exciting project. I wish there were some examples to look at. I'm also a bit reluctant to dig into it since it says that I should study how React works first. I don't know anything about React, and I'd rather learn from an example that's relevant to Emacs. But I hope it turns out to be a useful project.

I'm starting to think that it might be a good idea to implement interfaces using the embedded WebKit, but allow users to use and configure them via Emacs Lisp.

D:

Please, ping me if at some point you find a good solution.

If you haven't looked at ewoc.el, take a look. It's built-in to Emacs, but it seems that not much uses it. Maybe you could get some ideas from it, at least.

[baygeldin]

could you explain how it does this at a basic, high level?

If we have a view that uses a component (say the component is called foo), at every point in time the view keeps memoized only these values for the foo component:

1. Values that the foo component produced by explicit calculation during the last time we called the view.
2. Values that the foo component produced by returning a memoized value during the last time we called the view.
3. Values that the foo component produced at some point in time and that were used to calculate values that other components produced during the last time we called the view.

In order to understand why 3 is important, let's look at the example in README:

After the initial invocation of (view), it should have these values memoized:

• For the component todos:
  • EAT|hack|repeat for parameters '("eat" "hack" "repeat") 0
• For the component todo:
  • EAT for parameters "eat" t
  • hack for parameters "hack" nil
  • repeat for parameters "repeat" nil

After we increment todo-selected and call the (view) again, it should have these values memoized:

• For the component todos:
  • eat|HACK|repeat for parameters '("eat" "hack" "repeat") 1
• For the component todo:
  • eat for parameters "eat" nil
  • HACK for parameters "hack" t
  • repeat for parameters "repeat" nil

After we call the (view) yet again (without changing the data), it should have the exact same values memoized. But without 3 it would have only these values memoized:

• For the component todos:
  • eat|HACK|repeat for parameters '("eat" "hack" "repeat") 1

This is bad because if we add a new entry at this point these values for the todo component will be recalculated (although we didn't change any entries, we just added a new one):

• eat for parameters "eat" nil
• HACK for parameters "hack" t
• repeat for parameters "repeat" nil

Sorry for the long answer, I'm just trying to be as explicit as possible.

If you haven't looked at ewoc.el, take a look. It's built-in to Emacs, but it seems that not much uses it. Maybe you could get some ideas from it, at least.

I've looked into it after reading your first message and indeed it looks nice! From reading the source code it seems that it would perfectly fit your use case. Also, while investigating it I came up with an idea of how to possibly make rem.el suitable for the same use case. I'll try to play around with it on the weekend and ping you if I come up with something interesting.

[alphapapa]

Thanks for your detailed answer. You might consider adding that to the readme. :)

Also, while investigating it I came up with an idea of how to possibly make rem.el suitable for the same use case. I'll try to play around with it on the weekend and ping you if I come up with something interesting.

I look forward to it!