Allow to use large backend list filter options

[trepmag]

If you have thousands (or more) of items as a input for the backend list filter options then your system will fails later or sooner...

The solution is annoted here octobercms/october#5470 (comment) which mention that modules/system/assets/ui/js/filter.js needs to be fixed for this.

Here is an attempts to fix filter.js: trepmag/wn-system-module@caaf76a then the filter group must be defined as follow:

config_filter.yaml:

scopes:
  contact:
    label: Contact
    modelClass: Foo\Bar\Model\Contact
    options: getOptions
    conditions: contact_id in (:filtered)

Contact.php model:

<php
...
class Contact extends Model
{
    ...
    public static function getOptions()
    {
        // Build query
        $query = self::orderBy('name')
            ->take(50);

        // Lookup for user options search key
        if ($keywords = Input::get('search')) {
            $keywords_a = preg_split('/\s+/', $keywords);
            foreach ($keywords_a as $word) {
                $query->orWhere(function ($query) use ($word) {
                    $query->where('name', 'LIKE', '%' . strtolower($word) . '%');
                    $query->orWhere('first_name', 'LIKE', '%' . strtolower($word) . '%');
                    $query->orWhere('company', 'LIKE', '%' . strtolower($word) . '%');
                });
            }
        }
        // Required by wintercms modules/system/assets/ui/js/filter.js which lookup for available options...
        else if (Input::get('filterScope') && Input::get('options') !== 'null') {
            $options = json_decode(Input::get('options'), true);
            foreach ($options as $option) {
                if ($value = $option['id']) {
                    $query->orWhere('id', $value);
                }
            }
        }

        return $query->pluck('name', 'id')->toArray();
    }
    ...
}

Related issues:

• Fix filter type group using modelClass when 500+ options are available. #27
• Fix: Group filter type with modelClass does not work when 500+ options are available octobercms/october#5470 (comment)

[bennothommo]

@trepmag unfortunately, your suggested fix as well as the linked fix on the October repo both don't resolve the issue but instead just band-aid it by trying to limit the amount of options retrieved and displayed at any one time.

As I mentioned in the comments on the October PR, the best way to fix this is to implement lazy-loading, where we load up results in batches of 100 (perhaps dropping the previous 100 once scrolled through to prevent too much HTML from being loaded) which should give us both the same fix that you have proposed and simultaneously allowing someone to show as many results as they want without the performance of the filter slowing down.

[danielbidala]

@trepmag the linked PR #5470 was merged to October CMS v2 and is working as expected. It's incomprehensible why not merged to Winter CMS since it does not introduces breaking changes...

[LukeTowers]

@danielbidala can you resubmit your pr against winter?

[bennothommo]

@danielbidala

It's incomprehensible why not merged to Winter CMS since it does not introduces breaking changes

Because it doesn't fix the issue. It still presents a UI/DOM performance problem. Even Sam's response when he (blindly) merged it in admits that there's still UI issues with it, which is as close as you're going to get to him admitting anything.

If @LukeTowers is willing to accept the PR, then that's fine, but I would rather a proper fix than a band-aid solution.

[LukeTowers]

@bennothommo can you refresh my memory on what the current behaviour is and what the proposed fix changes it to? If the current behaviour is completely broken and the proposed fix is less broken without causing future headaches for fixing the whole thing outright it would be better to make the small improvement now unless someone has some time to fully fix the issue in its entirety (which it sounds like no one does atm)

[bennothommo]

@LukeTowers there's two separate issues:

• One, currently the dropdown filter widget allows only a maximum of 500 options if these are retrieved directly from the model, in which further options are clipped at the query level. There have been people who have reported that as an issue because they have more than 500 options. It's workable in that people can enter a query and it will do a search and then limit to 500 records that match the search. There's apparently also a UI issue where if 500 records are displayed, and someone adds another one through the UI, it doesn't show.
• Two, forcing more than 500 items seems to result in a performance issue in the browser due to the amount of DOM elements and JavaScript in action, because there's currently no lazy-loading or management of visible items so it is showing the entire list in the browser and just using CSS overflow to hide other options.

It is indeed possible that the performance issue doesn't exist any longer (I don't have any sites that have anywhere close to 500 items in the filter), but my suggestion at the time (and still counts now) is that it would be better to remove the limit but try and stream options through and only keep a certain number of items visible in DOM at the same time, which would allow an unlimited number of items to be handled efficiently.

@danielbidala mentioned in the original PR that they didn't have time to work on such a feature, which is fair enough, but I still believe if we do this, we do this right, and the alternative is simply to recommend people do not use that number of options until such a time that we do fix it, which even the October founders alluded to in their comment for the group filter:

• The 'group' scope has trouble supporting more than 500 records at a time
• @todo Introduce a more advanced version with robust list support.

If @danielbidala believes their fix is iterative enough to fix the problem, then again, fair enough.

[danielbidala]

@bennothommo

Because it doesn't fix the issue.

The issue: You can't filter the lists when more than 500 options are present in group filter. With the mentioned PR: you can => Issue fixed!

It still presents a UI/DOM performance problem.

We use this PR in production with tens of thousands of related records, and we do not experience any performance problem. Sam talks about UX issue not UI/DOM performance issue. But I can't imagine bigger UX issue than something is not working as expected! With the merged PR in October the model query is limited to 200 items when you open the group filter and also when you search in it. In Winter it is limited to 500 items. Doesn't that mean a performance increase rather?

[bennothommo]

Alright then, with your permission, I'll merge the same fix into Winter, and take a look at improving the UI at a later stage. Is that alright to use your fix?

[danielbidala]

@bennothommo of course, feel free to use it.