Editor render is very slow when using many fields

tony845

I'm having an issue when my Editor can have 100-250 fields to display, the page load takes between 4-20 seconds. The issue seems to be the Editor.add() function which gets called iteratively in a linear fashion for each field to create the corresponding DOM elements. The performance graph below is for an Editor with 100 fields, and the time increases as the number of fields grows. We are using the fields property of the Editor initialization, which in turn is what calls add(). Our design requires the Editor to always be displayed, starting from page load, which is why this is an issue. Is there a way around this, for the Editor to load quicker or earlier somehow?

This link is the best I could do to create an example to show the problem. It's a dummy dataTable with 100 records, and presumably the 100 field Editor would display the slowdown when drawn, however the buttons aren't working in this setup which are necessary to open the Editor.

http://live.datatables.net/xupusodu/1/edit

tangerine

This is remarkably similar to this thread:
https://datatables.net/forums/discussion/64816/many-fields-in-editor-table-cause-screen-to-refresh-too-slow#latest

Is there some connection?

allan

It is quite an old version of DataTables and Buttons you have in your test test there, which is why the form doesn't show. Here is it updated with the current releases: http://live.datatables.net/xupusodu/2/edit .

With that it isn't instant for me, but it is close! Chrome shows the initialisation taking 380mS on my computer:

Allan

Yang.Song

Allan, is there any way I can send you a more accurate html to illustrate the issue?
https://datatables.net/forums/discussion/64816/many-fields-in-editor-table-cause-screen-to-refresh-too-slow#latest. I know that js bin does not run as is. But it does run when you merge the HTML and js together into an HTML file and you'd see the slowness in creating the editor with 100 fields. Could you take a look that and try and see if you still get the 380ms load time? Thanks!

allan

Can you give me a link to such a page please? The live page I linked to does run and shows all the fields if you click the 'New' button.

Allan

Yang.Song

This is the link. http://live.datatables.net/dunexoti/1/edit

The issue is the edit buttons don't show up on the page. I don't know why. When I put the js in the html and save it as an HTML page, the page does load with buttons. Below is the performance data and result page.
When it does not load with buttons, I don't know if it's working correctly.Thanks!

tony845

Yes, Alan that other thread is from my coworker on the same issue so the two threads can be merged, sorry for the duplication. Thank you for the corrected live page example, I can see the editor load instantly there. As Yang details above, we're not seeing the same performance however in our real-world application where more field attributes and an always displayed Editor are used.

Also, we're using DT v1.10.20 and Editor v1.9.1, is there any reason those versions may cause a slowdown?

allan

I don't believe that DT1.10.20 and E1.9.1 would be fundamentally slower - but it would be worth updating to the latest releases as the first port of call.

Regarding the example given above, here is it working: http://live.datatables.net/dunexoti/7/edit (there is something funny going on with the styling of the custom template causing the lightbox to be small, but that's a different issue - let's come back to that).

Does that load up quickly for you? It does appear to for me.

Allan

Yang.Song

Allan, thanks for getting this to load. I ran the profiler when loading this example. It looks like it took 3.62 seconds to load the datatable. While I understand this time also depends on how fast the machine is, you see the difference between 380ms to this one. and this is 100 fields. For a screen with 250 fields, it'd more than double the response time.

allan

Yes, an order of magnitude difference would not be expected!

What kind of computer are you using? Processor and OS? Also, what browser are you using?

Given how fast it is for me, I don't know why it would be so slow for you. I'm using Chrome on Ubuntu, with a Xeon processor which is now 3 generations old.

Allan

Yang.Song

Chrome on Window 7 professional. Intel i5-6300U, which is about 5 generations back. This is a common setting for us. I have 8G RAM, which is twice as much as most of my coworkers have. We have no control over configuration for us or for our clients. Do you have any suggestion that could boost response time on our machine?

colin

This section of the FAQ may help, it discusses various techniques to improve performance,

Cheers,

Colin

Yang.Song

Colin, Thanks!. But this is editor initialization slowness due to the number of fields involved. We create the editor table at the same time we create the datatable. There isn't much of anything we are doing other than making the constructor call to create the editor. The performance data suggests that chunck time is in the editor creation add method (this late pink box in the performance graph). Please advise.

allan

Many thanks. I see a way that I think I can improve the performance by at least half - possibly more depending on the specifics of how the browser works.

In your local copy of Editor, can you search for Editor.prototype.add and replace the function with the following:

Editor.prototype.add = function ( cfg, after, reorder )
{
    // Allow multiple fields to be added at the same time
    if ( Array.isArray( cfg ) ) {
        // Do it in reverse to allow fields to appear in the same order given, otherwise,
        // the would appear in reverse if given an `after`
        if ( after !== undefined ) {
            cfg.reverse();
        }

        for ( var i=0 ; i<cfg.length ; i++ ) {
            this.add( cfg[i], after );
        }

        this._displayReorder( this.order() );

        return this;
    }

    var name = cfg.name;

    if ( name === undefined ) {
        throw "Error adding field. The field requires a `name` option";
    }

    if ( this.s.fields[ name ] ) {
        throw "Error adding field '"+name+"'. A field already exists with this name";
    }

    // Allow the data source to add / modify the field properties
    // Dev: would this be better as an event `preAddField`? And have the
    // data sources init only once, but can listen for such events? More
    // complexity, but probably more flexible...
    this._dataSource( 'initField', cfg );

    var field = new Editor.Field( cfg, this.classes.field, this );

    // If in an editing mode, we need to set the field up for the data
    if ( this.s.mode ) {
        var editFields = this.s.editFields;
        field.multiReset();

        $.each( editFields, function ( idSrc, edit ) {
            var val;
            if ( edit.data ) {
                val = field.valFromData( edit.data );
            }

            field.multiSet( idSrc, val !== undefined ?
                val :
                field.def()
            );
        } );
    }

    this.s.fields[ name ] = field;

    if ( after === undefined ) {
        this.s.order.push( name );
    }
    else if ( after === null ) {
        this.s.order.unshift( name );
    }
    else {
        var idx = $.inArray( after, this.s.order );
        this.s.order.splice( idx+1, 0, name );
    }

    if ( reorder !== false ) {
        this._displayReorder( this.order() );
    }

    return this;
};

Hopefully that will help a bit!

Allan

Yang.Song

Allan, thank you! I made the change. The "Editor" time went from 3.51 to 1.97, which is about 46% improvement. Is there similar changes that can be done to Editor._displayOrder? Thanks!

Yang.Song

Allan, it looks like the Editor.prototype._displayOrder() call is repeated for each field as more fields join, e.g., if I have 100 fields, it's called in a loop with field0, then field0, field1, then field0, field1, field2... etc. until it's field100. Can we skip the first 99 calls and only call the last one with one 100 element array and .forEach() call? Please take a look. Thanks!

Yang.Song

Allan, I think if we make this one change in your Editor.prototype.add code, we can eliminate all the unnecessary _displayOrdder call. Add the check for reorder!== undefined before the false check in line 69 in the above code.

In our case, all fields are placed in order. No need to reorder at any given point of time. The editor now renders quick as expected. Please let me know if you see an issue with this change. Thanks

if ( reorder!== undefined && reorder !== false ) {

    this._displayReorder( this.order() );

}

Yang.Song

Sorry. Forgot to format the source code. Here it is. line 69

 if ( reorder !== undefined && reorder !== false ) {
        this._displayReorder( this.order() );
    }

allan

Gah - what it is, on line 12 in my code above it should be:

this.add( cfg[i], after, false );

I wrote it originally on a branch and then had to move it over and dropped that. With that change it should be nice and fast!

Allan

Yang.Song

I see. That works too. THANK YOU!!!

Yang.Song

Allan, I was wondering what your thoughts are about removing two _displayOrder calls in the code. The first one is in .open() method. The second one is in _edit(). Since _displayOrder() is already called upon editor creation, do we still need these two? When there are many fields in editor, these two calls takes some chunk time. Please let me know. Thank you!

allan

Yes, it is possible using event callbacks such as preOpen to alter the ordering.

How much time is each call to that function taking for you?

Allan

Yang.Song

When I was testing, it was taking about 700 - 800 milliseconds each. Upon load, both were called thus added about 1.5 second to the response time. Once they are removed, the page showed up even more quickly. The 100 field page load is now about 1.5 seconds and the 250 field load is about 3.5 seconds. Before your changes, it was coming up in the 10 seconds and 25 seconds range roughly. It felt like forever.

I looked at the preOpen documentation here. https://editor.datatables.net/reference/event/preOpen. It is not intuitive to me how I can pass in parameters to change the order of fields displayed in the editor. "field" parameter is passed in for constructor. Which method takes in a new field array? Is it possible to add a Boolean so that by default, it skips the _displayOrder() call in both functions but can make the call if desired? For our screens, the field order on the editor never changes once created.

allan

Wow - as much as nearly a second? That's slow! No wonder it was causing problems before.

I will look at ways to make the second call optional or better yet, automatic only if needed.

Allan