Create multiple rows with multiSet().

svendber

Hi !
I've got a question about editor.multiset() - see the code below.

function impNmapDataMultiSet (dataNmap, j) {
        customElement.text(j + "/" + dataNmap.length);
        tableCust=   $('#customer').DataTable();

        editorNessus.create(dataNmap.length, false)
            .multiSet('ip', dataNmap[j]['ip'])
            .multiSet('mac', dataNmap[j]['mac'])
            .multiSet('netbiosname', dataNmap[j]['netbiosname'])
            .multiSet('os', dataNmap[j]['os'])
            .multiSet('start', dataNmap[j]['start'])
            .multiSet('stop', dataNmap[j]['stop'])
            .multiSet('plugin_id', dataNmap[j]['plugin_id'])
            .multiSet('plugin_name', dataNmap[j]['plugin_name'])
            .multiSet('port', dataNmap[j]['port'])
            .multiSet('protocol', dataNmap[j]['protocol'])
            .multiSet('cve', dataNmap[j]['cve'])
            .multiSet('cvss_base_score', dataNmap[j]['cvss_base_score'])
            .multiSet('service_name', dataNmap[j]['service_name'])
            .multiSet('description', dataNmap[j]['description'])
            .multiSet('solution', dataNmap[j]['solution'])
            .multiSet('kundenr', dataNmap[j]['kundenr']);
        
                console.log(dataNmap[j]);
                j++;
                if (j < dataNmap.length) {
                    setTimeout(function () {
                        impNmapDataMultiSet(dataNmap, j);
                    }, 100);
                }
                else {
                    editorNessus.submit();
                    $('#nmapTable').busyLoad("hide");
                    $.busyLoadFull("hide");
                }
        tableCust.draw();
    }  

I'm trying to use multiSet() to create several rows imported from a file. The code above works, but it creates the same data for all the rows.. I've read some other posts which says that I need to have the Count of rows to create, as I've done with the lenght of my array (dataNmap.lenght), in my create method (editorNessus.create(dataNmap.length, false)).

What am I missing?

Have a great day!

allan

I think the issue here is your setTimeout and more specifically calling the impNmapDataMultiSet function in it. That is causing the create() method to be called without a submit N-1 times. Only on the last one will it submit (and it will take the values from that last one).

What I would suggest instead is that you do:

function impNmapDataMultiSet (dataNmap) {
        customElement.text(j + "/" + dataNmap.length);
        tableCust=   $('#customer').DataTable();
 
        editorNessus.create(dataNmap.length, false)

        for ( var i=0, ien=dataNmap.length ; i<ien ; i++ ) {
           editorNessus.multiSet( ... );
        }

        editorNessus.submit();
}

That will be a lot quicker and I think it should actually work properly .

Allan

svendber

Hi @allan
Ty for answer!

It does work, but only if I import a file with 14 rows MAX. If the files contains more then that (They normally contain around 200+ rows), then I get the following errors:

It loads the first 13 super fast, but then it continues slowly 14..... 15...... 16...... 17....
Then Aw, Snap! It crashes and nothing is stored in the database..

Do you have any idea how to fix this?

Have a great day!

svendber

I really thought that it wouldn't take that much memory, but it does? That's quite unfortunate.

allan

I'm surprised by that as well. But the a variable there, it looks like it has 22 million entires in the array. Does it!?

Allan

svendber

oh my! Yes it does! I'm not sure where it comes from.. The file that I'm using as a test only has 49 rows and the code looks like this:

  function impDataMultiSet (dataNmap) {
    
             editorVulnerability.create(dataNmap.length, false);

            for(i=0, ien= dataNmap.length; i<ien; i++   ){
      
                editorVulnerability.multiSet('crapper_vulnerability.ip', dataNmap[i]['ip']);
                editorVulnerability.multiSet('crapper_vulnerability.mac', dataNmap[i]['mac']);
                editorVulnerability .multiSet('crapper_vulnerability.netbiosname', dataNmap[i]['netbiosname']);
                editorVulnerability .multiSet('crapper_vulnerability.os', dataNmap[i]['os']);
                editorVulnerability .multiSet('crapper_vulnerability.start', dataNmap[i]['start']);
                editorVulnerability  .multiSet('crapper_vulnerability.stop', dataNmap[i]['stop']);
                editorVulnerability   .multiSet('crapper_vulnerability.plugin_id', dataNmap[i]['plugin_id']);
                editorVulnerability   .multiSet('crapper_vulnerability.plugin_name', dataNmap[i]['plugin_name']);
                editorVulnerability.multiSet('crapper_vulnerability.port', dataNmap[i]['port']);
                editorVulnerability .multiSet('crapper_vulnerability.protocol', dataNmap[i]['protocol']);
                editorVulnerability .multiSet('crapper_vulnerability.cve', dataNmap[i]['cve']);
                editorVulnerability .multiSet('crapper_vulnerability.cvss_base_score', dataNmap[i]['cvss_base_score']);
                editorVulnerability  .multiSet('crapper_vulnerability.service_name', dataNmap[i]['service_name']);
                editorVulnerability  .multiSet('crapper_vulnerability.description', dataNmap[i]['description']);
                editorVulnerability  .multiSet('crapper_vulnerability.solution', dataNmap[i]['solution']);
                editorVulnerability    .multiSet('crapper_vulnerability.kundenr', dataNmap[i]['kundenr']);
                console.log(i);
            }

In the call stack, the function is triggered by nessusSaveClicked, which you can see only has 48 rows.

in the call stack the "each" and "multiSet" does record the 22 millions entries..?

tangerine

for(i=0, ien= dataNmap.length; i<ien; i++   ){

Initialize "ien" before the loop, not inside it.

svendber

Doesn't work, even if I do like this:

function impDataMultiSet (dataNmap) {
   
           editorVulnerability.create(20, false);
 
          for(i=0;  i<20;  i++   ){
     
              editorVulnerability.multiSet('crapper_vulnerability.ip', dataNmap[i]['ip']);
              editorVulnerability.multiSet('crapper_vulnerability.mac', dataNmap[i]['mac']);
              editorVulnerability .multiSet('crapper_vulnerability.netbiosname', dataNmap[i]['netbiosname']);
              editorVulnerability .multiSet('crapper_vulnerability.os', dataNmap[i]['os']);
              editorVulnerability .multiSet('crapper_vulnerability.start', dataNmap[i]['start']);
              editorVulnerability  .multiSet('crapper_vulnerability.stop', dataNmap[i]['stop']);
              editorVulnerability   .multiSet('crapper_vulnerability.plugin_id', dataNmap[i]['plugin_id']);
              editorVulnerability   .multiSet('crapper_vulnerability.plugin_name', dataNmap[i]['plugin_name']);
              editorVulnerability.multiSet('crapper_vulnerability.port', dataNmap[i]['port']);
              editorVulnerability .multiSet('crapper_vulnerability.protocol', dataNmap[i]['protocol']);
              editorVulnerability .multiSet('crapper_vulnerability.cve', dataNmap[i]['cve']);
              editorVulnerability .multiSet('crapper_vulnerability.cvss_base_score', dataNmap[i]['cvss_base_score']);
              editorVulnerability  .multiSet('crapper_vulnerability.service_name', dataNmap[i]['service_name']);
              editorVulnerability  .multiSet('crapper_vulnerability.description', dataNmap[i]['description']);
              editorVulnerability  .multiSet('crapper_vulnerability.solution', dataNmap[i]['solution']);
              editorVulnerability    .multiSet('crapper_vulnerability.kundenr', dataNmap[i]['kundenr']);
              console.log(i);
          }
editorVulnerability.submit(); 

tangerine

Sorry - I didn't think that through well enough.
Your scenario does suggest a faulty loop, though. What does your console.log of "i" look like?

svendber

@tangerine
It loads the first 13 super fast, but then it continues slowly 14..... 15...... 16...... 17....
After that it get paused before potential out-of-memory crash.

Can I do anything to help debug further?

colin

Unless I'm missing something, that loop doesn't include/modify an array variable called a. Is it being created or modified elsewhere?

C

svendber

@colin
I create the array like so:

function nessusSaveClicked() {

        tableVuln=   $('#vulnTable2').DataTable();

        var rows = tableVuln.rows().data().toArray();
        dataarray = [];


        for (i=0; i< rows.length ; i++) {
            if (rows[i]['rapport'] === true) {
                dataarray.push(rows[i]);
            }
        }
        console.log(dataarray);
         impDataMultiSet(dataarray);
    }

I don't modify it anywhere else. If I can help debug further, please let me know!

colin

Hi @svendber ,

Yep, that's where you create the dataarray array - but the problem that Allan pointed out is a local variable called a. You should follow your code through and see where that is being created.

Cheers,

Colin

svendber

@colin @allan
Hi again,
I've been looking through it, and it appears when it comes to multiSet, in the dataTables.editor.min.js. Look at the pictures below.

I do not have a variable called a in my code. I might not be the best programmer, or even close to, but at this I'm 99.99% sure that this happens somewhere in the datatables library.

Have a great day!

allan

From those variable names, that's probably in the minified code. Are you able to give me a link to your page so I can trace through what is going on please? Something is leaking into that a array, and I'd need to be able to trace it through to figure out what.

Thanks,
Allan

svendber

@allan unfortunately not. But I'll make a test program within the next few days and test it out.
Have a great day!

svendber

@allan
Hi again,
I made a test project, and the same thing happened!

I even upgraded from Editor 1.7.4 -> Editor 1.8.1, only the necessary library etc.
Button to select file and call function imDataMultiSet

You don't have any working examples or something like that do you?

Have a great day.
//Svendber

allan

Any chance of a link to your page? It's not immediately clear what the issue would be there.

That said, you aren't using i in your loop for multiSet - should it be dataNessus[i] rather than j?

Allan

svendber

@allan
Woops, the 2nd should've been the following:

But nevertheless the outcome was still the same..
My project is running on a local server at my workplace, so I sadly can't give you any link to it.

As far as I know, the problem starts at multiset -> datatables.editor.min.js

And contiues over to the each -> jquery-1.11.3.min.js

Is there something else I can do since I can't provide a link?

Have a great day.
//Svendber

svendber

@allan
Do you have or know any working projects with a large import using mulitset?

colin

Hi @svendber ,

We don't have any examples, other than those in the docs for multiSet(). As you're not able to link to your page, would you be able to modify this example here to demonstrate the problem, please.

Cheers,

Colin