Create multiple rows with multiSet().

Create multiple rows with multiSet().

svendbersvendber Posts: 27Questions: 8Answers: 0

Hi !
I've got a question about editor.multiset() - see the code below.

function impNmapDataMultiSet (dataNmap, j) {
        customElement.text(j + "/" + dataNmap.length);
        tableCust=   $('#customer').DataTable();

        editorNessus.create(dataNmap.length, false)
            .multiSet('ip', dataNmap[j]['ip'])
            .multiSet('mac', dataNmap[j]['mac'])
            .multiSet('netbiosname', dataNmap[j]['netbiosname'])
            .multiSet('os', dataNmap[j]['os'])
            .multiSet('start', dataNmap[j]['start'])
            .multiSet('stop', dataNmap[j]['stop'])
            .multiSet('plugin_id', dataNmap[j]['plugin_id'])
            .multiSet('plugin_name', dataNmap[j]['plugin_name'])
            .multiSet('port', dataNmap[j]['port'])
            .multiSet('protocol', dataNmap[j]['protocol'])
            .multiSet('cve', dataNmap[j]['cve'])
            .multiSet('cvss_base_score', dataNmap[j]['cvss_base_score'])
            .multiSet('service_name', dataNmap[j]['service_name'])
            .multiSet('description', dataNmap[j]['description'])
            .multiSet('solution', dataNmap[j]['solution'])
            .multiSet('kundenr', dataNmap[j]['kundenr']);
        
                console.log(dataNmap[j]);
                j++;
                if (j < dataNmap.length) {
                    setTimeout(function () {
                        impNmapDataMultiSet(dataNmap, j);
                    }, 100);
                }
                else {
                    editorNessus.submit();
                    $('#nmapTable').busyLoad("hide");
                    $.busyLoadFull("hide");
                }
        tableCust.draw();
    }  

I'm trying to use multiSet() to create several rows imported from a file. The code above works, but it creates the same data for all the rows.. I've read some other posts which says that I need to have the Count of rows to create, as I've done with the lenght of my array (dataNmap.lenght), in my create method (editorNessus.create(dataNmap.length, false)).

What am I missing?

Have a great day!

This question has an accepted answers - jump to answer

Answers

  • allanallan Posts: 61,436Questions: 1Answers: 10,049 Site admin

    I think the issue here is your setTimeout and more specifically calling the impNmapDataMultiSet function in it. That is causing the create() method to be called without a submit N-1 times. Only on the last one will it submit (and it will take the values from that last one).

    What I would suggest instead is that you do:

    function impNmapDataMultiSet (dataNmap) {
            customElement.text(j + "/" + dataNmap.length);
            tableCust=   $('#customer').DataTable();
     
            editorNessus.create(dataNmap.length, false)
    
            for ( var i=0, ien=dataNmap.length ; i<ien ; i++ ) {
               editorNessus.multiSet( ... );
            }
    
            editorNessus.submit();
    }
    

    That will be a lot quicker and I think it should actually work properly :).

    Allan

  • svendbersvendber Posts: 27Questions: 8Answers: 0

    Hi @allan
    Ty for answer!

    It does work, but only if I import a file with 14 rows MAX. If the files contains more then that (They normally contain around 200+ rows), then I get the following errors:

    It loads the first 13 super fast, but then it continues slowly 14..... 15...... 16...... 17....
    Then Aw, Snap! It crashes and nothing is stored in the database.. :(

    Do you have any idea how to fix this?

    Have a great day!

  • svendbersvendber Posts: 27Questions: 8Answers: 0

    I really thought that it wouldn't take that much memory, but it does? That's quite unfortunate.

  • allanallan Posts: 61,436Questions: 1Answers: 10,049 Site admin

    I'm surprised by that as well. But the a variable there, it looks like it has 22 million entires in the array. Does it!?

    Allan

  • svendbersvendber Posts: 27Questions: 8Answers: 0

    oh my! Yes it does! I'm not sure where it comes from.. The file that I'm using as a test only has 49 rows and the code looks like this:

      function impDataMultiSet (dataNmap) {
        
                 editorVulnerability.create(dataNmap.length, false);
    
                for(i=0, ien= dataNmap.length; i<ien; i++   ){
          
                    editorVulnerability.multiSet('crapper_vulnerability.ip', dataNmap[i]['ip']);
                    editorVulnerability.multiSet('crapper_vulnerability.mac', dataNmap[i]['mac']);
                    editorVulnerability .multiSet('crapper_vulnerability.netbiosname', dataNmap[i]['netbiosname']);
                    editorVulnerability .multiSet('crapper_vulnerability.os', dataNmap[i]['os']);
                    editorVulnerability .multiSet('crapper_vulnerability.start', dataNmap[i]['start']);
                    editorVulnerability  .multiSet('crapper_vulnerability.stop', dataNmap[i]['stop']);
                    editorVulnerability   .multiSet('crapper_vulnerability.plugin_id', dataNmap[i]['plugin_id']);
                    editorVulnerability   .multiSet('crapper_vulnerability.plugin_name', dataNmap[i]['plugin_name']);
                    editorVulnerability.multiSet('crapper_vulnerability.port', dataNmap[i]['port']);
                    editorVulnerability .multiSet('crapper_vulnerability.protocol', dataNmap[i]['protocol']);
                    editorVulnerability .multiSet('crapper_vulnerability.cve', dataNmap[i]['cve']);
                    editorVulnerability .multiSet('crapper_vulnerability.cvss_base_score', dataNmap[i]['cvss_base_score']);
                    editorVulnerability  .multiSet('crapper_vulnerability.service_name', dataNmap[i]['service_name']);
                    editorVulnerability  .multiSet('crapper_vulnerability.description', dataNmap[i]['description']);
                    editorVulnerability  .multiSet('crapper_vulnerability.solution', dataNmap[i]['solution']);
                    editorVulnerability    .multiSet('crapper_vulnerability.kundenr', dataNmap[i]['kundenr']);
                    console.log(i);
                }
    

    In the call stack, the function is triggered by nessusSaveClicked, which you can see only has 48 rows.

    in the call stack the "each" and "multiSet" does record the 22 millions entries..?

  • tangerinetangerine Posts: 3,342Questions: 35Answers: 394
    for(i=0, ien= dataNmap.length; i<ien; i++   ){
    

    Initialize "ien" before the loop, not inside it.

  • svendbersvendber Posts: 27Questions: 8Answers: 0

    Doesn't work, even if I do like this:

    function impDataMultiSet (dataNmap) {
       
               editorVulnerability.create(20, false);
     
              for(i=0;  i<20;  i++   ){
         
                  editorVulnerability.multiSet('crapper_vulnerability.ip', dataNmap[i]['ip']);
                  editorVulnerability.multiSet('crapper_vulnerability.mac', dataNmap[i]['mac']);
                  editorVulnerability .multiSet('crapper_vulnerability.netbiosname', dataNmap[i]['netbiosname']);
                  editorVulnerability .multiSet('crapper_vulnerability.os', dataNmap[i]['os']);
                  editorVulnerability .multiSet('crapper_vulnerability.start', dataNmap[i]['start']);
                  editorVulnerability  .multiSet('crapper_vulnerability.stop', dataNmap[i]['stop']);
                  editorVulnerability   .multiSet('crapper_vulnerability.plugin_id', dataNmap[i]['plugin_id']);
                  editorVulnerability   .multiSet('crapper_vulnerability.plugin_name', dataNmap[i]['plugin_name']);
                  editorVulnerability.multiSet('crapper_vulnerability.port', dataNmap[i]['port']);
                  editorVulnerability .multiSet('crapper_vulnerability.protocol', dataNmap[i]['protocol']);
                  editorVulnerability .multiSet('crapper_vulnerability.cve', dataNmap[i]['cve']);
                  editorVulnerability .multiSet('crapper_vulnerability.cvss_base_score', dataNmap[i]['cvss_base_score']);
                  editorVulnerability  .multiSet('crapper_vulnerability.service_name', dataNmap[i]['service_name']);
                  editorVulnerability  .multiSet('crapper_vulnerability.description', dataNmap[i]['description']);
                  editorVulnerability  .multiSet('crapper_vulnerability.solution', dataNmap[i]['solution']);
                  editorVulnerability    .multiSet('crapper_vulnerability.kundenr', dataNmap[i]['kundenr']);
                  console.log(i);
              }
    editorVulnerability.submit(); 
    
  • tangerinetangerine Posts: 3,342Questions: 35Answers: 394

    Sorry - I didn't think that through well enough.
    Your scenario does suggest a faulty loop, though. What does your console.log of "i" look like?

  • svendbersvendber Posts: 27Questions: 8Answers: 0

    @tangerine
    It loads the first 13 super fast, but then it continues slowly 14..... 15...... 16...... 17....
    After that it get paused before potential out-of-memory crash.

    Can I do anything to help debug further?

  • colincolin Posts: 15,112Questions: 1Answers: 2,583

    Unless I'm missing something, that loop doesn't include/modify an array variable called a. Is it being created or modified elsewhere?

    C

  • svendbersvendber Posts: 27Questions: 8Answers: 0

    @colin
    I create the array like so:

    function nessusSaveClicked() {
    
            tableVuln=   $('#vulnTable2').DataTable();
    
            var rows = tableVuln.rows().data().toArray();
            dataarray = [];
    
    
            for (i=0; i< rows.length ; i++) {
                if (rows[i]['rapport'] === true) {
                    dataarray.push(rows[i]);
                }
            }
            console.log(dataarray);
             impDataMultiSet(dataarray);
        }
    

    I don't modify it anywhere else. If I can help debug further, please let me know!

  • colincolin Posts: 15,112Questions: 1Answers: 2,583

    Hi @svendber ,

    Yep, that's where you create the dataarray array - but the problem that Allan pointed out is a local variable called a. You should follow your code through and see where that is being created.

    Cheers,

    Colin

  • svendbersvendber Posts: 27Questions: 8Answers: 0

    @colin @allan
    Hi again,
    I've been looking through it, and it appears when it comes to multiSet, in the dataTables.editor.min.js. Look at the pictures below.

    I do not have a variable called a in my code. I might not be the best programmer, or even close to, but at this I'm 99.99% sure that this happens somewhere in the datatables library.

    Have a great day! :smile:





  • allanallan Posts: 61,436Questions: 1Answers: 10,049 Site admin

    From those variable names, that's probably in the minified code. Are you able to give me a link to your page so I can trace through what is going on please? Something is leaking into that a array, and I'd need to be able to trace it through to figure out what.

    Thanks,
    Allan

  • svendbersvendber Posts: 27Questions: 8Answers: 0

    @allan unfortunately not. But I'll make a test program within the next few days and test it out.
    Have a great day!

  • svendbersvendber Posts: 27Questions: 8Answers: 0

    @allan
    Hi again,
    I made a test project, and the same thing happened!

    I even upgraded from Editor 1.7.4 -> Editor 1.8.1, only the necessary library etc.
    Button to select file and call function imDataMultiSet

    You don't have any working examples or something like that do you?

    Have a great day.
    //Svendber

  • allanallan Posts: 61,436Questions: 1Answers: 10,049 Site admin

    Any chance of a link to your page? It's not immediately clear what the issue would be there.

    That said, you aren't using i in your loop for multiSet - should it be dataNessus[i] rather than j?

    Allan

  • svendbersvendber Posts: 27Questions: 8Answers: 0

    @allan
    Woops, the 2nd should've been the following:

    But nevertheless the outcome was still the same.. :/
    My project is running on a local server at my workplace, so I sadly can't give you any link to it.

    As far as I know, the problem starts at multiset -> datatables.editor.min.js

    And contiues over to the each -> jquery-1.11.3.min.js

    Is there something else I can do since I can't provide a link?

    Have a great day.
    //Svendber

  • svendbersvendber Posts: 27Questions: 8Answers: 0

    @allan
    Do you have or know any working projects with a large import using mulitset?

  • colincolin Posts: 15,112Questions: 1Answers: 2,583
    Answer ✓

    Hi @svendber ,

    We don't have any examples, other than those in the docs for multiSet(). As you're not able to link to your page, would you be able to modify this example here to demonstrate the problem, please.

    Cheers,

    Colin

This discussion has been closed.