Loading a large file into a datatable in ASP.NET-MVC

dec4

Hi,

I'm trying to load a datatable with the data from a large file (containing around 300k lines, size : ~700 Mo) containing mulitple types of lines (for example, in a line, from position 1 to 7, I have one property, etc. and it differs between the different types of lines existing in the file).

When I try to do it with a relatively small file, it works, but when I try with a much larger file like the one mentionned above, it doesn't. It processes the data for a few minutes and then I get an out of memory exception.

The datatable code looks like that:

<script>
    $(document).ready(function () {
        $('#myDataTable').DataTable({
            "scrollY": 500,
            "scrollX": true,
            "scrollCollapse": true,
            "scroller": {
                loadingIndicator: true
            }
                }
            }
        });
    });
</script>

<div class="container">
    <table id="myDataTable" class="table table-bordered">
        <thead>
            <tr>
                @foreach (var item in criSelectedColumn.Columns)
                {
                    <th>
                        @item
                    </th>
                }
            </tr>
        </thead>

        <tbody>
            @foreach (var item in criSelectedColumn.Data)
            {
                <tr>
                    @foreach (var itemItem in @item.GetType().GetFields().Where(x => !x.Name.Contains("Set") && !x.Name.Contains("Get")))
                    {
                        <td>
                            @itemItem.GetValue(item)
                        </td>
                    }
                </tr>
            }
        </tbody>
    </table>
</div>

I tried removing the "Where" part but it doesn't really help.

Could you please advise me on how I should proceed to manage to load and display a large file's data with datatable ? Possibly without having to store the file's data in a database ?

Thanks in advance for your help!

colin

Hi @dec4 ,

That's a lot of lines without any server-side support. You could look at this FAQ entry here - it has some suggestions on how to speed things up.

Cheers,

Colin