Skip to content

Export Processor Limitation on Processing Large Amount of Records

Release Date: 5/13/2021
Version: 8.0 & Up

QI exported all invoice header records from our production company using Export Processor. No filters were used. As you can imagine, this took a long time. No errors were encountered while it was processing either the process or export jobs.

It said it exported 1,621,017 records which seems about right I guess. In the actual exported file, when it hit record 1,242,387, it started repeating the same exact row of data until the end of the file.  See sample screen below.


It also takes a very long time to close the Export Processor window after clicking the X button. I don't know if some cleanup is happening or what, but it took much longer than expected.

A - I think the problem you experienced is because the system ran out of memory.

In the first phase of Export Processor, you will use “get records” in the Process tab. The Process tab will read all qualified invoice records and put them in the “List View.” The handling of the List View memory is actually done by the OS, not by us. All we do is just keep on pushing records to the List View and let the OS obtain more memory to handle our request. Elliott is a 32-bit application, and at some point, the OS will run out of memory for a 32-bit application. I know the maximum memory that a 32-bit application can access is 2GB. I don’t know if the OS will further limit that in a List View.

If you like, you can use Export Processor and choose “get records” again, to see how the Elliott memory grows in the Task Manager and where it stops. That will give you some clue. My test shows each 100,000 records takes about 100MB of memory. I did not test to go beyond a million rows. But you can see my sample screen below to get an idea:



When you try to exit Export Processor, it needs to release all the memory used by the application, so it takes a long time. This is a normal behavior.

If this limitation is because of memory, I think the strategy for you is to break it up into two batches to export and append the file when done.

Just FYI, Excel has a limitation of 1,048,576 rows.

EMK

Feedback and Knowledge Base