How to export millions of pieces of data to EXCEL without reporting OOM exceptions?

POI is used in the Java project to export millions of pieces of data to Excel, but a memory overflow exception occurs.

there are the following problems to consider

  1. the number of POI exports is limited to 6w +
  2. A large amount of data will lead to memory overflow

the current practice is to split every 6w pieces of data to create a new sheet, but this is very slow

        List<List<Object>> result = new ArrayList<List<Object>>();
        List<Object> dataList = new ArrayList<Object>();
        if (resultList != null) {
            for (int i = 0; i < resultList.size(); iPP) {
                Map<String, Object> map = getDataByClass(resultList.get(i));
                dataList.add(map);
                if (dataList.size() % 60000 == 0 && dataList.size() != 0) {
                    result.add(dataList);
                    dataList = new ArrayList<Object>();
                }
            }
            if(dataList.size()!=0){
                result.add(dataList);
            }
        }

I hope you can give some suggestions and think of a better solution to this problem ~

May.15,2021

there is no good way to speed. If you have memory, set the startup parameter of Java-Xmx to increase some memory.
then my suggestion is that if you can accept csv, export csv, it will be much faster, not even poi.


The

sheet does not need to create a new one, and the
query does not look it up all at once.

should be:
check 6w, write once sheet,
check 6w, write once sheet,
end, send

MySQL Query : SELECT * FROM `codeshelper`.`v9_news` WHERE status=99 AND catid='6' ORDER BY rand() LIMIT 5
MySQL Error : Disk full (/tmp/#sql-temptable-64f5-1b3a304-2c220.MAI); waiting for someone to free some space... (errno: 28 "No space left on device")
MySQL Errno : 1021
Message : Disk full (/tmp/#sql-temptable-64f5-1b3a304-2c220.MAI); waiting for someone to free some space... (errno: 28 "No space left on device")
Need Help?