Export streaming for large MongoDB collections

When we export large MongoDB collections, the problem is it consumes a lot of memory and execution time because it will load all query results in memory at once.

For example, below code is exporting all users.

const users = await User.find({}).exec();
const xlsx = exportXlsx(users); // this takes a lot of memory and processing time

In this case, we can use MongoDB cursor. Cursor returns one document as readable stream so you can transform the cursor object into Xlsx writable stream.

const query = Customer.find(profileQuery);

const queryStream = query.stream({
  transform
});

const packer = new XlsxExport({
  ...options,
  stream: queryStream
});

packer.pipe(res); // this pipes the stream directly into response

Here we used xlsx-exportnpm package for transforming MongoDB stream into Xlsx writable pipe.

This reduces a lot of time from export process.