Exporting large amounts of data in Laravel

Generally exporting large amounts of data should be done using a queue backend job service such as Redis or if it can’t be helped then a command that runs during times of less traffic.

When doing so via a command, it can be helpful to process Eloquent records using the chunk function, this preserves RAM. Filtered data can then be saved in memory as an array or collection, such as the below:

$modelsCount = Model::count();
$modelsReachingExpectation = collect();
$bar = $this->output->createProgressBar($modelsCount);

$bar->start();

Model::each(function (Model $model) use (&$bar, &$modelsReachingExpectation) {
	$bar->advance();

	// Condition to skip record.
	if (true) {
		continue;
	}

	$modelsReachingExpectation->push($model->getKey());
});

$bar->finish();

$this->line("\n");

$this->table([
	'Models reaching expectation',
	'Models not reaching expectation',
], [
	[
		$modelsReachingExpectation->count(),
		$modelsCount - $modelsReachingExpectation->count(),
	],
]);

// Perform export

$this->line("Exported models.\n");

In performing the export logic, the array of filtered results could then be passed to a dedicated job. An example could look like the below:

ModelProcessExport::dispatch($modelKeys);

use Illuminate\Bus\Queueable;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Bus\Dispatchable;
use Illuminate\Queue\InteractsWithQueue;
use Illuminate\Queue\SerializesModels;

class ModelProcessExport implements ShouldQueue
{
    use Dispatchable, InteractsWithQueue, Queueable, SerializesModels;
 
    /**
     * Create a new job instance.
     */
    public function __construct(
        public array $modelKeys,
    ) {}
 
    /**
     * Execute the job.
     */
    public function handle(): void
    {
		Model::findMany($this->modelKeys);
		
		// Perform export action...
    }
}