Create LazyCollection from API resource
Laravel 6 introduced the LazyCollection, a different kind of collection which exploits php generators to save memory usage.
While data fetching from database is natively managed by eloquent, Web API-based data fetching is not provided.
This article describes how to use a paginated api endpoint to fetch data to a LazyCollection instance.
With this approach, client is not required to deal with page numbers or subsequent api calls. In addition, by using generators, api calls are made transparently behind the scenes only when they are actually required, resulting in memory and time savings.
Steps
- Create test application
- Create dummy data
- Create an API endpoint
- Create API client
Create test application
Let's start by creating a blank laravel application:
1laravel new TestGenerators
Then, since we will use an http client, include the GuzzleHttp library:
1composer require guzzlehttp/guzzle
Make test model
This step is required to create test data. Create a new database for the test application:
1create database TestGenerators;
Generate a test Model along with migrations:
1php artisan make:model Test -m
Create Test model migration as follows
1 public function up()
2 {
3 Schema::create('tests', function (Blueprint $table) {
4 $table->bigIncrements('id');
5 $table->string('value');
6 $table->timestamps();
7 });
8 }
Run migrations:
1php artisan migrate
Now, generate a factory for the Test model:
1php artisan make:factory TestFactory --model=Test
and define TestFactory as follows:
1$factory->define(Test::class, function (Faker $faker) {
2 return [
3 'value' => $faker->word,
4 ];
5});
In order not to incur the MassAssignmentException error, open App\Test.php and make all the field mass assignable:
1class Test extends Model
2{
3 protected $guarded = [];
4}
Finally create a lot of dummy data:
1php artisan tinker
1factory('App\Test', 100000)->create();
Create an API endpoint
This step is required to create an endpoint for fetching dummy data.
Open routes/api.php and add the following endpoint:
1Route::get('/test', function () {
2
3 return \App\Test::paginate();
4
5});
this will create an api endpoint for retrieving data.
Note: you would need to disable the throttle middleware from app/Http/Kernel.php to avoid endpoint protection from too many requests:
1 'api' => [
2// 'throttle:60,1',
3 'bindings',
4 ],
To test the endpoint, simply point your browser to http://<your-host>/api/test, you should see the first page of data.
Create Repository interface and implementations
Before moving to generators, let's create a normal collection. The idea is to load all the api pages, one after the other and merge all the results in a single array.
So, we will implement an API repository, with the following interface:
1<?php
2
3namespace App;
4
5
6interface TestRepositoryInterface
7{
8 public function cursor($url, $options = null);
9}
Create a class with a single public method cursor($url) which will return a Collection instance:
1class TestRepository implements TestRepositoryInterface
2{
3
4 public function cursor($url, $options = null)
5 {
6 if ($options == null) {
7 $options = [
8 'timeout' => 2.0,
9 ];
10 }
11
12 $nextPage = 1;
13 $lastPage = 1;
14
15 $result = [];
16
17 $client = new Client($options);
18
19 while ($nextPage <= $lastPage) {
20 list($data, $nextPage, $lastPage) = $this->getNextPage($client, $nextPage, $url);
21 $result = array_merge($result, $data);
22 }
23
24 return Collection::make($result);
25 }
26
27
28 private function getNextPage(Client $client, int $nextPage, string $url): array
29 {
30 $response = $client->request('GET', $url . '?page=' . $nextPage);
31 $data = json_decode($response->getBody());
32
33 $nextPage = $data->current_page + 1;
34 $lastPage = $data->last_page;
35
36 return array($data->data, $nextPage, $lastPage);
37 }
38}
The core of the class is the while loop, which gets data from one page and merge to create a final $result array.
Data are thus stored in memory and this will become easily unmanageable whene there are lot of data.
In order to exploit generators features, the only modifications required are these:
1 public function cursor($url)
2 {
3 return LazyCollection::make(function () use ($url) {
4 $nextPage = 1;
5 $lastPage = 1;
6
7 $client = new Client([
8 'base_uri' => url('api') . '/',
9 'timeout' => 2.0,
10 ]);
11
12 while ($nextPage <= $lastPage) {
13 list($data, $nextPage, $lastPage) = $this->getNextPage($client, $nextPage, $url);
14
15 yield from $data;
16 }
17 });
18 }
The logic is wrapped in a callback for a LazyCollection generation, and, instead of merging data and returning a final value, each page is yielded into the generator. Now, you can assume you have the full dataset. However, what's most important is that data pages are loaded only if they are actually required.
Testing
For testing this approach, create a new command:
1php artisan make:command TestApiCommand
1<?php
2
3namespace App\Console\Commands;
4
5use App\TestRepositoryGenerators;
6use Illuminate\Console\Command;
7
8class TestApiCommand extends Command
9{
10 /**
11 * The name and signature of the console command.
12 *
13 * @var string
14 */
15 protected $signature = 'test:api';
16
17 /**
18 * The console command description.
19 *
20 * @var string
21 */
22 protected $description = 'Tests the generator api repository';
23
24 /**
25 * Create a new command instance.
26 *
27 * @return void
28 */
29 public function __construct()
30 {
31 parent::__construct();
32 }
33
34 /**
35 * Execute the console command.
36 *
37 * @return mixed
38 */
39 public function handle()
40 {
41 $iterator = new TestRepository();
42 $data = $iterator->cursor(url('api/test'));
43
44 dump($data->first());
45 dump($data->skip(1000)->first());
46 }
47}
You should see in your console the first and tenth element. Note that unused pages would never be loaded.
Final notes
If you have doubts or request, please leave a comment below.