X-RateLimit-Remaining decreasing when cache should be used

[jrean]

Hi,

I scratch my head with this issue. My cache seems to work correctly but I don't understand why for this case it is not...

For instance:

1- I fetch all files within the given path.
2- For each file I fetch its commit(s).

public function testRateLimit()
{
    $contents = $this->client->api('repo')->contents()->show('KnpLabs', 'php-github-api', 'lib/Github/HttpClient/Cache');

    foreach ($contents as $content) {
        $commits = $this->client->api('repo')->commits()->all('KnpLabs', 'php-github-api', array('sha' => 'master', 'path' => $content['path']));
    }
}

Then wherever you want run the following test:

        for ($i = 0; $i < 31; $i++) {
            var_dump("Test $i");
            var_dump($this->testRateLimit());

            if ($i == 10 || $i == 20 || $i == 30) {
                $response = $this->client->getHttpClient()->get("rate_limit");
                var_dump(ResponseMediator::getContent($response)['rate']['remaining'] . ' remaining calls');
            }
        }

As you can see the Github Api rate remaining decrease...

This is how I instantiated the client:

            $cachedClient = new CachedHttpClient;
            $cachedClient->setCache(new FilesystemCache('/tmp/github-api-cache'));

            $client = new Client($cachedClient);

            $client->authenticate('my-token', null, Client::AUTH_HTTP_TOKEN);

            return $client;

Why the cache is not used?

Thank you very much for your great work and your help.

[GrahamCampbell]

As you can see the Github Api rate remaining decrease...

That's because the api limit is cached in the response, then we literally just unserialize the cached response and get the old api limit. This is not actually the current api limit from the original request.

[jrean]

Thank you Graham,

So you mean in reality my api limit remaining is not moving and I’m correctly using the cache even when I’m fetching the commits inside the foreach? :-)

On 26 Apr 2015, at 19:59, Graham Campbell notifications@github.com wrote:

As you can see the Github Api rate remaining decrease...

That's because the api limit is cached in the response, then we literally just unserialize the cached response and get the old api limit. This is not actually the current api limit from the original request.

—
Reply to this email directly or view it on GitHub #262 (comment).

[GrahamCampbell]

Yes. You are just getting the api limit as it was when the original request was made. :)

[jrean]

Ok I got it!
But the number is still decreasing and it shouldn’t! Because the response should be cached and no file(s) change(s) btw each request.

When I fetch the content (files) my api rate remaining doesn’t move because I use the cache. But as soon as I start fetching the commit(s) for each file(s) the api rate remaining seriously decrease for each fetch commits call!

It should not because of the cache !

I’m sorry but I don’t understand :)

On 26 Apr 2015, at 20:40, Graham Campbell notifications@github.com wrote:

Yes. You are just getting the api limit as it was when the original request was made. :)

—
Reply to this email directly or view it on GitHub #262 (comment).

[GrahamCampbell]

remaining seriously decrease for each fetch commits call!

It should decrease with each iteration, because the original requests made where one after another.

[jrean]

Yes you are right, it decreases after each iteration.
But why does it decrease whereas it is the same content requested?

I think I don’t understand something :(

Logically if the content requested is the same the status code should be == 304 and so the cache should be used. Instead a new request is made against the api and my remaining hits decrease.

Let me update the test:

public function testRateLimit()
{
    $contents = $this->client->api('repo')->contents()->show('KnpLabs', 'php-github-api', 'lib/Github/HttpClient/Cache');
}

for ($i = 0; $i < 11; $i++) {
    var_dump("Test $i");
    var_dump($this->testRateLimit());
    $response = $this->client->getHttpClient()->get("rate_limit");
    var_dump(ResponseMediator::getContent($response)['rate']['remaining'] . ' remaining calls');
}

This test outputs:

string(6) "Test 0"
string(4) "Done"
string(20) "4964 remaining calls"
string(6) "Test 1"
string(4) "Done"
string(20) "4964 remaining calls"
string(6) "Test 2"
string(4) "Done"
string(20) "4964 remaining calls"
string(6) "Test 3"
string(4) "Done"
string(20) "4964 remaining calls"
string(6) "Test 4"
string(4) "Done"
string(20) "4964 remaining calls"
string(6) "Test 5"
string(4) "Done"
string(20) "4964 remaining calls"
string(6) "Test 6"
string(4) "Done"
string(20) "4964 remaining calls"
string(6) "Test 7"
string(4) "Done"
string(20) "4964 remaining calls"
string(6) "Test 8"
string(4) "Done"
string(20) "4964 remaining calls"
string(6) "Test 9"
string(4) "Done"
string(20) "4964 remaining calls"
string(7) "Test 10"
string(4) "Done"
string(20) "4964 remaining calls"

As we can see, the remaining calls number don't move because the cache is used for this request.

Now I update the test by adding the sub requests

public function testRateLimit()
{
    $contents = $this->client->api('repo')->contents()->show('KnpLabs', 'php-github-api', 'lib/Github/HttpClient/Cache');

    foreach ($contents as $content) {
        $commits = $this->github->api('repo')->commits()->all('KnpLabs', 'php-github-api', array('sha' => 'master', 'path' => $content['path']));
    }
}

for ($i = 0; $i < 11; $i++) {
    var_dump("Test $i");
    var_dump($this->testRateLimit());
    $response = $this->client->getHttpClient()->get("rate_limit");
    var_dump(ResponseMediator::getContent($response)['rate']['remaining'] . ' remaining calls');
}

This test outputs:

string(6) "Test 0"
string(4) "Done"
string(20) "4961 remaining calls"
string(6) "Test 1"
string(4) "Done"
string(20) "4958 remaining calls"
string(6) "Test 2"
string(4) "Done"
string(20) "4955 remaining calls"
string(6) "Test 3"
string(4) "Done"
string(20) "4952 remaining calls"
string(6) "Test 4"
string(4) "Done"
string(20) "4949 remaining calls"
string(6) "Test 5"
string(4) "Done"
string(20) "4946 remaining calls"
string(6) "Test 6"
string(4) "Done"
string(20) "4943 remaining calls"
string(6) "Test 7"
string(4) "Done"
string(20) "4940 remaining calls"
string(6) "Test 8"
string(4) "Done"
string(20) "4937 remaining calls"
string(6) "Test 9"
string(4) "Done"
string(20) "4934 remaining calls"
string(7) "Test 10"
string(4) "Done"
string(20) "4931 remaining calls"

As we can see, now the remaining calls number decrease as soon as I introduce the sub requests with the foreach. And of course I don't understand why it is not cached.

What do I miss?

Thank you for your answers Graham.

On 26 Apr 2015, at 20:56, Graham Campbell notifications@github.com wrote:

remaining seriously decrease for each fetch commits call!

It should decrease with each iteration, because the original requests made where one after another.

—
Reply to this email directly or view it on GitHub #262 (comment).

[jrean]

I updated my test:

public function testRateLimit()
{
    $contents = $this->client->api('repo')->contents()->show('KnpLabs', 'php-github-api', 'lib/Github/HttpClient/Cache');

    var_dump("Last Request => Response Status Code : " . $this->client->getHttpClient()->getLastRequest()->getResponse()->getStatusCode());
    var_dump("Last Request => Response Rate Limit Remaining : " . $this->client->getHttpClient()->getLastRequest()->getResponse()->getHeader('X-RateLimit-Remaining'));

    $response = $this->client->getHttpClient()->get("rate_limit");
    var_dump("New Github Request => Rate Limit Remaining : " .    ResponseMediator::getContent($response)['rate']['remaining']);


    foreach ($contents as $key => $content) {
        var_dump("foreach iteration $key");

        $commits = $this->client->api('repo')->commits()->all('KnpLabs', 'php-github-api', array('sha' => 'master', 'path' => $content['path']));

        var_dump("Last Request => Response Status Code : " . $this->client->getHttpClient()->getLastRequest()->getResponse()->getStatusCode());

        var_dump("Last Request => Response Rate Limit Remaining : " . $this->client->getHttpClient()->getLastRequest()->getResponse()->getHeader('X-RateLimit-Remaining'));

        $response = $this->client->getHttpClient()->get("rate_limit");
        var_dump("New Github Request => Rate Limit Remaining : " . ResponseMediator::getContent($response)['rate']['remaining']);
        var_dump("/foreach iteration $key");
    }
}

for ($i = 0; $i < 11; $i++) {
    var_dump("Test $i");

    $this->testRateLimit();

    var_dump("/Test $i");
    var_dump("----");
}

The output is clear:

string(6) "Test 0"
string(42) "Last Request => Response Status Code : 304"
string(52) "Last Request => Response Rate Limit Remaining : 4924"
string(49) "New Github Request => Rate Limit Remaining : 4924"
string(19) "foreach iteration 0"
string(42) "Last Request => Response Status Code : 200"
string(52) "Last Request => Response Rate Limit Remaining : 4923"
string(49) "New Github Request => Rate Limit Remaining : 4923"
string(20) "/foreach iteration 0"
string(19) "foreach iteration 1"
string(42) "Last Request => Response Status Code : 200"
string(52) "Last Request => Response Rate Limit Remaining : 4922"
string(49) "New Github Request => Rate Limit Remaining : 4922"
string(20) "/foreach iteration 1"
string(19) "foreach iteration 2"
string(42) "Last Request => Response Status Code : 200"
string(52) "Last Request => Response Rate Limit Remaining : 4921"
string(49) "New Github Request => Rate Limit Remaining : 4921"
string(20) "/foreach iteration 2"
string(7) "/Test 0"
string(4) "----"
string(6) "Test 1"
string(42) "Last Request => Response Status Code : 304"
string(52) "Last Request => Response Rate Limit Remaining : 4921"
string(49) "New Github Request => Rate Limit Remaining : 4921"
string(19) "foreach iteration 0"
string(42) "Last Request => Response Status Code : 200"
string(52) "Last Request => Response Rate Limit Remaining : 4920"
string(49) "New Github Request => Rate Limit Remaining : 4920"
string(20) "/foreach iteration 0"
string(19) "foreach iteration 1"
string(42) "Last Request => Response Status Code : 200"
string(52) "Last Request => Response Rate Limit Remaining : 4919"
string(49) "New Github Request => Rate Limit Remaining : 4919"
string(20) "/foreach iteration 1"
string(19) "foreach iteration 2"
....

The first request has a 304 status code.
Each iteration requests return a 200 status code. So the cache is not used and the X-RateLimit-Remaining decreasing.

@GrahamCampbell please help me. I'm going crazy because I don't understand why these requests inside the iteration are not cached and don't return a 304!

Maybe @pilot?

:)

Another test:

        $response = $this->client->getHttpClient()->get('rate_limit');
        var_dump($response->getHeader('X-RateLimit-Remaining'));

        $commits = $this->client->api('repo')->commits()->all('KnpLabs', 'php-github-api', array('sha' => 'master', 'path' => 'lib/Github/HttpClient/Cache/CacheInterface.php'));

        $response = $this->client->getHttpClient()->getLastRequest()->getResponse()->getStatusCode();

        var_dump($response); // 304

        $response = $this->client->getHttpClient()->get('rate_limit');
        var_dump($response->getHeader('X-RateLimit-Remaining'));

Status code is 304

I update a second commits query:

        $response = $this->client->getHttpClient()->get('rate_limit');
        var_dump($response->getHeader('X-RateLimit-Remaining'));

        $commits = $this->client->api('repo')->commits()->all('KnpLabs', 'php-github-api', array('sha' => 'master', 'path' => 'lib/Github/HttpClient/Cache/CacheInterface.php'));
        $commits = $this->client->api('repo')->commits()->all('KnpLabs', 'php-github-api', array('sha' => 'master', 'path' => 'lib/Github/HttpClient/Cache/FilesystemCache.php'));

        $response = $this->client->getHttpClient()->getLastRequest()->getResponse()->getStatusCode();

        var_dump($response); // 200

        $response = $this->client->getHttpClient()->get('rate_limit');
        var_dump($response->getHeader('X-RateLimit-Remaining'));

Status code is always 200 and the X-RateLimit-Remaining decreases by 2.

[jrean]

#265