Predis Error on Horizon - unknown command 'SENTINEL'

[jab1000]

I have a Redis Sentinel cluster for our production servers -- the web servers have been running for the past year and have had zero issues. Tried originally to setup Horizon to use our cluster but wasn't ever able to originally, but decided it was time to try again.

official error: production.ERROR: ERR unknown command 'SENTINEL' {"exception":"[object] (Predis\Response\ServerException(code: 0): ERR unknown command 'SENTINEL' at /var/www/tennispoint/vendor/predis/predis/src/Connection/Aggregate/SentinelReplication.php:332)

Yes, we are running Sentinel on all 3 Redis Servers, but interesting thing is we can't run a "sentinel command" from CLI either. I CAN though on the CLI run it with Redis-cli first like this: redis-cli -p 26379 sentinel slaves mymaster

If it is a "redis sentinel server setup" weird though production servers never complain for cache, session, etc stored there.

From what I can tell all is configured correctly for our "Horizon server" and since our "non-horizon laravel" servers aren't having in issue with Redis Sentinel wondering if something else?

To be sure here is our configs overall - Laravel 6.x & latest laravel-redis-sentinel package

My .env file contains:

REDIS_DRIVER=redis-sentinel
QUEUE_CONNECTION=redis-sentinel

REDIS_HORIZON_HOST=ip1,slave-ip1,slave-ip2
REDIS_HORIZON_PORT=26379
REDIS_HORIZON_DB=3
REDIS_QUEUE_DATABASE=3
REDIS_SENTINEL_SERVICE=mymaster
HORIZON_DRIVER=redis-sentinel
REDIS_CLUSTER=redis-sentinel

My database.php file contains:

'redis' => [

        'client' => env('REDIS_CLIENT', 'phpredis'),

        'options' => [
            'cluster' => env('REDIS_CLUSTER', 'redis'),
            'prefix' => env('REDIS_PREFIX', Str::slug(env('APP_NAME', 'laravel'), '_').'_database_'),
        ],

        'default' => [
            'url' => env('REDIS_URL'), /* not used */
            'host' => env('REDIS_HORIZON_HOST', '127.0.0.1'),
            'password' => env('REDIS_HORIZON_PASSWORD', null),
            'port' => env('REDIS_HORIZON_PORT', 6379),
            'database' => env('REDIS_HORIZON_DB', 0),
        ],

My Horizon.php file contains:

'use' => 'default',
'waits' => [
        'redis-sentinel:default' => 60,
 ],
'driver' => 'redis-sentinel',

'environments' => [
        'production' => [
            'supervisor-1' => [
                'connection' => 'redis-sentinel',
                'queue' => ['emailSend', 'pushNotify'],
                'balance' => 'auto',
                'processes' => 10,
                'tries' => 2,
                'retry_after' => 2000,
                'timeout' => 600,
            ],

My Queue.php file contains the following:

'connections' => [
        'redis' => [
            'driver' => 'redis-sentinel',
            'connection' => 'default',
            'queue' => env('REDIS_QUEUE', 'default'),
            'retry_after' => 90,
            'block_for' => null,
        ],
        'redis-sentinel' => [
            'driver' => 'redis-sentinel',
            'connection' => 'default',
            'queue' => 'default',
            'retry_after' => 3666,
            'expire' => 3666,
        ],
    ],

Just in case, here is the full stack trace:

[2020-01-25 19:24:51] production.ERROR: ERR unknown command 'SENTINEL' {"exception":"[object] (Predis\\Response\\ServerException(code: 0): ERR unknown command 'SENTINEL' at /var/www/tennispoint/vendor/predis/predis/src/Connection/Aggregate/SentinelReplication.php:332)
[stacktrace]
#0 /var/www/tennispoint/vendor/predis/predis/src/Connection/Aggregate/SentinelReplication.php(355): Predis\\Connection\\Aggregate\\SentinelReplication->handleSentinelErrorResponse(Object(Predis\\Connection\\StreamConnection), Object(Predis\\Response\\Error))
#1 /var/www/tennispoint/vendor/predis/predis/src/Connection/Aggregate/SentinelReplication.php(427): Predis\\Connection\\Aggregate\\SentinelReplication->querySentinelForMaster(Object(Predis\\Connection\\StreamConnection), 'mymaster')
#2 /var/www/tennispoint/vendor/predis/predis/src/Connection/Aggregate/SentinelReplication.php(498): Predis\\Connection\\Aggregate\\SentinelReplication->getMaster()
#3 /var/www/tennispoint/vendor/predis/predis/src/Connection/Aggregate/SentinelReplication.php(536): Predis\\Connection\\Aggregate\\SentinelReplication->getConnectionInternal(Object(Predis\\Command\\ServerEval))
#4 /var/www/tennispoint/vendor/predis/predis/src/Connection/Aggregate/SentinelReplication.php(658): Predis\\Connection\\Aggregate\\SentinelReplication->getConnection(Object(Predis\\Command\\ServerEval))
#5 /var/www/tennispoint/vendor/predis/predis/src/Connection/Aggregate/SentinelReplication.php(698): Predis\\Connection\\Aggregate\\SentinelReplication->retryCommandOnFailure(Object(Predis\\Command\\ServerEval), 'executeCommand')
#6 /var/www/tennispoint/vendor/predis/predis/src/Client.php(331): Predis\\Connection\\Aggregate\\SentinelReplication->executeCommand(Object(Predis\\Command\\ServerEval))
#7 /var/www/tennispoint/vendor/predis/predis/src/Client.php(314): Predis\\Client->executeCommand(Object(Predis\\Command\\ServerEval))
#8 /var/www/tennispoint/vendor/laravel/framework/src/Illuminate/Redis/Connections/Connection.php(116): Predis\\Client->__call('eval', Array)
#9 /var/www/tennispoint/vendor/laravel/framework/src/Illuminate/Redis/Connections/Connection.php(220): Illuminate\\Redis\\Connections\\Connection->command('eval', Array)
#10 /var/www/tennispoint/vendor/laravel/framework/src/Illuminate/Queue/RedisQueue.php(105): Illuminate\\Redis\\Connections\\Connection->__call('eval', Array)
#11 /var/www/tennispoint/vendor/laravel/framework/src/Illuminate/Queue/RedisQueue.php(91): Illuminate\\Queue\\RedisQueue->pushRaw('{\"displayName\":...', 'emailTrack')
#12 /var/www/tennispoint/vendor/laravel/framework/src/Illuminate/Queue/Queue.php(44): Illuminate\\Queue\\RedisQueue->push(Object(App\\Jobs\\EmailOpened), '', 'emailTrack')
#13 /var/www/tennispoint/vendor/laravel/framework/src/Illuminate/Bus/Dispatcher.php(177): Illuminate\\Queue\\Queue->pushOn('emailTrack', Object(App\\Jobs\\EmailOpened))
#14 /var/www/tennispoint/vendor/laravel/framework/src/Illuminate/Bus/Dispatcher.php(160): Illuminate\\Bus\\Dispatcher->pushCommandToQueue(Object(Illuminate\\Queue\\RedisQueue), Object(App\\Jobs\\EmailOpened))
#15 /var/www/tennispoint/vendor/laravel/framework/src/Illuminate/Bus/Dispatcher.php(73): Illuminate\\Bus\\Dispatcher->dispatchToQueue(Object(App\\Jobs\\EmailOpened))
#16 /var/www/tennispoint/vendor/laravel/framework/src/Illuminate/Foundation/Bus/DispatchesJobs.php(17): Illuminate\\Bus\\Dispatcher->dispatch(Object(App\\Jobs\\EmailOpened))
#17 /var/www/tennispoint/app/Http/Controllers/SesMessageLoggingController.php(358): App\\Http\\Controllers\\Controller->dispatch(Object(App\\Jobs\\EmailOpened))
#18 [internal function]: App\\Http\\Controllers\\SesMessageLoggingController->opened('20950422', 'eyJQSUQiOiIwIiw...', Object(Illuminate\\Http\\Request))
#19 /var/www/tennispoint/vendor/laravel/framework/src/Illuminate/Routing/Controller.php(54): call_user_func_array(Array, Array)
#20 /var/www/tennispoint/vendor/laravel/framework/src/Illuminate/Routing/ControllerDispatcher.php(45): Illuminate\\Routing\\Controller->callAction('opened', Array)
#21 /var/www/tennispoint/vendor/laravel/framework/src/Illuminate/Routing/Route.php(219): Illuminate\\Routing\\ControllerDispatcher->dispatch(Object(Illuminate\\Routing\\Route), Object(App\\Http\\Controllers\\SesMessageLoggingController), 'opened')
#22 /var/www/tennispoint/vendor/laravel/framework/src/Illuminate/Routing/Route.php(176): Illuminate\\Routing\\Route->runController()
#23 /var/www/tennispoint/vendor/laravel/framework/src/Illuminate/Routing/Router.php(681): Illuminate\\Routing\\Route->run()
#24 /var/www/tennispoint/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(130): Illuminate\\Routing\\Router->Illuminate\\Routing\\{closure}(Object(Illuminate\\Http\\Request))
#25 /var/www/tennispoint/vendor/laravel/framework/src/Illuminate/Routing/Middleware/SubstituteBindings.php(41): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}(Object(Illuminate\\Http\\Request))
#26 /var/www/tennispoint/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(171): Illuminate\\Routing\\Middleware\\SubstituteBindings->handle(Object(Illuminate\\Http\\Request), Object(Closure))
#27 /var/www/tennispoint/vendor/laravel/framework/src/Illuminate/View/Middleware/ShareErrorsFromSession.php(49): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}(Object(Illuminate\\Http\\Request))
#28 /var/www/tennispoint/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(171): Illuminate\\View\\Middleware\\ShareErrorsFromSession->handle(Object(Illuminate\\Http\\Request), Object(Closure))
#29 /var/www/tennispoint/vendor/laravel/framework/src/Illuminate/Session/Middleware/StartSession.php(56): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}(Object(Illuminate\\Http\\Request))
#30 /var/www/tennispoint/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(171): Illuminate\\Session\\Middleware\\StartSession->handle(Object(Illuminate\\Http\\Request), Object(Closure))
#31 /var/www/tennispoint/vendor/laravel/framework/src/Illuminate/Cookie/Middleware/AddQueuedCookiesToResponse.php(37): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}(Object(Illuminate\\Http\\Request))
#32 /var/www/tennispoint/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(171): Illuminate\\Cookie\\Middleware\\AddQueuedCookiesToResponse->handle(Object(Illuminate\\Http\\Request), Object(Closure))
#33 /var/www/tennispoint/vendor/laravel/framework/src/Illuminate/Cookie/Middleware/EncryptCookies.php(66): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}(Object(Illuminate\\Http\\Request))
#34 /var/www/tennispoint/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(171): Illuminate\\Cookie\\Middleware\\EncryptCookies->handle(Object(Illuminate\\Http\\Request), Object(Closure))
#35 /var/www/tennispoint/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(105): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}(Object(Illuminate\\Http\\Request))
#36 /var/www/tennispoint/vendor/laravel/framework/src/Illuminate/Routing/Router.php(683): Illuminate\\Pipeline\\Pipeline->then(Object(Closure))
#37 /var/www/tennispoint/vendor/laravel/framework/src/Illuminate/Routing/Router.php(658): Illuminate\\Routing\\Router->runRouteWithinStack(Object(Illuminate\\Routing\\Route), Object(Illuminate\\Http\\Request))
#38 /var/www/tennispoint/vendor/laravel/framework/src/Illuminate/Routing/Router.php(624): Illuminate\\Routing\\Router->runRoute(Object(Illuminate\\Http\\Request), Object(Illuminate\\Routing\\Route))
#39 /var/www/tennispoint/vendor/laravel/framework/src/Illuminate/Routing/Router.php(613): Illuminate\\Routing\\Router->dispatchToRoute(Object(Illuminate\\Http\\Request))
#40 /var/www/tennispoint/vendor/laravel/framework/src/Illuminate/Foundation/Http/Kernel.php(170): Illuminate\\Routing\\Router->dispatch(Object(Illuminate\\Http\\Request))
#41 /var/www/tennispoint/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(130): Illuminate\\Foundation\\Http\\Kernel->Illuminate\\Foundation\\Http\\{closure}(Object(Illuminate\\Http\\Request))
#42 /var/www/tennispoint/vendor/fideloper/proxy/src/TrustProxies.php(57): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}(Object(Illuminate\\Http\\Request))
#43 /var/www/tennispoint/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(171): Fideloper\\Proxy\\TrustProxies->handle(Object(Illuminate\\Http\\Request), Object(Closure))
#44 /var/www/tennispoint/vendor/laravel/framework/src/Illuminate/Foundation/Http/Middleware/TransformsRequest.php(21): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}(Object(Illuminate\\Http\\Request))
#45 /var/www/tennispoint/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(171): Illuminate\\Foundation\\Http\\Middleware\\TransformsRequest->handle(Object(Illuminate\\Http\\Request), Object(Closure))
#46 /var/www/tennispoint/vendor/laravel/framework/src/Illuminate/Foundation/Http/Middleware/TransformsRequest.php(21): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}(Object(Illuminate\\Http\\Request))
#47 /var/www/tennispoint/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(171): Illuminate\\Foundation\\Http\\Middleware\\TransformsRequest->handle(Object(Illuminate\\Http\\Request), Object(Closure))
#48 /var/www/tennispoint/vendor/laravel/framework/src/Illuminate/Foundation/Http/Middleware/ValidatePostSize.php(27): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}(Object(Illuminate\\Http\\Request))
#49 /var/www/tennispoint/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(171): Illuminate\\Foundation\\Http\\Middleware\\ValidatePostSize->handle(Object(Illuminate\\Http\\Request), Object(Closure))
#50 /var/www/tennispoint/vendor/laravel/framework/src/Illuminate/Foundation/Http/Middleware/CheckForMaintenanceMode.php(63): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}(Object(Illuminate\\Http\\Request))
#51 /var/www/tennispoint/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(171): Illuminate\\Foundation\\Http\\Middleware\\CheckForMaintenanceMode->handle(Object(Illuminate\\Http\\Request), Object(Closure))
#52 /var/www/tennispoint/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(105): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}(Object(Illuminate\\Http\\Request))
#53 /var/www/tennispoint/vendor/laravel/framework/src/Illuminate/Foundation/Http/Kernel.php(145): Illuminate\\Pipeline\\Pipeline->then(Object(Closure))
#54 /var/www/tennispoint/vendor/laravel/framework/src/Illuminate/Foundation/Http/Kernel.php(110): Illuminate\\Foundation\\Http\\Kernel->sendRequestThroughRouter(Object(Illuminate\\Http\\Request))
#55 /var/www/tennispoint/public/index.php(55): Illuminate\\Foundation\\Http\\Kernel->handle(Object(Illuminate\\Http\\Request))
#56 {main}
"} 

[cyrossignol]

Hi @jab1000,

It looks like you want to use Sentinel connections for your application's queues and for Horizon. There are some problems with the configuration that you posted. To enable the Sentinel connection for the queue, we need these directives in .env:

QUEUE_CONNECTION=redis-sentinel

REDIS_DRIVER=redis-sentinel
REDIS_HOST=sentinel-ip1,sentinel-ip2,sentinel-ip3,...
REDIS_PORT=26379
REDIS_SENTINEL_SERVICE=mymaster
REDIS_QUEUE_DATABASE=3

We use the REDIS_HOST and REDIS_PORT environment variables here instead of REDIS_SENTINEL_HOST and REDIS_SENTINEL_PORT because you set REDIS_DRIVER=redis-sentinel, so I assume that you want the whole application to use Sentinel connections. Note that we want to provide the IP addresses or hostnames of the Sentinel servers, not the Redis servers that they monitor (set the master in your sentinel.conf file).

The package will automatically set up a Sentinel connection for the queue named database.redis-sentinel.queue. It will also define the queue.connections.redis-sentinel queue connection for us, but since you're using some non-default options, we need to override that connection in config/queue.php:

'connections' => [
    ...
    'redis-sentinel' => [
        'driver' => 'redis-sentinel',
        'connection' => env('QUEUE_REDIS_CONNECTION', 'queue'),
        'queue' => 'default',
        'retry_after' => 3666,
        'expire' => 3666, // Legacy, Laravel < 5.4.30
    ],
],

The queue will now run using Sentinel connections. Since you set REDIS_DRIVER=redis-sentinel, Horizon will also use Sentinel connections automatically, but we need to define a specific connection for Horizon in config/database.php because you want a separate database number for Horizon:

'redis-sentinel' => [

    'queue' => [ ... ],

    'horizon' => [
        [
            'host' => env('REDIS_HOST', 'localhost'),
            'port' => env('REDIS_PORT', 26379),
        ], 
        'options' => [
            'service' => env('REDIS_SENTINEL_SERVICE', 'mymaster'),
            'parameters' => [
                'password' => env('REDIS_PASSWORD', null),
                'database' => env('REDIS_HORIZON_DB', 0),
            ],
        ],
    ],

],

Note that we also need to define the queue connection here now because adding an explicit connection configuration block for the package's Sentinel connections disables the automatic configuration.

Change 'use' => 'default' to 'horizon' in config/horizon.php and add a matching connection to the 'redis' connections array in config/database.php (this is currently a limitation of Horizon):

'redis' => [
    ...
    'horizon' => [ ... ],
],

Finally, your configuration contains some options for Redis Cluster. Be aware that this is a different clustering system than Redis Sentinel, and that this package does not target Redis Cluster. This directive is invalid and has no effect:

REDIS_CLUSTER=redis-sentinel

---

...but interesting thing is we can't run a "sentinel command" from CLI either. I CAN though on the CLI run it with Redis-cli first like this: redis-cli -p 26379 sentinel slaves mymaster

This package's API provides an interface to the Redis servers monitored by Sentinel (the master and replicas that contain the data), not to the Sentinel servers themselves. We can send commands to the Sentinels directly by obtaining a connection to a Sentinel server from the package's connection manager:

use Predis\Command\RawCommand;
...
$sentinel = app('redis-sentinel')->client()->getConnection()->getSentinelConnection();
$command = RawCommand::create('SENTINEL', 'slaves', 'mymaster');

$sentinel->executeCommand($command);

I'm working on an API for low-level Sentinel commands that may become part of this package in the future.

[cyrossignol]

Closing—please comment if you believe this is still an issue.