Sentinel: better time desynchronization.

Sentinels are now desynchronized in a better way changing the time
handler frequency between 10 and 20 HZ. This way on average a
desynchronization of 25 milliesconds is produced that should be larger
enough compared to network latency, avoiding most split-brain condition
during the vote.

Now that the clocks are desynchronized, to have larger random delays when
performing operations can be easily achieved in the following way.
Take as example the function that starts the failover, that is
called with a frequency between 10 and 20 HZ and will start the
failover every time there are the conditions. By just adding as an
additional condition something like rand()%4 == 0, we can amplify the
desynchronization between Sentinel instances easily.

See issue #1419.
This commit is contained in:
antirez 2013-12-02 12:22:17 +01:00
parent 6fa42b7507
commit dffebbc904

View File

@ -2733,12 +2733,9 @@ char *sentinelVoteLeader(sentinelRedisInstance *master, uint64_t req_epoch, char
master->leader, (unsigned long long) master->leader_epoch);
/* If we did not voted for ourselves, set the master failover start
* time to now, in order to force a delay before we can start a
* failover for the same master.
*
* The random addition is useful to desynchronize a bit the slaves
* and reduce the chance that no slave gets majority. */
* failover for the same master. */
if (strcasecmp(master->leader,server.runid))
master->failover_start_time = mstime() + rand() % 2000;
master->failover_start_time = mstime();
}
*leader_epoch = master->leader_epoch;
@ -3389,5 +3386,13 @@ void sentinelTimer(void) {
sentinelRunPendingScripts();
sentinelCollectTerminatedScripts();
sentinelKillTimedoutScripts();
/* We continuously change the frequency of the Redis "timer interrupt"
* in order to desynchronize every Sentinel from every other.
* This non-determinism avoids that Sentinels started at the same time
* exactly continue to stay synchronized asking to be voted at the
* same time again and again (resulting in nobody likely winning the
* election because of split brain voting). */
server.hz = REDIS_DEFAULT_HZ + rand() % REDIS_DEFAULT_HZ;
}