rpc: Our rpcClient should make an attempt to reconnect. (#3221)

rpcClient should attempt a reconnect if the call fails
with 'rpc.ErrShutdown' this is needed since at times when
the servers are taken down and brought back up.

The hijacked connection from net.Dial is usually closed.

So upon first attempt rpcClient might falsely indicate that
disk to be down, to avoid this state make another dial attempt
to really fail.

Fixes #3206
Fixes #3205
This commit is contained in:
Harshavardhana
2016-11-10 07:44:41 -08:00
committed by GitHub
parent cf2fb30ac7
commit 2f7fb78692
9 changed files with 31 additions and 56 deletions

View File

@@ -142,17 +142,26 @@ func (rpcClient *RPCClient) Call(serviceMethod string, args interface{}, reply i
// rpc.Client for a subsequent reconnect.
err := rpcLocalStack.Call(serviceMethod, args, reply)
if err != nil {
if err.Error() == rpc.ErrShutdown.Error() {
// Reset rpcClient.rpc to nil to trigger a reconnect in future
// and close the underlying connection.
rpcClient.clearRPCClient()
// Any errors other than rpc.ErrShutdown just return quickly.
if err != rpc.ErrShutdown {
return err
} // else rpc.ErrShutdown returned by rpc.Call
// Close the underlying connection.
rpcLocalStack.Close()
// Reset the underlying rpc connection before
// moving to reconnect.
rpcClient.clearRPCClient()
// Set rpc error as rpc.ErrShutdown type.
err = rpc.ErrShutdown
// Close the underlying connection before reconnect.
rpcLocalStack.Close()
// Try once more to re-connect.
rpcLocalStack, err = rpcClient.dialRPCClient()
if err != nil {
return err
}
// Attempt the rpc.Call once again, upon any error now just give up.
err = rpcLocalStack.Call(serviceMethod, args, reply)
}
return err
}