Add new ReadFileWithVerify storage-layer API (#4349)

This is an enhancement to the XL/distributed-XL mode. FS mode is
unaffected.

The ReadFileWithVerify storage-layer call is similar to ReadFile with
the additional functionality of performing bit-rot checking. It
accepts additional parameters for a hashing algorithm to use and the
expected hex-encoded hash string.

This patch provides significant performance improvement because:

1. combines the step of reading the file (during
erasure-decoding/reconstruction) with bit-rot verification;

2. limits the number of file-reads; and

3. avoids transferring the file over the network for bit-rot
verification.

ReadFile API is implemented as ReadFileWithVerify with empty hashing
arguments.

Credits to AB and Harsha for the algorithmic improvement.

Fixes #4236.

cmd/storage-rpc-client.go

@@ -262,6 +262,35 @@ func (n *networkStorage) ReadFile(volume string, path string, offset int64, buff
 	return int64(len(result)), toStorageErr(err)
 }
 
+// ReadFileWithVerify - reads a file at remote path and fills the buffer.
+func (n *networkStorage) ReadFileWithVerify(volume string, path string, offset int64,
+	buffer []byte, algo HashAlgo, expectedHash string) (m int64, err error) {
+
+	defer func() {
+		if r := recover(); r != nil {
+			// Recover any panic from allocation, and return error.
+			err = bytes.ErrTooLarge
+		}
+	}() // Do not crash the server.
+
+	var result []byte
+	err = n.rpcClient.Call("Storage.ReadFileWithVerifyHandler",
+		&ReadFileWithVerifyArgs{
+			Vol:          volume,
+			Path:         path,
+			Offset:       offset,
+			Buffer:       buffer,
+			Algo:         algo,
+			ExpectedHash: expectedHash,
+		}, &result)
+
+	// Copy results to buffer.
+	copy(buffer, result)
+
+	// Return length of result, err if any.
+	return int64(len(result)), toStorageErr(err)
+}
+
 // ListDir - list all entries at prefix.
 func (n *networkStorage) ListDir(volume, path string) (entries []string, err error) {
 	if err = n.rpcClient.Call("Storage.ListDirHandler", &ListDirArgs{