MyFavMirrors
/
minio
mirror of https://github.com/minio/minio.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
minio/internal/hash/reader_test.go

316 lines
10 KiB
Go
Raw Normal View History

update license change for MinIO Signed-off-by: Harshavardhana <harsha@minio.io>
2021-04-18 15:41:13 -04:00
// Copyright (c) 2015-2021 MinIO, Inc.
//
// This file is part of MinIO Object Storage stack
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU Affero General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU Affero General Public License for more details.
//
// You should have received a copy of the GNU Affero General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
Simplify data verification with HashReader. (#5071) Verify() was being called by caller after the data has been successfully read after io.EOF. This disconnection opens a race under concurrent access to such an object. Verification is not necessary outside of Read() call, we can simply just do checksum verification right inside Read() call at io.EOF. This approach simplifies the usage.
2017-10-22 01:30:34 -04:00
package hash
import (
"bytes"
fix: under FanOut API avoid repeated md5sum calculation (#17572) md5sum calculation has a high CPU overhead, avoid calculating it repeatedly for similar fanOut calls. To fix following CPU profiler result ``` (pprof) top10 Showing nodes accounting for 678.68s, 84.67% of 801.54s total Dropped 1072 nodes (cum <= 4.01s) Showing top 10 nodes out of 156 flat flat% sum% cum cum% 332.54s 41.49% 41.49% 332.54s 41.49% runtime/internal/syscall.Syscall6 228.39s 28.49% 69.98% 228.39s 28.49% crypto/md5.block 48.07s 6.00% 75.98% 48.07s 6.00% runtime.memmove 28.91s 3.61% 79.59% 28.91s 3.61% github.com/minio/highwayhash.updateAVX2 8.25s 1.03% 80.61% 8.25s 1.03% runtime.futex 8.25s 1.03% 81.64% 10.81s 1.35% runtime.step 6.99s 0.87% 82.52% 22.35s 2.79% runtime.pcvalue 6.67s 0.83% 83.35% 38.90s 4.85% runtime.mallocgc 5.77s 0.72% 84.07% 32.61s 4.07% runtime.gentraceback 4.84s 0.6% 84.67% 10.49s 1.31% runtime.lock2 ```
2023-07-05 06:16:05 -04:00
"encoding/base64"
Simplify data verification with HashReader. (#5071) Verify() was being called by caller after the data has been successfully read after io.EOF. This disconnection opens a race under concurrent access to such an object. Verification is not necessary outside of Read() call, we can simply just do checksum verification right inside Read() call at io.EOF. This approach simplifies the usage.
2017-10-22 01:30:34 -04:00
"encoding/hex"
merge nested hash readers (#9582) The `ioutil.NopCloser(reader)` was hiding nested hash readers. We make it an `io.Closer` so it can be attached without wrapping and allows for nesting, by merging the requests.
2020-05-14 17:01:31 -04:00
"fmt"
Simplify data verification with HashReader. (#5071) Verify() was being called by caller after the data has been successfully read after io.EOF. This disconnection opens a race under concurrent access to such an object. Verification is not necessary outside of Read() call, we can simply just do checksum verification right inside Read() call at io.EOF. This approach simplifies the usage.
2017-10-22 01:30:34 -04:00
"io"
"testing"
Signed trailers for signature v4 (#16484)
2023-05-05 22:53:12 -04:00
"github.com/minio/minio/internal/ioutil"
Simplify data verification with HashReader. (#5071) Verify() was being called by caller after the data has been successfully read after io.EOF. This disconnection opens a race under concurrent access to such an object. Verification is not necessary outside of Read() call, we can simply just do checksum verification right inside Read() call at io.EOF. This approach simplifies the usage.
2017-10-22 01:30:34 -04:00
)
// Tests functions like Size(), MD5*(), SHA256*()
func TestHashReaderHelperMethods(t *testing.T) {
pkg/etag: add new package for S3 ETag handling (#11577) This commit adds a new package `etag` for dealing with S3 ETags. Even though ETag is often viewed as MD5 checksum of an object, handling S3 ETags correctly is a surprisingly complex task. While it is true that the ETag corresponds to the MD5 for the most basic S3 API operations, there are many exceptions in case of multipart uploads or encryption. In worse, some S3 clients expect very specific behavior when it comes to ETags. For example, some clients expect that the ETag is a double-quoted string and fail otherwise. Non-AWS compliant ETag handling has been a source of many bugs in the past. Therefore, this commit adds a dedicated `etag` package that provides functionality for parsing, generating and converting S3 ETags. Further, this commit removes the ETag computation from the `hash` package. Instead, the `hash` package (i.e. `hash.Reader`) should focus only on computing and verifying the content-sha256. One core feature of this commit is to provide a mechanism to communicate a computed ETag from a low-level `io.Reader` to a high-level `io.Reader`. This problem occurs when an S3 server receives a request and has to compute the ETag of the content. However, the server may also wrap the initial body with several other `io.Reader`, e.g. when encrypting or compressing the content: ``` reader := Encrypt(Compress(ETag(content))) ``` In such a case, the ETag should be accessible by the high-level `io.Reader`. The `etag` provides a mechanism to wrap `io.Reader` implementations such that the `ETag` can be accessed by a type-check. This technique is applied to the PUT, COPY and Upload handlers.
2021-02-23 15:31:53 -05:00
r, err := NewReader(bytes.NewReader([]byte("abcd")), 4, "e2fc714c4727ee9395f324cd2e7f331f", "88d4266fd4e6338d13b845fcf289579d209c897823b9217da3e161936f031589", 4)
Simplify data verification with HashReader. (#5071) Verify() was being called by caller after the data has been successfully read after io.EOF. This disconnection opens a race under concurrent access to such an object. Verification is not necessary outside of Read() call, we can simply just do checksum verification right inside Read() call at io.EOF. This approach simplifies the usage.
2017-10-22 01:30:34 -04:00
if err != nil {
t.Fatal(err)
}
Remove deprecated io/ioutil (#15707)
2022-09-19 14:05:16 -04:00
_, err = io.Copy(io.Discard, r)
Simplify data verification with HashReader. (#5071) Verify() was being called by caller after the data has been successfully read after io.EOF. This disconnection opens a race under concurrent access to such an object. Verification is not necessary outside of Read() call, we can simply just do checksum verification right inside Read() call at io.EOF. This approach simplifies the usage.
2017-10-22 01:30:34 -04:00
if err != nil {
t.Fatal(err)
}
fix: under FanOut API avoid repeated md5sum calculation (#17572) md5sum calculation has a high CPU overhead, avoid calculating it repeatedly for similar fanOut calls. To fix following CPU profiler result ``` (pprof) top10 Showing nodes accounting for 678.68s, 84.67% of 801.54s total Dropped 1072 nodes (cum <= 4.01s) Showing top 10 nodes out of 156 flat flat% sum% cum cum% 332.54s 41.49% 41.49% 332.54s 41.49% runtime/internal/syscall.Syscall6 228.39s 28.49% 69.98% 228.39s 28.49% crypto/md5.block 48.07s 6.00% 75.98% 48.07s 6.00% runtime.memmove 28.91s 3.61% 79.59% 28.91s 3.61% github.com/minio/highwayhash.updateAVX2 8.25s 1.03% 80.61% 8.25s 1.03% runtime.futex 8.25s 1.03% 81.64% 10.81s 1.35% runtime.step 6.99s 0.87% 82.52% 22.35s 2.79% runtime.pcvalue 6.67s 0.83% 83.35% 38.90s 4.85% runtime.mallocgc 5.77s 0.72% 84.07% 32.61s 4.07% runtime.gentraceback 4.84s 0.6% 84.67% 10.49s 1.31% runtime.lock2 ```
2023-07-05 06:16:05 -04:00
md5sum := r.MD5Current()
if hex.EncodeToString(md5sum) != "e2fc714c4727ee9395f324cd2e7f331f" {
t.Errorf("Expected md5hex \"e2fc714c4727ee9395f324cd2e7f331f\", got %s", hex.EncodeToString(md5sum))
Simplify data verification with HashReader. (#5071) Verify() was being called by caller after the data has been successfully read after io.EOF. This disconnection opens a race under concurrent access to such an object. Verification is not necessary outside of Read() call, we can simply just do checksum verification right inside Read() call at io.EOF. This approach simplifies the usage.
2017-10-22 01:30:34 -04:00
}
if r.SHA256HexString() != "88d4266fd4e6338d13b845fcf289579d209c897823b9217da3e161936f031589" {
t.Errorf("Expected sha256hex \"88d4266fd4e6338d13b845fcf289579d209c897823b9217da3e161936f031589\", got %s", r.SHA256HexString())
}
fix: under FanOut API avoid repeated md5sum calculation (#17572) md5sum calculation has a high CPU overhead, avoid calculating it repeatedly for similar fanOut calls. To fix following CPU profiler result ``` (pprof) top10 Showing nodes accounting for 678.68s, 84.67% of 801.54s total Dropped 1072 nodes (cum <= 4.01s) Showing top 10 nodes out of 156 flat flat% sum% cum cum% 332.54s 41.49% 41.49% 332.54s 41.49% runtime/internal/syscall.Syscall6 228.39s 28.49% 69.98% 228.39s 28.49% crypto/md5.block 48.07s 6.00% 75.98% 48.07s 6.00% runtime.memmove 28.91s 3.61% 79.59% 28.91s 3.61% github.com/minio/highwayhash.updateAVX2 8.25s 1.03% 80.61% 8.25s 1.03% runtime.futex 8.25s 1.03% 81.64% 10.81s 1.35% runtime.step 6.99s 0.87% 82.52% 22.35s 2.79% runtime.pcvalue 6.67s 0.83% 83.35% 38.90s 4.85% runtime.mallocgc 5.77s 0.72% 84.07% 32.61s 4.07% runtime.gentraceback 4.84s 0.6% 84.67% 10.49s 1.31% runtime.lock2 ```
2023-07-05 06:16:05 -04:00
if base64.StdEncoding.EncodeToString(md5sum) != "4vxxTEcn7pOV8yTNLn8zHw==" {
t.Errorf("Expected md5base64 \"4vxxTEcn7pOV8yTNLn8zHw==\", got \"%s\"", base64.StdEncoding.EncodeToString(md5sum))
Add base64 encoded MD5 output for Hash Reader (#5315) - Use it to send the Content-MD5 header correctly encoded to S3 Gateway - Fixes a bug in PutObject (including anonymous PutObject) and PutObjectPart with S3 Gateway found when testing with Mint.
2017-12-21 20:27:33 -05:00
}
Simplify data verification with HashReader. (#5071) Verify() was being called by caller after the data has been successfully read after io.EOF. This disconnection opens a race under concurrent access to such an object. Verification is not necessary outside of Read() call, we can simply just do checksum verification right inside Read() call at io.EOF. This approach simplifies the usage.
2017-10-22 01:30:34 -04:00
if r.Size() != 4 {
t.Errorf("Expected size 4, got %d", r.Size())
}
Add object compression support (#6292) Add support for streaming (golang/LZ77/snappy) compression.
2018-09-27 23:36:17 -04:00
if r.ActualSize() != 4 {
t.Errorf("Expected size 4, got %d", r.ActualSize())
}
Simplify data verification with HashReader. (#5071) Verify() was being called by caller after the data has been successfully read after io.EOF. This disconnection opens a race under concurrent access to such an object. Verification is not necessary outside of Read() call, we can simply just do checksum verification right inside Read() call at io.EOF. This approach simplifies the usage.
2017-10-22 01:30:34 -04:00
expectedMD5, err := hex.DecodeString("e2fc714c4727ee9395f324cd2e7f331f")
if err != nil {
t.Fatal(err)
}
Set etag properly to calculated value if available (#5106) Fixes #5100
2017-10-24 15:25:42 -04:00
if !bytes.Equal(r.MD5Current(), expectedMD5) {
t.Errorf("Expected md5hex \"e2fc714c4727ee9395f324cd2e7f331f\", got %s", hex.EncodeToString(r.MD5Current()))
}
Simplify data verification with HashReader. (#5071) Verify() was being called by caller after the data has been successfully read after io.EOF. This disconnection opens a race under concurrent access to such an object. Verification is not necessary outside of Read() call, we can simply just do checksum verification right inside Read() call at io.EOF. This approach simplifies the usage.
2017-10-22 01:30:34 -04:00
expectedSHA256, err := hex.DecodeString("88d4266fd4e6338d13b845fcf289579d209c897823b9217da3e161936f031589")
Introduce staticcheck for stricter builds (#7035)
2019-02-13 07:59:36 -05:00
if err != nil {
t.Fatal(err)
}
Simplify data verification with HashReader. (#5071) Verify() was being called by caller after the data has been successfully read after io.EOF. This disconnection opens a race under concurrent access to such an object. Verification is not necessary outside of Read() call, we can simply just do checksum verification right inside Read() call at io.EOF. This approach simplifies the usage.
2017-10-22 01:30:34 -04:00
if !bytes.Equal(r.SHA256(), expectedSHA256) {
t.Errorf("Expected md5hex \"88d4266fd4e6338d13b845fcf289579d209c897823b9217da3e161936f031589\", got %s", r.SHA256HexString())
}
}
// Tests hash reader checksum verification.
func TestHashReaderVerification(t *testing.T) {
testCases := []struct {
merge nested hash readers (#9582) The `ioutil.NopCloser(reader)` was hiding nested hash readers. We make it an `io.Closer` so it can be attached without wrapping and allows for nesting, by merging the requests.
2020-05-14 17:01:31 -04:00
desc string
Simplify data verification with HashReader. (#5071) Verify() was being called by caller after the data has been successfully read after io.EOF. This disconnection opens a race under concurrent access to such an object. Verification is not necessary outside of Read() call, we can simply just do checksum verification right inside Read() call at io.EOF. This approach simplifies the usage.
2017-10-22 01:30:34 -04:00
src io.Reader
size int64
Add object compression support (#6292) Add support for streaming (golang/LZ77/snappy) compression.
2018-09-27 23:36:17 -04:00
actualSize int64
Simplify data verification with HashReader. (#5071) Verify() was being called by caller after the data has been successfully read after io.EOF. This disconnection opens a race under concurrent access to such an object. Verification is not necessary outside of Read() call, we can simply just do checksum verification right inside Read() call at io.EOF. This approach simplifies the usage.
2017-10-22 01:30:34 -04:00
md5hex, sha256hex string
err error
}{
Signed trailers for signature v4 (#16484)
2023-05-05 22:53:12 -04:00
0: {
merge nested hash readers (#9582) The `ioutil.NopCloser(reader)` was hiding nested hash readers. We make it an `io.Closer` so it can be attached without wrapping and allows for nesting, by merging the requests.
2020-05-14 17:01:31 -04:00
desc: "Success, no checksum verification provided.",
Add object compression support (#6292) Add support for streaming (golang/LZ77/snappy) compression.
2018-09-27 23:36:17 -04:00
src: bytes.NewReader([]byte("abcd")),
size: 4,
actualSize: 4,
Simplify data verification with HashReader. (#5071) Verify() was being called by caller after the data has been successfully read after io.EOF. This disconnection opens a race under concurrent access to such an object. Verification is not necessary outside of Read() call, we can simply just do checksum verification right inside Read() call at io.EOF. This approach simplifies the usage.
2017-10-22 01:30:34 -04:00
},
{
merge nested hash readers (#9582) The `ioutil.NopCloser(reader)` was hiding nested hash readers. We make it an `io.Closer` so it can be attached without wrapping and allows for nesting, by merging the requests.
2020-05-14 17:01:31 -04:00
desc: "Failure md5 mismatch.",
Add object compression support (#6292) Add support for streaming (golang/LZ77/snappy) compression.
2018-09-27 23:36:17 -04:00
src: bytes.NewReader([]byte("abcd")),
size: 4,
actualSize: 4,
md5hex: "d41d8cd98f00b204e9800998ecf8427f",
Simplify data verification with HashReader. (#5071) Verify() was being called by caller after the data has been successfully read after io.EOF. This disconnection opens a race under concurrent access to such an object. Verification is not necessary outside of Read() call, we can simply just do checksum verification right inside Read() call at io.EOF. This approach simplifies the usage.
2017-10-22 01:30:34 -04:00
err: BadDigest{
"d41d8cd98f00b204e9800998ecf8427f",
"e2fc714c4727ee9395f324cd2e7f331f",
},
},
{
merge nested hash readers (#9582) The `ioutil.NopCloser(reader)` was hiding nested hash readers. We make it an `io.Closer` so it can be attached without wrapping and allows for nesting, by merging the requests.
2020-05-14 17:01:31 -04:00
desc: "Failure sha256 mismatch.",
Add object compression support (#6292) Add support for streaming (golang/LZ77/snappy) compression.
2018-09-27 23:36:17 -04:00
src: bytes.NewReader([]byte("abcd")),
size: 4,
actualSize: 4,
sha256hex: "88d4266fd4e6338d13b845fcf289579d209c897823b9217da3e161936f031580",
Simplify data verification with HashReader. (#5071) Verify() was being called by caller after the data has been successfully read after io.EOF. This disconnection opens a race under concurrent access to such an object. Verification is not necessary outside of Read() call, we can simply just do checksum verification right inside Read() call at io.EOF. This approach simplifies the usage.
2017-10-22 01:30:34 -04:00
err: SHA256Mismatch{
"88d4266fd4e6338d13b845fcf289579d209c897823b9217da3e161936f031580",
"88d4266fd4e6338d13b845fcf289579d209c897823b9217da3e161936f031589",
},
},
merge nested hash readers (#9582) The `ioutil.NopCloser(reader)` was hiding nested hash readers. We make it an `io.Closer` so it can be attached without wrapping and allows for nesting, by merging the requests.
2020-05-14 17:01:31 -04:00
{
desc: "Nested hash reader NewReader() should merge.",
pkg/etag: add new package for S3 ETag handling (#11577) This commit adds a new package `etag` for dealing with S3 ETags. Even though ETag is often viewed as MD5 checksum of an object, handling S3 ETags correctly is a surprisingly complex task. While it is true that the ETag corresponds to the MD5 for the most basic S3 API operations, there are many exceptions in case of multipart uploads or encryption. In worse, some S3 clients expect very specific behavior when it comes to ETags. For example, some clients expect that the ETag is a double-quoted string and fail otherwise. Non-AWS compliant ETag handling has been a source of many bugs in the past. Therefore, this commit adds a dedicated `etag` package that provides functionality for parsing, generating and converting S3 ETags. Further, this commit removes the ETag computation from the `hash` package. Instead, the `hash` package (i.e. `hash.Reader`) should focus only on computing and verifying the content-sha256. One core feature of this commit is to provide a mechanism to communicate a computed ETag from a low-level `io.Reader` to a high-level `io.Reader`. This problem occurs when an S3 server receives a request and has to compute the ETag of the content. However, the server may also wrap the initial body with several other `io.Reader`, e.g. when encrypting or compressing the content: ``` reader := Encrypt(Compress(ETag(content))) ``` In such a case, the ETag should be accessible by the high-level `io.Reader`. The `etag` provides a mechanism to wrap `io.Reader` implementations such that the `ETag` can be accessed by a type-check. This technique is applied to the PUT, COPY and Upload handlers.
2021-02-23 15:31:53 -05:00
src: mustReader(t, bytes.NewReader([]byte("abcd")), 4, "", "", 4),
merge nested hash readers (#9582) The `ioutil.NopCloser(reader)` was hiding nested hash readers. We make it an `io.Closer` so it can be attached without wrapping and allows for nesting, by merging the requests.
2020-05-14 17:01:31 -04:00
size: 4,
actualSize: 4,
},
{
desc: "Incorrect sha256, nested",
pkg/etag: add new package for S3 ETag handling (#11577) This commit adds a new package `etag` for dealing with S3 ETags. Even though ETag is often viewed as MD5 checksum of an object, handling S3 ETags correctly is a surprisingly complex task. While it is true that the ETag corresponds to the MD5 for the most basic S3 API operations, there are many exceptions in case of multipart uploads or encryption. In worse, some S3 clients expect very specific behavior when it comes to ETags. For example, some clients expect that the ETag is a double-quoted string and fail otherwise. Non-AWS compliant ETag handling has been a source of many bugs in the past. Therefore, this commit adds a dedicated `etag` package that provides functionality for parsing, generating and converting S3 ETags. Further, this commit removes the ETag computation from the `hash` package. Instead, the `hash` package (i.e. `hash.Reader`) should focus only on computing and verifying the content-sha256. One core feature of this commit is to provide a mechanism to communicate a computed ETag from a low-level `io.Reader` to a high-level `io.Reader`. This problem occurs when an S3 server receives a request and has to compute the ETag of the content. However, the server may also wrap the initial body with several other `io.Reader`, e.g. when encrypting or compressing the content: ``` reader := Encrypt(Compress(ETag(content))) ``` In such a case, the ETag should be accessible by the high-level `io.Reader`. The `etag` provides a mechanism to wrap `io.Reader` implementations such that the `ETag` can be accessed by a type-check. This technique is applied to the PUT, COPY and Upload handlers.
2021-02-23 15:31:53 -05:00
src: mustReader(t, bytes.NewReader([]byte("abcd")), 4, "", "", 4),
merge nested hash readers (#9582) The `ioutil.NopCloser(reader)` was hiding nested hash readers. We make it an `io.Closer` so it can be attached without wrapping and allows for nesting, by merging the requests.
2020-05-14 17:01:31 -04:00
size: 4,
actualSize: 4,
sha256hex: "50d858e0985ecc7f60418aaf0cc5ab587f42c2570a884095a9e8ccacd0f6545c",
err: SHA256Mismatch{
ExpectedSHA256: "50d858e0985ecc7f60418aaf0cc5ab587f42c2570a884095a9e8ccacd0f6545c",
CalculatedSHA256: "88d4266fd4e6338d13b845fcf289579d209c897823b9217da3e161936f031589",
},
},
Signed trailers for signature v4 (#16484)
2023-05-05 22:53:12 -04:00
5: {
merge nested hash readers (#9582) The `ioutil.NopCloser(reader)` was hiding nested hash readers. We make it an `io.Closer` so it can be attached without wrapping and allows for nesting, by merging the requests.
2020-05-14 17:01:31 -04:00
desc: "Correct sha256, nested",
pkg/etag: add new package for S3 ETag handling (#11577) This commit adds a new package `etag` for dealing with S3 ETags. Even though ETag is often viewed as MD5 checksum of an object, handling S3 ETags correctly is a surprisingly complex task. While it is true that the ETag corresponds to the MD5 for the most basic S3 API operations, there are many exceptions in case of multipart uploads or encryption. In worse, some S3 clients expect very specific behavior when it comes to ETags. For example, some clients expect that the ETag is a double-quoted string and fail otherwise. Non-AWS compliant ETag handling has been a source of many bugs in the past. Therefore, this commit adds a dedicated `etag` package that provides functionality for parsing, generating and converting S3 ETags. Further, this commit removes the ETag computation from the `hash` package. Instead, the `hash` package (i.e. `hash.Reader`) should focus only on computing and verifying the content-sha256. One core feature of this commit is to provide a mechanism to communicate a computed ETag from a low-level `io.Reader` to a high-level `io.Reader`. This problem occurs when an S3 server receives a request and has to compute the ETag of the content. However, the server may also wrap the initial body with several other `io.Reader`, e.g. when encrypting or compressing the content: ``` reader := Encrypt(Compress(ETag(content))) ``` In such a case, the ETag should be accessible by the high-level `io.Reader`. The `etag` provides a mechanism to wrap `io.Reader` implementations such that the `ETag` can be accessed by a type-check. This technique is applied to the PUT, COPY and Upload handlers.
2021-02-23 15:31:53 -05:00
src: mustReader(t, bytes.NewReader([]byte("abcd")), 4, "", "", 4),
merge nested hash readers (#9582) The `ioutil.NopCloser(reader)` was hiding nested hash readers. We make it an `io.Closer` so it can be attached without wrapping and allows for nesting, by merging the requests.
2020-05-14 17:01:31 -04:00
size: 4,
actualSize: 4,
sha256hex: "88d4266fd4e6338d13b845fcf289579d209c897823b9217da3e161936f031589",
},
{
desc: "Correct sha256, nested, truncated",
pkg/etag: add new package for S3 ETag handling (#11577) This commit adds a new package `etag` for dealing with S3 ETags. Even though ETag is often viewed as MD5 checksum of an object, handling S3 ETags correctly is a surprisingly complex task. While it is true that the ETag corresponds to the MD5 for the most basic S3 API operations, there are many exceptions in case of multipart uploads or encryption. In worse, some S3 clients expect very specific behavior when it comes to ETags. For example, some clients expect that the ETag is a double-quoted string and fail otherwise. Non-AWS compliant ETag handling has been a source of many bugs in the past. Therefore, this commit adds a dedicated `etag` package that provides functionality for parsing, generating and converting S3 ETags. Further, this commit removes the ETag computation from the `hash` package. Instead, the `hash` package (i.e. `hash.Reader`) should focus only on computing and verifying the content-sha256. One core feature of this commit is to provide a mechanism to communicate a computed ETag from a low-level `io.Reader` to a high-level `io.Reader`. This problem occurs when an S3 server receives a request and has to compute the ETag of the content. However, the server may also wrap the initial body with several other `io.Reader`, e.g. when encrypting or compressing the content: ``` reader := Encrypt(Compress(ETag(content))) ``` In such a case, the ETag should be accessible by the high-level `io.Reader`. The `etag` provides a mechanism to wrap `io.Reader` implementations such that the `ETag` can be accessed by a type-check. This technique is applied to the PUT, COPY and Upload handlers.
2021-02-23 15:31:53 -05:00
src: mustReader(t, bytes.NewReader([]byte("abcd-more-stuff-to-be ignored")), 4, "", "", 4),
merge nested hash readers (#9582) The `ioutil.NopCloser(reader)` was hiding nested hash readers. We make it an `io.Closer` so it can be attached without wrapping and allows for nesting, by merging the requests.
2020-05-14 17:01:31 -04:00
size: 4,
actualSize: -1,
sha256hex: "88d4266fd4e6338d13b845fcf289579d209c897823b9217da3e161936f031589",
Signed trailers for signature v4 (#16484)
2023-05-05 22:53:12 -04:00
err: ioutil.ErrOverread,
merge nested hash readers (#9582) The `ioutil.NopCloser(reader)` was hiding nested hash readers. We make it an `io.Closer` so it can be attached without wrapping and allows for nesting, by merging the requests.
2020-05-14 17:01:31 -04:00
},
Signed trailers for signature v4 (#16484)
2023-05-05 22:53:12 -04:00
7: {
merge nested hash readers (#9582) The `ioutil.NopCloser(reader)` was hiding nested hash readers. We make it an `io.Closer` so it can be attached without wrapping and allows for nesting, by merging the requests.
2020-05-14 17:01:31 -04:00
desc: "Correct sha256, nested, truncated, swapped",
pkg/etag: add new package for S3 ETag handling (#11577) This commit adds a new package `etag` for dealing with S3 ETags. Even though ETag is often viewed as MD5 checksum of an object, handling S3 ETags correctly is a surprisingly complex task. While it is true that the ETag corresponds to the MD5 for the most basic S3 API operations, there are many exceptions in case of multipart uploads or encryption. In worse, some S3 clients expect very specific behavior when it comes to ETags. For example, some clients expect that the ETag is a double-quoted string and fail otherwise. Non-AWS compliant ETag handling has been a source of many bugs in the past. Therefore, this commit adds a dedicated `etag` package that provides functionality for parsing, generating and converting S3 ETags. Further, this commit removes the ETag computation from the `hash` package. Instead, the `hash` package (i.e. `hash.Reader`) should focus only on computing and verifying the content-sha256. One core feature of this commit is to provide a mechanism to communicate a computed ETag from a low-level `io.Reader` to a high-level `io.Reader`. This problem occurs when an S3 server receives a request and has to compute the ETag of the content. However, the server may also wrap the initial body with several other `io.Reader`, e.g. when encrypting or compressing the content: ``` reader := Encrypt(Compress(ETag(content))) ``` In such a case, the ETag should be accessible by the high-level `io.Reader`. The `etag` provides a mechanism to wrap `io.Reader` implementations such that the `ETag` can be accessed by a type-check. This technique is applied to the PUT, COPY and Upload handlers.
2021-02-23 15:31:53 -05:00
src: mustReader(t, bytes.NewReader([]byte("abcd-more-stuff-to-be ignored")), 4, "", "", -1),
merge nested hash readers (#9582) The `ioutil.NopCloser(reader)` was hiding nested hash readers. We make it an `io.Closer` so it can be attached without wrapping and allows for nesting, by merging the requests.
2020-05-14 17:01:31 -04:00
size: 4,
actualSize: -1,
sha256hex: "88d4266fd4e6338d13b845fcf289579d209c897823b9217da3e161936f031589",
Signed trailers for signature v4 (#16484)
2023-05-05 22:53:12 -04:00
err: ioutil.ErrOverread,
merge nested hash readers (#9582) The `ioutil.NopCloser(reader)` was hiding nested hash readers. We make it an `io.Closer` so it can be attached without wrapping and allows for nesting, by merging the requests.
2020-05-14 17:01:31 -04:00
},
{
desc: "Incorrect MD5, nested",
pkg/etag: add new package for S3 ETag handling (#11577) This commit adds a new package `etag` for dealing with S3 ETags. Even though ETag is often viewed as MD5 checksum of an object, handling S3 ETags correctly is a surprisingly complex task. While it is true that the ETag corresponds to the MD5 for the most basic S3 API operations, there are many exceptions in case of multipart uploads or encryption. In worse, some S3 clients expect very specific behavior when it comes to ETags. For example, some clients expect that the ETag is a double-quoted string and fail otherwise. Non-AWS compliant ETag handling has been a source of many bugs in the past. Therefore, this commit adds a dedicated `etag` package that provides functionality for parsing, generating and converting S3 ETags. Further, this commit removes the ETag computation from the `hash` package. Instead, the `hash` package (i.e. `hash.Reader`) should focus only on computing and verifying the content-sha256. One core feature of this commit is to provide a mechanism to communicate a computed ETag from a low-level `io.Reader` to a high-level `io.Reader`. This problem occurs when an S3 server receives a request and has to compute the ETag of the content. However, the server may also wrap the initial body with several other `io.Reader`, e.g. when encrypting or compressing the content: ``` reader := Encrypt(Compress(ETag(content))) ``` In such a case, the ETag should be accessible by the high-level `io.Reader`. The `etag` provides a mechanism to wrap `io.Reader` implementations such that the `ETag` can be accessed by a type-check. This technique is applied to the PUT, COPY and Upload handlers.
2021-02-23 15:31:53 -05:00
src: mustReader(t, bytes.NewReader([]byte("abcd")), 4, "", "", 4),
merge nested hash readers (#9582) The `ioutil.NopCloser(reader)` was hiding nested hash readers. We make it an `io.Closer` so it can be attached without wrapping and allows for nesting, by merging the requests.
2020-05-14 17:01:31 -04:00
size: 4,
actualSize: 4,
md5hex: "0773da587b322af3a8718cb418a715ce",
err: BadDigest{
ExpectedMD5: "0773da587b322af3a8718cb418a715ce",
CalculatedMD5: "e2fc714c4727ee9395f324cd2e7f331f",
},
},
{
desc: "Correct sha256, truncated",
src: bytes.NewReader([]byte("abcd-morethan-4-bytes")),
size: 4,
actualSize: 4,
sha256hex: "88d4266fd4e6338d13b845fcf289579d209c897823b9217da3e161936f031589",
Signed trailers for signature v4 (#16484)
2023-05-05 22:53:12 -04:00
err: ioutil.ErrOverread,
merge nested hash readers (#9582) The `ioutil.NopCloser(reader)` was hiding nested hash readers. We make it an `io.Closer` so it can be attached without wrapping and allows for nesting, by merging the requests.
2020-05-14 17:01:31 -04:00
},
{
desc: "Correct MD5, nested",
pkg/etag: add new package for S3 ETag handling (#11577) This commit adds a new package `etag` for dealing with S3 ETags. Even though ETag is often viewed as MD5 checksum of an object, handling S3 ETags correctly is a surprisingly complex task. While it is true that the ETag corresponds to the MD5 for the most basic S3 API operations, there are many exceptions in case of multipart uploads or encryption. In worse, some S3 clients expect very specific behavior when it comes to ETags. For example, some clients expect that the ETag is a double-quoted string and fail otherwise. Non-AWS compliant ETag handling has been a source of many bugs in the past. Therefore, this commit adds a dedicated `etag` package that provides functionality for parsing, generating and converting S3 ETags. Further, this commit removes the ETag computation from the `hash` package. Instead, the `hash` package (i.e. `hash.Reader`) should focus only on computing and verifying the content-sha256. One core feature of this commit is to provide a mechanism to communicate a computed ETag from a low-level `io.Reader` to a high-level `io.Reader`. This problem occurs when an S3 server receives a request and has to compute the ETag of the content. However, the server may also wrap the initial body with several other `io.Reader`, e.g. when encrypting or compressing the content: ``` reader := Encrypt(Compress(ETag(content))) ``` In such a case, the ETag should be accessible by the high-level `io.Reader`. The `etag` provides a mechanism to wrap `io.Reader` implementations such that the `ETag` can be accessed by a type-check. This technique is applied to the PUT, COPY and Upload handlers.
2021-02-23 15:31:53 -05:00
src: mustReader(t, bytes.NewReader([]byte("abcd")), 4, "", "", 4),
merge nested hash readers (#9582) The `ioutil.NopCloser(reader)` was hiding nested hash readers. We make it an `io.Closer` so it can be attached without wrapping and allows for nesting, by merging the requests.
2020-05-14 17:01:31 -04:00
size: 4,
actualSize: 4,
md5hex: "e2fc714c4727ee9395f324cd2e7f331f",
},
{
desc: "Correct MD5, truncated",
src: bytes.NewReader([]byte("abcd-morethan-4-bytes")),
size: 4,
actualSize: 4,
sha256hex: "",
md5hex: "e2fc714c4727ee9395f324cd2e7f331f",
Signed trailers for signature v4 (#16484)
2023-05-05 22:53:12 -04:00
err: ioutil.ErrOverread,
merge nested hash readers (#9582) The `ioutil.NopCloser(reader)` was hiding nested hash readers. We make it an `io.Closer` so it can be attached without wrapping and allows for nesting, by merging the requests.
2020-05-14 17:01:31 -04:00
},
{
desc: "Correct MD5, nested, truncated",
pkg/etag: add new package for S3 ETag handling (#11577) This commit adds a new package `etag` for dealing with S3 ETags. Even though ETag is often viewed as MD5 checksum of an object, handling S3 ETags correctly is a surprisingly complex task. While it is true that the ETag corresponds to the MD5 for the most basic S3 API operations, there are many exceptions in case of multipart uploads or encryption. In worse, some S3 clients expect very specific behavior when it comes to ETags. For example, some clients expect that the ETag is a double-quoted string and fail otherwise. Non-AWS compliant ETag handling has been a source of many bugs in the past. Therefore, this commit adds a dedicated `etag` package that provides functionality for parsing, generating and converting S3 ETags. Further, this commit removes the ETag computation from the `hash` package. Instead, the `hash` package (i.e. `hash.Reader`) should focus only on computing and verifying the content-sha256. One core feature of this commit is to provide a mechanism to communicate a computed ETag from a low-level `io.Reader` to a high-level `io.Reader`. This problem occurs when an S3 server receives a request and has to compute the ETag of the content. However, the server may also wrap the initial body with several other `io.Reader`, e.g. when encrypting or compressing the content: ``` reader := Encrypt(Compress(ETag(content))) ``` In such a case, the ETag should be accessible by the high-level `io.Reader`. The `etag` provides a mechanism to wrap `io.Reader` implementations such that the `ETag` can be accessed by a type-check. This technique is applied to the PUT, COPY and Upload handlers.
2021-02-23 15:31:53 -05:00
src: mustReader(t, bytes.NewReader([]byte("abcd-morestuff")), -1, "", "", -1),
merge nested hash readers (#9582) The `ioutil.NopCloser(reader)` was hiding nested hash readers. We make it an `io.Closer` so it can be attached without wrapping and allows for nesting, by merging the requests.
2020-05-14 17:01:31 -04:00
size: 4,
actualSize: 4,
md5hex: "e2fc714c4727ee9395f324cd2e7f331f",
Signed trailers for signature v4 (#16484)
2023-05-05 22:53:12 -04:00
err: ioutil.ErrOverread,
merge nested hash readers (#9582) The `ioutil.NopCloser(reader)` was hiding nested hash readers. We make it an `io.Closer` so it can be attached without wrapping and allows for nesting, by merging the requests.
2020-05-14 17:01:31 -04:00
},
Simplify data verification with HashReader. (#5071) Verify() was being called by caller after the data has been successfully read after io.EOF. This disconnection opens a race under concurrent access to such an object. Verification is not necessary outside of Read() call, we can simply just do checksum verification right inside Read() call at io.EOF. This approach simplifies the usage.
2017-10-22 01:30:34 -04:00
}
for i, testCase := range testCases {
merge nested hash readers (#9582) The `ioutil.NopCloser(reader)` was hiding nested hash readers. We make it an `io.Closer` so it can be attached without wrapping and allows for nesting, by merging the requests.
2020-05-14 17:01:31 -04:00
t.Run(fmt.Sprintf("case-%d", i+1), func(t *testing.T) {
pkg/etag: add new package for S3 ETag handling (#11577) This commit adds a new package `etag` for dealing with S3 ETags. Even though ETag is often viewed as MD5 checksum of an object, handling S3 ETags correctly is a surprisingly complex task. While it is true that the ETag corresponds to the MD5 for the most basic S3 API operations, there are many exceptions in case of multipart uploads or encryption. In worse, some S3 clients expect very specific behavior when it comes to ETags. For example, some clients expect that the ETag is a double-quoted string and fail otherwise. Non-AWS compliant ETag handling has been a source of many bugs in the past. Therefore, this commit adds a dedicated `etag` package that provides functionality for parsing, generating and converting S3 ETags. Further, this commit removes the ETag computation from the `hash` package. Instead, the `hash` package (i.e. `hash.Reader`) should focus only on computing and verifying the content-sha256. One core feature of this commit is to provide a mechanism to communicate a computed ETag from a low-level `io.Reader` to a high-level `io.Reader`. This problem occurs when an S3 server receives a request and has to compute the ETag of the content. However, the server may also wrap the initial body with several other `io.Reader`, e.g. when encrypting or compressing the content: ``` reader := Encrypt(Compress(ETag(content))) ``` In such a case, the ETag should be accessible by the high-level `io.Reader`. The `etag` provides a mechanism to wrap `io.Reader` implementations such that the `ETag` can be accessed by a type-check. This technique is applied to the PUT, COPY and Upload handlers.
2021-02-23 15:31:53 -05:00
r, err := NewReader(testCase.src, testCase.size, testCase.md5hex, testCase.sha256hex, testCase.actualSize)
merge nested hash readers (#9582) The `ioutil.NopCloser(reader)` was hiding nested hash readers. We make it an `io.Closer` so it can be attached without wrapping and allows for nesting, by merging the requests.
2020-05-14 17:01:31 -04:00
if err != nil {
t.Fatalf("Test %q: Initializing reader failed %s", testCase.desc, err)
}
Remove deprecated io/ioutil (#15707)
2022-09-19 14:05:16 -04:00
_, err = io.Copy(io.Discard, r)
merge nested hash readers (#9582) The `ioutil.NopCloser(reader)` was hiding nested hash readers. We make it an `io.Closer` so it can be attached without wrapping and allows for nesting, by merging the requests.
2020-05-14 17:01:31 -04:00
if err != nil {
Signed trailers for signature v4 (#16484)
2023-05-05 22:53:12 -04:00
if testCase.err == nil {
t.Errorf("Test %q; got unexpected error: %v", testCase.desc, err)
return
}
merge nested hash readers (#9582) The `ioutil.NopCloser(reader)` was hiding nested hash readers. We make it an `io.Closer` so it can be attached without wrapping and allows for nesting, by merging the requests.
2020-05-14 17:01:31 -04:00
if err.Error() != testCase.err.Error() {
t.Errorf("Test %q: Expected error %s, got error %s", testCase.desc, testCase.err, err)
}
Simplify data verification with HashReader. (#5071) Verify() was being called by caller after the data has been successfully read after io.EOF. This disconnection opens a race under concurrent access to such an object. Verification is not necessary outside of Read() call, we can simply just do checksum verification right inside Read() call at io.EOF. This approach simplifies the usage.
2017-10-22 01:30:34 -04:00
}
merge nested hash readers (#9582) The `ioutil.NopCloser(reader)` was hiding nested hash readers. We make it an `io.Closer` so it can be attached without wrapping and allows for nesting, by merging the requests.
2020-05-14 17:01:31 -04:00
})
}
}
pkg/etag: add new package for S3 ETag handling (#11577) This commit adds a new package `etag` for dealing with S3 ETags. Even though ETag is often viewed as MD5 checksum of an object, handling S3 ETags correctly is a surprisingly complex task. While it is true that the ETag corresponds to the MD5 for the most basic S3 API operations, there are many exceptions in case of multipart uploads or encryption. In worse, some S3 clients expect very specific behavior when it comes to ETags. For example, some clients expect that the ETag is a double-quoted string and fail otherwise. Non-AWS compliant ETag handling has been a source of many bugs in the past. Therefore, this commit adds a dedicated `etag` package that provides functionality for parsing, generating and converting S3 ETags. Further, this commit removes the ETag computation from the `hash` package. Instead, the `hash` package (i.e. `hash.Reader`) should focus only on computing and verifying the content-sha256. One core feature of this commit is to provide a mechanism to communicate a computed ETag from a low-level `io.Reader` to a high-level `io.Reader`. This problem occurs when an S3 server receives a request and has to compute the ETag of the content. However, the server may also wrap the initial body with several other `io.Reader`, e.g. when encrypting or compressing the content: ``` reader := Encrypt(Compress(ETag(content))) ``` In such a case, the ETag should be accessible by the high-level `io.Reader`. The `etag` provides a mechanism to wrap `io.Reader` implementations such that the `ETag` can be accessed by a type-check. This technique is applied to the PUT, COPY and Upload handlers.
2021-02-23 15:31:53 -05:00
func mustReader(t *testing.T, src io.Reader, size int64, md5Hex, sha256Hex string, actualSize int64) *Reader {
r, err := NewReader(src, size, md5Hex, sha256Hex, actualSize)
merge nested hash readers (#9582) The `ioutil.NopCloser(reader)` was hiding nested hash readers. We make it an `io.Closer` so it can be attached without wrapping and allows for nesting, by merging the requests.
2020-05-14 17:01:31 -04:00
if err != nil {
t.Fatal(err)
Simplify data verification with HashReader. (#5071) Verify() was being called by caller after the data has been successfully read after io.EOF. This disconnection opens a race under concurrent access to such an object. Verification is not necessary outside of Read() call, we can simply just do checksum verification right inside Read() call at io.EOF. This approach simplifies the usage.
2017-10-22 01:30:34 -04:00
}
merge nested hash readers (#9582) The `ioutil.NopCloser(reader)` was hiding nested hash readers. We make it an `io.Closer` so it can be attached without wrapping and allows for nesting, by merging the requests.
2020-05-14 17:01:31 -04:00
return r
Simplify data verification with HashReader. (#5071) Verify() was being called by caller after the data has been successfully read after io.EOF. This disconnection opens a race under concurrent access to such an object. Verification is not necessary outside of Read() call, we can simply just do checksum verification right inside Read() call at io.EOF. This approach simplifies the usage.
2017-10-22 01:30:34 -04:00
}
// Tests NewReader() constructor with invalid arguments.
func TestHashReaderInvalidArguments(t *testing.T) {
testCases := []struct {
merge nested hash readers (#9582) The `ioutil.NopCloser(reader)` was hiding nested hash readers. We make it an `io.Closer` so it can be attached without wrapping and allows for nesting, by merging the requests.
2020-05-14 17:01:31 -04:00
desc string
Simplify data verification with HashReader. (#5071) Verify() was being called by caller after the data has been successfully read after io.EOF. This disconnection opens a race under concurrent access to such an object. Verification is not necessary outside of Read() call, we can simply just do checksum verification right inside Read() call at io.EOF. This approach simplifies the usage.
2017-10-22 01:30:34 -04:00
src io.Reader
size int64
Add object compression support (#6292) Add support for streaming (golang/LZ77/snappy) compression.
2018-09-27 23:36:17 -04:00
actualSize int64
Simplify data verification with HashReader. (#5071) Verify() was being called by caller after the data has been successfully read after io.EOF. This disconnection opens a race under concurrent access to such an object. Verification is not necessary outside of Read() call, we can simply just do checksum verification right inside Read() call at io.EOF. This approach simplifies the usage.
2017-10-22 01:30:34 -04:00
md5hex, sha256hex string
success bool
}{
{
pkg/etag: add new package for S3 ETag handling (#11577) This commit adds a new package `etag` for dealing with S3 ETags. Even though ETag is often viewed as MD5 checksum of an object, handling S3 ETags correctly is a surprisingly complex task. While it is true that the ETag corresponds to the MD5 for the most basic S3 API operations, there are many exceptions in case of multipart uploads or encryption. In worse, some S3 clients expect very specific behavior when it comes to ETags. For example, some clients expect that the ETag is a double-quoted string and fail otherwise. Non-AWS compliant ETag handling has been a source of many bugs in the past. Therefore, this commit adds a dedicated `etag` package that provides functionality for parsing, generating and converting S3 ETags. Further, this commit removes the ETag computation from the `hash` package. Instead, the `hash` package (i.e. `hash.Reader`) should focus only on computing and verifying the content-sha256. One core feature of this commit is to provide a mechanism to communicate a computed ETag from a low-level `io.Reader` to a high-level `io.Reader`. This problem occurs when an S3 server receives a request and has to compute the ETag of the content. However, the server may also wrap the initial body with several other `io.Reader`, e.g. when encrypting or compressing the content: ``` reader := Encrypt(Compress(ETag(content))) ``` In such a case, the ETag should be accessible by the high-level `io.Reader`. The `etag` provides a mechanism to wrap `io.Reader` implementations such that the `ETag` can be accessed by a type-check. This technique is applied to the PUT, COPY and Upload handlers.
2021-02-23 15:31:53 -05:00
desc: "Invalid md5sum NewReader() will fail.",
src: bytes.NewReader([]byte("abcd")),
size: 4,
actualSize: 4,
md5hex: "invalid-md5",
success: false,
Simplify data verification with HashReader. (#5071) Verify() was being called by caller after the data has been successfully read after io.EOF. This disconnection opens a race under concurrent access to such an object. Verification is not necessary outside of Read() call, we can simply just do checksum verification right inside Read() call at io.EOF. This approach simplifies the usage.
2017-10-22 01:30:34 -04:00
},
{
pkg/etag: add new package for S3 ETag handling (#11577) This commit adds a new package `etag` for dealing with S3 ETags. Even though ETag is often viewed as MD5 checksum of an object, handling S3 ETags correctly is a surprisingly complex task. While it is true that the ETag corresponds to the MD5 for the most basic S3 API operations, there are many exceptions in case of multipart uploads or encryption. In worse, some S3 clients expect very specific behavior when it comes to ETags. For example, some clients expect that the ETag is a double-quoted string and fail otherwise. Non-AWS compliant ETag handling has been a source of many bugs in the past. Therefore, this commit adds a dedicated `etag` package that provides functionality for parsing, generating and converting S3 ETags. Further, this commit removes the ETag computation from the `hash` package. Instead, the `hash` package (i.e. `hash.Reader`) should focus only on computing and verifying the content-sha256. One core feature of this commit is to provide a mechanism to communicate a computed ETag from a low-level `io.Reader` to a high-level `io.Reader`. This problem occurs when an S3 server receives a request and has to compute the ETag of the content. However, the server may also wrap the initial body with several other `io.Reader`, e.g. when encrypting or compressing the content: ``` reader := Encrypt(Compress(ETag(content))) ``` In such a case, the ETag should be accessible by the high-level `io.Reader`. The `etag` provides a mechanism to wrap `io.Reader` implementations such that the `ETag` can be accessed by a type-check. This technique is applied to the PUT, COPY and Upload handlers.
2021-02-23 15:31:53 -05:00
desc: "Invalid sha256 NewReader() will fail.",
src: bytes.NewReader([]byte("abcd")),
size: 4,
actualSize: 4,
sha256hex: "invalid-sha256",
success: false,
Simplify data verification with HashReader. (#5071) Verify() was being called by caller after the data has been successfully read after io.EOF. This disconnection opens a race under concurrent access to such an object. Verification is not necessary outside of Read() call, we can simply just do checksum verification right inside Read() call at io.EOF. This approach simplifies the usage.
2017-10-22 01:30:34 -04:00
},
{
merge nested hash readers (#9582) The `ioutil.NopCloser(reader)` was hiding nested hash readers. We make it an `io.Closer` so it can be attached without wrapping and allows for nesting, by merging the requests.
2020-05-14 17:01:31 -04:00
desc: "Nested hash reader NewReader() should merge.",
pkg/etag: add new package for S3 ETag handling (#11577) This commit adds a new package `etag` for dealing with S3 ETags. Even though ETag is often viewed as MD5 checksum of an object, handling S3 ETags correctly is a surprisingly complex task. While it is true that the ETag corresponds to the MD5 for the most basic S3 API operations, there are many exceptions in case of multipart uploads or encryption. In worse, some S3 clients expect very specific behavior when it comes to ETags. For example, some clients expect that the ETag is a double-quoted string and fail otherwise. Non-AWS compliant ETag handling has been a source of many bugs in the past. Therefore, this commit adds a dedicated `etag` package that provides functionality for parsing, generating and converting S3 ETags. Further, this commit removes the ETag computation from the `hash` package. Instead, the `hash` package (i.e. `hash.Reader`) should focus only on computing and verifying the content-sha256. One core feature of this commit is to provide a mechanism to communicate a computed ETag from a low-level `io.Reader` to a high-level `io.Reader`. This problem occurs when an S3 server receives a request and has to compute the ETag of the content. However, the server may also wrap the initial body with several other `io.Reader`, e.g. when encrypting or compressing the content: ``` reader := Encrypt(Compress(ETag(content))) ``` In such a case, the ETag should be accessible by the high-level `io.Reader`. The `etag` provides a mechanism to wrap `io.Reader` implementations such that the `ETag` can be accessed by a type-check. This technique is applied to the PUT, COPY and Upload handlers.
2021-02-23 15:31:53 -05:00
src: mustReader(t, bytes.NewReader([]byte("abcd")), 4, "", "", 4),
merge nested hash readers (#9582) The `ioutil.NopCloser(reader)` was hiding nested hash readers. We make it an `io.Closer` so it can be attached without wrapping and allows for nesting, by merging the requests.
2020-05-14 17:01:31 -04:00
size: 4,
actualSize: 4,
success: true,
},
{
pkg/etag: add new package for S3 ETag handling (#11577) This commit adds a new package `etag` for dealing with S3 ETags. Even though ETag is often viewed as MD5 checksum of an object, handling S3 ETags correctly is a surprisingly complex task. While it is true that the ETag corresponds to the MD5 for the most basic S3 API operations, there are many exceptions in case of multipart uploads or encryption. In worse, some S3 clients expect very specific behavior when it comes to ETags. For example, some clients expect that the ETag is a double-quoted string and fail otherwise. Non-AWS compliant ETag handling has been a source of many bugs in the past. Therefore, this commit adds a dedicated `etag` package that provides functionality for parsing, generating and converting S3 ETags. Further, this commit removes the ETag computation from the `hash` package. Instead, the `hash` package (i.e. `hash.Reader`) should focus only on computing and verifying the content-sha256. One core feature of this commit is to provide a mechanism to communicate a computed ETag from a low-level `io.Reader` to a high-level `io.Reader`. This problem occurs when an S3 server receives a request and has to compute the ETag of the content. However, the server may also wrap the initial body with several other `io.Reader`, e.g. when encrypting or compressing the content: ``` reader := Encrypt(Compress(ETag(content))) ``` In such a case, the ETag should be accessible by the high-level `io.Reader`. The `etag` provides a mechanism to wrap `io.Reader` implementations such that the `ETag` can be accessed by a type-check. This technique is applied to the PUT, COPY and Upload handlers.
2021-02-23 15:31:53 -05:00
desc: "Mismatching sha256",
src: mustReader(t, bytes.NewReader([]byte("abcd")), 4, "", "88d4266fd4e6338d13b845fcf289579d209c897823b9217da3e161936f031589", 4),
size: 4,
actualSize: 4,
sha256hex: "50d858e0985ecc7f60418aaf0cc5ab587f42c2570a884095a9e8ccacd0f6545c",
success: false,
merge nested hash readers (#9582) The `ioutil.NopCloser(reader)` was hiding nested hash readers. We make it an `io.Closer` so it can be attached without wrapping and allows for nesting, by merging the requests.
2020-05-14 17:01:31 -04:00
},
{
desc: "Correct sha256",
pkg/etag: add new package for S3 ETag handling (#11577) This commit adds a new package `etag` for dealing with S3 ETags. Even though ETag is often viewed as MD5 checksum of an object, handling S3 ETags correctly is a surprisingly complex task. While it is true that the ETag corresponds to the MD5 for the most basic S3 API operations, there are many exceptions in case of multipart uploads or encryption. In worse, some S3 clients expect very specific behavior when it comes to ETags. For example, some clients expect that the ETag is a double-quoted string and fail otherwise. Non-AWS compliant ETag handling has been a source of many bugs in the past. Therefore, this commit adds a dedicated `etag` package that provides functionality for parsing, generating and converting S3 ETags. Further, this commit removes the ETag computation from the `hash` package. Instead, the `hash` package (i.e. `hash.Reader`) should focus only on computing and verifying the content-sha256. One core feature of this commit is to provide a mechanism to communicate a computed ETag from a low-level `io.Reader` to a high-level `io.Reader`. This problem occurs when an S3 server receives a request and has to compute the ETag of the content. However, the server may also wrap the initial body with several other `io.Reader`, e.g. when encrypting or compressing the content: ``` reader := Encrypt(Compress(ETag(content))) ``` In such a case, the ETag should be accessible by the high-level `io.Reader`. The `etag` provides a mechanism to wrap `io.Reader` implementations such that the `ETag` can be accessed by a type-check. This technique is applied to the PUT, COPY and Upload handlers.
2021-02-23 15:31:53 -05:00
src: mustReader(t, bytes.NewReader([]byte("abcd")), 4, "", "88d4266fd4e6338d13b845fcf289579d209c897823b9217da3e161936f031589", 4),
merge nested hash readers (#9582) The `ioutil.NopCloser(reader)` was hiding nested hash readers. We make it an `io.Closer` so it can be attached without wrapping and allows for nesting, by merging the requests.
2020-05-14 17:01:31 -04:00
size: 4,
actualSize: 4,
sha256hex: "88d4266fd4e6338d13b845fcf289579d209c897823b9217da3e161936f031589",
success: true,
},
{
pkg/etag: add new package for S3 ETag handling (#11577) This commit adds a new package `etag` for dealing with S3 ETags. Even though ETag is often viewed as MD5 checksum of an object, handling S3 ETags correctly is a surprisingly complex task. While it is true that the ETag corresponds to the MD5 for the most basic S3 API operations, there are many exceptions in case of multipart uploads or encryption. In worse, some S3 clients expect very specific behavior when it comes to ETags. For example, some clients expect that the ETag is a double-quoted string and fail otherwise. Non-AWS compliant ETag handling has been a source of many bugs in the past. Therefore, this commit adds a dedicated `etag` package that provides functionality for parsing, generating and converting S3 ETags. Further, this commit removes the ETag computation from the `hash` package. Instead, the `hash` package (i.e. `hash.Reader`) should focus only on computing and verifying the content-sha256. One core feature of this commit is to provide a mechanism to communicate a computed ETag from a low-level `io.Reader` to a high-level `io.Reader`. This problem occurs when an S3 server receives a request and has to compute the ETag of the content. However, the server may also wrap the initial body with several other `io.Reader`, e.g. when encrypting or compressing the content: ``` reader := Encrypt(Compress(ETag(content))) ``` In such a case, the ETag should be accessible by the high-level `io.Reader`. The `etag` provides a mechanism to wrap `io.Reader` implementations such that the `ETag` can be accessed by a type-check. This technique is applied to the PUT, COPY and Upload handlers.
2021-02-23 15:31:53 -05:00
desc: "Mismatching MD5",
src: mustReader(t, bytes.NewReader([]byte("abcd")), 4, "e2fc714c4727ee9395f324cd2e7f331f", "", 4),
size: 4,
actualSize: 4,
md5hex: "0773da587b322af3a8718cb418a715ce",
success: false,
Simplify data verification with HashReader. (#5071) Verify() was being called by caller after the data has been successfully read after io.EOF. This disconnection opens a race under concurrent access to such an object. Verification is not necessary outside of Read() call, we can simply just do checksum verification right inside Read() call at io.EOF. This approach simplifies the usage.
2017-10-22 01:30:34 -04:00
},
{
merge nested hash readers (#9582) The `ioutil.NopCloser(reader)` was hiding nested hash readers. We make it an `io.Closer` so it can be attached without wrapping and allows for nesting, by merging the requests.
2020-05-14 17:01:31 -04:00
desc: "Correct MD5",
pkg/etag: add new package for S3 ETag handling (#11577) This commit adds a new package `etag` for dealing with S3 ETags. Even though ETag is often viewed as MD5 checksum of an object, handling S3 ETags correctly is a surprisingly complex task. While it is true that the ETag corresponds to the MD5 for the most basic S3 API operations, there are many exceptions in case of multipart uploads or encryption. In worse, some S3 clients expect very specific behavior when it comes to ETags. For example, some clients expect that the ETag is a double-quoted string and fail otherwise. Non-AWS compliant ETag handling has been a source of many bugs in the past. Therefore, this commit adds a dedicated `etag` package that provides functionality for parsing, generating and converting S3 ETags. Further, this commit removes the ETag computation from the `hash` package. Instead, the `hash` package (i.e. `hash.Reader`) should focus only on computing and verifying the content-sha256. One core feature of this commit is to provide a mechanism to communicate a computed ETag from a low-level `io.Reader` to a high-level `io.Reader`. This problem occurs when an S3 server receives a request and has to compute the ETag of the content. However, the server may also wrap the initial body with several other `io.Reader`, e.g. when encrypting or compressing the content: ``` reader := Encrypt(Compress(ETag(content))) ``` In such a case, the ETag should be accessible by the high-level `io.Reader`. The `etag` provides a mechanism to wrap `io.Reader` implementations such that the `ETag` can be accessed by a type-check. This technique is applied to the PUT, COPY and Upload handlers.
2021-02-23 15:31:53 -05:00
src: mustReader(t, bytes.NewReader([]byte("abcd")), 4, "e2fc714c4727ee9395f324cd2e7f331f", "", 4),
merge nested hash readers (#9582) The `ioutil.NopCloser(reader)` was hiding nested hash readers. We make it an `io.Closer` so it can be attached without wrapping and allows for nesting, by merging the requests.
2020-05-14 17:01:31 -04:00
size: 4,
actualSize: 4,
md5hex: "e2fc714c4727ee9395f324cd2e7f331f",
success: true,
},
{
desc: "Nothing, all ok",
Add object compression support (#6292) Add support for streaming (golang/LZ77/snappy) compression.
2018-09-27 23:36:17 -04:00
src: bytes.NewReader([]byte("abcd")),
size: 4,
actualSize: 4,
success: true,
Simplify data verification with HashReader. (#5071) Verify() was being called by caller after the data has been successfully read after io.EOF. This disconnection opens a race under concurrent access to such an object. Verification is not necessary outside of Read() call, we can simply just do checksum verification right inside Read() call at io.EOF. This approach simplifies the usage.
2017-10-22 01:30:34 -04:00
},
merge nested hash readers (#9582) The `ioutil.NopCloser(reader)` was hiding nested hash readers. We make it an `io.Closer` so it can be attached without wrapping and allows for nesting, by merging the requests.
2020-05-14 17:01:31 -04:00
{
pkg/etag: add new package for S3 ETag handling (#11577) This commit adds a new package `etag` for dealing with S3 ETags. Even though ETag is often viewed as MD5 checksum of an object, handling S3 ETags correctly is a surprisingly complex task. While it is true that the ETag corresponds to the MD5 for the most basic S3 API operations, there are many exceptions in case of multipart uploads or encryption. In worse, some S3 clients expect very specific behavior when it comes to ETags. For example, some clients expect that the ETag is a double-quoted string and fail otherwise. Non-AWS compliant ETag handling has been a source of many bugs in the past. Therefore, this commit adds a dedicated `etag` package that provides functionality for parsing, generating and converting S3 ETags. Further, this commit removes the ETag computation from the `hash` package. Instead, the `hash` package (i.e. `hash.Reader`) should focus only on computing and verifying the content-sha256. One core feature of this commit is to provide a mechanism to communicate a computed ETag from a low-level `io.Reader` to a high-level `io.Reader`. This problem occurs when an S3 server receives a request and has to compute the ETag of the content. However, the server may also wrap the initial body with several other `io.Reader`, e.g. when encrypting or compressing the content: ``` reader := Encrypt(Compress(ETag(content))) ``` In such a case, the ETag should be accessible by the high-level `io.Reader`. The `etag` provides a mechanism to wrap `io.Reader` implementations such that the `ETag` can be accessed by a type-check. This technique is applied to the PUT, COPY and Upload handlers.
2021-02-23 15:31:53 -05:00
desc: "Nested, size mismatch",
src: mustReader(t, bytes.NewReader([]byte("abcd-morestuff")), 4, "", "", -1),
size: 2,
actualSize: -1,
success: false,
merge nested hash readers (#9582) The `ioutil.NopCloser(reader)` was hiding nested hash readers. We make it an `io.Closer` so it can be attached without wrapping and allows for nesting, by merging the requests.
2020-05-14 17:01:31 -04:00
},
Simplify data verification with HashReader. (#5071) Verify() was being called by caller after the data has been successfully read after io.EOF. This disconnection opens a race under concurrent access to such an object. Verification is not necessary outside of Read() call, we can simply just do checksum verification right inside Read() call at io.EOF. This approach simplifies the usage.
2017-10-22 01:30:34 -04:00
}
for i, testCase := range testCases {
merge nested hash readers (#9582) The `ioutil.NopCloser(reader)` was hiding nested hash readers. We make it an `io.Closer` so it can be attached without wrapping and allows for nesting, by merging the requests.
2020-05-14 17:01:31 -04:00
t.Run(fmt.Sprintf("case-%d", i+1), func(t *testing.T) {
pkg/etag: add new package for S3 ETag handling (#11577) This commit adds a new package `etag` for dealing with S3 ETags. Even though ETag is often viewed as MD5 checksum of an object, handling S3 ETags correctly is a surprisingly complex task. While it is true that the ETag corresponds to the MD5 for the most basic S3 API operations, there are many exceptions in case of multipart uploads or encryption. In worse, some S3 clients expect very specific behavior when it comes to ETags. For example, some clients expect that the ETag is a double-quoted string and fail otherwise. Non-AWS compliant ETag handling has been a source of many bugs in the past. Therefore, this commit adds a dedicated `etag` package that provides functionality for parsing, generating and converting S3 ETags. Further, this commit removes the ETag computation from the `hash` package. Instead, the `hash` package (i.e. `hash.Reader`) should focus only on computing and verifying the content-sha256. One core feature of this commit is to provide a mechanism to communicate a computed ETag from a low-level `io.Reader` to a high-level `io.Reader`. This problem occurs when an S3 server receives a request and has to compute the ETag of the content. However, the server may also wrap the initial body with several other `io.Reader`, e.g. when encrypting or compressing the content: ``` reader := Encrypt(Compress(ETag(content))) ``` In such a case, the ETag should be accessible by the high-level `io.Reader`. The `etag` provides a mechanism to wrap `io.Reader` implementations such that the `ETag` can be accessed by a type-check. This technique is applied to the PUT, COPY and Upload handlers.
2021-02-23 15:31:53 -05:00
_, err := NewReader(testCase.src, testCase.size, testCase.md5hex, testCase.sha256hex, testCase.actualSize)
merge nested hash readers (#9582) The `ioutil.NopCloser(reader)` was hiding nested hash readers. We make it an `io.Closer` so it can be attached without wrapping and allows for nesting, by merging the requests.
2020-05-14 17:01:31 -04:00
if err != nil && testCase.success {
t.Errorf("Test %q: Expected success, but got error %s instead", testCase.desc, err)
}
if err == nil && !testCase.success {
t.Errorf("Test %q: Expected error, but got success", testCase.desc)
}
})
Simplify data verification with HashReader. (#5071) Verify() was being called by caller after the data has been successfully read after io.EOF. This disconnection opens a race under concurrent access to such an object. Verification is not necessary outside of Read() call, we can simply just do checksum verification right inside Read() call at io.EOF. This approach simplifies the usage.
2017-10-22 01:30:34 -04:00
}
}