minio/pkg
Andreas Auernhammer d4b822d697
pkg/etag: add new package for S3 ETag handling (#11577)
This commit adds a new package `etag` for dealing
with S3 ETags.

Even though ETag is often viewed as MD5 checksum of
an object, handling S3 ETags correctly is a surprisingly
complex task. While it is true that the ETag corresponds
to the MD5 for the most basic S3 API operations, there are
many exceptions in case of multipart uploads or encryption.

In worse, some S3 clients expect very specific behavior when
it comes to ETags. For example, some clients expect that the
ETag is a double-quoted string and fail otherwise.
Non-AWS compliant ETag handling has been a source of many bugs
in the past.

Therefore, this commit adds a dedicated `etag` package that provides
functionality for parsing, generating and converting S3 ETags.
Further, this commit removes the ETag computation from the `hash`
package. Instead, the `hash` package (i.e. `hash.Reader`) should
focus only on computing and verifying the content-sha256.

One core feature of this commit is to provide a mechanism to
communicate a computed ETag from a low-level `io.Reader` to
a high-level `io.Reader`.

This problem occurs when an S3 server receives a request and
has to compute the ETag of the content. However, the server
may also wrap the initial body with several other `io.Reader`,
e.g. when encrypting or compressing the content:
```
   reader := Encrypt(Compress(ETag(content)))
```
In such a case, the ETag should be accessible by the high-level
`io.Reader`.

The `etag` provides a mechanism to wrap `io.Reader` implementations
such that the `ETag` can be accessed by a type-check.
This technique is applied to the PUT, COPY and Upload handlers.
2021-02-23 12:31:53 -08:00
..
argon2 fix: refactor locks to apply them uniquely per node (#11052) 2020-12-10 07:28:37 -08:00
auth fix: remove LDAP groups claim and store them on server (#9637) 2020-05-20 11:33:35 -07:00
bandwidth Add basic bandwidth monitoring for replication. (#10501) 2020-10-09 20:36:00 -07:00
bpool Replace Minio refs in docs with MinIO and links (#7494) 2019-04-09 11:39:42 -07:00
bucket lifecycle: Fix expiration header in some cases (#11565) 2021-02-17 14:51:29 -08:00
certs feat: treat /var/run/secrets/ on k8s as system cert directory (#11123) 2020-12-16 18:24:12 -08:00
cgroup Replace Minio refs in docs with MinIO and links (#7494) 2019-04-09 11:39:42 -07:00
color Bring in safe mode support (#8478) 2019-11-09 09:27:23 -08:00
console add _MINIO_SERVER_DEBUG env for enabling debug messages (#11128) 2020-12-17 16:52:47 -08:00
csvparser sql, csv: Cache some values between Read() calls to gain performance (#9645) 2020-05-22 10:15:08 -07:00
disk fix: for containers use root-disk detection cleverly (#11593) 2021-02-22 10:32:21 -08:00
dsync fix: improper ticker usage in goroutines (#11468) 2021-02-05 19:23:48 -08:00
ellipses Replace Minio refs in docs with MinIO and links (#7494) 2019-04-09 11:39:42 -07:00
env azure: Use default upload parameters to avoid consuming too much memory (#11251) 2021-01-11 22:48:09 -08:00
etag pkg/etag: add new package for S3 ETag handling (#11577) 2021-02-23 12:31:53 -08:00
event check if kafka producer is connected (#11578) 2021-02-18 11:14:27 -08:00
handlers List v1/versions routes based on source IP if found (#10603) 2020-09-30 13:38:27 -07:00
hash pkg/etag: add new package for S3 ETag handling (#11577) 2021-02-23 12:31:53 -08:00
iam/policy default to common conditions if conditions not present (#11546) 2021-02-16 11:56:45 -08:00
ioutil fix: use buffers only when necessary for io.Copy() (#11229) 2021-01-06 09:36:55 -08:00
licverifier update license verifier to use updated keys (#11197) 2021-01-06 10:17:05 -08:00
lock fix: Allow Walk to honor load balanced drives (#10610) 2020-10-01 20:24:34 -07:00
lsync delayed locks until we have started reading the body (#10474) 2020-09-14 15:57:13 -07:00
madmin Skip printing error if empty for reporting bandwidth (#11606) 2021-02-22 13:41:40 -08:00
mimedb Replace Minio refs in docs with MinIO and links (#7494) 2019-04-09 11:39:42 -07:00
mountinfo Cache Windows mount point info (#8572) 2019-11-27 20:10:13 +05:30
net Unwrap network errors (#10934) 2020-11-20 22:55:35 -08:00
pubsub tracing: NumSubscribers() to use atomic instead of mutex (#11219) 2021-01-04 09:40:30 -08:00
quick fix: etcd import paths again depend on v3.4.14 release (#11020) 2020-12-03 11:35:18 -08:00
rpc Upgrade compress and pgzip package (#10992) 2020-11-27 10:10:15 -08:00
s3select fix: missing data on multiple columns reading parquet (#11499) 2021-02-10 08:49:48 -08:00
safe Replace Minio refs in docs with MinIO and links (#7494) 2019-04-09 11:39:42 -07:00
smart add NVMe drive info [model num, serial num, drive temp. etc.] (#10613) 2020-10-04 10:18:46 -07:00
sync/errgroup fix: align atomic variables for 32bit arch (#11475) 2021-02-08 08:51:12 -08:00
sys Use hw.physmem64 instead of hw.physmem for NetBSD in pkg/sys/getHwPhysmem (#10907) 2020-11-16 20:53:47 -08:00
trace feat: time to bring back http2.0 support (#10230) 2020-08-10 09:02:29 -07:00
trie fix: [fs] CompleteMultipart use trie structure for partMatch (#10522) 2020-09-21 01:18:13 -07:00
wildcard Simplify cast of string to rune slice in wildcard matching (#9577) 2020-05-14 08:20:13 -07:00
words Replace Minio refs in docs with MinIO and links (#7494) 2019-04-09 11:39:42 -07:00