Commit Graph

3740 Commits

Author SHA1 Message Date
Krishnan Parthasarathi
a1df230518
Add a 'free' version to track deletion of tiered object content (#12470) 2021-06-30 19:32:07 -07:00
Aditya Manthramurthy
dc6958b6a1
fix: enhance openid claim missing error (#12608)
The error implies an expected claim is missing even when 
the claim is present. Added an additional error message 
to clarify the problem.
2021-06-30 17:11:23 -07:00
Harshavardhana
5a33d3f05a fix: set the correct IDP salt/passphrase 2021-06-30 16:45:52 -07:00
Harshavardhana
4781e7580b
add support for customizing redirect_uri for IDP (#12607) 2021-06-30 16:08:20 -07:00
Poorna Krishnamoorthy
a3f0288262
Use multipart call for replication (#12535)
if object was uploaded with multipart. This is to ensure that
GetObject calls with partNumber in URI request parameters
have same behavior on source and replication target.
2021-06-30 07:44:24 -07:00
Harshavardhana
a6ad965799
fix: use NumVersions for list resolver (#12599)
also do not incorrectly double count
objExists unless its selected and it
matches with previous entry.

Bonus: change listQuorum to match with
AskDisks to ensure that we atleast by
default choose all the "drives" that
we asked is consistent.
2021-06-30 07:43:19 -07:00
Shireesh Anjal
4575291f8a
fix: drive state not added in health report (#12603)
In case of FS mode, the drive state was not being added in the health
report. Fixed by hard coding it to "ok".
2021-06-30 07:42:38 -07:00
Harshavardhana
ec8d93f756
fix: add missing readTriggerCh close (#12593) 2021-06-29 08:47:15 -07:00
Poorna Krishnamoorthy
a69c2a2fb3
Change replication to use read lock instead of writelock (#12581)
Fixes #12573

This PR also adding audit logging for replication activity
2021-06-28 23:58:08 -07:00
Poorna Krishnamoorthy
c20fbcd842
fix:(replication/bandwidth) add missing validation in monitored reader (#12582) 2021-06-27 10:11:13 -07:00
Poorna Krishnamoorthy
d00783c923
Use rate.Limiter for bandwidth monitoring (#12506)
Bonus: fixes a hang when bandwidth caps are enabled for
synchronous replication
2021-06-24 18:29:30 -07:00
Harshavardhana
8d1bc65757
allow resetting and reapply config on broken clusters (#12554)
Bonus: remove kms_kes as sub-system, since its ENV only.
- also fixes a crash with etcd cluster without KMS
configured and also if KMS decryption is missing.
2021-06-24 16:24:12 -07:00
Harshavardhana
cc5656f6d5
allow OPA fallback for STS requests (#12568)
fixes #12547
2021-06-24 12:00:06 -07:00
Harshavardhana
41caf89cf4
fix: apply pre-conditions first on object metadata (#12545)
This change in error flow complies with AWS S3 behavior
for applications depending on specific error conditions.

fixes #12543
2021-06-24 09:44:00 -07:00
Emir İşman
9bf1ac0bb6
[deps]: update jwt-go dependency (#12544)
jwt-go has been renamed to jwt and has a new home. See
https://github.com/dgrijalva/jwt-go/issues/462
2021-06-24 08:41:04 -07:00
Poorna Krishnamoorthy
ba6e9682e5
Clean up targets properly on bucket deletion (#12565) 2021-06-24 08:39:58 -07:00
Harshavardhana
b4bf82c751
do not heal "backend-encrypted" out-of-band with migration (#12556)
backend-encrypted doesn't need to be explicitly healed anymore
since this file is deleted upon upgrade and migration to the
KMS based encrypted config/IAM credentials.
2021-06-23 12:09:10 -07:00
zxxxhonest
3a0ca7af8c
panic: unaligned 64-bit atomic operation (#12559)
goroutine 1 [running]:
runtime/internal/atomic.panicUnaligned()
        /usr/local/go/src/runtime/internal/atomic/unaligned.go:8 +0x24

golang doc:
// BUG(rsc): On x86-32, the 64-bit functions use instructions unavailable before the Pentium MMX.
//
// On non-Linux ARM, the 64-bit functions use instructions unavailable before the ARMv6k core.
//
// On ARM, x86-32, and 32-bit MIPS,
// it is the caller's responsibility to arrange for 64-bit
// alignment of 64-bit words accessed atomically. The first word in a
// variable or in an allocated struct, array, or slice can be relied upon to be
// 64-bit aligned.
2021-06-23 07:15:43 -07:00
Harshavardhana
bcea72ee8a
fix: ForceUnlock should unlock everywhere (#12542) 2021-06-21 17:47:35 -07:00
Klaus Post
14bb969782
lock on multi pool object creation (#12541)
Create write lock on PutObject and CopyObject when on multi-pool setup.

Use the same lock as NewMultipartUpload so all creation calls share the same lock.
2021-06-21 09:25:10 -07:00
Harshavardhana
8f1fe3b761
fix: --console-address when specified endpoints missing (#12534)
Additionally upgrade console dependency for reading
environment variables properly.
2021-06-20 23:04:47 -07:00
Klaus Post
42ad12ebf1
fs: Handle no buckets (#12531)
Since total are sent after each bucket, deleting all would 
not make usage go to zero.
2021-06-18 10:41:36 -07:00
Harshavardhana
cdeccb5510
feat: Deprecate embedded browser and import console (#12460)
This feature also changes the default port where
the browser is running, now the port has moved
to 9001 and it can be configured with

```
--console-address ":9001"
```
2021-06-17 20:27:04 -07:00
Klaus Post
ef99438695
fs: Return faster on no ListObjects results (#12525)
When no results are sent `result.end` is never sent, so the list becomes hot until the list is full.

Break immediately when channel is closed.

Fixes #12518
2021-06-17 08:16:31 -07:00
Harshavardhana
180eabaa8e fix: rename(tmp, tmp-old) is necessary
previous PR incorrectly changed rename() from
tmp to -> tmp/.trash/uuid, since it is self
referential - to clear this up make sure its
renamed to a separate folder and deleted
in background - just like before.
2021-06-16 16:19:26 -07:00
Harshavardhana
4669d19f2a
fix: simplify diskMap usage to keep certain checks predictable (#12519)
Bonus: also make sure that we Sanitize() the drives only during
startup of the server, but not during disk reconnects.
2021-06-16 14:26:26 -07:00
Klaus Post
a6cbfc3600
fs: fix stale bucket counts in data usage (#12521)
In FS mode bucket count would be incorrect. Children were not removed.

Other totals is correct, though.

Fixes #12512
2021-06-16 14:22:55 -07:00
Klaus Post
33cee9f38a
Improve multipart upload (#12514)
Each multipart upload is holding a read lock for the entire upload 
duration of each part.

This makes it impossible for other parts to complete until all currently 
uploading parts have released their locks.

It will also make it impossible for new parts to start as long as the 
write lock is still being requested, essentially deadlocking uploads 
until all that may have been granted a read lock has been completed.

Refactor to only hold the upload id lock while reading and writing 
the metadata, but hold a part id lock while the part is being uploaded.
2021-06-16 13:21:36 -07:00
Harshavardhana
951877f576
fix: root credentials should be able to create users (#12511) 2021-06-15 18:52:01 -07:00
Anis Elleuch
7722b91e1d
s3: Force a prefix removal using a special header (#12504)
An S3 client can send `x-minio-force-delete: true` to remove a prefix.
2021-06-15 18:43:14 -07:00
Anis Elleuch
f30c996d48
trace: Add bucket/prefix to WalkDir() tracing (#12510)
Bonus, replace os.* API with os-instrumented.go
2021-06-15 14:34:26 -07:00
Harshavardhana
da74e2f167
move internal/net to pkg/net package (#12505) 2021-06-14 14:54:37 -07:00
Harshavardhana
0d1fb10940
fix: use madmin.Credentials for gateway interface (#12493)
the main reason is to de-couple the project from
depending on MinIO's internal/auth package, other
changes will subsequently follow.
2021-06-14 12:53:49 -07:00
Klaus Post
b89c0beea4
fix a crash from unstable sort for > 2 pools (#12501)
Fix in https://github.com/minio/minio/pull/12487 assumes that slices with 
tiebreaks are sorted equally. That is only the case for "stable"  sort versions.
2021-06-14 11:00:13 -07:00
Harshavardhana
31971906ff
fix: force-delete should just rename to .trash (#12499)
avoid blocking call for force-delete, instead
treat it lazily and delete in background.
2021-06-14 08:04:37 -07:00
Andreas Auernhammer
a43f2097ff
admin: add KMS status API (#12429)
This commit adds an admin API for fetching
the KMS status information (default key ID, endpoints, ...).

With this commit the server exposes REST endpoint:
```
GET <admin-api>/kms/status
```

Signed-off-by: Andreas Auernhammer <hi@aead.dev>
2021-06-11 15:04:26 -07:00
Anis Elleuch
f982303b4b
webui: Assign parent user with the new temporary account (#12489)
Web Handlers can generate STS tokens but forgot to create a parent user
and save it along with the temporary access account. This commit fixes
this.

fixes #12381
2021-06-11 08:34:27 -07:00
Harshavardhana
0385ecbf34
fix: serve always only the latest objects (#12487)
due to a historic bug, it is possible that
some objects might exist on multiple pools,
rely on ModTime to return the correct pool.
2021-06-10 23:07:16 -07:00
ebozduman
b154581b65
fix: partially defined cred env vars cause "minio gateway s3" to fail (#12228)
Both credential env vars not needed to start s3 gateway
2021-06-10 22:28:09 -07:00
Anis Elleuch
ba5fb2365c
feat: support of ZIP list/get/head as S3 extension (#12267)
When enabled, it is possible to list/get files
inside a zip file without uncompressing it.

Signed-off-by: Anis Elleuch <anis@min.io>
2021-06-10 08:17:03 -07:00
Harshavardhana
a93aa2eac1
fix: upon failure attempt an undo for all calls in DeleteBucket() (#12480)
its possible that, version might exist on second pool such that
upon deleteBucket() might have deleted the bucket on pool1 successfully
since it doesn't have any objects, undo such operations properly in
all any error scenario.

Also delete bucket metadata from pool layer rather than sets layer.
2021-06-09 17:13:00 -07:00
Harshavardhana
0980554725
fix: getServerPoolsAvailableSpace() shouldn't crash (#12478)
if one of the disk is offline then DiskInfo can be `nil`
and crash in server pool.
2021-06-09 11:14:47 -07:00
Anis Elleuch
8e9e028c0c
fix: safe update of the audit objectErasureMap (#12477)
objectErasureMap in the audit holds information about the objects
involved in the current S3 operation such as pool index, set an index,
and disk endpoints. One user saw a crash due to a concurrent update of
objectErasureMap information. Use sync.Map to prevent a crash.
2021-06-09 10:51:19 -07:00
Harshavardhana
af6366e102 fix: allow GetBucketLifecycle in NAS gateway 2021-06-09 08:48:07 -07:00
Harshavardhana
66d549c05d
remove support for deprecated MINIO_KMS_MASTER_KEY (#12463) 2021-06-08 18:50:14 -07:00
Anis Elleuch
6c8be64cdb
rest: healthcheck should not update failure metrics (#12458)
Otherwise, we can see high numbers of networking issues when a node is
down.
2021-06-08 14:09:26 -07:00
Klaus Post
9a2102f5ed
Always get actual size in CopyObjectPart (#12466)
Always use `GetActualSize` to get the part size, not just when encrypted.

Fixes mint test io.minio.MinioClient.uploadPartCopy, 
error "Range specified is not valid for source object".
2021-06-08 09:51:55 -07:00
Harshavardhana
542fe4ea2e
fix: legacy objects with 10MiB blockSize should use right buffers (#12459)
healing code was using incorrect buffers to heal older
objects with 10MiB erasure blockSize, incorrect calculation
of such buffers can lead to incorrect premature closure of
io.Pipe() during healing.

fixes #12410
2021-06-07 10:06:06 -07:00
Harshavardhana
dd2831c1a0
fix: remove parent dirs in RenameData upon failure (#12452)
- it is possible that during I/O failures we might
  leave partially written directories, make sure
  we purge them after.

- rename current data-dir (null) versionId only after
  the newer xl.meta has been written fully.

- attempt removal once for minioMetaTmpBucket/uuid/
  as this folder is empty if all previous operations
  were successful, this allows avoiding recursive os.Remove()
2021-06-07 09:35:08 -07:00
Klaus Post
403f4b9c84
Improve disk usage calculation (#12376)
- for single pool setups usage is not checked.
- for pools, only check the "set" in which it would be placed.
- keep a minimum number of inodes (when we know it).
- ignore for `.minio.sys`.
2021-06-07 08:13:15 -07:00