Commit Graph

8493 Commits

Author SHA1 Message Date
Harshavardhana
3383a311b8 build containers to ship FIPS compatible MinIO 2021-06-18 09:11:55 -07:00
Harshavardhana
cdeccb5510
feat: Deprecate embedded browser and import console (#12460)
This feature also changes the default port where
the browser is running, now the port has moved
to 9001 and it can be configured with

```
--console-address ":9001"
```
2021-06-17 20:27:04 -07:00
Klaus Post
ef99438695
fs: Return faster on no ListObjects results (#12525)
When no results are sent `result.end` is never sent, so the list becomes hot until the list is full.

Break immediately when channel is closed.

Fixes #12518
2021-06-17 08:16:31 -07:00
Harshavardhana
e4fbc6a9ff update to RELEASE.2021-06-17T00-10-46Z 2021-06-16 17:50:32 -07:00
Harshavardhana
180eabaa8e fix: rename(tmp, tmp-old) is necessary
previous PR incorrectly changed rename() from
tmp to -> tmp/.trash/uuid, since it is self
referential - to clear this up make sure its
renamed to a separate folder and deleted
in background - just like before.
2021-06-16 16:19:26 -07:00
Harshavardhana
4669d19f2a
fix: simplify diskMap usage to keep certain checks predictable (#12519)
Bonus: also make sure that we Sanitize() the drives only during
startup of the server, but not during disk reconnects.
2021-06-16 14:26:26 -07:00
Klaus Post
a6cbfc3600
fs: fix stale bucket counts in data usage (#12521)
In FS mode bucket count would be incorrect. Children were not removed.

Other totals is correct, though.

Fixes #12512
2021-06-16 14:22:55 -07:00
Klaus Post
33cee9f38a
Improve multipart upload (#12514)
Each multipart upload is holding a read lock for the entire upload 
duration of each part.

This makes it impossible for other parts to complete until all currently 
uploading parts have released their locks.

It will also make it impossible for new parts to start as long as the 
write lock is still being requested, essentially deadlocking uploads 
until all that may have been granted a read lock has been completed.

Refactor to only hold the upload id lock while reading and writing 
the metadata, but hold a part id lock while the part is being uploaded.
2021-06-16 13:21:36 -07:00
Harshavardhana
951877f576
fix: root credentials should be able to create users (#12511) 2021-06-15 18:52:01 -07:00
Anis Elleuch
7722b91e1d
s3: Force a prefix removal using a special header (#12504)
An S3 client can send `x-minio-force-delete: true` to remove a prefix.
2021-06-15 18:43:14 -07:00
Anis Elleuch
f30c996d48
trace: Add bucket/prefix to WalkDir() tracing (#12510)
Bonus, replace os.* API with os-instrumented.go
2021-06-15 14:34:26 -07:00
Harshavardhana
da74e2f167
move internal/net to pkg/net package (#12505) 2021-06-14 14:54:37 -07:00
Harshavardhana
ed6cc66cf4
fix: remove unnecessary limit for queueStore (#12491)
There is no good reason to limit ourselves
to max_open_fd for queue_store

Bonus: Support for publisher confirms
2021-06-14 13:28:44 -07:00
Harshavardhana
0d1fb10940
fix: use madmin.Credentials for gateway interface (#12493)
the main reason is to de-couple the project from
depending on MinIO's internal/auth package, other
changes will subsequently follow.
2021-06-14 12:53:49 -07:00
Klaus Post
0d1d26a4ea
Simplify s3zip example (#12500)
There is no need to create a custom transport wrapper.

Just set the header in the options.
2021-06-14 11:04:14 -07:00
Klaus Post
b89c0beea4
fix a crash from unstable sort for > 2 pools (#12501)
Fix in https://github.com/minio/minio/pull/12487 assumes that slices with 
tiebreaks are sorted equally. That is only the case for "stable"  sort versions.
2021-06-14 11:00:13 -07:00
Harshavardhana
31971906ff
fix: force-delete should just rename to .trash (#12499)
avoid blocking call for force-delete, instead
treat it lazily and delete in background.
2021-06-14 08:04:37 -07:00
Harshavardhana
264ee97219 update to RELEASE.2021-06-14T01-29-23Z release 2021-06-13 20:58:56 -07:00
Poorna Krishnamoorthy
92e4c8be10
Update replication docs to fix mc reference (#12490)
Signed-off-by: Poorna Krishnamoorthy <poorna@minio.io>
2021-06-13 11:37:22 -07:00
Harshavardhana
9713f59d24 add support for quay.io mirror 2021-06-11 16:23:14 -07:00
Andreas Auernhammer
a43f2097ff
admin: add KMS status API (#12429)
This commit adds an admin API for fetching
the KMS status information (default key ID, endpoints, ...).

With this commit the server exposes REST endpoint:
```
GET <admin-api>/kms/status
```

Signed-off-by: Andreas Auernhammer <hi@aead.dev>
2021-06-11 15:04:26 -07:00
Anis Elleuch
f982303b4b
webui: Assign parent user with the new temporary account (#12489)
Web Handlers can generate STS tokens but forgot to create a parent user
and save it along with the temporary access account. This commit fixes
this.

fixes #12381
2021-06-11 08:34:27 -07:00
Harshavardhana
0385ecbf34
fix: serve always only the latest objects (#12487)
due to a historic bug, it is possible that
some objects might exist on multiple pools,
rely on ModTime to return the correct pool.
2021-06-10 23:07:16 -07:00
ebozduman
b154581b65
fix: partially defined cred env vars cause "minio gateway s3" to fail (#12228)
Both credential env vars not needed to start s3 gateway
2021-06-10 22:28:09 -07:00
Poorna Krishnamoorthy
f2a3872301
Update design.md for replication (#12486)
Fixes #12483
2021-06-10 16:05:39 -07:00
Anis Elleuch
ba5fb2365c
feat: support of ZIP list/get/head as S3 extension (#12267)
When enabled, it is possible to list/get files
inside a zip file without uncompressing it.

Signed-off-by: Anis Elleuch <anis@min.io>
2021-06-10 08:17:03 -07:00
Harshavardhana
c221633a8a remove unused docker_kms_master_encryption_env 2021-06-10 02:39:11 -07:00
Harshavardhana
a93aa2eac1
fix: upon failure attempt an undo for all calls in DeleteBucket() (#12480)
its possible that, version might exist on second pool such that
upon deleteBucket() might have deleted the bucket on pool1 successfully
since it doesn't have any objects, undo such operations properly in
all any error scenario.

Also delete bucket metadata from pool layer rather than sets layer.
2021-06-09 17:13:00 -07:00
Minio Trusted
22c58be421 Update yaml files to latest version RELEASE.2021-06-09T18-51-39Z 2021-06-09 19:21:21 +00:00
Harshavardhana
0980554725
fix: getServerPoolsAvailableSpace() shouldn't crash (#12478)
if one of the disk is offline then DiskInfo can be `nil`
and crash in server pool.
2021-06-09 11:14:47 -07:00
Anis Elleuch
8e9e028c0c
fix: safe update of the audit objectErasureMap (#12477)
objectErasureMap in the audit holds information about the objects
involved in the current S3 operation such as pool index, set an index,
and disk endpoints. One user saw a crash due to a concurrent update of
objectErasureMap information. Use sync.Map to prevent a crash.
2021-06-09 10:51:19 -07:00
iternity-dotcom
dd5e9493f1
docs: add missing admin actions (#12442) 2021-06-09 10:37:20 -07:00
Harshavardhana
af6366e102 fix: allow GetBucketLifecycle in NAS gateway 2021-06-09 08:48:07 -07:00
Nitish Tiwari
32017454ee
fix typo in Grafana dashboard json (#12471) 2021-06-09 08:04:12 -07:00
Harshavardhana
66d549c05d
remove support for deprecated MINIO_KMS_MASTER_KEY (#12463) 2021-06-08 18:50:14 -07:00
Anis Elleuch
6c8be64cdb
rest: healthcheck should not update failure metrics (#12458)
Otherwise, we can see high numbers of networking issues when a node is
down.
2021-06-08 14:09:26 -07:00
Klaus Post
9a2102f5ed
Always get actual size in CopyObjectPart (#12466)
Always use `GetActualSize` to get the part size, not just when encrypted.

Fixes mint test io.minio.MinioClient.uploadPartCopy, 
error "Range specified is not valid for source object".
2021-06-08 09:51:55 -07:00
Minio Trusted
1bf80a6eea Update yaml files to latest version RELEASE.2021-06-07T21-40-51Z 2021-06-07 22:13:41 +00:00
Nitish Tiwari
00c5d7e1b3
Add healing related metrics in official dashboard (#12456) 2021-06-07 12:46:54 -07:00
Shireesh Anjal
000656ac6b
Do not send 'sensitive' flag to client (#12461)
As it is server specific and is not required on client side.
2021-06-07 12:43:21 -07:00
Harshavardhana
542fe4ea2e
fix: legacy objects with 10MiB blockSize should use right buffers (#12459)
healing code was using incorrect buffers to heal older
objects with 10MiB erasure blockSize, incorrect calculation
of such buffers can lead to incorrect premature closure of
io.Pipe() during healing.

fixes #12410
2021-06-07 10:06:06 -07:00
Harshavardhana
dd2831c1a0
fix: remove parent dirs in RenameData upon failure (#12452)
- it is possible that during I/O failures we might
  leave partially written directories, make sure
  we purge them after.

- rename current data-dir (null) versionId only after
  the newer xl.meta has been written fully.

- attempt removal once for minioMetaTmpBucket/uuid/
  as this folder is empty if all previous operations
  were successful, this allows avoiding recursive os.Remove()
2021-06-07 09:35:08 -07:00
Klaus Post
403f4b9c84
Improve disk usage calculation (#12376)
- for single pool setups usage is not checked.
- for pools, only check the "set" in which it would be placed.
- keep a minimum number of inodes (when we know it).
- ignore for `.minio.sys`.
2021-06-07 08:13:15 -07:00
Harshavardhana
8a9ff2b331 build edge off from latest release
remove complicated docker files just
use one for all situations, if need
to build master build locally using
Dockerfile.dev

fixes #12450
2021-06-05 17:17:36 -07:00
Anis Elleuch
810af07529
xl: Avoid multi-disks node to exit when one disk fails (#12423)
It makes sense that a node that has multiple disks starts when one
disk fails, returning an i/o error for example. This commit will make this
faulty tolerance available in this specific use case.
2021-06-05 09:10:32 -07:00
Harshavardhana
e448dbbabf upgrade madmin-go to v1.0.11 2021-06-04 12:51:04 -07:00
Poorna Krishnamoorthy
f199afcd6c
tiering: add aws role support for s3 (#12424)
Signed-off-by: Poorna Krishnamoorthy <poorna@minio.io>
2021-06-04 12:47:00 -07:00
Harshavardhana
36b2f6d11d
fix: etcd IAM encryption fails due to incorrect kms.Context (#12431)
Due to incorrect KMS context constructed, we need to add
additional fallbacks and also fix the original root cause
to fix already migrated deployments.

Bonus remove double migration is avoided in gateway mode
for etcd, instead do it once in iam.Init(), also simplify
the migration by not migrating STS users instead let the
clients regenerate them.
2021-06-04 11:15:13 -07:00
Harshavardhana
c0e41356f5
add error level logger similar to Info() for console (#12445) 2021-06-04 11:11:30 -07:00
Klaus Post
d524544494
Fix nil disk check in parity upgrade feature (#12444)
Fixes #12443
2021-06-04 09:38:19 -07:00