Commit Graph

9280 Commits

Author SHA1 Message Date
Harshavardhana
0d1fb10940 fix: use madmin.Credentials for gateway interface (#12493)
the main reason is to de-couple the project from
depending on MinIO's internal/auth package, other
changes will subsequently follow.
2021-06-14 12:53:49 -07:00
Klaus Post
0d1d26a4ea Simplify s3zip example (#12500)
There is no need to create a custom transport wrapper.

Just set the header in the options.
2021-06-14 11:04:14 -07:00
Klaus Post
b89c0beea4 fix a crash from unstable sort for > 2 pools (#12501)
Fix in https://github.com/minio/minio/pull/12487 assumes that slices with 
tiebreaks are sorted equally. That is only the case for "stable"  sort versions.
2021-06-14 11:00:13 -07:00
Harshavardhana
31971906ff fix: force-delete should just rename to .trash (#12499)
avoid blocking call for force-delete, instead
treat it lazily and delete in background.
2021-06-14 08:04:37 -07:00
Harshavardhana
264ee97219 update to RELEASE.2021-06-14T01-29-23Z release 2021-06-13 20:58:56 -07:00
Poorna Krishnamoorthy
92e4c8be10 Update replication docs to fix mc reference (#12490)
Signed-off-by: Poorna Krishnamoorthy <poorna@minio.io>
RELEASE.2021-06-14T01-29-23Z
2021-06-13 11:37:22 -07:00
Harshavardhana
9713f59d24 add support for quay.io mirror 2021-06-11 16:23:14 -07:00
Andreas Auernhammer
a43f2097ff admin: add KMS status API (#12429)
This commit adds an admin API for fetching
the KMS status information (default key ID, endpoints, ...).

With this commit the server exposes REST endpoint:
```
GET <admin-api>/kms/status
```

Signed-off-by: Andreas Auernhammer <hi@aead.dev>
2021-06-11 15:04:26 -07:00
Anis Elleuch
f982303b4b webui: Assign parent user with the new temporary account (#12489)
Web Handlers can generate STS tokens but forgot to create a parent user
and save it along with the temporary access account. This commit fixes
this.

fixes #12381
2021-06-11 08:34:27 -07:00
Harshavardhana
0385ecbf34 fix: serve always only the latest objects (#12487)
due to a historic bug, it is possible that
some objects might exist on multiple pools,
rely on ModTime to return the correct pool.
2021-06-10 23:07:16 -07:00
ebozduman
b154581b65 fix: partially defined cred env vars cause "minio gateway s3" to fail (#12228)
Both credential env vars not needed to start s3 gateway
2021-06-10 22:28:09 -07:00
Poorna Krishnamoorthy
f2a3872301 Update design.md for replication (#12486)
Fixes #12483
2021-06-10 16:05:39 -07:00
Anis Elleuch
ba5fb2365c feat: support of ZIP list/get/head as S3 extension (#12267)
When enabled, it is possible to list/get files
inside a zip file without uncompressing it.

Signed-off-by: Anis Elleuch <anis@min.io>
2021-06-10 08:17:03 -07:00
Harshavardhana
c221633a8a remove unused docker_kms_master_encryption_env 2021-06-10 02:39:11 -07:00
Harshavardhana
a93aa2eac1 fix: upon failure attempt an undo for all calls in DeleteBucket() (#12480)
its possible that, version might exist on second pool such that
upon deleteBucket() might have deleted the bucket on pool1 successfully
since it doesn't have any objects, undo such operations properly in
all any error scenario.

Also delete bucket metadata from pool layer rather than sets layer.
2021-06-09 17:13:00 -07:00
Minio Trusted
22c58be421 Update yaml files to latest version RELEASE.2021-06-09T18-51-39Z 2021-06-09 19:21:21 +00:00
Harshavardhana
0980554725 fix: getServerPoolsAvailableSpace() shouldn't crash (#12478)
if one of the disk is offline then DiskInfo can be `nil`
and crash in server pool.
RELEASE.2021-06-09T18-51-39Z
2021-06-09 11:14:47 -07:00
Anis Elleuch
8e9e028c0c fix: safe update of the audit objectErasureMap (#12477)
objectErasureMap in the audit holds information about the objects
involved in the current S3 operation such as pool index, set an index,
and disk endpoints. One user saw a crash due to a concurrent update of
objectErasureMap information. Use sync.Map to prevent a crash.
2021-06-09 10:51:19 -07:00
iternity-dotcom
dd5e9493f1 docs: add missing admin actions (#12442) 2021-06-09 10:37:20 -07:00
Harshavardhana
af6366e102 fix: allow GetBucketLifecycle in NAS gateway 2021-06-09 08:48:07 -07:00
Nitish Tiwari
32017454ee fix typo in Grafana dashboard json (#12471) 2021-06-09 08:04:12 -07:00
Harshavardhana
66d549c05d remove support for deprecated MINIO_KMS_MASTER_KEY (#12463) 2021-06-08 18:50:14 -07:00
Anis Elleuch
6c8be64cdb rest: healthcheck should not update failure metrics (#12458)
Otherwise, we can see high numbers of networking issues when a node is
down.
2021-06-08 14:09:26 -07:00
Klaus Post
9a2102f5ed Always get actual size in CopyObjectPart (#12466)
Always use `GetActualSize` to get the part size, not just when encrypted.

Fixes mint test io.minio.MinioClient.uploadPartCopy, 
error "Range specified is not valid for source object".
2021-06-08 09:51:55 -07:00
Minio Trusted
1bf80a6eea Update yaml files to latest version RELEASE.2021-06-07T21-40-51Z 2021-06-07 22:13:41 +00:00
Nitish Tiwari
00c5d7e1b3 Add healing related metrics in official dashboard (#12456) RELEASE.2021-06-07T21-40-51Z 2021-06-07 12:46:54 -07:00
Shireesh Anjal
000656ac6b Do not send 'sensitive' flag to client (#12461)
As it is server specific and is not required on client side.
2021-06-07 12:43:21 -07:00
Harshavardhana
542fe4ea2e fix: legacy objects with 10MiB blockSize should use right buffers (#12459)
healing code was using incorrect buffers to heal older
objects with 10MiB erasure blockSize, incorrect calculation
of such buffers can lead to incorrect premature closure of
io.Pipe() during healing.

fixes #12410
2021-06-07 10:06:06 -07:00
Harshavardhana
dd2831c1a0 fix: remove parent dirs in RenameData upon failure (#12452)
- it is possible that during I/O failures we might
  leave partially written directories, make sure
  we purge them after.

- rename current data-dir (null) versionId only after
  the newer xl.meta has been written fully.

- attempt removal once for minioMetaTmpBucket/uuid/
  as this folder is empty if all previous operations
  were successful, this allows avoiding recursive os.Remove()
2021-06-07 09:35:08 -07:00
Klaus Post
403f4b9c84 Improve disk usage calculation (#12376)
- for single pool setups usage is not checked.
- for pools, only check the "set" in which it would be placed.
- keep a minimum number of inodes (when we know it).
- ignore for `.minio.sys`.
2021-06-07 08:13:15 -07:00
Harshavardhana
8a9ff2b331 build edge off from latest release
remove complicated docker files just
use one for all situations, if need
to build master build locally using
Dockerfile.dev

fixes #12450
2021-06-05 17:17:36 -07:00
Anis Elleuch
810af07529 xl: Avoid multi-disks node to exit when one disk fails (#12423)
It makes sense that a node that has multiple disks starts when one
disk fails, returning an i/o error for example. This commit will make this
faulty tolerance available in this specific use case.
2021-06-05 09:10:32 -07:00
Harshavardhana
e448dbbabf upgrade madmin-go to v1.0.11 2021-06-04 12:51:04 -07:00
Poorna Krishnamoorthy
f199afcd6c tiering: add aws role support for s3 (#12424)
Signed-off-by: Poorna Krishnamoorthy <poorna@minio.io>
2021-06-04 12:47:00 -07:00
Harshavardhana
36b2f6d11d fix: etcd IAM encryption fails due to incorrect kms.Context (#12431)
Due to incorrect KMS context constructed, we need to add
additional fallbacks and also fix the original root cause
to fix already migrated deployments.

Bonus remove double migration is avoided in gateway mode
for etcd, instead do it once in iam.Init(), also simplify
the migration by not migrating STS users instead let the
clients regenerate them.
2021-06-04 11:15:13 -07:00
Harshavardhana
c0e41356f5 add error level logger similar to Info() for console (#12445) 2021-06-04 11:11:30 -07:00
Klaus Post
d524544494 Fix nil disk check in parity upgrade feature (#12444)
Fixes #12443
2021-06-04 09:38:19 -07:00
Harshavardhana
c0e79e28b2 fix: close the channel appropriately for dataUsageEntry (#12432)
Bonus: initialize dataScanner routines after server
config has initialized.

fixes #12430
2021-06-03 19:18:59 -07:00
Anis Elleuch
3109441258 s3: Return correct error XML tag in case of copy object (#12427)
In Copy Object S3 API, the server does not return correct bucket &
object names when the source bucket/object does not exist, this commit
fixes it.
2021-06-03 17:25:31 -07:00
Aditya Manthramurthy
30a3921d3e [Tiering] Support remote tiers with object versioning (#12342)
- Adds versioning support for S3 based remote tiers that have versioning
enabled. This ensures that when reading or deleting we specify the specific
version ID of the object. In case of deletion, this is important to ensure that
the object version is actually deleted instead of simply being marked for
deletion.

- Stores the remote object's version id in the tier-journal. Tier-journal file
version is not bumped up as serializing the new struct version is
compatible with old journals without the remote object version id.

- `storageRESTVersion` is bumped up as FileInfo struct now includes a
`TransitionRemoteVersionID` member.

- Azure and GCS support for this feature will be added subsequently.

Co-authored-by: Krishnan Parthasarathi <krisis@users.noreply.github.com>
2021-06-03 14:26:51 -07:00
Harshavardhana
41d4d650e4 simplify dockerfiles and remove duplication (#12419)
also add additional packages such as 'iproute',
'iputils' for `ping` and `ip addr` commands.
2021-06-03 08:16:06 -07:00
Shireesh Anjal
fb140c146b Redact sensitive values from config in health data (#12421)
The health api returns the server configuration details. Redact
sensitive values from the config values like URLs and credentials.
2021-06-03 08:15:44 -07:00
Harshavardhana
7a3b5235bf remove deprecated kms_vault unused key name 2021-06-03 00:10:11 -07:00
Harshavardhana
67ce920fea remove ruleguard rules 2021-06-01 20:44:01 -07:00
Poorna Krishnamoorthy
dbea8d2ee0 Add support for existing object replication. (#12109)
Also adding an API to allow resyncing replication when
existing object replication is enabled and the remote target
is entirely lost. With the `mc replicate reset` command, the
objects that are eligible for replication as per the replication
config will be resynced to target if existing object replication
is enabled on the rule.
2021-06-01 19:59:11 -07:00
Harshavardhana
1f262daf6f rename all remaining packages to internal/ (#12418)
This is to ensure that there are no projects
that try to import `minio/minio/pkg` into
their own repo. Any such common packages should
go to `https://github.com/minio/pkg`
2021-06-01 14:59:40 -07:00
Harshavardhana
bf87c4b1e4 fix: no need to proxy if IAM not initialized (#12416)
IAM not initialized doesn't mean we can't still
read the content from the disk, we should just
allow the request to go-through if object layer
is initialized.
2021-06-01 12:23:13 -07:00
Harshavardhana
7148c2490a avoid metrics not meant for single drive mode (#12415)
fixes #12414
2021-06-01 12:22:42 -07:00
Bala FA
120951d9e9 Refactor health data structure (#11914)
This feature comes with simplified data structures and versioning support.

Signed-off-by: Bala.FA <bala.gluster@gmail.com>
2021-06-01 08:55:49 -07:00
Anis Elleuch
8347db8be3 sts: Map parent user to the STS access key policy (#12411) 2021-06-01 08:37:42 -07:00