Commit Graph

3685 Commits

Author SHA1 Message Date
Anis Elleuch
3109441258
s3: Return correct error XML tag in case of copy object (#12427)
In Copy Object S3 API, the server does not return correct bucket &
object names when the source bucket/object does not exist, this commit
fixes it.
2021-06-03 17:25:31 -07:00
Aditya Manthramurthy
30a3921d3e
[Tiering] Support remote tiers with object versioning (#12342)
- Adds versioning support for S3 based remote tiers that have versioning
enabled. This ensures that when reading or deleting we specify the specific
version ID of the object. In case of deletion, this is important to ensure that
the object version is actually deleted instead of simply being marked for
deletion.

- Stores the remote object's version id in the tier-journal. Tier-journal file
version is not bumped up as serializing the new struct version is
compatible with old journals without the remote object version id.

- `storageRESTVersion` is bumped up as FileInfo struct now includes a
`TransitionRemoteVersionID` member.

- Azure and GCS support for this feature will be added subsequently.

Co-authored-by: Krishnan Parthasarathi <krisis@users.noreply.github.com>
2021-06-03 14:26:51 -07:00
Shireesh Anjal
fb140c146b
Redact sensitive values from config in health data (#12421)
The health api returns the server configuration details. Redact
sensitive values from the config values like URLs and credentials.
2021-06-03 08:15:44 -07:00
Harshavardhana
7a3b5235bf remove deprecated kms_vault unused key name 2021-06-03 00:10:11 -07:00
Poorna Krishnamoorthy
dbea8d2ee0
Add support for existing object replication. (#12109)
Also adding an API to allow resyncing replication when
existing object replication is enabled and the remote target
is entirely lost. With the `mc replicate reset` command, the
objects that are eligible for replication as per the replication
config will be resynced to target if existing object replication
is enabled on the rule.
2021-06-01 19:59:11 -07:00
Harshavardhana
1f262daf6f
rename all remaining packages to internal/ (#12418)
This is to ensure that there are no projects
that try to import `minio/minio/pkg` into
their own repo. Any such common packages should
go to `https://github.com/minio/pkg`
2021-06-01 14:59:40 -07:00
Harshavardhana
bf87c4b1e4
fix: no need to proxy if IAM not initialized (#12416)
IAM not initialized doesn't mean we can't still
read the content from the disk, we should just
allow the request to go-through if object layer
is initialized.
2021-06-01 12:23:13 -07:00
Harshavardhana
7148c2490a
avoid metrics not meant for single drive mode (#12415)
fixes #12414
2021-06-01 12:22:42 -07:00
Bala FA
120951d9e9
Refactor health data structure (#11914)
This feature comes with simplified data structures and versioning support.

Signed-off-by: Bala.FA <bala.gluster@gmail.com>
2021-06-01 08:55:49 -07:00
Anis Elleuch
8347db8be3
sts: Map parent user to the STS access key policy (#12411) 2021-06-01 08:37:42 -07:00
Poorna Krishnamoorthy
3690de0c6b
Drop Pending size and count from replication metrics (#12378)
Real-time metrics calculated in-memory rely on the initial
replication metrics saved with data usage. However, this can
lag behind the actual state of the cluster at the time of server 
restart leading to inaccurate Pending size/counts reported to
Prometheus. Dropping the Pending metrics as this can be more 
reliably monitored by applications with replication notifications.

Signed-off-by: Poorna Krishnamoorthy <poorna@minio.io>
2021-05-31 20:26:52 -07:00
Harshavardhana
fdc2020b10
move to iam, bucket policy from minio/pkg (#12400) 2021-05-29 21:16:42 -07:00
Harshavardhana
3350dbc50d
always indent and reply policy JSON (#12399) 2021-05-29 09:22:22 -07:00
Harshavardhana
81d5688d56
move the dependency to minio/pkg for common libraries (#12397) 2021-05-28 15:17:01 -07:00
Poorna Krishnamoorthy
547bb7d0a1
replication: Init worker kill channel correctly (#12379)
Signed-off-by: Poorna Krishnamoorthy <poorna@minio.io>
2021-05-28 13:28:37 -07:00
Harshavardhana
4444ba13a4
support ldap:username for policy substitution (#12390)
LDAPusername is the simpler form of LDAPUser (userDN),
using a simpler version is convenient from policy
conditions point of view, since these are unique id's
used for LDAP login.
2021-05-28 10:33:07 -07:00
Harshavardhana
fa8e3151bc
fix: move to new etcd imports (#12391) 2021-05-28 10:31:42 -07:00
Harshavardhana
89bb9f17d7
fix: when parityDrives hits > len(storageDisks)/2, keep maxParity (#12387)
Additionally move out `x-minio-internal-erasure-upgraded` from HTTP headers
list, as its an internal header, rename elsewhere accordingly.
2021-05-27 13:38:04 -07:00
Klaus Post
acc452b7ce
Add more erasure codes on degraded systems. (#11852)
In cases where a cluster is degraded, we do not uphold our consistency 
guarantee and we will write fewer erasure codes and rely on healing 
to recreate the missing shards.

In some cases replacing known bad disks in practice take days.
We want to change the behavior of a known degraded system to keep
the erasure code promise of the storage class for each object.

This will create the objects with the same confidence as a fully 
functional cluster. The tradeoff will be that objects created 
during a partial outage will take up slightly more space.

This means that when the storage class is EC:4, there should 
always be written 4 parity shards, even if some disks are unavailable.

When an object is created on a set, the disks are immediately 
checked. If any disks are unavailable additional parity shards 
will be made for each offline disk, up to 50% of the number of disks.

We add an internal metadata field with the actual and intended 
erasure code level, this can optionally be picked up later by 
the scanner if we decide that data like this should be re-sharded.
2021-05-27 11:38:09 -07:00
Harshavardhana
be541dba8a
feat: introduce listUsers, listPolicies for any bucket (#12372)
Bonus change LDAP settings such as user, group mappings
are now listed as part of `mc admin user list` and
`mc admin group list`

Additionally this PR also deprecates the `/v2` API
that is no longer in use.
2021-05-27 10:15:02 -07:00
Harshavardhana
b5ebfd35b4
fix: always prefer DataBlocks present in FileInfo (#12386) 2021-05-27 10:11:50 -07:00
Anis Elleuch
530b703902
audit/logger: Increase http request timeout (#12385)
A configured audit logger or HTTP logger is validated during MinIO
server startup. Relax the timeout to 10 seconds in that case, otherwise,
both loggers won't be used.

1 second could be too low for a busy HTTP endpoint.
2021-05-27 09:54:10 -07:00
Andreas Auernhammer
e8a12cbfdd
etag: compute ETag as MD5 for compressed single-part objects (#12375)
This commit fixes a bug causing the MinIO server to compute
the ETag of a single-part object as MD5 of the compressed
content - not as MD5 of the actual content.

This usually does not affect clients since the MinIO appended
a `-1` to indicate that the ETag belongs to a multipart object.
However, this behavior was problematic since:
 - A S3 client being very strict should reject such an ETag since
   the client uploaded the object via single-part API but got
   a multipart ETag that is not the content MD5.
 - The MinIO server leaks (via the ETag) that it compressed the
   object.

This commit addresses both cases. Now, the MinIO server returns
an ETag equal to the content MD5 for single-part objects that got
compressed.

Signed-off-by: Andreas Auernhammer <aead@mail.de>
2021-05-27 08:18:41 -07:00
Anis Elleuch
e63908c391
Update bloom module (#12383)
To fix dependency import issues when importing madmin-go v0.7.1
2021-05-27 08:02:39 -07:00
Harshavardhana
b251ae5f3d fix: update default values for listing, replication workers 2021-05-26 11:55:46 -07:00
Anis Elleuch
0e80b5fe63
tests: Add test for upload of the same object inlined and not inlined (#12374)
Upload an object smaller than small file threshold and upload another file bigger
than small file threshold and tries to read it.
2021-05-26 08:09:23 -07:00
Harshavardhana
225d8c51fd
fix: missing path in admin trace (#12373)
PR #12360 introduced a change which seems to have
added a regression, the RawPath in r.URL seems to
be empty, if it is fallback to r.URL.Path instead.
2021-05-26 08:04:12 -07:00
Klaus Post
3fff50120b
Revert heal locks (#12365)
A lot of healing is likely to be on non-existing objects and 
locks are very expensive and will slow down scanning 
significantly.

In cases where all are valid or, all are broken allow 
rejection without locking.

Keep the existing behavior, but move the check for 
dangling objects to after the lock has been acquired.

```
	_, err = getLatestFileInfo(ctx, partsMetadata, errs)
	if err != nil {
		return er.purgeObjectDangling(ctx, bucket, object, versionID, partsMetadata, errs, []error{}, opts)
	}
```

Revert "heal: Hold lock when reading xl.meta from disks (#12362)"

This reverts commit abd32065aa
2021-05-25 17:02:06 -07:00
Harshavardhana
4840974d7a
fix: inline data upon overwrites should be readable (#12369)
This PR fixes two bugs

- Remove fi.Data upon overwrite of objects from inlined-data to non-inlined-data
- Workaround for an existing bug on disk with latest releases to ignore fi.Data
  and instead read from the disk for non-inlined-data
- Addtionally add a reserved metadata header to indicate data is inlined for
  a given version.
2021-05-25 16:33:06 -07:00
Harshavardhana
4fd1378242
fix: lint errors after upgrading golangci-lint (#12368) 2021-05-25 14:17:33 -07:00
Harshavardhana
ed4941a5f3
fix: calculate dataBlocks properly in healing (#12364) 2021-05-25 09:34:27 -07:00
Harshavardhana
cacdeca8cc
fix: return error for unexpected quorum in pickValidFileInfo (#12363) 2021-05-24 18:31:56 -07:00
Anis Elleuch
abd32065aa
heal: Hold lock when reading xl.meta from disks (#12362)
Lock is hold in healObject() after reading xl.meta from disks the first
time. This commit will held the lock since the beginning of HealObject()

Co-authored-by: Anis Elleuch <anis@min.io>
2021-05-24 13:39:38 -07:00
Harshavardhana
2baabd455b docs: fix per tenant limits docs formatting 2021-05-24 09:37:17 -07:00
Harshavardhana
ebf75ef10d
fix: remove all unused code (#12360) 2021-05-24 09:28:19 -07:00
Klaus Post
f01820a4ee
fix: invalid multipart offset when compressed+encrypted. (#12340)
Fixes `testSSES3EncryptedGetObjectReadSeekFunctional` mint test.

```
{
  "args": {
    "bucketName": "minio-go-test-w53hbpat649nhvws",
    "objectName": "6mdswladz4vfpp2oit1pkn3qd11te5"
  },
  "duration": 7537,
  "error": "We encountered an internal error, please try again.: cause(The requested range \"bytes 251717932 -> -116384170 of 135333762\" is not satisfiable.)",
  "function": "GetObject(bucketName, objectName)",
  "message": "CopyN failed",
  "name": "minio-go: testSSES3EncryptedGetObjectReadSeekFunctional",
  "status": "FAIL"
}
```

Compressed files always start at the beginning of a part so no additional offset should be added.
2021-05-21 14:07:16 -07:00
Harshavardhana
0287711dc9
fix: implement readMetadata common function for re-use (#12353)
Previous PR #12351 added functions to read from the reader
stream to reduce memory usage, use the same technique in
few other places where we are not interested in reading the
data part.
2021-05-21 11:41:25 -07:00
Klaus Post
9d1b6fb37d
Add XL reader without data (#12351)
Add XL metadata reader that reads metadata only on larger files.

Use for scanning and listing for now.
2021-05-21 09:10:54 -07:00
Harshavardhana
32d8a48d4e
reduce memory usage in metacache reader (#12334) 2021-05-20 09:00:11 -07:00
Harshavardhana
6060b755c6
fix: migrate users properly from older releases to newer (#12333) 2021-05-19 19:25:44 -07:00
Krishnan Parthasarathi
cfa94cc35c
Simplify remote tier validation in lifecycle rule validation (#12329) 2021-05-19 18:51:23 -07:00
Klaus Post
2ca9c533ef
feat: implement in-progress partial bucket updates (#12279) 2021-05-19 14:38:30 -07:00
Anis Elleuch
866593fd94
heal: Ignore disks with non quorum modtime and dataDir (#12328) 2021-05-19 12:04:08 -07:00
Harshavardhana
ecb5525c91
fix: muxing order for rejected APIs (#12321) 2021-05-19 09:21:34 -07:00
Klaus Post
c2c803dd30
Fix list entry deduplication (#12325)
File infos would always be the same.

Add numversions as a final tiebreaker.
2021-05-19 09:21:18 -07:00
Harshavardhana
4f5d75f22b
fix: speed up drive mux registration (#12319)
in setups with lots of drives the server
startup is slow, initialize all local drives
in parallel before registering with muxer.

this speeds up when there are multiple pools
and large collection of drives.
2021-05-18 17:25:00 -07:00
Harshavardhana
bb7fbcdc09
fix: generating service accounts for group only LDAP accounts (#12318)
fixes #12315
2021-05-18 15:19:20 -07:00
Andreas Auernhammer
82c53ac260
sse-kms: set KMS key ID response header (#12316)
This commit adds the `X-Amz-Server-Side-Encryption-Aws-Kms-Key-Id`
response header to the GET, HEAD, PUT and Download API.

Based on AWS documentation [1] AWS S3 returns the KMS key ID as part
of the response headers.

[1] https://docs.aws.amazon.com/AmazonS3/latest/userguide/specifying-kms-encryption.html

Signed-off-by: Andreas Auernhammer <aead@mail.de>
2021-05-18 14:21:20 -07:00
Harshavardhana
a70e0da19e
use direntPool, direntNamePool for reusable buffers (#12314)
- in readDirFn re-use buffers from direntPool()
- in readDirN use separate dirent name buffer direntNamePool()
2021-05-18 10:29:50 -07:00
Harshavardhana
c6b7dc012a
fix: use key.Ciphertext for DecryptKey in KeyStatus (#12313)
enhance GlobalKMS.Stat() for kes to actually perform
a network call to check Version() of kes and also
implicitly that its reachable.
2021-05-18 07:22:31 -07:00