Commit Graph

9247 Commits

Author SHA1 Message Date
Klaus Post aca6dfbd60
Check for nil RPC in listing (#13917)
Fixes #13915
2021-12-15 09:19:11 -08:00
Harshavardhana 5f7e6d03ff
copy bucket slice to avoid skipping .minio.sys/buckets (#13912)
healing was skipping `.minio.sys/buckets` path so
essentially not healing `.usage.json` - fix this
by making a copy of `buckets` slice.
2021-12-15 09:18:09 -08:00
Harshavardhana 88ad742da0
fix: error handling cases in site-replication (#13901)
- Allow proper SRError to be propagated to
  handlers and converted appropriately.

- Make sure to enable object locking on buckets
  when requested in MakeBucketHook.

- When DNSConfig is enabled attempt to delete it
  first before deleting buckets locally.
2021-12-14 14:09:57 -08:00
Klaus Post a8d4042853
select: Add IS (NOT) operators (#13906)
Add `IS` and `IS NOT` as comparison operators.

This may be a bit wider than the S3 spec, but we can rather 
easily remove the forwarding.
2021-12-14 09:54:50 -08:00
Krishnan Parthasarathi 44a9339c0a
Newer noncurrent versions (#13815)
- Rename MaxNoncurrentVersions tag to NewerNoncurrentVersions

Note: We apply overlapping NewerNoncurrentVersions rules such that 
we honor the highest among applicable limits. e.g if 2 overlapping rules 
are configured with 2 and 3 noncurrent versions to be retained, we 
will retain 3.

- Expire newer noncurrent versions after noncurrent days
- MinIO extension: allow noncurrent days to be zero, allowing expiry 
  of noncurrent version as soon as more than configured 
  NewerNoncurrentVersions are present.
- Allow NewerNoncurrentVersions rules on object-locked buckets
- No x-amz-expiration when NewerNoncurrentVersions configured
- ComputeAction should skip rules with NewerNoncurrentVersions > 0
- Add unit tests for lifecycle.ComputeAction
- Support lifecycle rules with MaxNoncurrentVersions
- Extend ExpectedExpiryTime to work with zero days
- Fix all-time comparisons to be relative to UTC
2021-12-14 09:41:44 -08:00
Harshavardhana 113c7ff49a
add code to parse secrets natively instead of shell scripts (#13883) 2021-12-13 18:23:31 -08:00
fpaupier 40dbe243d9
update: keycloak binding documentation (#13894) 2021-12-13 18:23:16 -08:00
Poorna K d422d24278
replication: warn if insufficient workers (#13899)
This should give an early warning if configured replication 
workers are insufficient to meet application workload.
2021-12-13 18:22:56 -08:00
Harshavardhana 109c927dad docs/debug: log any corruption and continue debug tool 2021-12-13 17:45:34 -08:00
Aditya Manthramurthy de400f3473
Allow setting non-existent policy on a user/group (#13898) 2021-12-13 15:55:52 -08:00
Harshavardhana 8144a125ce
check for update in background (#13889) 2021-12-13 09:43:03 -08:00
Minio Trusted 3e34e41a5a update helm to 3.3.4 2021-12-11 09:59:34 -08:00
Minio Trusted 2270887d43 update helm to 3.3.3 2021-12-11 09:28:02 -08:00
Minio Trusted c6431f9a04 update helm to 3.3.2
resolves #13887
2021-12-11 09:26:01 -08:00
jiangfucheng 88c0d0120c
update heal object unit test (#13886) 2021-12-11 09:04:07 -08:00
Aditya Manthramurthy 44fefe5b9f
Add option to policy info API to return create/mod timestamps (#13796)
- This introduces a new admin API with a query parameter (v=2) to return a
response with the timestamps

- Older API still works for compatibility/smooth transition in console
2021-12-11 09:03:39 -08:00
Minio Trusted 878d368cea Update yaml files to latest version RELEASE.2021-12-10T23-03-39Z 2021-12-11 00:02:59 -08:00
Aditya Manthramurthy f2bd026d0e
Allow OIDC user to query user info if policies permit (#13882) 2021-12-10 15:03:39 -08:00
Klaus Post 518612492c
xl-meta: Add header titles (#13880)
Add type for headers and create custom marshal to make 
it easier to read. Group headers and metadata.

Restore functionality that will read `xl.meta` in the current dir with no params.

Before:
```
{
  "Headers": [
    [
      "8M04bTiYRDmEMQGeAsk1yg==",
      1639150471630100400,
      "rLD1Rw==",
      1,
      6
    ],
  ],
    "Versions": [
    {
      "Type": 1,
      "V2Obj": {
        "CSumAlgo": 1,
        "DDir": "oC1Xpg4tRfW03g8o8w7Bzg==",
        "EcAlgo": 1,
        "EcBSize": 1048576,
        "EcDist": [
          7,
          8,
          1,
          2,
          3,
          4,
          5,
          6
        ],
        "EcIndex": 1,
        "EcM": 4,
        "EcN": 4,
        "ID": "8M04bTiYRDmEMQGeAsk1yg==",
        "MTime": 1639150471630100400,
        "MetaSys": {
          "x-minio-internal-inline-data": "dHJ1ZQ=="
        },
        "MetaUsr": {
          "content-type": "application/octet-stream",
          "etag": "b8252c86fad2d8937300aa92b467a3aa"
        },
        "PartASizes": [
          1000
        ],
        "PartETags": null,
        "PartNums": [
          1
        ],
        "PartSizes": [
          1000
        ],
        "Size": 1000
      }
    }
  ]
}
```

After:
```
{
  "Versions": [
    {
      "Header": {
        "Flags": 6,
        "ModTime": "2021-12-10T16:34:31.6301004+01:00",
        "Signature": "acb0f547",
        "Type": 1,
        "VersionID": "f0cd386d389844398431019e02c935ca"
      },
      "Idx": 0,
      "Metadata": {
        "Type": 1,
        "V2Obj": {
          "CSumAlgo": 1,
          "DDir": "oC1Xpg4tRfW03g8o8w7Bzg==",
          "EcAlgo": 1,
          "EcBSize": 1048576,
          "EcDist": [
            7,
            8,
            1,
            2,
            3,
            4,
            5,
            6
          ],
          "EcIndex": 1,
          "EcM": 4,
          "EcN": 4,
          "ID": "8M04bTiYRDmEMQGeAsk1yg==",
          "MTime": 1639150471630100400,
          "MetaSys": {
            "x-minio-internal-inline-data": "dHJ1ZQ=="
          },
          "MetaUsr": {
            "content-type": "application/octet-stream",
            "etag": "b8252c86fad2d8937300aa92b467a3aa"
          },
          "PartASizes": [
            1000
          ],
          "PartETags": null,
          "PartNums": [
            1
          ],
          "PartSizes": [
            1000
          ],
          "Size": 1000
        }
      }
    }
  ]
}
```
2021-12-10 15:03:25 -08:00
Klaus Post 81e43b87c2
Don't zero buffer if big enough (#13877)
Only append zeroed bytes when we don't have enough space anyway.
2021-12-10 13:08:10 -08:00
Aditya Manthramurthy a02e17f15c
Add tests to ensure that OIDC user can create IAM users (#13881) 2021-12-10 13:04:21 -08:00
Ravind Kumar c76f86fdbd
Clarify example for Standalone Docker instructions (#13879)
Closes #13868 

Also points users to the web documentation for docker installation.
2021-12-10 09:43:02 -08:00
Harshavardhana 5b7c00ff52
add more tests to cover areas for weird object names (#13873)
continuation of #13858 to add more tests and also validate the 
written object data.
2021-12-09 17:52:53 -08:00
Aditya Manthramurthy b9f0046ee7
Allow STS credentials to create users (#13874)
- allow any regular user to change their own password
- allow STS credentials to create users if permissions allow

Bonus: do not allow changes to sts/service account credentials (via add user API)
2021-12-09 17:48:51 -08:00
Harshavardhana 3b79f7e4ae
ignore if volume exists in MakeVolBulk, return other errors (#13866) 2021-12-09 15:55:42 -08:00
Aditya Manthramurthy 85d2df02b9
fix: user listing with LDAP (#13872)
Users listing was showing just a weird policy 
mapping output which does not make sense here.
2021-12-09 15:55:28 -08:00
Harshavardhana 2f1e8ba612
add more directory marker tests and fix a bug (#13871)
ListObjects() should never list a delete-marked folder
if latest is delete marker and delimiter is not provided.

ListObjectVersions() should list a delete-marked folder
even if latest is delete marker and delimiter is not
provided.

Enhance further versioning listing on the buckets
2021-12-09 14:59:23 -08:00
Anis Elleuch 84c690cb07
storage: Use request.Form and avoid mux matching (#13858)
request.Form uses less memory allocation and avoids gorilla mux matching
with weird characters in parameters such as '\n'

- Remove Queries() to avoid matching
- Ensure r.ParseForm is called to populate fields
- Add a unit test for object names with '\n'
2021-12-09 08:38:46 -08:00
Harshavardhana 239bbad7ab
add test to expect prefix without a directory object (#13865)
Motivation is to cover more areas
2021-12-09 08:36:54 -08:00
Minio Trusted 4be8023408 Update yaml files to latest version RELEASE.2021-12-09T06-19-41Z 2021-12-09 08:40:46 +00:00
Harshavardhana 83e8da57b8 update console to release v0.12.8 2021-12-08 22:19:41 -08:00
Harshavardhana dcff6c996d
fix: do not list delete-marked objects (#13864)
delete marked objects should not be considered
for listing when listing is delimited, this issue
as introduced in PR #13804 which was mainly to
address listing of directories in listing when
delimited.

This PR fixes this properly and adds tests to
ensure that we behave in accordance with how
an S3 API behaves for ListObjects() without
versions.
2021-12-08 17:34:52 -08:00
Poorna K 0a66a6f1e5
Avoid cache GC of writebacks before commit syncs (#13860)
Save part.1 for writebacks in a separate folder
and move it to cache dir atomically while saving
the cache metadata. This is to avoid GC mistaking
part.1 as orphaned cache entries and purging them.

This PR also fixes object size being overwritten during
retries for write-back mode.
2021-12-08 14:52:31 -08:00
Harshavardhana e82a5c5c54
fix: site replication issues and add tests (#13861)
- deleting policies was deleting all LDAP
  user mapping, this was a regression introduced
  in #13567

- deleting of policies is properly sent across
  all sites.

- remove unexpected errors instead embed the real
  errors as part of the 500 error response.
2021-12-08 11:50:15 -08:00
Harshavardhana 92fdcafb66
add verification tests for ETag on replicated content (#13857) 2021-12-07 10:08:26 -08:00
Harshavardhana b9aae1aaae
fix: speedtest should exit upon errors cleanly (#13851)
- deleteBucket() should be called for cleanup
  if client abruptly disconnects

- out of disk errors should be sent to client
  properly and also cancel the calls

- limit concurrency to available MAXPROCS not
  32 for auto-tuned setup, if procs are beyond
  32 then continue normally. this is to handle
  smaller setups.

fixes #13834
2021-12-06 16:36:14 -08:00
Harshavardhana 7d70afc937
fix: potential crash in diskCache when fileScorer is empty (#13850)
```
goroutine 115 [running]:
github.com/minio/minio/cmd.(*diskCache).purge.func3({0xc007a10a40, 0x40}, 0x40)
   github.com/minio/minio/cmd/disk-cache-backend.go:430 +0x90d
```
2021-12-06 15:55:29 -08:00
Aditya Manthramurthy 12b63061c2
Fix LDAP service account creation (#13849)
- when a user has only group permissions
- fixes regression from ac74237f0 (#13657)
- fixes https://github.com/minio/console/issues/1291
2021-12-06 15:55:11 -08:00
Klaus Post 038fdeea83
snowball: return errors on failures (#13836)
Return errors when untar fails at once.

Current error handling was quite a mess. Errors are written 
to the stream, but processing continues.

Instead, return errors when they occur and transform 
internal errors to bad request errors, since it is likely a 
problem with the input.

Fixes #13832
2021-12-06 09:45:23 -08:00
Anis Elleuch 0b6225bcc3
Better error msg when version mismatch of internode API (#13845)
Sometimes, we see an error message like "Server expects 'storage' API
version 'v41', instead found 'v41'" shows a more generic error message
with the path of the REST call.
2021-12-06 09:44:48 -08:00
Anis Elleuch f286ef8e17
isMultipart to test on parts sizes only if object is encrypted (#13839)
ObjectInfo.isMultipart() is testing if parts sizes are compatible with
encrypted parts but this only can be done if the object is encrypted.
2021-12-06 09:43:43 -08:00
Harshavardhana b120bcb60a
validate if cached value is empty before use (#13830)
fixes a crash reproduced while running hadoop tests

```
goroutine 201564 [running]:
github.com/minio/minio/cmd.metaCacheEntries.resolve({0xc0206ab7a0, 0x4, 0xc0015b1908}, 0xc0212a7040)
	github.com/minio/minio/cmd/metacache-entries.go:352 +0x58a
```

Bonus: HeadBucket() should always provide content-type
2021-12-06 02:59:51 -08:00
Harshavardhana be34fc9134
fix: kms-id header should have arn:aws:kms: prefix (#13833)
arn:aws:kms: is a must for KMS keyID.
2021-12-06 00:39:32 -08:00
Harshavardhana 8591d17d82
return appropriate errors upon parseErrors (#13831) 2021-12-05 11:36:26 -08:00
Harshavardhana f6190d6751
Add single drive support for directory prefixes in Listing (#13829)
This fixes the compatibility issue with Hadoop 3.3.1

fixes #13710
2021-12-03 18:08:40 -08:00
Harshavardhana f0fc77fded update CREDITS file with new deps 2021-12-03 13:24:49 -08:00
Klaus Post f56cac6381
jwt: Parse standard claims faster (#13821)
* Use structless/allocationless decoding for header (note "typ" isn't used)
* Create custom unmarshal code using jsonparser for StandardClaims.

Before/After:

```
BenchmarkParseJWTStandardClaims-32    	 4270724	       294.0 ns/op	     706 B/op	      16 allocs/op
BenchmarkParseJWTStandardClaims-32    	 5634847	       214.7 ns/op	     544 B/op	       9 allocs/op

BenchmarkParseJWTMapClaims-32    	 2763045	       428.6 ns/op	    1251 B/op	      29 allocs/op
BenchmarkParseJWTMapClaims-32    	 2839455	       410.9 ns/op	    1219 B/op	      26 allocs/op
```
2021-12-03 13:19:38 -08:00
Aditya Manthramurthy 4f35054d29
Ensure that role ARNs don't collide (#13817)
This is to prepare for multiple providers enhancement.
2021-12-03 13:15:56 -08:00
Shireesh Anjal d29df6714a
Introduce new config `subnet api_key` (#13793)
The earlier approach of using a license token for 
communicating with SUBNET is being replaced 
with a simpler mechanism of API keys. Unlike the 
license which is a JWT token, these API keys will 
be simple UUID tokens and don't have any embedded 
information in them. SUBNET would generate the 
API key on cluster registration, and then it would 
be saved in this config, to be used for subsequent 
communication with SUBNET.
2021-12-03 09:32:11 -08:00
jiangfucheng 7460fb8349
fix padding error and compatible with uploaded objects (#13803) 2021-12-03 09:26:30 -08:00