Commit Graph

9071 Commits

Author SHA1 Message Date
Minio Trusted f3602d7d08 Update yaml files to latest version RELEASE.2021-12-20T22-07-16Z 2021-12-20 23:13:48 +00:00
Aditya Manthramurthy 526e10a2e0
Fix regression in STS permissions via group in internal IDP (#13955)
- When using MinIO's internal IDP, STS credential permissions did not check the
groups of a user.

- Also fix bug in policy checking in AccountInfo call
2021-12-20 14:07:16 -08:00
Harshavardhana 0b21734571 update helm chart link 2021-12-20 13:20:59 -08:00
Harshavardhana 499872f31d
Add configurable channel queue_size for audit/logger webhook targets (#13819)
Also log all the missed events and logs instead of silently
swallowing the events.

Bonus: Extend the logger webhook to support mTLS
similar to audit webhook target.
2021-12-20 13:16:53 -08:00
Anis Elleuch 5cc16e098c
env: Remove quotes when parsing a config env file (#13953)
The code parsing the config environment file does not remove 
quotes of environment variables values. This commit adds this 
capability.
2021-12-20 13:13:06 -08:00
Minio Trusted 364e27d5f2 add support for etcd secrets and relabel configs
helm release v3.4.1
2021-12-20 13:11:50 -08:00
Aditya Manthramurthy 1f4e0bd17c
fix: access for root user's STS credential (#13947)
add a test to cover this case
2021-12-19 23:05:20 -08:00
Minio Trusted 0557e18472 update helm v3.4.0 2021-12-19 14:32:49 -08:00
Minio Trusted cfd66ab8c3 Update yaml files to latest version RELEASE.2021-12-18T04-42-33Z 2021-12-19 22:19:30 +00:00
Harshavardhana 691b763613 update console v0.12.9 2021-12-17 20:42:33 -08:00
Krishnan Parthasarathi 3ddb501190
Add docs for NewerNoncurrentVersions (#13944) 2021-12-17 19:32:29 -08:00
Aditya Manthramurthy 997e808088
fix; race in bucket replication stats (#13942)
- r.ulock was not locked when r.UsageCache was being modified

Bonus:

- simplify code by removing some unnecessary clone methods - we can 
do this because go arrays are values (not pointers/references) that are 
automatically copied on assignment.

- remove some unnecessary map allocation calls
2021-12-17 15:33:13 -08:00
Shireesh Anjal 13441ad0f8
Add IsKubernetes and IsDocker to health data (#13936) 2021-12-17 14:46:54 -08:00
Harshavardhana aa508591c1
cache only metrics served from the disks (#13940)
do not need to cache in-memory instant metrics
2021-12-17 11:40:09 -08:00
Harshavardhana 818f0201fc
re-implement prometheus metrics endpoint to be simpler (#13922)
data-structures were repeatedly initialized
this causes GC pressure, instead re-use the
collectors.

Initialize collectors in `init()`, also make
sure to honor the cache semantics for performance
requirements.

Avoid a global map and a global lock for metrics
lookup instead let them all be lock-free unless
the cache is being invalidated.
2021-12-17 10:11:04 -08:00
Aditya Manthramurthy 890f43ffa5
Map policy to parent for STS (#13884)
When STS credentials are created for a user, a unique (hopefully stable) parent
user value exists for the credential, which corresponds to the user for whom the
credentials are created. The access policy is mapped to this parent-user and is
persisted. This helps ensure that all STS credentials of a user have the same
policy assignment at all times.

Before this change, for an OIDC STS credential, when the policy claim changes in
the provider (when not using RoleARNs), the change would not take effect on
existing credentials, but only on new ones.

To support existing STS credentials without parent-user policy mappings, we
lookup the policy in the policy claim value. This behavior should be deprecated
when such support is no longer required, as it can still lead to stale
policy mappings.

Additionally this change also simplifies the implementation for all non-RoleARN
STS credentials. Specifically, for AssumeRole (internal IDP) STS credentials,
policies are picked up from the parent user's policies; for
AssumeRoleWithCertificate STS credentials, policies are picked up from the
parent user mapping created when the STS credential is generated.
AssumeRoleWithLDAP already picks up policies mapped to the virtual parent user.
2021-12-17 00:46:30 -08:00
Poorna K e270ab65b3
fix: healing of replication delete markers (#13933)
A corner case can occur where the delete-marker was propagated 
but the metadata could not be updated on the primary. Sending 
a RemoveObject call with the Delete marker version would end 
up permanently deleting the version on target. Instead, perform 
a Stat on the delete-marker version on target and redo replication 
only if the delete-marker is missing on target.
2021-12-16 15:34:55 -08:00
Anis Elleuch 926373f9c1
Run the data scanner routine in a loop (#13928)
After the introduction of Refresh logic in locks, the data scanner can
quit when the data scanner lock is not able to get refreshed. In that
case, the context of the data scanner will get canceled and
runDataScanner() will quit. Another server would pick the scanning
routine but after some time, all nodes can just have all scanning
routine aborted, as described above.

This fix will just run the data scanner in a loop.
2021-12-16 08:32:15 -08:00
Poorna K 111c6177d2
Deprecate caching for erasure/distributed mode (#13909)
Fixes: #13907

Also removing default value of `writethrough` for cache commit
which was interfering with cache_after setting
2021-12-15 16:48:34 -08:00
Klaus Post 91f72f25ab
select: Return early from bool AND, OR (#13914)
Return as soon as an AND fails and whenever an OR succeeds. Faster and more flexible.

For example makes `select * from S3object where _2 != '' AND _2 > 1` able to operate on empty fields.

Followup to #13900
2021-12-15 16:47:21 -08:00
Harshavardhana da540ccf8c
update selfupdate dependency to fix signature verification (#13919)
minisign v0.10.0 tool broke compatibility, that leads
to our library failing to parse the newer signatures.

This PR
fixes - https://github.com/minio/operator/issues/913
fixes - https://github.com/minio/minio/issues/13824

A workaround for users facing this problem is to unset

```
MINIO_UPDATE_MINISIGN_PUBKEY
```

or set it to `empty` string then signature verification
is skipped automatically.
2021-12-15 15:05:23 -08:00
Daryl Hughes d6396f82fe
support GCS gateway on vanilla helm chart (#13810)
These changes have been migrated from the previous chart: https://github.com/helm/charts/tree/master/stable/minio

Added `GCS` support for gateway mode in the helm chart.

Added a new GCS block under the gateway key to the house 
the GCS-specific variables.

The gateway-deployment template now sets the env var: GOOGLE_APPLICATION_CREDENTIALS as a path to the 
service-account-file.json

The service-account-file.json can be added to the MinIO 
the secret if an existingSecret is not specified.
2021-12-15 12:30:13 -08:00
Harshavardhana 4fa250a6a1
add hash-set debugging currently supports SIPMOD (#13911)
fixes a bug in s3-check-md5 for single part
uploaded multipart objects.
2021-12-15 12:07:15 -08:00
Poorna K b42cfcea60
Disallow versioning/replication change in cluster replication setup (#13910) 2021-12-15 10:37:08 -08:00
Klaus Post aca6dfbd60
Check for nil RPC in listing (#13917)
Fixes #13915
2021-12-15 09:19:11 -08:00
Harshavardhana 5f7e6d03ff
copy bucket slice to avoid skipping .minio.sys/buckets (#13912)
healing was skipping `.minio.sys/buckets` path so
essentially not healing `.usage.json` - fix this
by making a copy of `buckets` slice.
2021-12-15 09:18:09 -08:00
Harshavardhana 88ad742da0
fix: error handling cases in site-replication (#13901)
- Allow proper SRError to be propagated to
  handlers and converted appropriately.

- Make sure to enable object locking on buckets
  when requested in MakeBucketHook.

- When DNSConfig is enabled attempt to delete it
  first before deleting buckets locally.
2021-12-14 14:09:57 -08:00
Klaus Post a8d4042853
select: Add IS (NOT) operators (#13906)
Add `IS` and `IS NOT` as comparison operators.

This may be a bit wider than the S3 spec, but we can rather 
easily remove the forwarding.
2021-12-14 09:54:50 -08:00
Krishnan Parthasarathi 44a9339c0a
Newer noncurrent versions (#13815)
- Rename MaxNoncurrentVersions tag to NewerNoncurrentVersions

Note: We apply overlapping NewerNoncurrentVersions rules such that 
we honor the highest among applicable limits. e.g if 2 overlapping rules 
are configured with 2 and 3 noncurrent versions to be retained, we 
will retain 3.

- Expire newer noncurrent versions after noncurrent days
- MinIO extension: allow noncurrent days to be zero, allowing expiry 
  of noncurrent version as soon as more than configured 
  NewerNoncurrentVersions are present.
- Allow NewerNoncurrentVersions rules on object-locked buckets
- No x-amz-expiration when NewerNoncurrentVersions configured
- ComputeAction should skip rules with NewerNoncurrentVersions > 0
- Add unit tests for lifecycle.ComputeAction
- Support lifecycle rules with MaxNoncurrentVersions
- Extend ExpectedExpiryTime to work with zero days
- Fix all-time comparisons to be relative to UTC
2021-12-14 09:41:44 -08:00
Harshavardhana 113c7ff49a
add code to parse secrets natively instead of shell scripts (#13883) 2021-12-13 18:23:31 -08:00
fpaupier 40dbe243d9
update: keycloak binding documentation (#13894) 2021-12-13 18:23:16 -08:00
Poorna K d422d24278
replication: warn if insufficient workers (#13899)
This should give an early warning if configured replication 
workers are insufficient to meet application workload.
2021-12-13 18:22:56 -08:00
Harshavardhana 109c927dad docs/debug: log any corruption and continue debug tool 2021-12-13 17:45:34 -08:00
Aditya Manthramurthy de400f3473
Allow setting non-existent policy on a user/group (#13898) 2021-12-13 15:55:52 -08:00
Harshavardhana 8144a125ce
check for update in background (#13889) 2021-12-13 09:43:03 -08:00
Minio Trusted 3e34e41a5a update helm to 3.3.4 2021-12-11 09:59:34 -08:00
Minio Trusted 2270887d43 update helm to 3.3.3 2021-12-11 09:28:02 -08:00
Minio Trusted c6431f9a04 update helm to 3.3.2
resolves #13887
2021-12-11 09:26:01 -08:00
jiangfucheng 88c0d0120c
update heal object unit test (#13886) 2021-12-11 09:04:07 -08:00
Aditya Manthramurthy 44fefe5b9f
Add option to policy info API to return create/mod timestamps (#13796)
- This introduces a new admin API with a query parameter (v=2) to return a
response with the timestamps

- Older API still works for compatibility/smooth transition in console
2021-12-11 09:03:39 -08:00
Minio Trusted 878d368cea Update yaml files to latest version RELEASE.2021-12-10T23-03-39Z 2021-12-11 00:02:59 -08:00
Aditya Manthramurthy f2bd026d0e
Allow OIDC user to query user info if policies permit (#13882) 2021-12-10 15:03:39 -08:00
Klaus Post 518612492c
xl-meta: Add header titles (#13880)
Add type for headers and create custom marshal to make 
it easier to read. Group headers and metadata.

Restore functionality that will read `xl.meta` in the current dir with no params.

Before:
```
{
  "Headers": [
    [
      "8M04bTiYRDmEMQGeAsk1yg==",
      1639150471630100400,
      "rLD1Rw==",
      1,
      6
    ],
  ],
    "Versions": [
    {
      "Type": 1,
      "V2Obj": {
        "CSumAlgo": 1,
        "DDir": "oC1Xpg4tRfW03g8o8w7Bzg==",
        "EcAlgo": 1,
        "EcBSize": 1048576,
        "EcDist": [
          7,
          8,
          1,
          2,
          3,
          4,
          5,
          6
        ],
        "EcIndex": 1,
        "EcM": 4,
        "EcN": 4,
        "ID": "8M04bTiYRDmEMQGeAsk1yg==",
        "MTime": 1639150471630100400,
        "MetaSys": {
          "x-minio-internal-inline-data": "dHJ1ZQ=="
        },
        "MetaUsr": {
          "content-type": "application/octet-stream",
          "etag": "b8252c86fad2d8937300aa92b467a3aa"
        },
        "PartASizes": [
          1000
        ],
        "PartETags": null,
        "PartNums": [
          1
        ],
        "PartSizes": [
          1000
        ],
        "Size": 1000
      }
    }
  ]
}
```

After:
```
{
  "Versions": [
    {
      "Header": {
        "Flags": 6,
        "ModTime": "2021-12-10T16:34:31.6301004+01:00",
        "Signature": "acb0f547",
        "Type": 1,
        "VersionID": "f0cd386d389844398431019e02c935ca"
      },
      "Idx": 0,
      "Metadata": {
        "Type": 1,
        "V2Obj": {
          "CSumAlgo": 1,
          "DDir": "oC1Xpg4tRfW03g8o8w7Bzg==",
          "EcAlgo": 1,
          "EcBSize": 1048576,
          "EcDist": [
            7,
            8,
            1,
            2,
            3,
            4,
            5,
            6
          ],
          "EcIndex": 1,
          "EcM": 4,
          "EcN": 4,
          "ID": "8M04bTiYRDmEMQGeAsk1yg==",
          "MTime": 1639150471630100400,
          "MetaSys": {
            "x-minio-internal-inline-data": "dHJ1ZQ=="
          },
          "MetaUsr": {
            "content-type": "application/octet-stream",
            "etag": "b8252c86fad2d8937300aa92b467a3aa"
          },
          "PartASizes": [
            1000
          ],
          "PartETags": null,
          "PartNums": [
            1
          ],
          "PartSizes": [
            1000
          ],
          "Size": 1000
        }
      }
    }
  ]
}
```
2021-12-10 15:03:25 -08:00
Klaus Post 81e43b87c2
Don't zero buffer if big enough (#13877)
Only append zeroed bytes when we don't have enough space anyway.
2021-12-10 13:08:10 -08:00
Aditya Manthramurthy a02e17f15c
Add tests to ensure that OIDC user can create IAM users (#13881) 2021-12-10 13:04:21 -08:00
Ravind Kumar c76f86fdbd
Clarify example for Standalone Docker instructions (#13879)
Closes #13868 

Also points users to the web documentation for docker installation.
2021-12-10 09:43:02 -08:00
Harshavardhana 5b7c00ff52
add more tests to cover areas for weird object names (#13873)
continuation of #13858 to add more tests and also validate the 
written object data.
2021-12-09 17:52:53 -08:00
Aditya Manthramurthy b9f0046ee7
Allow STS credentials to create users (#13874)
- allow any regular user to change their own password
- allow STS credentials to create users if permissions allow

Bonus: do not allow changes to sts/service account credentials (via add user API)
2021-12-09 17:48:51 -08:00
Harshavardhana 3b79f7e4ae
ignore if volume exists in MakeVolBulk, return other errors (#13866) 2021-12-09 15:55:42 -08:00
Aditya Manthramurthy 85d2df02b9
fix: user listing with LDAP (#13872)
Users listing was showing just a weird policy 
mapping output which does not make sense here.
2021-12-09 15:55:28 -08:00