Commit Graph

60 Commits

Author SHA1 Message Date
Klaus Post
470553ff5d
Tweak readall allocation and renameData buffer reuse (#13108)
Use a single allocation for reading the file, not the growing buffer of `io.ReadAll`.

Reuse the write buffer if we can when writing metadata in RenameData.
2021-08-30 08:38:11 -07:00
Anis Elleuch
e05886561d
lock: Fix Refresh logic with multi resources lock (#13092)
A multi resources lock is a single lock UID with multiple associated
resources. This is created for example by multi objects delete
operation. This commit changes the behavior of Refresh() to iterate over
all locks having the same UID and refresh them.

Bonus: Fix showing top locks for multi delete objects
2021-08-27 13:07:55 -07:00
Anis Elleuch
06b71c99ee
locks: Ensure local lock removal after a failed refresh (#12979)
In the event when a lock is not refreshed in the cluster, this latter
will be automatically removed in the subsequent cleanup of non 
refreshed locks routine, but it forgot to clean the local server, 
hence having the same weird stale locks present.

This commit will remove the lock locally also in remote nodes, if
removing a lock from a remote node will fail, it will be anyway 
removed later in the locks cleanup routine.
2021-08-27 08:59:36 -07:00
Harshavardhana
c11a2ac396
refactor healing to remove certain structs (#13079)
- remove sourceCh usage from healing
  we already have tasks and resp channel

- use read locks to lookup globalHealConfig

- fix healing resolver to pick candidates quickly
  that need healing, without this resolver was
  unexpectedly skipping.
2021-08-26 14:06:04 -07:00
Harshavardhana
0559f46bbb
fix: make healObject() make non-blocking (#13071)
healObject() should be non-blocking to ensure
that scanner is not blocked for a long time,
this adversely affects performance of the scanner
and also affects the way usage is updated
subsequently.

This PR allows for a non-blocking behavior for
healing, dropping operations that cannot be queued
anymore.
2021-08-25 17:46:20 -07:00
Klaus Post
8315bcd0d8
Fix TrafficMeter data race (#13041)
When reading `TrafficMeter` values, there was a value receiver.

This means that receivers are copied unsafely when invoked.

Fixes race seen with `-race` build.
2021-08-23 09:19:14 -07:00
Harshavardhana
202d0b64eb
fix: enable go1.17 github ci/cd (#12997) 2021-08-18 18:35:22 -07:00
Harshavardhana
a690772cc5
add support to set subnet license for embedded console (#12993) 2021-08-17 11:56:01 -07:00
Aditya Manthramurthy
2ca5ee026d
Remove default value of STSExpiry for LDAP (#12985)
This ensures that the deprecation warning is shown when the setting is actually
used in a configuration - instead of showing up whenever LDAP is enabled.
2021-08-17 02:25:05 -07:00
Anis Elleuch
47dfc1b1b0
ldap: Reevalute filter when searching for non eligible users (#12953)
The previous code removes SVC/STS accounts for ldap users that do not
exist anymore in LDAP server. This commit will actually re-evaluate
filter as well if it is changed and remove all local SVC/STS accounts
beloning to the ldap user if the latter is not eligible for the
search filter anymore.

For example: the filter selects enabled users among other criteras in
the LDAP database, if one ldap user changes his status to disabled
later, then associated SVC/STS accounts will be removed because that user
does not meet the filter search anymore.
2021-08-13 11:40:04 -07:00
Klaus Post
f31a00de01
fix: http stats race in traffic metering (#12956)
Traffic metering was not protected against concurrent updates.

```
WARNING: DATA RACE
Read at 0x00c02b0dace8 by goroutine 235:
  github.com/minio/minio/cmd.setHTTPStatsHandler.func1()
      d:/minio/minio/cmd/generic-handlers.go:360 +0x27d
  net/http.HandlerFunc.ServeHTTP()
...

Previous write at 0x00c02b0dace8 by goroutine 994:
  github.com/minio/minio/internal/http/stats.(*IncomingTrafficMeter).Read()
      d:/minio/minio/internal/http/stats/http-traffic-recorder.go:34 +0xd2

```
2021-08-13 07:30:03 -07:00
Harshavardhana
8f2a3efa85
disallow sub-credentials based on root credentials to gain priviledges (#12947)
This happens because of a change added where any sub-credential
with parentUser == rootCredential i.e (MINIO_ROOT_USER) will
always be an owner, you cannot generate credentials with lower
session policy to restrict their access.

This doesn't affect user service accounts created with regular
users, LDAP or OpenID
2021-08-12 18:07:08 -07:00
Ricardo Katz
a526ad2e80
Add headers into AMQP notifications (#12911)
Signed-off-by: Ricardo Katz <rkatz@vmware.com>
2021-08-11 22:24:19 -07:00
Krishnan Parthasarathi
65b6f4aa31
Add dynamic reconfiguration of number of transition workers (#12926) 2021-08-11 22:23:56 -07:00
Harshavardhana
3becee9e5d
use sync map instead of local DNS cache (#12925)
also enable PreferGo only resolver for FIPS builds
2021-08-10 21:20:09 -07:00
Anis Elleuch
37bef900fd
ilm: More compliance to spec related to expired delete markers (#12887)
<Days> tag can remove expired delete markers according to AWS S3 spec.

<NoncurrentVersionExpiration> cannot remove expired delete markers in
anyway.
2021-08-05 11:21:21 -07:00
Harshavardhana
ea64a9263c
add deprecation notice for LDAP username format (#12849) 2021-08-02 18:20:06 -07:00
Poorna Krishnamoorthy
b6cd54779c
Increase context timeout for bandwidth throttled reader (#12820)
increase default timeout up to one hour for toy setups.

fixes #12812
2021-07-28 15:20:01 -07:00
Harshavardhana
bfbdb8f0a8
fix: incorrect O_DIRECT behavior for reads (#12811)
O_DIRECT behavior was broken and it was still
caching all the reads, this change properly fixes
this behavior.
2021-07-28 11:20:16 -07:00
Harshavardhana
a78bc7bfdb rename all access token to id_tokens 2021-07-27 22:38:12 -07:00
Aditya Manthramurthy
de00b641da
[LDAP] Support syncing user-group memberships with LDAP service (#12785)
When configured in Lookup Bind mode, the server now periodically queries the
LDAP IDP service to find changes to a user's group memberships, and saves this
info to update the access policies for all temporary and service account
credentials belonging to LDAP users.
2021-07-24 11:57:36 -07:00
Aditya Manthramurthy
9a31030e74
DEPRECATION Warning: Avoid configuring default LDAP STS Expiry (#12781)
- Show notice when `MINIO_IDENTITY_LDAP_STS_EXPIRY` or the 
  corresponding to the configuration option is used at server startup.
- Once support is removed, the default will be fixed at 1 hour.
- Users may specify expiry directly in the STS API.
- Update docs and help message
- Adds example in ldap.go to configure expiry in STS API.
2021-07-22 16:43:57 -07:00
Aditya Manthramurthy
a3079a7de2
fix: Add support for DurationSeconds in LDAP STS API (#12778) 2021-07-22 12:13:21 -07:00
Krishnan Parthasarathi
209e6d00c6
Use ObjectInfo.ToLifecycleOpts instead of literal values (#12772)
Promote getLifecycleTransitionTier to a method on lifecycle.Lifecycle.
2021-07-21 19:12:44 -07:00
Harshavardhana
320e1533c4
use expected MinIO URLs for console (#12770)
when TLS is configured using IPs directly
might interfere and not work properly when
the server is configured with TLS certs but
the certs only have domain certs.

Also additionally allow users to specify
a public accessible URL for console to talk
to MinIO i.e `MINIO_SERVER_URL` this would
allow them to use an external ingress domain
to talk to MinIO. This internally fixes few
problems such as presigned URL generation on
the console UI etc.

This needs to be done additionally for any
MinIO deployments that might have a much more
stricter requirement when running in standalone
mode such as FS or standalone erasure code.
2021-07-21 14:51:16 -07:00
Aditya Manthramurthy
0db1c94e7d
[IDP:LDAP] Cleanup creds for removed LDAP user entries (#12759) 2021-07-20 23:33:12 -07:00
Krishnan Parthasarathi
d0963974a5
pkg/lifecycle: Add SetPredictionHeaders method (#12755)
This method is used to add expected expiration and transition time 
for an object in GET/HEAD Object response headers.

Also fixed bugs in lifecycle.PredictTransitionTime and
getLifecycleTransitionTier in handling current and 
non-current versions.
2021-07-20 17:36:55 -07:00
Harshavardhana
10fc30a989
fix crash when config is not properly initialized (#12714)
fixes #12709
2021-07-14 18:27:25 -07:00
Shireesh Anjal
4d6d4244f1
Anonymize hosts in subnet health report (#12627)
Ensure that hostnames / ip addresses are not printed in the subnet
health report. Anonymize them by replacing them with `servern` where `n`
represents the position of the server in the pool.

This is done by building a `host anonymizer` map that maps every
possible value containing the host e.g. host, host:port,
http://host:port, etc to the corresponding anonymized name and using
this map to replace the values at the time of health report generation.

A different logic is used to anonymize host names in the `procinfo`
data, as the host names are part of an ellipses pattern in the process
start command. Here we just replace the prefix/suffix of the ellipses
pattern with their hashes.
2021-07-14 00:23:22 -07:00
Harshavardhana
8d19efe7e0
feat: increase allowed maximum STS expiration timeout to 365 days (#12704) 2021-07-14 00:08:22 -07:00
Harshavardhana
e316873f84
feat: Add support for kakfa audit logger target (#12678) 2021-07-13 09:39:13 -07:00
Harshavardhana
da0fd5f056
fix: allow customizing console redirection (#12665)
MinIO might be running inside proxies, and
console while being on another port might not be
reachable on a specific port behind such proxies.

For such scenarios customize the redirect URL
such that console can be redirected to correct
proxy endpoint instead.

fixes #12661
2021-07-09 14:27:09 -07:00
Harshavardhana
bab72f6887
fix: allow authToken for webhook to support Splunk (#12663) 2021-07-09 11:47:04 -07:00
Harshavardhana
28adb29db3
feat: Add support to poll users on external SSO (#12592)
Additional support for vendor-specific admin API
integrations for OpenID, to ensure validity of
credentials on MinIO.

Every 5minutes check for validity of credentials
on MinIO with vendor specific IDP.
2021-07-09 11:17:21 -07:00
Harshavardhana
c99d399d09
fix: handle redirects for specific resources (#12629) 2021-07-07 12:04:16 -07:00
Harshavardhana
039978640f
fix: honor system umask for file creates (#12601)
use 0666 os.FileMode to honor system umask
2021-07-06 12:54:16 -07:00
Klaus Post
05aebc52c2
feat: Implement listing version 3.0 (#12605)
Co-authored-by: Harshavardhana <harsha@minio.io>
2021-07-05 15:34:41 -07:00
Lenin Alevski
0ad03908d0
LDAP Test connection was not using GlobalRootCAs (#12612)
Signed-off-by: Lenin Alevski <alevsk.8772@gmail.com>
2021-07-01 17:41:01 -07:00
Harshavardhana
4f6c74a257
simplify audit logging for replication and ILM (#12610)
auditLog should be attempted right before the
return of the function and not multiple times
per function, this ensures that we only trigger
it once per function call.
2021-07-01 14:02:44 -07:00
Harshavardhana
4781e7580b
add support for customizing redirect_uri for IDP (#12607) 2021-06-30 16:08:20 -07:00
Poorna Krishnamoorthy
c20fbcd842
fix:(replication/bandwidth) add missing validation in monitored reader (#12582) 2021-06-27 10:11:13 -07:00
Poorna Krishnamoorthy
d00783c923
Use rate.Limiter for bandwidth monitoring (#12506)
Bonus: fixes a hang when bandwidth caps are enabled for
synchronous replication
2021-06-24 18:29:30 -07:00
Harshavardhana
8d1bc65757
allow resetting and reapply config on broken clusters (#12554)
Bonus: remove kms_kes as sub-system, since its ENV only.
- also fixes a crash with etcd cluster without KMS
configured and also if KMS decryption is missing.
2021-06-24 16:24:12 -07:00
Emir İşman
9bf1ac0bb6
[deps]: update jwt-go dependency (#12544)
jwt-go has been renamed to jwt and has a new home. See
https://github.com/dgrijalva/jwt-go/issues/462
2021-06-24 08:41:04 -07:00
Harshavardhana
8f1fe3b761
fix: --console-address when specified endpoints missing (#12534)
Additionally upgrade console dependency for reading
environment variables properly.
2021-06-20 23:04:47 -07:00
Harshavardhana
cdeccb5510
feat: Deprecate embedded browser and import console (#12460)
This feature also changes the default port where
the browser is running, now the port has moved
to 9001 and it can be configured with

```
--console-address ":9001"
```
2021-06-17 20:27:04 -07:00
Anis Elleuch
7722b91e1d
s3: Force a prefix removal using a special header (#12504)
An S3 client can send `x-minio-force-delete: true` to remove a prefix.
2021-06-15 18:43:14 -07:00
Harshavardhana
da74e2f167
move internal/net to pkg/net package (#12505) 2021-06-14 14:54:37 -07:00
Harshavardhana
ed6cc66cf4
fix: remove unnecessary limit for queueStore (#12491)
There is no good reason to limit ourselves
to max_open_fd for queue_store

Bonus: Support for publisher confirms
2021-06-14 13:28:44 -07:00
ebozduman
b154581b65
fix: partially defined cred env vars cause "minio gateway s3" to fail (#12228)
Both credential env vars not needed to start s3 gateway
2021-06-10 22:28:09 -07:00