Commit Graph

454 Commits

Author SHA1 Message Date
Harshavardhana
202d0b64eb
fix: enable go1.17 github ci/cd (#12997) 2021-08-18 18:35:22 -07:00
Harshavardhana
a690772cc5
add support to set subnet license for embedded console (#12993) 2021-08-17 11:56:01 -07:00
Aditya Manthramurthy
2ca5ee026d
Remove default value of STSExpiry for LDAP (#12985)
This ensures that the deprecation warning is shown when the setting is actually
used in a configuration - instead of showing up whenever LDAP is enabled.
2021-08-17 02:25:05 -07:00
Anis Elleuch
47dfc1b1b0
ldap: Reevalute filter when searching for non eligible users (#12953)
The previous code removes SVC/STS accounts for ldap users that do not
exist anymore in LDAP server. This commit will actually re-evaluate
filter as well if it is changed and remove all local SVC/STS accounts
beloning to the ldap user if the latter is not eligible for the
search filter anymore.

For example: the filter selects enabled users among other criteras in
the LDAP database, if one ldap user changes his status to disabled
later, then associated SVC/STS accounts will be removed because that user
does not meet the filter search anymore.
2021-08-13 11:40:04 -07:00
Klaus Post
f31a00de01
fix: http stats race in traffic metering (#12956)
Traffic metering was not protected against concurrent updates.

```
WARNING: DATA RACE
Read at 0x00c02b0dace8 by goroutine 235:
  github.com/minio/minio/cmd.setHTTPStatsHandler.func1()
      d:/minio/minio/cmd/generic-handlers.go:360 +0x27d
  net/http.HandlerFunc.ServeHTTP()
...

Previous write at 0x00c02b0dace8 by goroutine 994:
  github.com/minio/minio/internal/http/stats.(*IncomingTrafficMeter).Read()
      d:/minio/minio/internal/http/stats/http-traffic-recorder.go:34 +0xd2

```
2021-08-13 07:30:03 -07:00
Harshavardhana
8f2a3efa85
disallow sub-credentials based on root credentials to gain priviledges (#12947)
This happens because of a change added where any sub-credential
with parentUser == rootCredential i.e (MINIO_ROOT_USER) will
always be an owner, you cannot generate credentials with lower
session policy to restrict their access.

This doesn't affect user service accounts created with regular
users, LDAP or OpenID
2021-08-12 18:07:08 -07:00
Ricardo Katz
a526ad2e80
Add headers into AMQP notifications (#12911)
Signed-off-by: Ricardo Katz <rkatz@vmware.com>
2021-08-11 22:24:19 -07:00
Krishnan Parthasarathi
65b6f4aa31
Add dynamic reconfiguration of number of transition workers (#12926) 2021-08-11 22:23:56 -07:00
Harshavardhana
3becee9e5d
use sync map instead of local DNS cache (#12925)
also enable PreferGo only resolver for FIPS builds
2021-08-10 21:20:09 -07:00
Anis Elleuch
37bef900fd
ilm: More compliance to spec related to expired delete markers (#12887)
<Days> tag can remove expired delete markers according to AWS S3 spec.

<NoncurrentVersionExpiration> cannot remove expired delete markers in
anyway.
2021-08-05 11:21:21 -07:00
Harshavardhana
ea64a9263c
add deprecation notice for LDAP username format (#12849) 2021-08-02 18:20:06 -07:00
Poorna Krishnamoorthy
b6cd54779c
Increase context timeout for bandwidth throttled reader (#12820)
increase default timeout up to one hour for toy setups.

fixes #12812
2021-07-28 15:20:01 -07:00
Harshavardhana
bfbdb8f0a8
fix: incorrect O_DIRECT behavior for reads (#12811)
O_DIRECT behavior was broken and it was still
caching all the reads, this change properly fixes
this behavior.
2021-07-28 11:20:16 -07:00
Harshavardhana
a78bc7bfdb rename all access token to id_tokens 2021-07-27 22:38:12 -07:00
Aditya Manthramurthy
de00b641da
[LDAP] Support syncing user-group memberships with LDAP service (#12785)
When configured in Lookup Bind mode, the server now periodically queries the
LDAP IDP service to find changes to a user's group memberships, and saves this
info to update the access policies for all temporary and service account
credentials belonging to LDAP users.
2021-07-24 11:57:36 -07:00
Aditya Manthramurthy
9a31030e74
DEPRECATION Warning: Avoid configuring default LDAP STS Expiry (#12781)
- Show notice when `MINIO_IDENTITY_LDAP_STS_EXPIRY` or the 
  corresponding to the configuration option is used at server startup.
- Once support is removed, the default will be fixed at 1 hour.
- Users may specify expiry directly in the STS API.
- Update docs and help message
- Adds example in ldap.go to configure expiry in STS API.
2021-07-22 16:43:57 -07:00
Aditya Manthramurthy
a3079a7de2
fix: Add support for DurationSeconds in LDAP STS API (#12778) 2021-07-22 12:13:21 -07:00
Krishnan Parthasarathi
209e6d00c6
Use ObjectInfo.ToLifecycleOpts instead of literal values (#12772)
Promote getLifecycleTransitionTier to a method on lifecycle.Lifecycle.
2021-07-21 19:12:44 -07:00
Harshavardhana
320e1533c4
use expected MinIO URLs for console (#12770)
when TLS is configured using IPs directly
might interfere and not work properly when
the server is configured with TLS certs but
the certs only have domain certs.

Also additionally allow users to specify
a public accessible URL for console to talk
to MinIO i.e `MINIO_SERVER_URL` this would
allow them to use an external ingress domain
to talk to MinIO. This internally fixes few
problems such as presigned URL generation on
the console UI etc.

This needs to be done additionally for any
MinIO deployments that might have a much more
stricter requirement when running in standalone
mode such as FS or standalone erasure code.
2021-07-21 14:51:16 -07:00
Aditya Manthramurthy
0db1c94e7d
[IDP:LDAP] Cleanup creds for removed LDAP user entries (#12759) 2021-07-20 23:33:12 -07:00
Krishnan Parthasarathi
d0963974a5
pkg/lifecycle: Add SetPredictionHeaders method (#12755)
This method is used to add expected expiration and transition time 
for an object in GET/HEAD Object response headers.

Also fixed bugs in lifecycle.PredictTransitionTime and
getLifecycleTransitionTier in handling current and 
non-current versions.
2021-07-20 17:36:55 -07:00
Harshavardhana
10fc30a989
fix crash when config is not properly initialized (#12714)
fixes #12709
2021-07-14 18:27:25 -07:00
Shireesh Anjal
4d6d4244f1
Anonymize hosts in subnet health report (#12627)
Ensure that hostnames / ip addresses are not printed in the subnet
health report. Anonymize them by replacing them with `servern` where `n`
represents the position of the server in the pool.

This is done by building a `host anonymizer` map that maps every
possible value containing the host e.g. host, host:port,
http://host:port, etc to the corresponding anonymized name and using
this map to replace the values at the time of health report generation.

A different logic is used to anonymize host names in the `procinfo`
data, as the host names are part of an ellipses pattern in the process
start command. Here we just replace the prefix/suffix of the ellipses
pattern with their hashes.
2021-07-14 00:23:22 -07:00
Harshavardhana
8d19efe7e0
feat: increase allowed maximum STS expiration timeout to 365 days (#12704) 2021-07-14 00:08:22 -07:00
Harshavardhana
e316873f84
feat: Add support for kakfa audit logger target (#12678) 2021-07-13 09:39:13 -07:00
Harshavardhana
da0fd5f056
fix: allow customizing console redirection (#12665)
MinIO might be running inside proxies, and
console while being on another port might not be
reachable on a specific port behind such proxies.

For such scenarios customize the redirect URL
such that console can be redirected to correct
proxy endpoint instead.

fixes #12661
2021-07-09 14:27:09 -07:00
Harshavardhana
bab72f6887
fix: allow authToken for webhook to support Splunk (#12663) 2021-07-09 11:47:04 -07:00
Harshavardhana
28adb29db3
feat: Add support to poll users on external SSO (#12592)
Additional support for vendor-specific admin API
integrations for OpenID, to ensure validity of
credentials on MinIO.

Every 5minutes check for validity of credentials
on MinIO with vendor specific IDP.
2021-07-09 11:17:21 -07:00
Harshavardhana
c99d399d09
fix: handle redirects for specific resources (#12629) 2021-07-07 12:04:16 -07:00
Harshavardhana
039978640f
fix: honor system umask for file creates (#12601)
use 0666 os.FileMode to honor system umask
2021-07-06 12:54:16 -07:00
Klaus Post
05aebc52c2
feat: Implement listing version 3.0 (#12605)
Co-authored-by: Harshavardhana <harsha@minio.io>
2021-07-05 15:34:41 -07:00
Lenin Alevski
0ad03908d0
LDAP Test connection was not using GlobalRootCAs (#12612)
Signed-off-by: Lenin Alevski <alevsk.8772@gmail.com>
2021-07-01 17:41:01 -07:00
Harshavardhana
4f6c74a257
simplify audit logging for replication and ILM (#12610)
auditLog should be attempted right before the
return of the function and not multiple times
per function, this ensures that we only trigger
it once per function call.
2021-07-01 14:02:44 -07:00
Harshavardhana
4781e7580b
add support for customizing redirect_uri for IDP (#12607) 2021-06-30 16:08:20 -07:00
Poorna Krishnamoorthy
c20fbcd842
fix:(replication/bandwidth) add missing validation in monitored reader (#12582) 2021-06-27 10:11:13 -07:00
Poorna Krishnamoorthy
d00783c923
Use rate.Limiter for bandwidth monitoring (#12506)
Bonus: fixes a hang when bandwidth caps are enabled for
synchronous replication
2021-06-24 18:29:30 -07:00
Harshavardhana
8d1bc65757
allow resetting and reapply config on broken clusters (#12554)
Bonus: remove kms_kes as sub-system, since its ENV only.
- also fixes a crash with etcd cluster without KMS
configured and also if KMS decryption is missing.
2021-06-24 16:24:12 -07:00
Emir İşman
9bf1ac0bb6
[deps]: update jwt-go dependency (#12544)
jwt-go has been renamed to jwt and has a new home. See
https://github.com/dgrijalva/jwt-go/issues/462
2021-06-24 08:41:04 -07:00
Harshavardhana
8f1fe3b761
fix: --console-address when specified endpoints missing (#12534)
Additionally upgrade console dependency for reading
environment variables properly.
2021-06-20 23:04:47 -07:00
Harshavardhana
cdeccb5510
feat: Deprecate embedded browser and import console (#12460)
This feature also changes the default port where
the browser is running, now the port has moved
to 9001 and it can be configured with

```
--console-address ":9001"
```
2021-06-17 20:27:04 -07:00
Anis Elleuch
7722b91e1d
s3: Force a prefix removal using a special header (#12504)
An S3 client can send `x-minio-force-delete: true` to remove a prefix.
2021-06-15 18:43:14 -07:00
Harshavardhana
da74e2f167
move internal/net to pkg/net package (#12505) 2021-06-14 14:54:37 -07:00
Harshavardhana
ed6cc66cf4
fix: remove unnecessary limit for queueStore (#12491)
There is no good reason to limit ourselves
to max_open_fd for queue_store

Bonus: Support for publisher confirms
2021-06-14 13:28:44 -07:00
ebozduman
b154581b65
fix: partially defined cred env vars cause "minio gateway s3" to fail (#12228)
Both credential env vars not needed to start s3 gateway
2021-06-10 22:28:09 -07:00
Harshavardhana
66d549c05d
remove support for deprecated MINIO_KMS_MASTER_KEY (#12463) 2021-06-08 18:50:14 -07:00
Anis Elleuch
6c8be64cdb
rest: healthcheck should not update failure metrics (#12458)
Otherwise, we can see high numbers of networking issues when a node is
down.
2021-06-08 14:09:26 -07:00
Shireesh Anjal
000656ac6b
Do not send 'sensitive' flag to client (#12461)
As it is server specific and is not required on client side.
2021-06-07 12:43:21 -07:00
Harshavardhana
c0e41356f5
add error level logger similar to Info() for console (#12445) 2021-06-04 11:11:30 -07:00
Shireesh Anjal
fb140c146b
Redact sensitive values from config in health data (#12421)
The health api returns the server configuration details. Redact
sensitive values from the config values like URLs and credentials.
2021-06-03 08:15:44 -07:00
Harshavardhana
7a3b5235bf remove deprecated kms_vault unused key name 2021-06-03 00:10:11 -07:00
Poorna Krishnamoorthy
dbea8d2ee0
Add support for existing object replication. (#12109)
Also adding an API to allow resyncing replication when
existing object replication is enabled and the remote target
is entirely lost. With the `mc replicate reset` command, the
objects that are eligible for replication as per the replication
config will be resynced to target if existing object replication
is enabled on the rule.
2021-06-01 19:59:11 -07:00
Harshavardhana
1f262daf6f
rename all remaining packages to internal/ (#12418)
This is to ensure that there are no projects
that try to import `minio/minio/pkg` into
their own repo. Any such common packages should
go to `https://github.com/minio/pkg`
2021-06-01 14:59:40 -07:00
Harshavardhana
988d39a5b6 Migrate to golang1.5 release with GO15VENDOREXPERIMENT=1 enabled 2015-08-22 18:35:37 -07:00
Harshavardhana
61175ef091 Migrate to govendor to avoid limitations of godep
- over the course of a project history every maintainer needs to update
  its dependency packages, the problem essentially with godep is manipulating
  GOPATH - this manipulation leads to static objects created at different locations
  which end up conflicting with the overall functionality of golang.

  This also leads to broken builds. There is no easier way out of this other than
  asking developers to do 'godep restore' all the time. Which perhaps as a practice
  doesn't sound like a clean solution. On the other hand 'godep restore' has its own
  set of problems.

- govendor is a right tool but a stop gap tool until we wait for golangs official
  1.5 version which fixes this vendoring issue once and for all.

- govendor provides consistency in terms of how import paths should be handled unlike
  manipulation GOPATH.

  This has advantages
    - no more compiled objects being referenced in GOPATH and build time GOPATH
      manging which leads to conflicts.
    - proper import paths referencing the exact package a project is dependent on.

 govendor is simple and provides the minimal necessary tooling to achieve this.

 For now this is the right solution.
2015-08-12 19:24:57 -07:00