Commit Graph

9826 Commits

Author SHA1 Message Date
Harshavardhana 661b263e77
add gocritic/ruleguard checks back again, cleanup code. (#13665)
- remove some duplicated code
- reported a bug, separately fixed in #13664
- using strings.ReplaceAll() when needed
- using filepath.ToSlash() use when needed
- remove all non-Go style comments from the codebase

Co-authored-by: Aditya Manthramurthy <donatello@users.noreply.github.com>
2021-11-16 09:28:29 -08:00
Aditya Manthramurthy 07c5e72cdb
add thread context in surrounding function into IAM functions (#13658) 2021-11-15 14:14:22 -08:00
Harshavardhana 7752cdbfaf
fix: restored object to preserve x-amz-meta properly (#13664)
with SelectRestoreRequest OutputLocation provides
additional metadata for the object, this is not
preserved due to argument order change.
2021-11-15 13:25:55 -08:00
Harshavardhana 4545ecad58
ignore swapped drives instead of throwing errors (#13655)
- add checks such that swapped disks are detected
  and ignored - never used for normal operations.

- implement `unrecognizedDisk` to be ignored with
  all operations returning `errDiskNotFound`.

- also add checks such that we do not load unexpected
  disks while connecting automatically.

- additionally humanize the values when printing the errors.

Bonus: fixes handling of non-quorum situations in
getLatestFileInfo(), that does not work when 2 drives
are down, currently this function would return errors
incorrectly.
2021-11-15 09:46:55 -08:00
Harshavardhana ac74237f01
add explicit deny support for service accounts (#13657)
creating service accounts is implicitly enabled
for all users, this PR however adds support to
reject creating service accounts, with an explicit
"Deny" policy.
2021-11-15 06:57:52 -08:00
Harshavardhana 5a36179c19 update go.mod to v0.12.3 2021-11-13 09:05:28 -08:00
Harshavardhana 82d73f387d
add tool to read healing.bin for debugging (#13650) 2021-11-12 16:18:53 -08:00
Aditya Manthramurthy e8c6314770
IAM: init IAM with Init() rather than InitStore() in tests (#13643)
- rename InitStore() to initStore() and fix tests

- Use IAMSys.Lock() only when IAMSys struct is being mutated
2021-11-11 21:03:02 -08:00
Aditya Manthramurthy 087c1b98dc
Add tests for OpenID STS creds and add to CI (#13638) 2021-11-11 11:23:30 -08:00
Harshavardhana 68c5ad83fb
fix: backend not reachable should be more descriptive (#13634) 2021-11-10 22:33:17 -08:00
Harshavardhana 5acc8c0134
add multi-site replication tests (#13631) 2021-11-10 18:18:09 -08:00
Klaus Post c897b6a82d
fix: missing entries on first list resume (#13627)
On first list resume or when specifying a custom markers entries could be missed in rare cases.

Do conservative truncation of entries when forwarding.

Replaces #13619
2021-11-10 10:41:21 -08:00
Shireesh Anjal d008e90d50
Support dynamic reset of minio config (#13626)
If a given MinIO config is dynamic (can be changed without restart),
ensure that it can be reset also without restart.

Signed-off-by: Shireesh Anjal <shireesh@minio.io>
2021-11-10 10:01:32 -08:00
Harshavardhana ea820b30bf
fix: use equalFold() instead of lower and compare (#13624) 2021-11-10 08:12:50 -08:00
Poorna K 03725dc015
Default multipart caching to writethrough (#13613)
when `MINIO_CACHE_COMMIT` is set.

- `writeback` caching applies only to single 
uploads. When cache commit mode is 
`writeback`, default multipart caching to be
synchronous.

- Add writethrough caching for single uploads
2021-11-10 08:12:03 -08:00
Harshavardhana 0a6f9bc1eb
allocate new highwayhash for each string hash (#13623)
fixes #13622
2021-11-09 15:28:08 -08:00
Aditya Manthramurthy 1946922de3
Add CI for etcd IAM backend (#13614)
Runs when ETCD_SERVER env var is set
2021-11-09 09:25:13 -08:00
Minio Trusted edf1f4233b Update yaml files to latest version RELEASE.2021-11-09T03-21-45Z 2021-11-09 04:51:05 +00:00
Harshavardhana f4b55ea7a7 update console to v0.12.2 2021-11-08 19:21:45 -08:00
Aditya Manthramurthy 8dfd1f03e9
fix: IAM initialization crash with etcd store (#13612) 2021-11-08 12:55:27 -08:00
Harshavardhana acf26c5ab7 re-arrange metacache struct to be optimal (#13609) 2021-11-08 10:26:08 -08:00
Klaus Post d9800c8135
fix: make sure to log panic in handlers (#13611) 2021-11-08 09:28:13 -08:00
Harshavardhana 02bef7560f add missing Copyright header 2021-11-08 09:13:15 -08:00
Daniel A. Ochoa 07dd0692b6
Fix hdfs gateway concurrent map writes (#13596)
Co-authored-by: Harshavardhana <harsha@minio.io>
2021-11-08 09:07:58 -08:00
Klaus Post 4f3317effe
Close stream on panic (#13605)
Always close streamHTTPResponse on panic on main thread to avoid 
write/flush after response handler has returned.
2021-11-08 08:41:27 -08:00
Klaus Post 9afdbe3648
fix: RLock UID memory leak (#13607)
UID were misnamed in RLock, leading to memory buildup.

Regression in #13430
2021-11-08 07:35:50 -08:00
Aditya Manthramurthy fe0df01448
fix: locking in some situations for IAM store (#13595)
- Fix a bug where read locks were taken instead of write locks in some situations
- Remove an unnecessary lock when updating based on notifications.
2021-11-07 17:42:32 -08:00
Harshavardhana 12e6907512
apply spelling checks for US locale (#13599) 2021-11-07 01:22:59 -08:00
Harshavardhana 5aef492b4c update disk-caching design guide 2021-11-07 01:21:34 -08:00
Harshavardhana 5d7ed8ff7d update S3 gateway limitation docs 2021-11-06 23:24:48 -07:00
Harshavardhana b1754fc5ff update go.mod and CREDITS 2021-11-06 11:39:17 -07:00
Harshavardhana 19bbf3e142 update CREDITS 2021-11-05 13:53:21 -07:00
jiangfucheng e1755275a0
resume heal from previous object instead of bucket after server restart (#13581) 2021-11-05 13:10:41 -07:00
Harshavardhana 520037e721
move to jwt-go v4 with correct releases (#13586) 2021-11-05 12:20:08 -07:00
Minio Trusted cbb0828ab8 Update yaml files to latest version RELEASE.2021-11-05T09-16-26Z 2021-11-05 10:03:56 +00:00
Andreas Auernhammer 8774d10bdf
sts: always verify the key usage of client certificates (#13583)
This commit makes the MinIO server behavior more consistent
w.r.t. key usage verification.

When MinIO verifies the client certificates it also checks
that the client certificate is valid of client authentication
(or any (i.e. wildcard) usage).

However, the MinIO server used to not verify the client key usage
when client certificate verification was disabled.
Now, the MinIO server verifies the client key usage even when
client certificate verification has been disabled. This makes
the MinIO behavior more consistent from a client's perspective.

Now, a client certificate has to be valid for client authentication
in all cases.

Signed-off-by: Andreas Auernhammer <hi@aead.dev>
2021-11-05 02:16:26 -07:00
Harshavardhana df9f479d58 update console to v0.12.1 2021-11-04 16:43:59 -07:00
Harshavardhana 8bb52c9c2a
fix: ignore disks that are available but not writable (#13585)
This is to allow replacing drives while some drives
while available are not writable.
2021-11-04 16:42:49 -07:00
Aditya Manthramurthy 947c423824
fix: user DN filtering that causes some unnecessary logs (#13584)
Additionally, remove the unnecessary `isUsingLookupBind` field in the LDAP struct
2021-11-04 13:11:20 -07:00
Harshavardhana c3d24fb26d
use single encoder for sending speedtest results (#13579)
Bonus: if runs have PUT higher then capture it anyways
to display an unexpected result, which provides a way
to understand what might be slowing things down on the
system.

For example on a Data24 WDC setup it is clearly visible
there is a bug in the hardware.

```
./mc admin speedtest wdc/
⠧ Running speedtest (With 64 MiB object size, 32 concurrency) PUT: 31 GiB/s GET: 24 GiB/s
⠹ Running speedtest (With 64 MiB object size, 48 concurrency) PUT: 38 GiB/s GET: 24 GiB/s

MinIO 2021-11-04T06:08:33Z, 6 servers, 48 drives
PUT: 38 GiB/s, 605 objs/s
GET: 24 GiB/s, 383 objs/s
```

Reads are almost 14GiB/sec slower than Writes which
is practically not possible.
2021-11-04 12:11:52 -07:00
Pavel M 112f9ae087
claim exp should be integer (#13582)
claim exp can be 

- float64
- json.Number

As per OIDC spec https://openid.net/specs/openid-connect-core-1_0.html#IDToken

Avoid using strings since the upstream library only supports these two types now.
2021-11-04 12:03:43 -07:00
Aditya Manthramurthy 01b9ff54d9
Add LDAP STS tests and workflow for CI (#13576)
Runs LDAP tests with openldap container on GH Actions
2021-11-04 08:16:30 -07:00
Aditya Manthramurthy 64a1904136
Remove unused GlobalServiceDoneCh (#13578) 2021-11-04 08:15:10 -07:00
Aditya Manthramurthy bce6864785
Add tests to verify default server policies (#13575)
Check that they are present and that they can be modified by user
2021-11-03 19:49:05 -07:00
Aditya Manthramurthy ecd54b4cba
Move all IAM storage functionality into iam store type (#13567)
This reverts commit 091a7ae359.

- Ensure all actions accessing storage lock properly.

- Behavior change: policies can be deleted only when they
  are not associated with any active credentials.

Also adds fix for accidental canned policy removal that was present in the
reverted version of the change.
2021-11-03 19:47:49 -07:00
Harshavardhana ca2b288a4b update console to v0.12.0 2021-11-03 16:23:45 -07:00
Harshavardhana 1016fbb8f9
feat: detect starting from windows explorer (#13570)
Windows users often click on the binary without
knowing MinIO is a command-line tool and should be
run from a terminal. Throw a message to guide them
on what to do.

Co-authored-by: Klaus Post <klauspost@gmail.com>
2021-11-03 14:22:13 -07:00
Harshavardhana be3f81c7ec
remove unused activeIOCount in single drive mode (#13574) 2021-11-03 12:29:45 -07:00
Minio Trusted 9f3c151c3c Update yaml files to latest version RELEASE.2021-11-03T03-36-36Z 2021-11-03 06:48:34 +00:00
Harshavardhana 9735f3d8f2 fix missing go.sum changes 2021-11-02 20:36:36 -07:00