Commit Graph

8977 Commits

Author SHA1 Message Date
Krishnan Parthasarathi
367cb48096
reduceErrs to handle context.Canceled errors (#13670)
With this change, reduceErrs will group all errors due to 
context cancelation as the same.

e.g, Following are errors due to context cancelation seen 
from 3 remote disks. Their error values are different but 
they are all caused due to the same context cancelation.

['Post
"http://minio2:9000/minio/storage/data1/v37/statvol?disk-id=101cbc99-f5d2-4a9d-b18b-97e872b3e4a7&volume=mybucket":
context canceled',
 'Post
 "http://minio3:9000/minio/storage/data1/v37/statvol?disk-id=7a84474b-a4fd-4b80-8aab-d308a587c280&volume=mybucket":
 context canceled',
 'Post
 "http://minio4:9000/minio/storage/data1/v37/statvol?disk-id=d60d571a-83c8-487d-9e14-beebc94682d2&volume=mybucket":
 context canceled']
2021-11-16 15:26:48 -08:00
Harshavardhana
661b263e77
add gocritic/ruleguard checks back again, cleanup code. (#13665)
- remove some duplicated code
- reported a bug, separately fixed in #13664
- using strings.ReplaceAll() when needed
- using filepath.ToSlash() use when needed
- remove all non-Go style comments from the codebase

Co-authored-by: Aditya Manthramurthy <donatello@users.noreply.github.com>
2021-11-16 09:28:29 -08:00
Aditya Manthramurthy
07c5e72cdb
add thread context in surrounding function into IAM functions (#13658) 2021-11-15 14:14:22 -08:00
Harshavardhana
7752cdbfaf
fix: restored object to preserve x-amz-meta properly (#13664)
with SelectRestoreRequest OutputLocation provides
additional metadata for the object, this is not
preserved due to argument order change.
2021-11-15 13:25:55 -08:00
Harshavardhana
4545ecad58
ignore swapped drives instead of throwing errors (#13655)
- add checks such that swapped disks are detected
  and ignored - never used for normal operations.

- implement `unrecognizedDisk` to be ignored with
  all operations returning `errDiskNotFound`.

- also add checks such that we do not load unexpected
  disks while connecting automatically.

- additionally humanize the values when printing the errors.

Bonus: fixes handling of non-quorum situations in
getLatestFileInfo(), that does not work when 2 drives
are down, currently this function would return errors
incorrectly.
2021-11-15 09:46:55 -08:00
Harshavardhana
ac74237f01
add explicit deny support for service accounts (#13657)
creating service accounts is implicitly enabled
for all users, this PR however adds support to
reject creating service accounts, with an explicit
"Deny" policy.
2021-11-15 06:57:52 -08:00
Harshavardhana
5a36179c19 update go.mod to v0.12.3 2021-11-13 09:05:28 -08:00
Harshavardhana
82d73f387d
add tool to read healing.bin for debugging (#13650) 2021-11-12 16:18:53 -08:00
Aditya Manthramurthy
e8c6314770
IAM: init IAM with Init() rather than InitStore() in tests (#13643)
- rename InitStore() to initStore() and fix tests

- Use IAMSys.Lock() only when IAMSys struct is being mutated
2021-11-11 21:03:02 -08:00
Aditya Manthramurthy
087c1b98dc
Add tests for OpenID STS creds and add to CI (#13638) 2021-11-11 11:23:30 -08:00
Harshavardhana
68c5ad83fb
fix: backend not reachable should be more descriptive (#13634) 2021-11-10 22:33:17 -08:00
Harshavardhana
5acc8c0134
add multi-site replication tests (#13631) 2021-11-10 18:18:09 -08:00
Klaus Post
c897b6a82d
fix: missing entries on first list resume (#13627)
On first list resume or when specifying a custom markers entries could be missed in rare cases.

Do conservative truncation of entries when forwarding.

Replaces #13619
2021-11-10 10:41:21 -08:00
Shireesh Anjal
d008e90d50
Support dynamic reset of minio config (#13626)
If a given MinIO config is dynamic (can be changed without restart),
ensure that it can be reset also without restart.

Signed-off-by: Shireesh Anjal <shireesh@minio.io>
2021-11-10 10:01:32 -08:00
Harshavardhana
ea820b30bf
fix: use equalFold() instead of lower and compare (#13624) 2021-11-10 08:12:50 -08:00
Poorna K
03725dc015
Default multipart caching to writethrough (#13613)
when `MINIO_CACHE_COMMIT` is set.

- `writeback` caching applies only to single 
uploads. When cache commit mode is 
`writeback`, default multipart caching to be
synchronous.

- Add writethrough caching for single uploads
2021-11-10 08:12:03 -08:00
Harshavardhana
0a6f9bc1eb
allocate new highwayhash for each string hash (#13623)
fixes #13622
2021-11-09 15:28:08 -08:00
Aditya Manthramurthy
1946922de3
Add CI for etcd IAM backend (#13614)
Runs when ETCD_SERVER env var is set
2021-11-09 09:25:13 -08:00
Minio Trusted
edf1f4233b Update yaml files to latest version RELEASE.2021-11-09T03-21-45Z 2021-11-09 04:51:05 +00:00
Harshavardhana
f4b55ea7a7 update console to v0.12.2 2021-11-08 19:21:45 -08:00
Aditya Manthramurthy
8dfd1f03e9
fix: IAM initialization crash with etcd store (#13612) 2021-11-08 12:55:27 -08:00
Harshavardhana
acf26c5ab7 re-arrange metacache struct to be optimal (#13609) 2021-11-08 10:26:08 -08:00
Klaus Post
d9800c8135
fix: make sure to log panic in handlers (#13611) 2021-11-08 09:28:13 -08:00
Harshavardhana
02bef7560f add missing Copyright header 2021-11-08 09:13:15 -08:00
Daniel A. Ochoa
07dd0692b6
Fix hdfs gateway concurrent map writes (#13596)
Co-authored-by: Harshavardhana <harsha@minio.io>
2021-11-08 09:07:58 -08:00
Klaus Post
4f3317effe
Close stream on panic (#13605)
Always close streamHTTPResponse on panic on main thread to avoid 
write/flush after response handler has returned.
2021-11-08 08:41:27 -08:00
Klaus Post
9afdbe3648
fix: RLock UID memory leak (#13607)
UID were misnamed in RLock, leading to memory buildup.

Regression in #13430
2021-11-08 07:35:50 -08:00
Aditya Manthramurthy
fe0df01448
fix: locking in some situations for IAM store (#13595)
- Fix a bug where read locks were taken instead of write locks in some situations
- Remove an unnecessary lock when updating based on notifications.
2021-11-07 17:42:32 -08:00
Harshavardhana
12e6907512
apply spelling checks for US locale (#13599) 2021-11-07 01:22:59 -08:00
Harshavardhana
5aef492b4c update disk-caching design guide 2021-11-07 01:21:34 -08:00
Harshavardhana
5d7ed8ff7d update S3 gateway limitation docs 2021-11-06 23:24:48 -07:00
Harshavardhana
b1754fc5ff update go.mod and CREDITS 2021-11-06 11:39:17 -07:00
Harshavardhana
19bbf3e142 update CREDITS 2021-11-05 13:53:21 -07:00
jiangfucheng
e1755275a0
resume heal from previous object instead of bucket after server restart (#13581) 2021-11-05 13:10:41 -07:00
Harshavardhana
520037e721
move to jwt-go v4 with correct releases (#13586) 2021-11-05 12:20:08 -07:00
Minio Trusted
cbb0828ab8 Update yaml files to latest version RELEASE.2021-11-05T09-16-26Z 2021-11-05 10:03:56 +00:00
Andreas Auernhammer
8774d10bdf
sts: always verify the key usage of client certificates (#13583)
This commit makes the MinIO server behavior more consistent
w.r.t. key usage verification.

When MinIO verifies the client certificates it also checks
that the client certificate is valid of client authentication
(or any (i.e. wildcard) usage).

However, the MinIO server used to not verify the client key usage
when client certificate verification was disabled.
Now, the MinIO server verifies the client key usage even when
client certificate verification has been disabled. This makes
the MinIO behavior more consistent from a client's perspective.

Now, a client certificate has to be valid for client authentication
in all cases.

Signed-off-by: Andreas Auernhammer <hi@aead.dev>
2021-11-05 02:16:26 -07:00
Harshavardhana
df9f479d58 update console to v0.12.1 2021-11-04 16:43:59 -07:00
Harshavardhana
8bb52c9c2a
fix: ignore disks that are available but not writable (#13585)
This is to allow replacing drives while some drives
while available are not writable.
2021-11-04 16:42:49 -07:00
Aditya Manthramurthy
947c423824
fix: user DN filtering that causes some unnecessary logs (#13584)
Additionally, remove the unnecessary `isUsingLookupBind` field in the LDAP struct
2021-11-04 13:11:20 -07:00
Harshavardhana
c3d24fb26d
use single encoder for sending speedtest results (#13579)
Bonus: if runs have PUT higher then capture it anyways
to display an unexpected result, which provides a way
to understand what might be slowing things down on the
system.

For example on a Data24 WDC setup it is clearly visible
there is a bug in the hardware.

```
./mc admin speedtest wdc/
⠧ Running speedtest (With 64 MiB object size, 32 concurrency) PUT: 31 GiB/s GET: 24 GiB/s
⠹ Running speedtest (With 64 MiB object size, 48 concurrency) PUT: 38 GiB/s GET: 24 GiB/s

MinIO 2021-11-04T06:08:33Z, 6 servers, 48 drives
PUT: 38 GiB/s, 605 objs/s
GET: 24 GiB/s, 383 objs/s
```

Reads are almost 14GiB/sec slower than Writes which
is practically not possible.
2021-11-04 12:11:52 -07:00
Pavel M
112f9ae087
claim exp should be integer (#13582)
claim exp can be 

- float64
- json.Number

As per OIDC spec https://openid.net/specs/openid-connect-core-1_0.html#IDToken

Avoid using strings since the upstream library only supports these two types now.
2021-11-04 12:03:43 -07:00
Aditya Manthramurthy
01b9ff54d9
Add LDAP STS tests and workflow for CI (#13576)
Runs LDAP tests with openldap container on GH Actions
2021-11-04 08:16:30 -07:00
Aditya Manthramurthy
64a1904136
Remove unused GlobalServiceDoneCh (#13578) 2021-11-04 08:15:10 -07:00
Aditya Manthramurthy
bce6864785
Add tests to verify default server policies (#13575)
Check that they are present and that they can be modified by user
2021-11-03 19:49:05 -07:00
Aditya Manthramurthy
ecd54b4cba
Move all IAM storage functionality into iam store type (#13567)
This reverts commit 091a7ae359.

- Ensure all actions accessing storage lock properly.

- Behavior change: policies can be deleted only when they
  are not associated with any active credentials.

Also adds fix for accidental canned policy removal that was present in the
reverted version of the change.
2021-11-03 19:47:49 -07:00
Harshavardhana
ca2b288a4b update console to v0.12.0 2021-11-03 16:23:45 -07:00
Harshavardhana
1016fbb8f9
feat: detect starting from windows explorer (#13570)
Windows users often click on the binary without
knowing MinIO is a command-line tool and should be
run from a terminal. Throw a message to guide them
on what to do.

Co-authored-by: Klaus Post <klauspost@gmail.com>
2021-11-03 14:22:13 -07:00
Harshavardhana
be3f81c7ec
remove unused activeIOCount in single drive mode (#13574) 2021-11-03 12:29:45 -07:00
Minio Trusted
9f3c151c3c Update yaml files to latest version RELEASE.2021-11-03T03-36-36Z 2021-11-03 06:48:34 +00:00