5ffb2a9605
Update yaml files to latest version RELEASE.2024-06-04T19-20-08Z
2024-06-04 22:25:53 +00:00
17fe91d6d1
chore: update all deps ( #19875 )
2024-06-04 12:20:08 -07:00
90a9f2dd70
fix: log diskerror when detect the disk space failed ( #19861 )
2024-06-04 09:42:03 -07:00
d5e48cfd65
fix: remove DriveOPTimeout for REST callers as they don't work properly ( #19873 )
...
Go's net/http is notoriously difficult to have a streaming
deadlines per READ/WRITE on the net.Conn if we add them they
interfere with the Go's internal requirements for a HTTP
connection.
Remove this support for now
fixes #19853
2024-06-04 08:12:57 -07:00
d274566463
race: Fix rare race detected by testing ( #19872 )
...
Below is the race warning:
```
WARNING: DATA RACE
Write at 0x00c02d3d27c0 by goroutine 1210:
github.com/minio/minio/cmd.(*healingTracker).bucketDone()
github.com/minio/minio/cmd/background-newdisks-heal-ops.go:273 +0x13a
github.com/minio/minio/cmd.(*erasureObjects).healErasureSet()
github.com/minio/minio/cmd/global-heal.go:525 +0x2158
github.com/minio/minio/cmd.healFreshDisk()
github.com/minio/minio/cmd/background-newdisks-heal-ops.go:450 +0x107e
github.com/minio/minio/cmd.monitorLocalDisksAndHeal.func1()
github.com/minio/minio/cmd/background-newdisks-heal-ops.go:528 +0x150
github.com/minio/minio/cmd.monitorLocalDisksAndHeal.gowrap2()
github.com/minio/minio/cmd/background-newdisks-heal-ops.go:538 +0x82
Previous read at 0x00c02d3d27c0 by goroutine 1446:
github.com/minio/minio/cmd.(*erasureObjects).healErasureSet.func5()
github.com/minio/minio/cmd/global-heal.go:232 +0xfd
```
2024-06-04 08:12:32 -07:00
39ac720826
Remove hardcoded `override` as not needed ( #19868 )
...
Fixes: https://github.com/minio/minio/issues/19867
Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
2024-06-04 06:24:37 -07:00
21b6204692
Test proxying of DEL marker for bucket replication ( #19870 )
...
Make sure to avoid proxying for DEL markers
Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
2024-06-04 04:38:26 -07:00
d98faeb26a
Check if LDAP User has attached policy before creating Service Account ( #19843 )
...
Check if ldap user has policy before creating
2024-06-03 12:58:48 -07:00
0a63dc199c
Add trace sizes to more trace types ( #19864 )
...
Add trace sizes to
* ILM traces
* Replication traces
* Healing traces
* Decommission traces
* Rebalance traces
* (s)ftp traces
* http traces.
2024-06-03 08:45:54 -07:00
3ba857dfa1
race: Fix detected test race in the internal audit code ( #19865 )
2024-06-03 08:44:50 -07:00
a8554c4022
Update madmin ( #19862 )
...
Make it include https://github.com/minio/madmin-go/pull/285
2024-06-03 05:00:14 -07:00
ba54b39c02
fix: crash when audit webhook queue_dir is not writable ( #19854 )
...
This is regression introduced in #19275 refactor
2024-06-01 20:03:39 -07:00
2a75225569
kafka: _MINIO_KAFKA_DEBUG to enable sarama debug messages ( #19849 )
2024-06-01 08:02:59 -07:00
e72429c79c
Add sizes to traces ( #19851 )
...
added to storage and grid traces. Can provide more context for traces that aren't HTTP. Others may apply.
2024-05-31 22:17:37 -07:00
c5b3f5553f
Add per connection RPC metrics ( #19852 )
...
Provides individual and aggregate stats for each RPC connection.
Example:
```
"rpc": {
"collectedAt": "2024-05-31T14:33:29.1373103+02:00",
"connected": 30,
"disconnected": 0,
"outgoingStreams": 69,
"incomingStreams": 0,
"outgoingBytes": 174822796,
"incomingBytes": 175821566,
"outgoingMessages": 768595,
"incomingMessages": 768589,
"outQueue": 0,
"lastPongTime": "2024-05-31T12:33:28Z",
"byDestination": {
"http://127.0.0.1:9001 ": {
"collectedAt": "2024-05-31T14:33:29.1373103+02:00",
"connected": 5,
"disconnected": 0,
"outgoingStreams": 2,
"incomingStreams": 0,
"outgoingBytes": 38432543,
"incomingBytes": 66604052,
"outgoingMessages": 229496,
"incomingMessages": 229575,
"outQueue": 0,
"lastPongTime": "2024-05-31T12:33:27Z"
},
"http://127.0.0.1:9002 ": {
"collectedAt": "2024-05-31T14:33:29.1373103+02:00",
"connected": 5,
"disconnected": 0,
"outgoingStreams": 6,
"incomingStreams": 0,
"outgoingBytes": 38215680,
"incomingBytes": 66121283,
"outgoingMessages": 228525,
"incomingMessages": 228510,
"outQueue": 0,
"lastPongTime": "2024-05-31T12:33:27Z"
},
...
```
2024-05-31 22:16:24 -07:00
d3ae0aaad3
Add max buffering to SFTP ( #19848 )
...
Prevent OOM by adversarial use of SFTP upload by setting a 100MB max upload buffer.
2024-05-31 14:28:07 -07:00
d67bccf861
Add xl-meta partial shard reconstruction ( #19841 )
...
Add partial shard reconstruction
* Add partial shard reconstruction
* Fix padding causing the last shard to be rejected
* Add md5 checks on single parts
* Move md5 verified to `verified/filename.ext`
* Move complete (without md5) to `complete/filename.ext.partno`
It's not pretty, but at least now the md5 gives some confidence it works correctly.
2024-05-31 07:49:23 -07:00
1277ad69a6
heal: Remove .healing.bin when all ES drives are healing ( #19846 )
...
In the very rare case when all drives in a erasure set need to be healed,
remove .healing.bin from all drives, otherwise it will be stuck in a
loop
Also, fix a unit test that fails sometimes due to wrong test.
2024-05-31 07:48:50 -07:00
8f93e81afb
change service account embedded policy size limit ( #19840 )
...
Bonus: trim-off all the unnecessary spaces to allow
for real 2048 characters in policies for STS handlers
and re-use the code in all STS handlers.
2024-05-30 11:10:41 -07:00
4af31e654b
avoid pre-populating buffers for deployments < 32GiB memory ( #19839 )
2024-05-30 04:58:12 -07:00
aad50579ba
fix: wire up ILM sub-system properly for help ( #19836 )
2024-05-30 01:14:58 -07:00
38d059b0ae
fix: single node multi-drive must register local drives properly ( #19832 )
...
since #19688 there was a regression introduced during drive
lookups for single node multi-drive setups, drive replacement
would not work correctly without this PR.
2024-05-29 13:12:44 -07:00
bd4eeb4522
Fix flipped EcM, EcN in metadata header ( #19831 )
...
Since this is a tuple encoded field we can just flip the struct members.
2024-05-29 12:14:09 -07:00
03e3493288
fix: correct parse the tagging error for PostPolicyBucketHandler ( #19825 )
2024-05-29 11:50:46 -07:00
64baedf5a4
fix: hide prefixes for Hadoop properly ( #19821 )
2024-05-28 15:53:15 -07:00
2f64d5f77e
Update yaml files to latest version RELEASE.2024-05-28T17-19-04Z
2024-05-28 19:23:04 +00:00
f79a4ef4d0
policy: More defensive code validating svc:DurationSeconds ( #19820 )
...
This does not fix any current issue, but merging https://github.com/minio/madmin-go/pull/282
can lose the validation of the service account expiration time.
Add more defensive code for now. In the future, we should avoid doing
validation in another library.
2024-05-28 10:19:04 -07:00
2d53854b19
Restrict access keys for users and groups to not allow '=' or ',' ( #19749 )
...
* initial commit
* Add UTF check
---------
Co-authored-by: Harshavardhana <harsha@minio.io>
2024-05-28 10:14:16 -07:00
e5c83535af
chore: upgrade deps ( #19819 )
...
Signed-off-by: Harshavardhana <harsha@minio.io>
2024-05-28 02:27:44 -07:00
c904ef966e
feat: support tags for PostPolicy upload ( #19816 )
2024-05-27 21:44:00 -07:00
8f266e0772
Update yaml files to latest version RELEASE.2024-05-27T19-17-46Z
2024-05-27 23:52:43 +00:00
e0fe7cc391
fix: information disclosure bug in preconditions GET ( #19810 )
...
precondition check was being honored before, validating
if anonymous access is allowed on the metadata of an
object, leading to metadata disclosure of the following
headers.
```
Last-Modified
Etag
x-amz-version-id
Expires:
Cache-Control:
```
although the information presented is minimal in nature,
and of opaque nature. It still simply discloses that an
object by a specific name exists or not without even having
enough permissions.
2024-05-27 12:17:46 -07:00
9d20dec56a
Revert "remove dataErrs from er.deleteIfDangling code"
...
This reverts commit 7d75b1e758
2024-05-26 11:13:29 -07:00
597a785253
fix: authenticate LDAP via actual DN instead of normalized DN ( #19805 )
...
fix: authenticate LDAP via actual DN instead of normalized DN
Normalized DN is only for internal representation, not for
external communication, any communication to LDAP must be
based on actual user DN. LDAP servers do not understand
normalized DN.
fixes #19757
2024-05-25 06:43:06 -07:00
7d75b1e758
remove dataErrs from er.deleteIfDangling code
...
avoid this until a comprehensive change is
merged such as https://github.com/minio/minio/pull/19797
2024-05-24 18:20:04 -07:00
5f78691fcf
ldap: Add user DN attributes list config param ( #19758 )
...
This change uses the updated ldap library in minio/pkg (bumped
up to v3). A new config parameter is added for LDAP configuration to
specify extra user attributes to load from the LDAP server and to store
them as additional claims for the user.
A test is added in sts_handlers.go that shows how to access the LDAP
attributes as a claim.
This is in preparation for adding SSH pubkey authentication to MinIO's SFTP
integration.
2024-05-24 16:05:23 -07:00
a591e06ae5
Add cluster scanner metrics in metrics-v3 ( #19517 )
...
endpoint: /minio/metrics/v3/cluster/scanner
metrics:
- bucket_scans_finished (counter)
- bucket_scans_started (counter)
- directories_scanned (counter)
- last_activity_nano_seconds (gauge)
- objects_scanned (counter)
- versions_scanned (counter)
2024-05-24 12:29:25 -07:00
443c93c634
compute time spent in ILM properly ( #19806 )
2024-05-24 12:28:51 -07:00
5659cddc84
Add cluster config metrics in metrics-v3 ( #19507 )
...
endpoint: /minio/metrics/v3/cluster/config
metrics:
- write_quorum
- rrs_parity
- standard_parity
2024-05-24 05:50:46 -07:00
2a03a34bde
Upgrade madmin-go to v3.0.52 ( #19798 )
...
This will ensure that content of /proc/cmdline from each server is
captured in the health report.
2024-05-24 05:34:57 -07:00
1654a9b7e6
Use point in time values for `gauge` metrics in graphs ( #19690 )
...
Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
2024-05-24 04:11:51 -07:00
673a521711
Change endpoint of v3 notification metrics ( #19804 )
...
from /cluster/notification to /notification
2024-05-24 04:10:24 -07:00
2e23076688
move windows runners to in-house ( #19800 )
...
GitHub CI runners for windows have gotten very slow,
moving them to our own hosted runners
2024-05-23 15:29:33 -07:00
b92ac55250
Add multipart combination to xl-meta ( #19780 )
...
Add combination of multiple parts.
Parts will be reconstructed and saved separately and can manually be combined to the complete object.
Parts will be named `(version_id)-(filename).(partnum).(in)complete`.
2024-05-23 09:37:31 -07:00
7981509cc8
Add cluster and bucket replication metrics in metrics-v3 ( #19546 )
...
endpoint: /minio/metrics/v3/cluster/replication
metrics:
- average_active_workers
- average_queued_bytes
- average_queued_count
- average_transfer_rate
- current_active_workers
- current_transfer_rate
- last_minute_queued_bytes
- last_minute_queued_count
- max_active_workers
- max_queued_bytes
- max_queued_count
- max_transfer_rate
- recent_backlog_count
endpoint: /minio/metrics/v3/api/bucket/replication
metrics:
- last_hour_failed_bytes
- last_hour_failed_count
- last_minute_failed_bytes
- last_minute_failed_count
- latency_ms
- proxied_delete_tagging_requests_total
- proxied_get_requests_failures
- proxied_get_requests_total
- proxied_get_tagging_requests_failures
- proxied_get_tagging_requests_total
- proxied_head_requests_failures
- proxied_head_requests_total
- proxied_put_tagging_requests_failures
- proxied_put_tagging_requests_total
- sent_bytes
- sent_count
- total_failed_bytes
- total_failed_count
- proxied_delete_tagging_requests_failures
2024-05-23 00:41:18 -07:00
6d5bc045bc
Disallow ExpiredObjectAllVersions with object lock ( #19792 )
...
Relaxes restrictions on Expiration and NoncurrentVersionExpiration
placed by https://github.com/minio/minio/pull/19785 .
ref: https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lock-managing.html#object-lock-managing-lifecycle
> Object lifecycle management configurations continue functioning
normally on protected objects, including placing delete markers.
However, a locked version of an object cannot be deleted by a S3
Lifecycle expiration policy. Object Lock is maintained regardless of
the object's storage class and throughout S3 Lifecycle
transitions between storage classes.
2024-05-22 18:12:48 -07:00
d38e020b29
remove errant logs for disconnected remote ( #19793 )
...
Signed-off-by: Harshavardhana <harsha@minio.io>
2024-05-22 18:12:23 -07:00
7d29030292
fix list results returned for spark max-keys=2 listing ( #19791 )
...
This PR continues fix #19725 for some unhandled cases
2024-05-22 16:16:34 -07:00
7c7650b7c3
Add sufficient deadlines and countermeasures to handle hung node scenario ( #19688 )
...
Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
Signed-off-by: Harshavardhana <harsha@minio.io>
2024-05-22 16:07:14 -07:00
ca80eced24
usage of deadline conn at Accept() breaks websocket ( #19789 )
...
fortunately not wired up to use, however if anyone
enables deadlines for conn then sporadically MinIO
startups fail.
2024-05-22 10:49:27 -07:00
Minio Trusted
Harshavardhana
jiuker
Anis Eleuch
Shubhendu
Taran Pelkey
Klaus Post
Aditya Manthramurthy
Shireesh Anjal
Krishnan Parthasarathi
Poorna