MyFavMirrors/minio
1
0
Fork 0
mirror of https://github.com/minio/minio.git synced 2025-11-09 05:34:56 -05:00
Code Issues Packages Projects Releases Wiki Activity
12,085 Commits 4 Branches 523 Tags
380233d646b3e02e8c9f389b8bb02d131d5a06c9
Commit Graph

12085 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Bala FA
7edc352d23 Add ILM metrics in metrics-v3 (#19539) 
Signed-off-by: Bala.FA <bala@minio.io>
 2024-06-06 02:36:25 -07:00
Poorna
850a84b08a simplify site replication multipart proxying (#19885) 2024-06-05 18:01:15 -07:00
Taran Pelkey
4148754ce0 Check both given and normalized group DN on LDAP policy detach requests (#19876) 2024-06-05 15:42:40 -07:00
Harshavardhana
2107722829 upgrade go-oidc to fix GO-2024-2631 (#19884) 2024-06-05 15:00:34 -07:00
jiuker
d326ba52e9 feat: support batchJob for windows (#19877) 2024-06-05 08:44:53 -07:00
Sveinn
91e1487de4 Add LDAP public key authentication to SFTP (#19833) 2024-06-05 00:51:13 -07:00
Minio Trusted
5ffb2a9605 Update yaml files to latest version RELEASE.2024-06-04T19-20-08Z 2024-06-04 22:25:53 +00:00
Harshavardhana
17fe91d6d1 chore: update all deps (#19875) RELEASE.2024-06-04T19-20-08Z 2024-06-04 12:20:08 -07:00
jiuker
90a9f2dd70 fix: log diskerror when detect the disk space failed (#19861) 2024-06-04 09:42:03 -07:00
Harshavardhana
d5e48cfd65 fix: remove DriveOPTimeout for REST callers as they don't work properly (#19873) 
Go's net/http is notoriously difficult to have a streaming
deadlines per READ/WRITE on the net.Conn if we add them they
interfere with the Go's internal requirements for a HTTP
connection.

Remove this support for now

fixes #19853
 2024-06-04 08:12:57 -07:00
Anis Eleuch
d274566463 race: Fix rare race detected by testing (#19872) 
Below is the race warning:

```
WARNING: DATA RACE
Write at 0x00c02d3d27c0 by goroutine 1210:
  github.com/minio/minio/cmd.(*healingTracker).bucketDone()
      github.com/minio/minio/cmd/background-newdisks-heal-ops.go:273 +0x13a
  github.com/minio/minio/cmd.(*erasureObjects).healErasureSet()
      github.com/minio/minio/cmd/global-heal.go:525 +0x2158
  github.com/minio/minio/cmd.healFreshDisk()
      github.com/minio/minio/cmd/background-newdisks-heal-ops.go:450 +0x107e
  github.com/minio/minio/cmd.monitorLocalDisksAndHeal.func1()
      github.com/minio/minio/cmd/background-newdisks-heal-ops.go:528 +0x150
  github.com/minio/minio/cmd.monitorLocalDisksAndHeal.gowrap2()
      github.com/minio/minio/cmd/background-newdisks-heal-ops.go:538 +0x82

Previous read at 0x00c02d3d27c0 by goroutine 1446:
  github.com/minio/minio/cmd.(*erasureObjects).healErasureSet.func5()
      github.com/minio/minio/cmd/global-heal.go:232 +0xfd
```
 2024-06-04 08:12:32 -07:00
Shubhendu
39ac720826 Remove hardcoded override as not needed (#19868) 
Fixes: https://github.com/minio/minio/issues/19867

Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
 2024-06-04 06:24:37 -07:00
Shubhendu
21b6204692 Test proxying of DEL marker for bucket replication (#19870) 
Make sure to avoid proxying for DEL markers

Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
 2024-06-04 04:38:26 -07:00
Taran Pelkey
d98faeb26a Check if LDAP User has attached policy before creating Service Account (#19843) 
Check if ldap user has policy before creating
 2024-06-03 12:58:48 -07:00
Klaus Post
0a63dc199c Add trace sizes to more trace types (#19864) 
Add trace sizes to

* ILM traces
* Replication traces
* Healing traces
* Decommission traces
* Rebalance traces
* (s)ftp traces
* http traces.
 2024-06-03 08:45:54 -07:00
Anis Eleuch
3ba857dfa1 race: Fix detected test race in the internal audit code (#19865) 2024-06-03 08:44:50 -07:00
Klaus Post
a8554c4022 Update madmin (#19862) 
Make it include https://github.com/minio/madmin-go/pull/285
 2024-06-03 05:00:14 -07:00
Harshavardhana
ba54b39c02 fix: crash when audit webhook queue_dir is not writable (#19854) 
This is regression introduced in #19275 refactor
 2024-06-01 20:03:39 -07:00
Anis Eleuch
2a75225569 kafka: _MINIO_KAFKA_DEBUG to enable sarama debug messages (#19849) 2024-06-01 08:02:59 -07:00
Klaus Post
e72429c79c Add sizes to traces (#19851) 
added to storage and grid traces. Can provide more context for traces that aren't HTTP. Others may apply.
 2024-05-31 22:17:37 -07:00
Klaus Post
c5b3f5553f Add per connection RPC metrics (#19852) 
Provides individual and aggregate stats for each RPC connection.

Example:

```
  "rpc": {
   "collectedAt": "2024-05-31T14:33:29.1373103+02:00",
   "connected": 30,
   "disconnected": 0,
   "outgoingStreams": 69,
   "incomingStreams": 0,
   "outgoingBytes": 174822796,
   "incomingBytes": 175821566,
   "outgoingMessages": 768595,
   "incomingMessages": 768589,
   "outQueue": 0,
   "lastPongTime": "2024-05-31T12:33:28Z",
   "byDestination": {
    "http://127.0.0.1:9001": {
     "collectedAt": "2024-05-31T14:33:29.1373103+02:00",
     "connected": 5,
     "disconnected": 0,
     "outgoingStreams": 2,
     "incomingStreams": 0,
     "outgoingBytes": 38432543,
     "incomingBytes": 66604052,
     "outgoingMessages": 229496,
     "incomingMessages": 229575,
     "outQueue": 0,
     "lastPongTime": "2024-05-31T12:33:27Z"
    },
    "http://127.0.0.1:9002": {
     "collectedAt": "2024-05-31T14:33:29.1373103+02:00",
     "connected": 5,
     "disconnected": 0,
     "outgoingStreams": 6,
     "incomingStreams": 0,
     "outgoingBytes": 38215680,
     "incomingBytes": 66121283,
     "outgoingMessages": 228525,
     "incomingMessages": 228510,
     "outQueue": 0,
     "lastPongTime": "2024-05-31T12:33:27Z"
    },
...
```
 2024-05-31 22:16:24 -07:00
Klaus Post
d3ae0aaad3 Add max buffering to SFTP (#19848) 
Prevent OOM by adversarial use of SFTP upload by setting a 100MB max upload buffer.
 2024-05-31 14:28:07 -07:00
Klaus Post
d67bccf861 Add xl-meta partial shard reconstruction (#19841) 
Add partial shard reconstruction

* Add partial shard reconstruction
* Fix padding causing the last shard to be rejected
* Add md5 checks on single parts
* Move md5 verified to `verified/filename.ext`
* Move complete (without md5) to `complete/filename.ext.partno`

It's not pretty, but at least now the md5 gives some confidence it works correctly.
 2024-05-31 07:49:23 -07:00
Anis Eleuch
1277ad69a6 heal: Remove .healing.bin when all ES drives are healing (#19846) 
In the very rare case when all drives in a erasure set need to be healed,
remove .healing.bin from all drives, otherwise it will be stuck in a
loop

Also, fix a unit test that fails sometimes due to wrong test.
 2024-05-31 07:48:50 -07:00
Harshavardhana
8f93e81afb change service account embedded policy size limit (#19840) 
Bonus: trim-off all the unnecessary spaces to allow
for real 2048 characters in policies for STS handlers
and re-use the code in all STS handlers.
 2024-05-30 11:10:41 -07:00
Harshavardhana
4af31e654b avoid pre-populating buffers for deployments < 32GiB memory (#19839) 2024-05-30 04:58:12 -07:00
Harshavardhana
aad50579ba fix: wire up ILM sub-system properly for help (#19836) 2024-05-30 01:14:58 -07:00
Harshavardhana
38d059b0ae fix: single node multi-drive must register local drives properly (#19832) 
since #19688 there was a regression introduced during drive
lookups for single node multi-drive setups, drive replacement
would not work correctly without this PR.
 2024-05-29 13:12:44 -07:00
Klaus Post
bd4eeb4522 Fix flipped EcM, EcN in metadata header (#19831) 
Since this is a tuple encoded field we can just flip the struct members.
 2024-05-29 12:14:09 -07:00
jiuker
03e3493288 fix: correct parse the tagging error for PostPolicyBucketHandler (#19825) 2024-05-29 11:50:46 -07:00
Harshavardhana
64baedf5a4 fix: hide prefixes for Hadoop properly (#19821) 2024-05-28 15:53:15 -07:00
Minio Trusted
2f64d5f77e Update yaml files to latest version RELEASE.2024-05-28T17-19-04Z 2024-05-28 19:23:04 +00:00
Anis Eleuch
f79a4ef4d0 policy: More defensive code validating svc:DurationSeconds (#19820) 
This does not fix any current issue, but merging https://github.com/minio/madmin-go/pull/282
can lose the validation of the service account expiration time.

Add more defensive code for now. In the future, we should avoid doing
validation in another library.
RELEASE.2024-05-28T17-19-04Z 		2024-05-28 10:19:04 -07:00
Taran Pelkey
2d53854b19 Restrict access keys for users and groups to not allow '=' or ',' (#19749) 
* initial commit

* Add UTF check

---------

Co-authored-by: Harshavardhana <harsha@minio.io>
 2024-05-28 10:14:16 -07:00
Harshavardhana
e5c83535af chore: upgrade deps (#19819) 
Signed-off-by: Harshavardhana <harsha@minio.io>
 2024-05-28 02:27:44 -07:00
jiuker
c904ef966e feat: support tags for PostPolicy upload (#19816) 2024-05-27 21:44:00 -07:00
Minio Trusted
8f266e0772 Update yaml files to latest version RELEASE.2024-05-27T19-17-46Z 2024-05-27 23:52:43 +00:00
Harshavardhana
e0fe7cc391 fix: information disclosure bug in preconditions GET (#19810) 
precondition check was being honored before, validating
if anonymous access is allowed on the metadata of an
object, leading to metadata disclosure of the following
headers.

```
Last-Modified
Etag
x-amz-version-id
Expires:
Cache-Control:
```

although the information presented is minimal in nature,
and of opaque nature. It still simply discloses that an
object by a specific name exists or not without even having
enough permissions.
RELEASE.2024-05-27T19-17-46Z 		2024-05-27 12:17:46 -07:00
Harshavardhana
9d20dec56a Revert "remove dataErrs from er.deleteIfDangling code" 
This reverts commit 7d75b1e758.

This fails multipart tests we need this code to handle
existing challenges, so wait for the comprehensive fix.
 2024-05-26 11:13:29 -07:00
Harshavardhana
597a785253 fix: authenticate LDAP via actual DN instead of normalized DN (#19805) 
fix: authenticate LDAP via actual DN instead of normalized DN

Normalized DN is only for internal representation, not for
external communication, any communication to LDAP must be
based on actual user DN. LDAP servers do not understand
normalized DN.

fixes #19757
 2024-05-25 06:43:06 -07:00
Harshavardhana
7d75b1e758 remove dataErrs from er.deleteIfDangling code 
avoid this until a comprehensive change is
merged such as https://github.com/minio/minio/pull/19797
 2024-05-24 18:20:04 -07:00
Aditya Manthramurthy
5f78691fcf ldap: Add user DN attributes list config param (#19758) 
This change uses the updated ldap library in minio/pkg (bumped
up to v3). A new config parameter is added for LDAP configuration to
specify extra user attributes to load from the LDAP server and to store
them as additional claims for the user.

A test is added in sts_handlers.go that shows how to access the LDAP
attributes as a claim.

This is in preparation for adding SSH pubkey authentication to MinIO's SFTP
integration.
 2024-05-24 16:05:23 -07:00
Shireesh Anjal
a591e06ae5 Add cluster scanner metrics in metrics-v3 (#19517) 
endpoint: /minio/metrics/v3/cluster/scanner
metrics:
 - bucket_scans_finished (counter)
 - bucket_scans_started (counter)
 - directories_scanned (counter)
 - last_activity_nano_seconds (gauge)
 - objects_scanned (counter)
 - versions_scanned (counter)
 2024-05-24 12:29:25 -07:00
Harshavardhana
443c93c634 compute time spent in ILM properly (#19806) 2024-05-24 12:28:51 -07:00
Shireesh Anjal
5659cddc84 Add cluster config metrics in metrics-v3 (#19507) 
endpoint: /minio/metrics/v3/cluster/config
metrics:
- write_quorum
- rrs_parity
- standard_parity
 2024-05-24 05:50:46 -07:00
Shireesh Anjal
2a03a34bde Upgrade madmin-go to v3.0.52 (#19798) 
This will ensure that content of /proc/cmdline from each server is
captured in the health report.
 2024-05-24 05:34:57 -07:00
Shubhendu
1654a9b7e6 Use point in time values for gauge metrics in graphs (#19690) 
Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
 2024-05-24 04:11:51 -07:00
Shireesh Anjal
673a521711 Change endpoint of v3 notification metrics (#19804) 
from /cluster/notification to /notification
 2024-05-24 04:10:24 -07:00
Harshavardhana
2e23076688 move windows runners to in-house (#19800) 
GitHub CI runners for windows have gotten very slow,
moving them to our own hosted runners
 2024-05-23 15:29:33 -07:00
Klaus Post
b92ac55250 Add multipart combination to xl-meta (#19780) 
Add combination of multiple parts.

Parts will be reconstructed and saved separately and can manually be combined to the complete object.

Parts will be named `(version_id)-(filename).(partnum).(in)complete`.
 2024-05-23 09:37:31 -07:00