MyFavMirrors/minio
1
0
Fork 0
mirror of https://github.com/minio/minio.git synced 2024-12-26 07:05:55 -05:00
Code Issues Packages Projects Releases Wiki Activity
12,033 Commits 4 Branches 506 Tags 189 MiB
2825294b7b
Commit Graph

6168 Commits

Author SHA1 Message Date
Harshavardhana
2825294b7b
allow server startup to come online with READ success (#19957) 2024-06-19 22:21:31 -07:00
Sveinn
bce93b5cfa
Removing timeout on shutdown (#19956) 2024-06-19 11:42:47 -07:00
Harshavardhana
7a4b250c8b
avoid waiting for quorum health while debugging (#19955) 2024-06-19 10:12:20 -07:00
Harshavardhana
69e41f87ef
compute localIPs only once per server startup() (#19951) 
repeatedly calling this function is not necessary,
on systems with lots of interfaces, including virtual
ones can make this reasonably delayed.
 2024-06-19 07:34:00 -07:00
Harshavardhana
ee48f9f206
perform healthchecks before initializing everything fully (#19953) 
adds more informative logs that provide details on which
erasure set is losing quorum etc.
 2024-06-19 07:33:40 -07:00
Sveinn
9ba39d7fad
Removing a channel that was not being used (#19948) 2024-06-19 01:59:39 -07:00
Harshavardhana
d2fb371f80
do not need response record body (#19949) 
since the connection is active, the
response recorder body can grow endlessly
causing leak, as this bytes buffer is
never given back to GC due to an goroutine.
 2024-06-19 01:59:21 -07:00
Klaus Post
2f9018f03b
Do regular checks for healing status while scanning (#19946) 2024-06-18 09:11:04 -07:00
Harshavardhana
bbb64eaade
skip healing properly in the scanner when a drive is hotplugged (#19939) 
skip healing properly in scanner when drive is hotplugged

due to how the state is passed around the SkipHealing
might not be the true state() of the system always, causing
a situation where we might healing from the scanner on the
same drive which is being. Due to this competing heals get
triggered that slow each other down.
 2024-06-17 16:39:11 -07:00
Harshavardhana
7bd1d899bc
remove overzealous check during HEAD() (#19940) 
due to a historic bug in CopyObject() where
an inlined object loses its metadata, the
check causes an incorrect fallback verifying
data-dir.

CopyObject() bug was fixed in ffa91f9794 however
the occurrence of this problem is historic, so
the aforementioned check is stretching too much.

Bonus: simplify fileInfoRaw() to read xl.json as well,
also recreate buckets properly.
 2024-06-17 07:29:18 -07:00
Harshavardhana
c91d1ec2e3
fix: avoid metadata cache without data for all callers (#19935) 2024-06-14 06:28:35 -07:00
Shubhendu
3bd3470d0b
Corrected names of node replication metrics (#19932) 
Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
 2024-06-13 15:26:54 -07:00
Harshavardhana
ba39ed9af7
loadUser() if not able to load() credential return error (#19931) 2024-06-13 15:26:38 -07:00
jiuker
62e6dc950d
fix: do not update metadata cache upon headObject() (#19929) 2024-06-13 08:42:02 -07:00
Klaus Post
ad04afe381
Fix SSEC multipart checksum replication (#19915) 
* Multipart SSEC checksums were not transferred.
* Remove key mismatch logging. This key is user-controlled with SSEC.
* If the source is SSEC and the destination reports ErrSSEEncryptedObject, 
  assume replication is good.
 2024-06-12 23:56:12 -07:00
Harshavardhana
d06b63d056
load credential for in-flights requests as singleflight (#19920) 
avoid concurrent callers for LoadUser() to even initiate
object read() requests, if an on-going operation is in progress.

this avoids many callers hitting the drives causing I/O
spikes, also allows for loading credentials faster.
 2024-06-12 13:47:56 -07:00
Harshavardhana
e3ac4035b9
decrement requests inqueue correctly after the request is processed (#19918) 2024-06-12 01:13:12 -07:00
Harshavardhana
d21b6daa49
fix: avoid crash when delete() returns an error in batch expiration (#19909) 2024-06-11 06:50:53 -07:00
Harshavardhana
55aa431578
fix: on windows avoid ':' as part of the object name (#19907) 
fixes #18865
avoid-colon
 2024-06-10 20:13:30 -07:00
Harshavardhana
614981e566
allow purge expired STS while loading credentials (#19905) 
the reason for this is to avoid STS mappings to be
purged without a successful load of other policies,
and all the credentials only loaded successfully
are properly handled.

This also avoids unnecessary cache store which was
implemented earlier for optimization.
 2024-06-10 11:45:50 -07:00
Klaus Post
d2eed44c78
Fix replication checksum transfer (#19906) 
Compression will be disabled by default if SSE-C is specified. So we can still honor SSE-C.
 2024-06-10 10:40:33 -07:00
Anis Eleuch
789cbc6fb2
heal: Dangling check to evaluate object parts separately (#19797) 2024-06-10 08:51:27 -07:00
jiuker
0662c90b5c
fix: copyObject restore with a specific version, update test cases (#19895) 2024-06-10 08:50:49 -07:00
Klaus Post
a2cab02554
Fix SSE-C checksums (#19896) 
Compression will be disabled by default if SSE-C is specified. So we can still honor SSE-C.
 2024-06-10 08:31:51 -07:00
Harshavardhana
6c7a21df6b
turn-off unexpected debug logging in List() calls (#19903) 2024-06-09 21:34:26 -07:00
Harshavardhana
29a25a538f
fix: make sure we list freeVersions like DEL marker with --versions (#19878) 
freeVersions() was being incorrectly skipped; list it as
valid objects properly.

Co-authored-by: Krishnan Parthasarathi <Krishnan Parthasarathi>
 2024-06-07 15:18:44 -07:00
Harshavardhana
2dd8faaedc remove unnecessary log in Listing() 2024-06-07 14:52:55 -07:00
Krishnan Parthasarathi
069c4015cd
Don't tier directory objects (#19891) 
Directory objects are used by applications that simulate the folder
structure of an on-disk filesystem. These are zero-byte objects with names
ending with '/'. They are only used to check whether a 'folder' exists in
the namespace.
 2024-06-07 08:43:17 -07:00
Shubhendu
2f6e03fb60
Calculate correct object size while replication (#19888) 
It was missing in case of `replicateObject` but was present for
`replicateAll` already

Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
 2024-06-06 12:31:01 -07:00
Klaus Post
0fbb945e13
Disable caching of encrypted objects (#19890) 
Don't write encrypted objects to cache, if configured.
 2024-06-06 11:39:18 -07:00
Anis Eleuch
b94dd835c9
decom: Fix CurrentSize output when generating the status (#19883) 
StartSize starts with the raw free space of all disks in the given pool,
however during the status, CurrentSize is not showing the current free
raw space, as expected at least by `mc admin decom status` since it was
written.
 2024-06-06 07:30:43 -07:00
Poorna
5aaef9790f
replication: pass checksum headers to replica (#19834) 2024-06-06 02:36:42 -07:00
Bala FA
7edc352d23
Add ILM metrics in metrics-v3 (#19539) 
Signed-off-by: Bala.FA <bala@minio.io>
 2024-06-06 02:36:25 -07:00
Poorna
850a84b08a
simplify site replication multipart proxying (#19885) 2024-06-05 18:01:15 -07:00
Taran Pelkey
4148754ce0
Check both given and normalized group DN on LDAP policy detach requests (#19876) 2024-06-05 15:42:40 -07:00
Harshavardhana
2107722829
upgrade go-oidc to fix GO-2024-2631 (#19884) 2024-06-05 15:00:34 -07:00
jiuker
d326ba52e9
feat: support batchJob for windows (#19877) 2024-06-05 08:44:53 -07:00
Sveinn
91e1487de4
Add LDAP public key authentication to SFTP (#19833) 2024-06-05 00:51:13 -07:00
jiuker
90a9f2dd70
fix: log diskerror when detect the disk space failed (#19861) 2024-06-04 09:42:03 -07:00
Harshavardhana
d5e48cfd65
fix: remove DriveOPTimeout for REST callers as they don't work properly (#19873) 
Go's net/http is notoriously difficult to have a streaming
deadlines per READ/WRITE on the net.Conn if we add them they
interfere with the Go's internal requirements for a HTTP
connection.

Remove this support for now

fixes #19853
 2024-06-04 08:12:57 -07:00
Anis Eleuch
d274566463
race: Fix rare race detected by testing (#19872) 
Below is the race warning:

```
WARNING: DATA RACE
Write at 0x00c02d3d27c0 by goroutine 1210:
  github.com/minio/minio/cmd.(*healingTracker).bucketDone()
      github.com/minio/minio/cmd/background-newdisks-heal-ops.go:273 +0x13a
  github.com/minio/minio/cmd.(*erasureObjects).healErasureSet()
      github.com/minio/minio/cmd/global-heal.go:525 +0x2158
  github.com/minio/minio/cmd.healFreshDisk()
      github.com/minio/minio/cmd/background-newdisks-heal-ops.go:450 +0x107e
  github.com/minio/minio/cmd.monitorLocalDisksAndHeal.func1()
      github.com/minio/minio/cmd/background-newdisks-heal-ops.go:528 +0x150
  github.com/minio/minio/cmd.monitorLocalDisksAndHeal.gowrap2()
      github.com/minio/minio/cmd/background-newdisks-heal-ops.go:538 +0x82

Previous read at 0x00c02d3d27c0 by goroutine 1446:
  github.com/minio/minio/cmd.(*erasureObjects).healErasureSet.func5()
      github.com/minio/minio/cmd/global-heal.go:232 +0xfd
```
 2024-06-04 08:12:32 -07:00
Shubhendu
21b6204692
Test proxying of DEL marker for bucket replication (#19870) 
Make sure to avoid proxying for DEL markers

Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
 2024-06-04 04:38:26 -07:00
Taran Pelkey
d98faeb26a
Check if LDAP User has attached policy before creating Service Account (#19843) 
Check if ldap user has policy before creating
 2024-06-03 12:58:48 -07:00
Klaus Post
0a63dc199c
Add trace sizes to more trace types (#19864) 
Add trace sizes to

* ILM traces
* Replication traces
* Healing traces
* Decommission traces
* Rebalance traces
* (s)ftp traces
* http traces.
 2024-06-03 08:45:54 -07:00
Klaus Post
e72429c79c
Add sizes to traces (#19851) 
added to storage and grid traces. Can provide more context for traces that aren't HTTP. Others may apply.
 2024-05-31 22:17:37 -07:00
Klaus Post
c5b3f5553f
Add per connection RPC metrics (#19852) 
Provides individual and aggregate stats for each RPC connection.

Example:

```
  "rpc": {
   "collectedAt": "2024-05-31T14:33:29.1373103+02:00",
   "connected": 30,
   "disconnected": 0,
   "outgoingStreams": 69,
   "incomingStreams": 0,
   "outgoingBytes": 174822796,
   "incomingBytes": 175821566,
   "outgoingMessages": 768595,
   "incomingMessages": 768589,
   "outQueue": 0,
   "lastPongTime": "2024-05-31T12:33:28Z",
   "byDestination": {
    "http://127.0.0.1:9001": {
     "collectedAt": "2024-05-31T14:33:29.1373103+02:00",
     "connected": 5,
     "disconnected": 0,
     "outgoingStreams": 2,
     "incomingStreams": 0,
     "outgoingBytes": 38432543,
     "incomingBytes": 66604052,
     "outgoingMessages": 229496,
     "incomingMessages": 229575,
     "outQueue": 0,
     "lastPongTime": "2024-05-31T12:33:27Z"
    },
    "http://127.0.0.1:9002": {
     "collectedAt": "2024-05-31T14:33:29.1373103+02:00",
     "connected": 5,
     "disconnected": 0,
     "outgoingStreams": 6,
     "incomingStreams": 0,
     "outgoingBytes": 38215680,
     "incomingBytes": 66121283,
     "outgoingMessages": 228525,
     "incomingMessages": 228510,
     "outQueue": 0,
     "lastPongTime": "2024-05-31T12:33:27Z"
    },
...
```
 2024-05-31 22:16:24 -07:00
Klaus Post
d3ae0aaad3
Add max buffering to SFTP (#19848) 
Prevent OOM by adversarial use of SFTP upload by setting a 100MB max upload buffer.
 2024-05-31 14:28:07 -07:00
Anis Eleuch
1277ad69a6
heal: Remove .healing.bin when all ES drives are healing (#19846) 
In the very rare case when all drives in a erasure set need to be healed,
remove .healing.bin from all drives, otherwise it will be stuck in a
loop

Also, fix a unit test that fails sometimes due to wrong test.
 2024-05-31 07:48:50 -07:00
Harshavardhana
8f93e81afb
change service account embedded policy size limit (#19840) 
Bonus: trim-off all the unnecessary spaces to allow
for real 2048 characters in policies for STS handlers
and re-use the code in all STS handlers.
 2024-05-30 11:10:41 -07:00
Harshavardhana
4af31e654b
avoid pre-populating buffers for deployments < 32GiB memory (#19839) 2024-05-30 04:58:12 -07:00