MyFavMirrors
/
minio
mirror of https://github.com/minio/minio.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
12,187 Commits 5 Branches 503 Tags 186 MiB
Commit Graph

6275 Commits

Author SHA1 Message Date
Aditya Manthramurthy 4bda4e4e2b
fix: check for disk-level O_DIRECT support (#18173) 
Disk level O_DIRECT support checking at xl storage initialization was
conditional on a config setting being enabled. (This never took effect
because config initialization happens after ObjectLayer is ready.) This
is not necessary as the config setting is dynamic - O_DIRECT should be
enabled via runtime config. So we need to do the disk level support
check regardless of the config setting.
 2023-10-05 20:54:49 -06:00
Harshavardhana 1971c54a50
update buffer channels for both trace and listen events (#18171) 
- Trace needs higher buffered channels than 4000 to ensure
  when we run `mc admin trace -a` it captures all information
  sufficiently.

- Listen event notification needs the event channel to be
  `apiRequestsMaxPerNode` * number of nodes
 2023-10-05 18:16:04 -06:00
Anis Eleuch b336e9a79f
fix: loading usage cache to not fail early when reading the backup fails (#18158) 
Currently, the retry is not fully used when there is no backup copy of
the data usage; use 5 retry attempts when we don't have any valid data, 
new or backup, unless we have seen an un-recognized error.
 2023-10-02 19:22:35 -07:00
Harshavardhana a2ab21e91c
add max-keys=2 optimization for spark workloads (#18154) 
comment in the code provides more detailed explanation
on what this PR entails and its assumptions.

this PR reduces the amount of listing() by an order
of magnitude, however there are other such calls that
still needs further optimization that shall be done
in subsequent PRs.
 2023-10-02 07:52:59 -06:00
Sveinn 603437e70f
Fix startup formatting (#18156) 
Percentages in root user names are used for formatting.

Before:
```
S3-API: http://192.168.50.21:9000  http://172.31.96.1:9000  http://127.0.0.1:9000
RootUser: "U4B6Zi!b75DXSPm%!!(MISSING)a(MISSING)vZb"
RootPass: "Q4#Q6y8G%!P(MISSING)x#npP4dudUobU#NBcGB7RMKV4ajYb"

Console: http://192.168.50.21:51915 http://172.31.96.1:51915 http://127.0.0.1:51915
RootUser: "U4B6Zi!b75DXSPm%!!(MISSING)a(MISSING)vZb"
RootPass: "Q4#Q6y8G%!P(MISSING)x#npP4dudUobU#NBcGB7RMKV4ajYb"

Command-line: https://min.io/docs/minio/linux/reference/minio-mc.html#quickstart
FORMAT: %117s MESSAGE: $ mc alias set myminio http://192.168.50.21:9000 "U4B6Zi!b75DXSPm%avZb" "Q4#Q6y8G%%Px#npP4dudUobU#NBcGB7RMKV4ajYb"
   $ mc alias set myminio http://192.168.50.21:9000 "U4B6Zi!b75DXSPm%!a(MISSING)vZb" "Q4#Q6y8G%Px#npP4dudUobU#NBcGB7RMKV4ajYb"
```

After:

```
Status:         1 Online, 0 Offline.
S3-API: http://192.168.50.21:9000  http://172.31.96.1:9000  http://127.0.0.1:9000
RootUser: "U4B6Zi!b75DXSPm%avZb"
RootPass: "Q4#Q6y8G%%Px#npP4dudUobU#NBcGB7RMKV4ajYb"

Console: http://192.168.50.21:52421 http://172.31.96.1:52421 http://127.0.0.1:52421
RootUser: "U4B6Zi!b75DXSPm%avZb"
RootPass: "Q4#Q6y8G%%Px#npP4dudUobU#NBcGB7RMKV4ajYb"

Command-line: https://min.io/docs/minio/linux/reference/minio-mc.html#quickstart
   $ mc alias set myminio http://192.168.50.21:9000 "U4B6Zi!b75DXSPm%avZb" "Q4#Q6y8G%%Px#npP4dudUobU#NBcGB7RMKV4ajYb"
```

No need for special Windows case. `mc` works just fine.
 2023-10-02 07:39:47 -06:00
Shireesh Anjal 6d20ec3bea
Add support for resource metrics (#18057) 
Add a new endpoint for "resource" metrics `/v2/metrics/resource`

This should return system metrics related to drives, network, CPU and
memory. Except for drives, other metrics should have corresponding "avg"
and "max" values also.

Reuse the real-time feature to capture the required data,
introducing CPU and memory metrics in it.

Collect the data every minute and keep updating the average and max values
accordingly, returning the latest values when the API is called.
 2023-09-30 13:40:20 -07:00
Anis Eleuch 22d2dbc4e6
decom: Fix infinite retry when the decom is canceled (#18143) 
Also, use rand.Float64() since it is thread-safe; otherwise go race
will complain.
 2023-09-30 00:02:29 -07:00
Harshavardhana d6446cb096
do not return an error in AbortMultipartUpload() (#18135) 
returning an error is a bit undefined in AWS S3
as it may return an error or not depending on the
time from AbortMultipartUpload().
 2023-09-29 10:28:19 -07:00
Harshavardhana c34bdc33fb
make sure to set Versioned field to ensure rename2 is not called (#18141) 
without this the rename2() can rename the previous dataDir
causing issues for different versions of the object, only
latest version is preserved due to this bug.

Added healing code to ensure recovery of such content.
 2023-09-29 09:08:24 -07:00
Anis Eleuch aec023f537
Avoid showing buckets without quorum in each pool (#18125) 2023-09-29 00:58:54 -07:00
Poorna e101eeeda9
fix: tier addition validation (#18136) 2023-09-28 22:33:24 -07:00
Harshavardhana 3c470a6b8b
fix: the inspect script to use scheme per deployment (#18118) 2023-09-27 08:22:50 -07:00
Poorna 6bc7d711b3
delete of a missing versionId return 204 (#18117) 2023-09-26 14:02:56 -07:00
Harshavardhana cdeab19673
fix: always check error upon w.Close() in Write() (#18111) 
not checking w.Close() can prematurely make us
think that the w.Write() actually succeeded, apparently
Write() may or may not return an error but sometimes
only during a Close() call to the fd we may see the
error from Write() propagate.

Fdatasync(w) on the FD would return an error requiring
Close() error handling is less of a concern, however it may
happen such that fdatasync() did not return an error, where
as Close() would.
 2023-09-26 11:04:00 -07:00
Anis Eleuch 22ee678136
tier: Avoid doing versioned operations since not required anymore (#18108) 
Currently, setting a new tiering target returns an error when a bucket
is versioned and the tiering credentials does not have authorization to
specify a version-id when reading or removing a specific version;

Since tiering does not require versioning anymore; avoid doing versioned
operations when performing checklist ops while adding a new tiering
configuration.
 2023-09-26 00:14:56 -07:00
Poorna 50a8f13e85
site replication: allow setting bandwidth default for bucket (#18062) 
This can still be overridden at the bucket level
 2023-09-25 15:50:52 -07:00
jiuker 6dec60b6e6
fix: check post policy like AWS S3 (#18074) 2023-09-25 12:35:25 -07:00
Harshavardhana ac3a19138a
fix: set scanning details locally to avoid cached values (#18092) 
atomic variable results such as scanning must not use
cached values, instead rely on real-time information.
 2023-09-25 08:26:29 -07:00
Klaus Post 21e8e071d7
Improve ListObject Compatibility (#18099) 
Do not error out when a provided marker is before or after the prefix, but instead just ignore it if before and return an empty list when after.

Fixes #18093
 2023-09-25 08:13:08 -07:00
Klaus Post 57f84a8b4c
Add abandoned folder scanning to metrics (#18076) 
Include object and versions heal scan times when checking non-empty abandoned folders.

Furthermore don't add delay between healing versions, instead do one per object wait.
 2023-09-24 22:15:31 -07:00
Aditya Manthramurthy 22041bbcc4
fix: Update policy mapping properly in notification (#18088) 
This is fixing a regression from an earlier change where STS account
loading was made lazy.
 2023-09-22 20:47:50 -07:00
Harshavardhana 91ebac0a00
fix: move abandoned parts check after healing not in ILM path (#18087) 2023-09-22 12:07:52 -07:00
Harshavardhana 3a90fb108c only look for metadata if batch replication asks for metadata filters (#18082) 
This PR changes the StatObject() to be must have for non-minio source
to being a conditional API call.

- Calls StatObject() when needed
- Calls GetObjectTagging() when needed

These calls if we do without these conditionals can cause a lot of
delays, so we avoid them if not needed in more common scenario.
 2023-09-22 11:31:57 -07:00
Shubhendu 74cfb207c1
Added check for mandatory MINIO_KMS_KES_KEY_NAME env var (#18077) 
If MinIO started with KMS enabled, MINIO_KMS_KES_KEY_NAME should
be set for server to start.

Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
 2023-09-21 10:37:37 -07:00
Harshavardhana 9788d85ea3
remove logging for invalid metadata values (#18068) 2023-09-20 15:49:55 -07:00
Anis Eleuch 69c0e18685
perf net: Add the endpoint name related to the perf net error (#18063) 
In a perf test, one node will run speed test with all nodes. If there is
an error with a peer node, the peer node name is not included in the
error hence confusing the user.

This commit will add the peer endpoint string to the netperf error.
 2023-09-19 22:41:06 -07:00
Aditya Manthramurthy 3cac927348
Load STS policy mappings periodically (#18061) 
To ensure that policy mappings are current for service accounts
belonging to (non-derived) STS accounts (like an LDAP user's service
account) we periodically reload such mappings.

This is primarily to handle a case where a policy mapping update
notification is missed by a minio node. Such a node would continue to
have the stale mapping in memory because STS creds/mappings were never
periodically scanned from storage.
 2023-09-19 17:57:42 -07:00
Harshavardhana 9081346c40 fix: more regressions listing policy mappings (#18060) 
also relax ListServiceAccounts() returning error if
no service accounts exist.
 2023-09-19 15:23:18 -07:00
Harshavardhana fcfadb0e51
fix: regression in loading LDAP users policy mappings (#18055) 
LDAP users are stored as STS users, we need to load
their policy mappings appropriately.

Fixes a regression caused by #17994
 2023-09-19 10:31:56 -07:00
Harshavardhana 2add57cfed
apply healing per object at 1024 cycles (#18050) 
- we already have MRF for most recent failures
- we trigger healing during HEAD/GET operation

These are enough, also change the default max wait
from 5sec to 1sec for default scanner speed.
 2023-09-19 09:24:22 -07:00
Poorna b73699fad8
replication: pass user tags while queueing (#18052) 
Continues from #18032 - otherwise replication will fail on tag based rules.
 2023-09-19 03:18:28 -07:00
Harshavardhana b8ebe54e53 Revert "skip tiered objects to GLACIER in batch replication (#18044)" 
This reverts commit fd421ddd6f.

MinIO already provides `filter` based on metadata that would work
in this scenario already.
 2023-09-19 00:05:40 -07:00
Harshavardhana c3d70e0795
cache usage, prefix-usage, and buckets for AccountInfo up to 10 secs (#18051) 
AccountInfo is quite frequently called by the Console UI 
login attempts, when many users are logging in it is important
that we provide them with better responsiveness.

- ListBuckets information is cached every second
- Bucket usage info is cached for up to 10 seconds
- Prefix usage (optional) info is cached for up to 10 secs

Failure to update after cache expiration, would still
allow login which would end up providing information
previously cached.

This allows for seamless responsiveness for the Console UI
logins, and overall responsiveness on a heavily loaded
system.
 2023-09-18 22:13:03 -07:00
Harshavardhana fd421ddd6f
skip tiered objects to GLACIER in batch replication (#18044) 
tiered objects to GLACIER are not readable until
they are restored, we skip these as unreadable
 2023-09-18 10:25:31 -07:00
jiuker 9947c01c8e
feat: SSE-KMS use uuid instead of read all data to md5. (#17958) 2023-09-18 10:00:54 -07:00
Eng Zer Jun a00db4267c
data-usage-cache: remove redundant nil check (#17970) 
From the Go specification:

  "3. If the map is nil, the number of iterations is 0." [1]

Therefore, an additional nil check for before the loop is unnecessary.

[1]: https://go.dev/ref/spec#For_range

Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
 2023-09-16 19:09:29 -07:00
Harshavardhana 36385010f5
use optimized pathJoin instead of path.Join (#18042) 
this avoids allocations in scanner routine, they are tiny but 
they allocate a lot over many cycles of the scanner.
 2023-09-16 19:08:59 -07:00
Harshavardhana fa6d082bfd
reduce all major allocations in replication path (#18032) 
- remove targetClient for passing around via replicationObjectInfo{}
- remove cloing to object info unnecessarily
- remove objectInfo from replicationObjectInfo{} (only require necessary fields)
 2023-09-16 02:28:06 -07:00
Poorna b733e6e83c
site replication turn off retry login for admin API calls (#18039) 
additionally also mark site offline if n/w is down
 2023-09-15 18:01:47 -07:00
Anis Eleuch 37aa5934a1
scanner: Fix loading data usage cache structure (#18037) 
Return an empty data usage cache structure when the data usage cache
file does not exist, otherwise, the scanner won't work.
 2023-09-15 13:11:08 -07:00
Harshavardhana 1647fc7edc
fix: optimize listMultipartUploads to serve via local disks (#18034) 
and remove unused getLoadBalancedDisks()
 2023-09-15 08:34:03 -07:00
Harshavardhana 7b92687397
remove generating presignedURLs with range header for lambda (#18033) 2023-09-14 21:58:17 -07:00
Alex dc48cd841a
Added MINIO_PROMETHEUS_AUTH_TOKEN env support (#18028) 
Signed-off-by: Benjamin Perez <benjamin@bexsoft.net>
 2023-09-14 17:28:21 -07:00
Anis Eleuch b0e1776d6d
Do not use a chain for S3 tiering to return better error messages (#18030) 
When using a chain provider all providers do not return a valid
access and secret key, an anonymous request is sent, which makes it hard
for users to figure out what is going on

In the case of S3 tiering, when AWS IAM temporary account generation returns
an error, an anonymous login will be used because of the chain provider.
Avoid this and use the AWS IAM provider directly to get a good error
message.
 2023-09-14 15:28:20 -07:00
Aditya Manthramurthy 7a7068ee47
Move IAM periodic ops to a single go routine (#18026) 
This helps reduce disk operations as these periodic routines would not
run concurrently any more.

Also add expired STS purging periodic operation: Since we do not scan
the on-disk STS credentials (and instead only load them on-demand) a
separate routine is needed to purge expired credentials from storage.
Currently this runs about a quarter as often as IAM refresh.

Also fix a bug where with etcd, STS accounts could get loaded into the
iamUsersMap instead of the iamSTSAccountsMap.
 2023-09-14 15:25:17 -07:00
Aditya Manthramurthy cbc0ef459b
Fix policy package import name (#18031) 
We do not need to rename the import of minio/pkg/v2/policy as iampolicy
any more.
 2023-09-14 14:50:16 -07:00
Harshavardhana a2aabfabd9
add backups for usage-caches to rely on upon error (#18029) 
This allows scanner to avoid lengthy scans, skip
things appropriately and also not lose metrics in
any manner.

reduce longer deadlines for usage-cache loads/saves
to match the disk timeout which is 2minutes now per
IOP.
 2023-09-14 11:53:52 -07:00
Harshavardhana 32890342ce
introduce MINIO_BROWSER_REDIRECT env to enable/disable auto-redirect (#18025) 2023-09-13 18:43:57 -07:00
Aditya Manthramurthy ed2c2a285f
Load STS accounts into IAM cache lazily (#17994) 
In situations with large number of STS credentials on disk, IAM load
time is high. To mitigate this, STS accounts will now be loaded into
memory only on demand - i.e. when the credential is used.

In each IAM cache (re)load we skip loading STS credentials and STS
policy mappings into memory. Since STS accounts only expire and cannot
be deleted, there is no risk of invalid credentials being reused,
because credential validity is checked when it is used.
 2023-09-13 12:43:46 -07:00
Poorna 18e23bafd9
replication resync: report only the on-disk status (#18017) 
Avoid reporting in-memory status since results can vary if different
nodes are queried, resync always runs at a single node.
 2023-09-13 10:58:38 -07:00
Harshavardhana 8b8be2695f
optimize mkdir calls to avoid base-dir `Mkdir` attempts (#18021) 
Currently we have IOPs of these patterns

```
[OS] os.Mkdir play.min.io:9000 /disk1 2.718µs
[OS] os.Mkdir play.min.io:9000 /disk1/data 2.406µs
[OS] os.Mkdir play.min.io:9000 /disk1/data/.minio.sys 4.068µs
[OS] os.Mkdir play.min.io:9000 /disk1/data/.minio.sys/tmp 2.843µs
[OS] os.Mkdir play.min.io:9000 /disk1/data/.minio.sys/tmp/d89c8ceb-f8d1-4cc6-b483-280f87c4719f 20.152µs
```

It can be seen that we can save quite Nx levels such as
if your drive is mounted at `/disk1/minio` you can simply
skip sending an `Mkdir /disk1/` and `Mkdir /disk1/minio`.

Since they are expected to exist already, this PR adds a way
for us to ignore all paths upto the mount or a directory which
ever has been provided to MinIO setup.
 2023-09-13 08:14:36 -07:00
Poorna 96fbf18201
replication: queue existing objects to same workers as incoming (#18020) 
Previously existing objects were queued to single worker and MRF re-queues
are also handled by same worker - this does not fully use the available
bandwidth in case there is no incoming workload.
 2023-09-12 21:59:15 -07:00
Harshavardhana c8a57a8fa2
fix: send content-md5 for AWS S3 proactively (#18018) 
fixes #17977
 2023-09-12 19:11:13 -07:00
Harshavardhana b1c2dacab3
fix: allow dynamic ports for API only in non-distributed setups (#18019) 
fixes #17998
 2023-09-12 19:10:49 -07:00
Harshavardhana 08b3a466e8
fix: allow concurrent SFTP connections (#18013) 
current implementation did not fully implement
the concurrent SFTP connection implementation,
this PR properly handles this.

fixes #17914
 2023-09-12 12:41:52 -07:00
Harshavardhana 1df5e31706
optimize MRF replication queue to avoid memory leaks (#18007) 2023-09-11 20:59:11 -07:00
Harshavardhana 9f7044aed0
fix: ignore transient errors in read path (#18006) 
Errors such as

```
returned an error (context deadline exceeded) (*fmt.wrapError)
```

```
(msgp: too few bytes left to read object) (*fmt.wrapError)
```
 2023-09-11 15:29:59 -07:00
Anis Eleuch 41de53996b
heal: calculate the number of workers based on NRRequests (#17945) 2023-09-11 14:48:54 -07:00
Harshavardhana 9878031cfd
fix: change DISK_ to DRIVE_ for some drive related envs (#18005) 2023-09-11 12:19:22 -07:00
Poorna 703ed46d79
fix: replication of tags while removing (#17989) 
A tag removal was not being replicated prior to this change
 2023-09-06 19:05:02 -07:00
Harshavardhana f7ca6c63c2
fix: bucket quota clear and honor existing quota config (#17988) 2023-09-06 19:03:58 -07:00
Harshavardhana ad69b9907f
fix: report bucket metrics for only existing buckets (#17987) 2023-09-06 12:50:46 -07:00
Shubhendu bfddbb8b40
Embed file in ZIP with custom permissions (#17954) 
This change enables embedding files in ZIP with custom permissions.
Also uses default creds for starting MinIO based on inspect data.

Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
 2023-09-06 09:24:01 -07:00
Poorna 13a2dc8485
replication resync: avoid blocking on results channel. (#17981) 
continues fix in #17775
 2023-09-05 20:22:39 -07:00
Harshavardhana 1e51424e8a
use syscall.Rename() directly instead of os.Rename() (#17982) 2023-09-05 20:22:23 -07:00
Harshavardhana 5b114b43f7
refactor bandwidth throttling for replication target (#17980) 
This refactor is to allow using the bandwidth throttling
for other purposes.
 2023-09-05 20:21:59 -07:00
Poorna 812f5a02d7
metrics: fix panic in replication stats reporting (#17979) 2023-09-05 10:26:18 -07:00
Aditya Manthramurthy 1c99fb106c
Update to minio/pkg/v2 (#17967) 2023-09-04 12:57:37 -07:00
Krishnan Parthasarathi 71c32e9b48
Return successorModTime in quorum when available (#17925) 2023-09-04 08:24:17 -07:00
Harshavardhana 380a59520b add missing testdata for benchmarking 2023-09-02 14:40:38 -07:00
Harshavardhana 3995355150
avoid repeated large allocations for large parts (#17968) 
objects with 10,000 parts and many of them can
cause a large memory spike which can potentially
lead to OOM due to lack of GC.

with previous PR reducing the memory usage significantly
in #17963, this PR reduces this further by 80% under
repeated calls.

Scanner sub-system has no use for the slice of Parts(),
it is better left empty.

```
benchmark                            old ns/op     new ns/op     delta
BenchmarkToFileInfo/ToFileInfo-8     295658        188143        -36.36%

benchmark                            old allocs     new allocs     delta
BenchmarkToFileInfo/ToFileInfo-8     61             60             -1.64%

benchmark                            old bytes     new bytes     delta
BenchmarkToFileInfo/ToFileInfo-8     1097210       227255        -79.29%
```
 2023-09-02 07:49:24 -07:00
Harshavardhana 8208bcb896
remove all unnecessary logging, logOnce when absolutely needed (#17965) 2023-09-01 16:19:18 -07:00
Poorna d665e855de
replication: remove check for empty version id (#17964) 2023-09-01 13:46:10 -07:00
Harshavardhana 18b3655c99
with xlv2 format we never had to fill in checksumInfo() (#17963) 
- this PR avoids sending a large ChecksumInfo slice
  when its not needed

- also for a file with XLV2 format there is no reason
  to allocate Checksum slice while reading
 2023-09-01 13:45:58 -07:00
Anis Eleuch 6a8d8f34a5
kafka: Do not require key when sending a message (#17962) 
Keys are helpful to ensure the strict ordering of messages, however currently the
code uses a random request id for every log, hence using the request-id
as a Kafka key is not serve any purpose;

This commit removes the usage of the key, to also fix the audit issue from
internal subsystem that does not have a request ID.
 2023-09-01 08:37:22 -07:00
Harshavardhana b1c1f02132
use buffers for pathJoin, to re-use buffers. (#17960) 
```
benchmark                        old ns/op     new ns/op     delta
BenchmarkPathJoin/PathJoin-8     79.6          55.3          -30.53%

benchmark                        old allocs     new allocs     delta
BenchmarkPathJoin/PathJoin-8     2              1              -50.00%

benchmark                        old bytes     new bytes     delta
BenchmarkPathJoin/PathJoin-8     48            24            -50.00%
```
 2023-08-31 17:58:48 -07:00
yangw b13fcaf666
fix: read atomic variable in clientDevNull round trip time (#17955) 2023-08-31 08:31:01 -07:00
Harshavardhana 9458485e43
avoid double logging from healing (#17950) 2023-08-30 18:46:04 -07:00
Poorna b48bbe08b2
Add additional info for replication metrics API (#17293) 
to track the replication transfer rate across different nodes,
number of active workers in use and in-queue stats to get
an idea of the current workload.

This PR also adds replication metrics to the site replication
status API. For site replication, prometheus metrics are
no longer at the bucket level - but at the cluster level.

Add prometheus metric to track credential errors since uptime
 2023-08-30 01:00:59 -07:00
Krishnan Parthasarathi 6a67c277eb
Reuse types for key-value, notification and retry (#17936) 2023-08-29 11:27:23 -07:00
Harshavardhana 7cafdc0512
fix: skip access checks further for known buckets (#17934) 2023-08-28 15:16:41 -07:00
Harshavardhana 8a57b6bced
use renameat2 Linux extension syscall (#17757) 
this is a faster and safer alternative
on newer kernel versions.
 2023-08-27 09:57:11 -07:00
Krishnan Parthasarathi 53abd25116
Don't log when object to be tiered is not found (#17924) 2023-08-25 23:34:16 -07:00
Harshavardhana 1ea7826c0e
do not have to consider replicationTimestamp for healing and quorum (#17922) 
replicationTimestamp might differ if there were retries
in replication and the retried attempt overwrote in
quorum but enough shards with newer timestamp causing
the existing timestamps on xl.meta to be invalid, we
do not rely on this value for anything external.

this is purely a hint for debugging purposes, but there
is no real value in it considering the object itself
is in-tact we do not have to spend time healing this
situation.

we may consider healing this situation in future but
that needs to be decoupled to make sure that we do not
over calculate how much we have to heal.
 2023-08-25 15:31:15 -07:00
Anis Eleuch 0cde37be50
Reduce the number of calls to import bucket metadata (#17899) 
For each bucket, save the bucket metadata 
once, call the site replication hook once
 2023-08-25 07:59:16 -07:00
jiuker 6aeca54ece
fix: replace context by timeout-context from parent-context when `selfSpeedTest` (#17906) 2023-08-25 07:58:38 -07:00
Harshavardhana 124e28578c
remove strict persistence requirements for List() .metacache objects (#17917) 
.metacache objects are transient in nature, and are better left to
use page-cache effectively to avoid using more IOPs on the disks.

this allows for incoming calls to be not taxed heavily due to
multiple large batch listings.
 2023-08-25 07:58:11 -07:00
Harshavardhana 62c9e500de
remove mTime requirement from pre-condition checks (#17916) 
given a versionId the mtime is always the same, it
can never be different than its original value.

versionIds also do not conflict, since they are uuid's
and unique practically forever.
 2023-08-24 14:33:58 -07:00
jiuker 02cc18ff29 refactor the perf client for TTFB and TotalResponseTime (#17901) 2023-08-24 10:21:08 -07:00
Harshavardhana ba4566e86d
add missing IAM node metrics to cluster and node endpoint (#17908) 2023-08-24 09:26:37 -07:00
Krishnan Parthasarathi 87cb0081ec
Retain current and upto NewerNoncurrentVersions versions (#17909) 
applyNewerNoncurrentVersionLimit method should pass along versions
unaffected by NewerNoncurrentVersions rule for further ILM evaluation.
 2023-08-24 09:26:29 -07:00
Poorna 4a6af93c83
mark replication target offline if network timeouts seen (#17907) 
regular target liveness check every 5 secs will toggle state back
as target returns online.
 2023-08-24 09:24:26 -07:00
Harshavardhana af564b8ba0
allow bootstrap to capture time-spent for each initializers (#17900) 2023-08-23 03:07:06 -07:00
Klaus Post 7c8746732b
Return cancelled storage calls as 499 (#17895) 
Make upstream cancels more visible - right now they are just reported as "forbidden".
 2023-08-22 11:10:41 -07:00
Klaus Post f506117edb
Reduce memory profiling rate (#17894) 
Change profiling from every 4KB to every 128K, reducing the lock contention by a factor of 32.
 2023-08-22 07:21:49 -07:00
Harshavardhana 1c5af7c31a
serialize queueMRFHeal(), add timeouts and avoid normal build-ups (#17886) 
we expect a certain level of IOPs and latency so this is okay.

fixes other miscellaneous bugs

- such as hanging on mrfCh <- when the context is canceled
- queuing MRF heal when the context is canceled
- remove unused saveStateCh channel
 2023-08-21 16:44:50 -07:00
Harshavardhana 3a0125fa1f
remove unexpected logging from peer calls (#17888) 
also make sure RequestID is set for system logs
 2023-08-21 14:25:24 -07:00
Daniel Valdivia 328cb0a076
Pass environment variable to control session length to console (#17885) 
Signed-off-by: Daniel Valdivia <18384552+dvaldivia@users.noreply.github.com>
 2023-08-21 11:55:43 -07:00
jiuker e3ea97c964
fix: replace req context by locker context (#17880) 2023-08-19 22:09:07 -07:00
Andreas Auernhammer 8f8f8854f0
update `minio/kes-go` dep to v0.2.0 (#17850) 
This commit updates the minio/kes-go dependency
to v0.2.0 and updates the existing code to work
with the new KES APIs.

The `SetPolicy` handler got removed since it
may not get implemented by KES at all and could
not have been used in the past since stateless KES
is read-only w.r.t. policies and identities.

Signed-off-by: Andreas Auernhammer <hi@aead.dev>
 2023-08-19 07:37:53 -07:00
Anis Eleuch 4c6869cd9a
ilm: Fix cleaning non current null versions (#17876) 2023-08-18 12:55:47 -07:00
Harshavardhana dde1a12819
fix: validate incoming uploadID to be base64 encoded (#17865) 
Bonus fixes include

- do not have to write final xl.meta (renameData) does this
  already, saves some IOPs.

- make sure to purge the multipart directory properly using
  a recursive delete, otherwise this can easily pile up and
  rely on the stale uploads cleanup.

fixes #17863
 2023-08-17 09:37:55 -07:00
Harshavardhana 9ebd10d3f4
Revert "Include SuccessorModTime for FileInfo quorum (#17732)" (#17860) 
This reverts commit bf3901342c.

This is to fix a regression caused when there are inconsistent
versions, but one version is in quorum. SuccessorModTime issue
must be fixed differently.
 2023-08-16 07:51:33 -07:00
Harshavardhana 3ba927edae
fix: batch status reporting after complete (#17852) 
batch status can perpetually wait after completion
due to a race between the MetricsHandler() returning
the active metrics in intervals of 1sec and delete
of metrics after job completion.

this PR ensures that we keep the 'status' around
for a while, i.e upto 24hrs for all the batch jobs.
 2023-08-15 12:22:30 -07:00
Harshavardhana c4ca0a5a57
add two more drive metrics when metrics is available (#17854) 2023-08-15 10:55:47 -07:00
Klaus Post 406ea4f281
Fix distributed listing not able to resume (#17855) 
Two fields in lifecycles made GOB encoding consistently fail with `gob: type lifecycle.Prefix has no exported fields`.

This meant that in distributed systems listings would never be able to continue and would restart on every call.

Fix issues and be sure to log these errors at least once per bucket. We may see some connectivity errors here, but we shouldn't hide them.
 2023-08-15 07:45:25 -07:00
Harshavardhana 64aa7feabd
allow specifying lower disks for Walk() (#17829) 
useful when you may want Walk() with
reduced quorum requirements.
 2023-08-14 21:32:39 -07:00
Poorna 875f4076ec
site replication: avoid retries when peer is offline (#17853) 2023-08-14 21:31:41 -07:00
Harshavardhana 4643efe6be
fix: add deadline worker pattern for local disk removers (#17845) 2023-08-14 12:28:13 -07:00
Harshavardhana b760137e1d
fix: add proxyByNode for batch jobs as part of their jobId (#17844) 2023-08-11 13:12:35 -07:00
Harshavardhana 5f56f441bf
fix: apply common notification code with content-type (#17843) 2023-08-11 11:34:43 -07:00
Klaus Post 96a22bfcbb
fix: wrapped io.EOF during ListObjects() (#17842) 
When listing getObjectFileInfo can return `io.EOF` if file is being written.

When we wrap the error it will *not* retry upstream, since `io.EOF` is a valid return value.

Allow one retry before returning errors and canceling the listing.
 2023-08-11 09:47:16 -07:00
Poorna dfaf735073
replication: fix queuing of large uploads (#17831) 
Fixes regression from #17687
 2023-08-10 15:48:42 -07:00
Anis Eleuch 7fcfde7f07
s3: Pick a pool with >85% if all other pools are in suspended state (#17826) 2023-08-10 11:06:31 -07:00
jiuker b1391d1991
feat: support perf client to show `TX` from client to server (#17718) 2023-08-10 07:14:46 -07:00
Harshavardhana eb55034dfe
optimize deletePrefix, use direct set location via object name (#17827) 
* optimize deletePrefix, use direct set location via object name

instead of fanning out the calls for an object force delete
we can assume the set location and not do fan-out calls

* Apply suggestions from code review

Co-authored-by: Krishnan Parthasarathi <krisis@users.noreply.github.com>

---------

Co-authored-by: Krishnan Parthasarathi <krisis@users.noreply.github.com>
 2023-08-09 16:30:22 -07:00
Harshavardhana c45bc32d98
skip disks under scanning when healing disks (#17822) 
Bonus:

- avoid calling DiskInfo() calls when missing blocks
  instead heal the object using MRF operation.

- change the max_sleep to 250ms beyond that we will
  not stop healing.
 2023-08-09 12:51:47 -07:00
Harshavardhana 6e860b6dc5
count all versions as part of DeleteAllVersionsAction (#17821) 2023-08-09 08:55:19 -07:00
Harshavardhana b732a673dc
reduce logging in bucket replication in retry scenarios (#17820) 2023-08-08 13:27:40 -07:00
Yang Wu 23e4895dfc
Create metrics slice when necessary (#17809) 2023-08-07 02:21:22 -07:00
Harshavardhana 8666c55ca6
fix: do not use PrefixEnabled() logic to ignore valid objects (#17677) 
ignoring valid objects with valid replication metadata
after the Prefix was disabled must still honor the older
metadata.

this can lead to unexpected results, allow it during
READ phase always.
 2023-08-05 13:56:01 -07:00
Anis Eleuch a3f00c5d5e
batch: Strict unmarshal yaml document to avoid user made typos (#17808) 
// UnmarshalStrict is like Unmarshal except that any fields that are found
// in the data that do not have corresponding struct members, or mapping
// keys that are duplicates, will result in
// an error.
 2023-08-05 13:51:48 -07:00
Poorna 26c23b30f4
replication: set context timeout for NewMultipartUpload calls (#17807) 2023-08-05 12:27:07 -07:00
Anis Eleuch a436fd513b
track client disconnections properly for all ListObjects calls (#17804) 
Currently ListObjects* calls were returning 200 OK for timed-out clients,
this makes debugging via `mc admin trace` very hard.
 2023-08-04 15:57:27 -07:00
Harshavardhana 533cd8d6df
fix: batch replication pull must preserve versionID (#17805) 
batch replication pull must preserve versionID regardless
of destination bucket versioning configuration.

This is similar to the issue with decommissioning and rebalancing
 2023-08-04 12:09:10 -07:00
Harshavardhana cb089dcb52
error out by default beyond 10000 versions per object (#17803) 
```
You've exceeded the limit on the number of versions you can create on this object
```
 2023-08-04 10:40:21 -07:00
Harshavardhana 239ccc9c40
fix: crash in globalTierJournal when TierConfig is not initialized (#17791) 2023-08-03 14:16:15 -07:00
Poorna b762fbaf21
sts: validate if iam subsystem initialized in handlers (#17796) 2023-08-03 13:24:25 -07:00
Praveen raj Mani 0285df5a02
fix: prioritize audit_webhook and logger_webhook ENVs over the config KVS (#17783) 2023-08-03 02:47:07 -07:00
Harshavardhana 45fb375c41
allow healing to prefer local disks over remote (#17788) 2023-08-03 02:18:18 -07:00
Harshavardhana 4a4950fe41
fix: honor requested allow origin settings properly (#17789) 
fixes #17778
 2023-08-02 20:41:21 -07:00
Anis Eleuch 1664fd8bb1
Avoid logging errors twice during transitioned objects expiration (#17782) 2023-08-02 09:06:03 -07:00
Harshavardhana 21cdd2bf5d
avoid overwriting metrics on success, save it in defer (#17780) 2023-08-01 22:19:56 -07:00
Harshavardhana 0153f96a20
add deadlines for readMetadata() in listing (#17776) 
Bonus: also skip spending time looking for xl.json

- Listing()
- Delete()
 2023-08-01 21:52:31 -07:00
Harshavardhana a7a7533190
add new errors for Disks with timeouts (#17770) 2023-08-01 12:47:50 -07:00
Poorna 311380f8cb
replication resync: fix queueing (#17775) 
Assign resync of all versions of object to the same worker to avoid locking
contention. Fixes parallel resync implementation in #16707
 2023-08-01 11:51:15 -07:00
Harshavardhana b0f0e53bba
fix: make sure to correctly initialize health checks (#17765) 
health checks were missing for drives replaced since

- HealFormat() would replace the drives without a health check
- disconnected drives when they reconnect via connectEndpoint()
  the loop also loses health checks for local disks and merges
  these into a single code.
- other than this separate cleanUp, health check variables to avoid
  overloading them with similar requirements.
- also ensure that we compete via context selector for disk monitoring
  such that the canceled disks don't linger around longer waiting for
  the ticker to trigger.
- allow disabling active monitoring.
 2023-08-01 10:54:26 -07:00
Klaus Post 004f1e2f66
Fix trailing header signature mismatch (#17774) 
Seems like clients may omit a newline at the end of the trailer chunk. Each header should end with a newline. Add that if missing.

Fixes #17662
 2023-08-01 08:45:57 -07:00
Harshavardhana 2fa561f22e
do not crash on invalid metric values (#17764) 
```
minio[1032735]: panic: label value "\xc0.\xc0." is not valid UTF-8
minio[1032735]: goroutine 1781101 [running]:
minio[1032735]: github.com/prometheus/client_golang/prometheus.MustNewConstMetric(...)
```

log such errors for investigation
 2023-08-01 00:55:39 -07:00
Harshavardhana 81be718674
fix: optimize DiskInfo() call avoid metrics when not needed (#17763) 2023-07-31 15:20:48 -07:00
Sho Ce 49a1e2f98e
update-notifier.go: misleading version age message (#17750) 2023-07-31 08:36:19 -07:00
Klaus Post 684c46369c
Send events for extracted objects (#17760) 
Fixes #17759
 2023-07-31 08:33:51 -07:00
Harshavardhana 73edd5b8fd
introduce 'mc admin config set alias/ api odirect=on' (#17753) 
change disable_odirect=off -> odirect=on to make it
easier to understand, instead of making it double
negative.
 2023-07-31 00:12:53 -07:00
Harshavardhana 5e5bdf5432
capture total errors data availability and any timeout errors (#17748) 2023-07-29 23:26:26 -07:00
Harshavardhana f13cfcb83e
allow disabling O_DIRECT for write ops (#17751) 
on really slow systems, O_DIRECT simply kills the drives
allow for a way to disable them.
 2023-07-29 15:17:56 -07:00
Harshavardhana 731e03fe5a
add ReadFileStream deadline for disk call (#17745) 
timeout the reader side if hung via disk max timeout
 2023-07-28 15:37:53 -07:00
Anis Eleuch 7057d00a28
s3: Return invalid bucket name the first thing in all S3 calls (#17742) 2023-07-28 10:49:20 -07:00
Harshavardhana 114fab4c70
export cluster health as prometheus metrics (#17741) 2023-07-28 01:16:53 -07:00
ruspaul013 a92cb66468
Get the signed headers in the order they were signed (#17690) 
use pSignValues to get signed headers in order
 2023-07-27 11:45:30 -07:00
ruspaul013 535f97ba61
check if metadata headers/url values are equal with signed headers (#17737) 2023-07-27 11:44:56 -07:00
drivebyer 14ebd82dbd
fix: missing disk metrics when query metric api from peer (#17738) 2023-07-27 11:44:13 -07:00
Harshavardhana 47dcfcbdd4
introduce deadlines on READ operations (#17724) 2023-07-27 07:33:05 -07:00
Krishnan Parthasarathi bf3901342c
Include SuccessorModTime for FileInfo quorum (#17732) 2023-07-26 17:04:16 -07:00
Harshavardhana b28bcad11b
avoid Access() calls on known bucket paths (#17719) 2023-07-26 11:31:40 -07:00
Harshavardhana a7c71e4c6b
protect disk monitoring to avoid busy loop configuration (#17723) 2023-07-25 20:02:22 -07:00
Poorna 1a42693d68
replication: limit larger uploads to a subset of workers (#17687) 
Limit large uploads (> 128MiB)  to a max of 10 workers, intent is to avoid
larger uploads from using all replication bandwidth, giving room for smaller
uploads to sync faster.
 2023-07-25 20:02:02 -07:00
Harshavardhana e7b60c4d65
Add slow drive timeouts to match with active disk monitoring (#17701) 
allow active disk-monitoring to be configurable, and use
these add deadlines in various call layers for various
syscalls.
 2023-07-25 16:58:31 -07:00
Poorna f95129894d
Use decrypted object size while computing object size summary (#17717) 
Corrects an issue with encrypted versioned objects being reported under
`unversioned` bin in the object version histogram
 2023-07-24 17:13:25 -07:00
Harshavardhana c32c71c836
allow DNS cache TTL to be configurable (#17709) 
this is added for now as a hidden variable
 2023-07-24 15:13:35 -07:00
Harshavardhana 14e1ace552
remove serializing WalkDir() across all buckets/prefixes on SSDs (#17707) 
slower drives get knocked off because they are too slow via 
active monitoring, we do not need to block calls arbitrarily.

Serializing adds latencies for already slow calls, remove
it for SSDs/NVMEs

Also, add a selection with context when writing to `out <-`
channel, to avoid any potential blocks.
 2023-07-24 09:30:19 -07:00
drivebyer a7fb3a3853
fix: Create metrics slice when necessary in getCacheMetrics() (#17711) 2023-07-24 08:40:21 -07:00
Klaus Post 2da4bd5f1a
Revert "don't error when asked for 0-based range on empty objects (#17708) (#17713) 
Revert "don't error when asked for 0-based range on empty objects (#17708)"

This reverts commit 7e76d66184.

There is no valid way to specify offsets in a 0-byte file. Blame it on the [RFC](https://datatracker.ietf.org/doc/html/rfc7233#section-4.4)

> The 416 (Range Not Satisfiable) status code indicates that none of the ranges in the 
> request's Range header field (Section 3.1) overlap the current extent of the selected resource...

A request for "bytes=0-" is a request for the first byte of a resource. If the resource is 0-length, 
the range [0,0] does not overlap the resource content and the server responds with an error.
 2023-07-24 07:56:28 -07:00
flisk 7e76d66184
don't error when asked for 0-based range on empty objects (#17708) 
In a reverse proxying setup, a proxy in front of MinIO may attempt to
request objects in slices for enhanced cache efficiency. Since such a
a proxy cannot have prior knowledge of how large a requested resource is,
it usually sends a header of the form:

        Range: 0-$slice_size

... and, depending on the size of the resource, expects either:

- an empty response, if $resource_size == 0
- a full response, if $resource_size <= $slice_size
- a partial response, if $resource_size > $slice_size

Prior to this change, MinIO would respond 416 Range Not Satisfiable if a
client tried to request a range on an empty resource. This behavior is
technically consistent with RFC9110[1] – However, it renders sliced
reverse proxying, such as implemented in Nginx, broken in the case of
empty files. Nginx itself seems to break this convention to enable
"useful" responses in these cases, and MinIO should probably do that
too.

[1]: https://www.rfc-editor.org/rfc/rfc9110#byte.ranges
 2023-07-23 00:10:03 -07:00
Harshavardhana 7764f4a8e3
return tags as part of Head/Get calls (#17635) 
AWS S3 only returns the number of tag
counts, along with that we must return
the tags as well to avoid another metadata
call to the server.
 2023-07-22 07:19:43 -07:00
Kaan Kabalak 6624f970c0
Fix spelling of 'already' across repository (#17703) 2023-07-21 08:45:08 -07:00
Harshavardhana 331bdc2245
fix: remove CompleteMultipartUpload() 200 OK response for blocking calls (#17699) 
sending whitespace character with CompleteMultipartUpload()
with 200 OK was an AWS S3 compatible implementation detail,
and it was expected that the client SDK must look for both
successful XML as well as error XML for 200 OK.

But this is not useful anymore on MinIO, since we do not
have any large delayed coalescing of parts anymore.
 2023-07-20 22:14:38 -07:00
Harshavardhana e12ab486a2
avoid using os.Getenv for internal code, use env.Get() instead (#17688) 2023-07-20 07:52:49 -07:00
Krishnan Parthasarathi 9eeee92d36
Add deletemarker_total metric (#17689) 2023-07-20 07:52:32 -07:00
Anis Eleuch 756d6aa729
fix: report correct pool/set/disk indexes for offline disks (#17695) 2023-07-20 07:48:21 -07:00
Harshavardhana bddd53d6d2
fix: retry listing in decommissioning if it fails perpetually (#17682) 2023-07-19 13:09:37 -07:00
jiuker a99cd825ab
fix: byHost realTime metrics API (#17681) 2023-07-18 23:50:30 -07:00
Harshavardhana 6426b74770
move bucket centric metrics to /minio/v2/metrics/bucket handlers (#17663) 
users/customers do not have a reasonable number of buckets anymore,
this is why we must avoid overpopulating cluster endpoints, instead
move the bucket monitoring to a separate endpoint.

some of it's a breaking change here for a couple of metrics, but
it is imperative that we do it to improve the responsiveness of
our Prometheus cluster endpoint.

Bonus: Added new cluster metrics for usage, objects and histograms
 2023-07-18 22:25:12 -07:00
Harshavardhana 4f257bf1e6
pick internode interface properly via globalLocalNodeName (#17680) 
current code will not pick the right interface name
if --address or --interface is not provided.
 2023-07-18 19:18:11 -07:00
Krishnan Parthasarathi 0120ff93bc
admin-info: add DeleteMarkers count (#17659) 2023-07-18 10:49:40 -07:00
Anis Eleuch 49638fa533
s3: Delete Bucket should not recreate bucket if it does not exist (#17676) 
Also return Bucket Not Found error in the same use case.
 2023-07-18 09:32:19 -07:00
Shubhendu 7a3a7b19e5
Added a start script to inspect command output (#17591) 
Using this script, post decrypt we should be able to bring up the
MinIO instance with same configuration.

Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
 2023-07-17 14:15:28 -07:00
Harshavardhana 24e86d0c59
avoid passing around poolIdx, setIdx instead pass the relevant disks (#17660) 2023-07-17 09:52:05 -07:00
jiuker d118031ed6
fix: when Origin: null is set return back '*' for allow origins (#17651) 2023-07-15 12:15:06 -07:00
Anis Eleuch 341a89c00d
return a descriptive error when loading any IAM item fails (#17654) 
Sometimes IAM fails to load certain items, which could be a user, 
a service account or a policy but with not enough information for 
us to debug.

This commit will create a more descriptive error to make it easier to
debug in such situations.
 2023-07-14 20:17:14 -07:00
Anis Eleuch df29d25e6b
return different status code for internode communication (#17655) 
mc admin trace -a will be able to quickly show
401 Unauthorized header to pinpoint trivial issues
between nodes, such as wrong root 
credentials and skewed time.
 2023-07-14 18:34:55 -07:00
Harshavardhana 3e196fa7b3
fix: ILM newer noncurrent version limit must return correct versions (#17652) 
objects/versions that are not expired via NewerNoncurrentVersions
must be properly returned to be applied under further ILM actions.

this would cause legitimately expired objects to be missed
from expiration.
 2023-07-14 16:42:35 -07:00
drivebyer 04c792476f
fix: provide a possible slice cap for heal failed metrics items (#17647) 
Signed-off-by: Wu <yang.wu@daocloud.io>
 2023-07-14 11:02:45 -07:00
Harshavardhana 005a4a275a
add more bootstrap messages to provide latency (#17650) 
- simplify refreshing bucket metadata, wait() to
  depend on how fast the bucket metadata can load.

- simplify resync to start resync in single pass.
 2023-07-14 04:00:29 -07:00
Harshavardhana bdddf597f6
shuffle buckets randomly before being scanned (#17644) 
this randomness is needed to avoid scanning
the same buckets across different erasure sets,
in the same order.

allow random buckets to be scanned instead
allowing a wider spread of ILM, replication
checks.

Additionally do not loop over twice to fill
the channel, fill the channel regardless of
having bucket new or old.
 2023-07-14 02:25:40 -07:00
Aditya Manthramurthy bb6921bf9c
Send AuditLog via new middleware fn for admin APIs (#17632) 
A new middleware function is added for admin handlers, including options
for modifying certain behaviors. This admin middleware:

- sets the handler context via reflection in the request and sends AuditLog
- checks for object API availability (skipping it if a flag is passed)
- enables gzip compression (skipping it if a flag is passed)
- enables header tracing (adding body tracing if a flag is passed)

While the new function is a middleware, due to the flags used for
conditional behavior modification, which is used in each route registration
call.

To try to ensure that no regressions are introduced, the following
changes were done mechanically mostly with `sed` and regexp:

- Remove defer logger.AuditLog in admin handlers
- Replace newContext() calls with r.Context()
- Update admin routes registration calls

Bonus: remove unused NetSpeedtestHandler

Since the new adminMiddleware function checks for object layer presence
by default, we need to pass the `noObjLayerFlag` explicitly to admin
handlers that should work even when it is not available. The following
admin handlers do not require it:

- ServerInfoHandler
- StartProfilingHandler
- DownloadProfilingHandler
- ProfileHandler
- SiteReplicationDevNull
- SiteReplicationNetPerf
- TraceHandler

For these handlers adminMiddleware does not check for the object layer
presence (disabled by passing the `noObjLayerFlag`), and for all other
handlers, the pre-check ensures that the handler is not called when the
object layer is not available - the client would get a
ErrServerNotInitialized and can retry later.

This `noObjLayerFlag` is added based on existing behavior for these
handlers only.
 2023-07-13 14:52:21 -07:00
Klaus Post 4f89e5bba9
Add active disk health checks (#17539) 
Add check every 2 minutes to see if a write+read operation can complete.

If disk is unresponsive for 2 minutes or returns errFaultyDisk, take it offline.
 2023-07-13 11:41:55 -07:00
jiuker 183428db03
fear: Implement 'mc support top net' (#17598) 2023-07-13 11:41:19 -07:00
Shireesh Anjal fc6d873758
Use os.ReadFile instead of ioutil.ReadFile (#17649) 
ioutil.ReadFile is deprecated and also doesn't work with certain kinds
of symlinks.
 2023-07-13 09:07:10 -07:00
Poorna 5e2f8d7a42
replication: Simplify mrf requeueing and add backlog handler (#17171) 
Simplify MRF queueing and add backlog handler

- Limit re-tries to 3 to avoid repeated re-queueing. Fall offs
to be re-tried when the scanner revisits this object or upon access.

- Change MRF to have each node process only its MRF entries.

- Collect MRF backlog by the node to allow for current backlog visibility
 2023-07-12 23:51:33 -07:00
Shubhendu 9b9871cfbb
Added `endpoint` and `versions` attributes to KMS details (#17350) 
Now it would list details of all KMS instances with additional
attributes `endpoint` and `version`. In the case of k8s-based
deployment the list would consist of a single entry.

Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
 2023-07-12 23:50:38 -07:00
guangwu f80b6926d3
chore: fix minor issues reported via staticcheck (#17639) 2023-07-12 20:33:11 -07:00
Shubhendu 6dc55fe5ed
Corrected the API name for audit logging purpose (#17642) 
This would better to record the correct API name so that
any verification around audit logs to figure out if required
APIs are called required no of times, would be correct.
Here in this case of policy attached, API `AttachDetachPolicyBuiltin`
would be called with `requestPath` as `/minio/admin/v3/idp/builtin/policy/attach`
and in case of detach policy the value would be `/minio/admin/v3/idp/builtin/policy/detach`

Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
 2023-07-12 15:38:49 -07:00
Harshavardhana 2d1cda2061
fix: do not os.Exit(1) while writing goroutines during shutdown (#17640) 
Also shutdown poll add jitter, to verify if the shutdown
sequence can finish before 500ms, this reduces the overall
time taken during "restart" of the service.

Provides speedup for `mc admin service restart` during
active I/O, also ensures that systemd doesn't treat the
returned 'error' as a failure, certain configurations in
systemd can cause it to 'auto-restart' the process by-itself
which can interfere with `mc admin service restart`.

It can be observed how now restarting the service is
much snappier.
 2023-07-12 07:18:30 -07:00
Harshavardhana a566bcf613
treat 0-byte objects to honor same quorum as delete marker (#17633) 
on unversioned buckets its possible that 0-byte objects
might lose quorum on flaky systems, allow them to be same
as DELETE markers. Since practically speak they have no
content.
 2023-07-11 21:53:49 -07:00
Klaus Post 9885a0a6af
Fix hasSpaceFor in SNSD setup (#17630) 
If drive is offline or filled we divide by 0.

Fixes #17629

Bonus: Reject when any valid disk exceeds minimum inode threshold.
 2023-07-11 14:29:34 -07:00
Kaan Kabalak f64d62b01d
Fix style of logOnceIf calls w/unique identifiers (#17631) 2023-07-11 13:17:45 -07:00
Harshavardhana 82075e8e3a
use strconv variants to improve on performance per 'op' (#17626) 
```
BenchmarkItoa
BenchmarkItoa-8         	673628088	         1.946 ns/op	       0 B/op	       0 allocs/op
BenchmarkFormatInt
BenchmarkFormatInt-8    	592919769	         2.012 ns/op	       0 B/op	       0 allocs/op
BenchmarkSprint
BenchmarkSprint-8       	26149144	        49.06 ns/op	       2 B/op	       1 allocs/op
BenchmarkSprintBool
BenchmarkSprintBool-8   	26440180	        45.92 ns/op	       4 B/op	       1 allocs/op
BenchmarkFormatBool
BenchmarkFormatBool-8   	1000000000	         0.2558 ns/op	       0 B/op	       0 allocs/op
```
 2023-07-11 07:46:58 -07:00
Harshavardhana 5b7c83341b
move per bucket metrics to peer location (#17627) 2023-07-11 07:46:24 -07:00
Poorna fb49aead9b
replication: add validation API (#17520) 
To check if replication is set up properly on a bucket.
 2023-07-10 20:09:20 -07:00
Aditya Manthramurthy 85f5700e4e
fix: missing audit logger call for some admin APIs (#17623) 2023-07-10 16:59:44 -07:00
Aditya Manthramurthy 43b3c093ef
Fix: set request id in trace context properly (#17622) 2023-07-10 15:40:44 -07:00
Kaan Kabalak bd6842d917
Further print log messages once per error (#17618) 2023-07-10 07:59:57 -07:00
Poorna e8c98c3246
Avoid extra GetObjectInfo call in DeleteObject API (#17599) 
Optimize DeleteObject API to avoid extra 
GetObjectInfo call on the replicating side.

For receiving side, it is just a regular
DeleteObject call.

Bonus: Fix a corner case where version purged is 
absent on target (either due to replication not yet
complete or target version already deleted in a
one-way replication or when replication was disabled). 

In such cases, mark version purge complete.
 2023-07-10 07:57:56 -07:00
Harshavardhana dfd7cca0d2
fix: allow cancel of decom only when its in progress (#17607) 2023-07-10 07:55:38 -07:00
Harshavardhana f6186965c3
honor DeleteAllVersions in list(), head() calls (#17604) 2023-07-08 15:42:10 -07:00
Harshavardhana 28a01f0320
update missing license header in files (#17603) 2023-07-08 10:42:05 -07:00
Anis Eleuch 6d0bc5ab1e
prometheus: Fix internode stats (#17594) 
Internode calculation was done inside S3 handlers, fix it by moving it
to internode handlers.

Remove admin stats since it is not used.
 2023-07-08 07:35:11 -07:00
Aditya Manthramurthy 7af78af1f0
fix: set request ID in tracing context key (#17602) 
Since `addCustomerHeaders` middleware was after the `httpTracer`
middleware, the request ID was not set in the http tracing context. By
reordering these middleware functions, the request ID header becomes
available. We also avoid setting the tracing context key again in
`newContext`.

Bonus: All middleware functions are renamed with a "Middleware" suffix
to avoid confusion with http Handler functions.
 2023-07-08 07:31:42 -07:00
Harshavardhana abb1f22057 Revert "change ttfb_distribution metrics to histogramMetric (#17115)" 
This reverts commit 9112ca4e29.
 2023-07-07 13:57:37 -07:00
Harshavardhana f41edb23e2
add variadic delays in peer notification retries (#17592) 
just adds more `jitter` in our retries to avoid
burst flooding for peer calls.
 2023-07-07 07:47:38 -07:00
Klaus Post e20aab25ec
Check for progress before we reach the limit (#17552) 2023-07-07 00:13:57 -07:00
Klaus Post ff5988f4e0
Reduce allocations (#17584) 
* Reduce allocations

* Add stringsHasPrefixFold which can compare string prefixes, while ignoring case and not allocating.
* Reuse all msgp.Readers
* Reuse metadata buffers when not reading data.

* Make type safe. Make buffer 4K instead of 8.

* Unslice
 2023-07-06 16:02:08 -07:00
jiuker c47ff44f5e
fix: disable site network test if site replication is disabled (#17579) 2023-07-06 09:19:14 -07:00
Harshavardhana 8af0773baf
remove deprecated Content-Security-Policy (#17580) 
https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Security-Policy/block-all-mixed-content
 2023-07-06 09:18:38 -07:00
jiuker 2dbb1cff4a
feat: support perf site replication (#17477) 2023-07-05 22:28:26 -07:00
Klaus Post 6efcf9c982
Do lockless last minute latency metrics (#17576) 
Collect metrics in one second and accumulate lockless before sending upstream.
 2023-07-05 10:40:45 -07:00
Harshavardhana 0bc34952eb
fix: under FanOut API avoid repeated md5sum calculation (#17572) 
md5sum calculation has a high CPU overhead, avoid calculating
it repeatedly for similar fanOut calls.

To fix following CPU profiler result
```
(pprof) top10
Showing nodes accounting for 678.68s, 84.67% of 801.54s total
Dropped 1072 nodes (cum <= 4.01s)
Showing top 10 nodes out of 156
      flat  flat%   sum%        cum   cum%
   332.54s 41.49% 41.49%    332.54s 41.49%  runtime/internal/syscall.Syscall6
   228.39s 28.49% 69.98%    228.39s 28.49%  crypto/md5.block
    48.07s  6.00% 75.98%     48.07s  6.00%  runtime.memmove
    28.91s  3.61% 79.59%     28.91s  3.61%  github.com/minio/highwayhash.updateAVX2
     8.25s  1.03% 80.61%      8.25s  1.03%  runtime.futex
     8.25s  1.03% 81.64%     10.81s  1.35%  runtime.step
     6.99s  0.87% 82.52%     22.35s  2.79%  runtime.pcvalue
     6.67s  0.83% 83.35%     38.90s  4.85%  runtime.mallocgc
     5.77s  0.72% 84.07%     32.61s  4.07%  runtime.gentraceback
     4.84s   0.6% 84.67%     10.49s  1.31%  runtime.lock2
```
 2023-07-05 03:16:05 -07:00
Harshavardhana e37c4efc6e
fix: upon DNS refresh() failure use previous values (#17561) 
DNS refresh() in-case of MinIO can safely re-use
the previous values on bare-metal setups, since
bare-metal arrangements do not change DNS in any 
manner commonly.

This PR simplifies that, we only ever need DNS caching
on bare-metal setups.

- On containerized setups do not enable DNS
  caching at all, as it may have adverse effects on
  the overall effectiveness of k8s DNS systems.

  k8s DNS systems are dynamic and expect applications
  to avoid managing DNS caching themselves, instead
  provide a cleaner container native caching
  implementations that must be used.

- update IsDocker() detection, including podman runtime

- move to minio/dnscache fork for a simpler package
 2023-07-03 12:30:51 -07:00
Anis Eleuch 15fd5ce2fa
fix: A typo in per pool make/delete bucket errs calculation (#17553) 2023-07-03 09:47:40 -07:00
Harshavardhana 7f782983ca
fix: for FTP server driver allow implicit trust of TLS (#17541) 
fixes #17535
 2023-06-30 08:04:13 -07:00
Aditya Manthramurthy 9d628346eb
fix: service account list for root user (#17547) 
Fixes https://github.com/minio/minio/issues/17545
 2023-06-30 08:02:12 -07:00
Aditya Manthramurthy bde533a9c7
fix: OpenID config initialization (#17544) 
This is due to a regression in the handling of the enable key in OpenID
configuration.
 2023-06-29 23:38:26 -07:00
Harshavardhana aae6846413
feat: allow expiration of all versions via ILM Expiration action (#17521) 
Following extension allows users to specify immediate purge of
all versions as soon as the latest version of this object has
expired.

```
<LifecycleConfiguration>
    <Rule>
        <ID>ClassADocRule</ID>
        <Filter>
           <Prefix>classA/</Prefix>
        </Filter>
        <Status>Enabled</Status>
        <Expiration>
             <Days>3650</Days>
	     <ExpiredObjectAllVersions>true</ExpiredObjectAllVersions>
        </Expiration>
    </Rule>
    ...
```
 2023-06-28 22:12:28 -07:00
Harshavardhana 5317a0b755
fix: support LDAP settings properly in ftp/sftp (#17536) 
Bonus this PR enhances and supports creating
buckets via ftp `mkdir`

fixes #17526
 2023-06-28 13:15:21 -07:00
Harshavardhana 73de721a63
fix: handle copyObjectPart encryption properly (#17530) 
- look for requested encryption while compressing
  not just via HTTP Headers, but also via multipart
  metadata

- look for SSE-S3 etag decryption not just via HTTP
  Headers, but also via multipart metadata

fixes #17519
 2023-06-28 09:43:50 -07:00
Harshavardhana d2f5c3621f
fix: add additional decommission traces for ILM expired content (#17522) 
current decommission traces were missing for

- Skipped ILM expired versions
- Skipped single DELETE marked version
- A success or failure in decommissioning DELETE marker
- allow additional info to be shared in DecomStatus() API
 2023-06-27 11:59:40 -07:00
Harshavardhana 1818764840
fix: bug in passing Versioned field set for getHealReplicationInfo() (#17498) 
Bonus: rejects prefix deletes on object-locked buckets earlier
 2023-06-27 09:45:50 -07:00
Harshavardhana d3e5e607a7
allow site-replication checks to work on non-distributed setups (#17524) 
fixes #17523
 2023-06-27 09:23:50 -07:00
Shireesh Anjal c1943ea3af
Capture realtime metrics in health report (#17516) 2023-06-27 01:39:18 -07:00
guangwu 87b6fb37d6
chore: pkg imported more than once (#17444) 2023-06-26 09:21:29 -07:00
Kaan Kabalak 21fbe88e1f
Print certain log messages once per error (#17484) 2023-06-24 20:29:13 -07:00
Harshavardhana 1f8b9b4bd5
fix: do not listAndHeal() inline with PutObject() (#17499) 
there is a possibility that slow drives can actually add latency
to the overall call, leading to a large spike in latency.

this can happen if there are other parallel listObjects()
calls to the same drive, in-turn causing each other to sort
of serialize.

this potentially improves performance and makes PutObject()
also non-blocking.
 2023-06-24 19:31:04 -07:00
Klaus Post 216069d0da
Remove 'null' version ID from directory object response (#17495) 
Fixes #17494

Regression from #17132
 2023-06-23 13:26:00 -07:00
Harshavardhana eefa047974
fix: keep decommission in a go-routine (#17496) 
This was removed by mistake in #17491
 2023-06-23 12:29:32 -07:00
Anis Eleuch d8dad5c9ea
s3: Make/Delete buckets to use error quorum per pool (#17467) 2023-06-23 11:48:23 -07:00
Klaus Post bf8a68879c
fix: Time ILM Actions for scanner info (#17493) 
ILM Actions were not timed fix it.
 2023-06-23 07:48:36 -07:00
Aditya Manthramurthy f3248a4b37
Redact all secrets from config viewing APIs (#17380) 
This change adds a `Secret` property to `HelpKV` to identify secrets
like passwords and auth tokens that should not be revealed by the server
in its configuration fetching APIs. Configuration reporting APIs now do
not return secrets.
 2023-06-23 07:45:27 -07:00
Harshavardhana d315d012a4
decom: during multiple pool decom preserve current pool status (#17491) 
removal of completed pools must retain pool status of other
pools in draining, to resume any remaining draining operations.
 2023-06-23 07:44:18 -07:00
Harshavardhana bd9bf3693f
lambda: negative duration for presigned URL default to 1H (#17489) 
fixes a bug where users created with Expiration as
timeSentinel is not rejected while generating the
presigned URL for lambda processing.
 2023-06-23 00:17:24 -07:00
Aditya Manthramurthy 82ce78a17c
Fix locking in policy attach API (#17426) 
For policy attach/detach API to work correctly the server should hold a
lock before reading existing policy mapping and until after writing the
updated policy mapping. This is fixed in this change.

A site replication bug, where LDAP policy attach/detach were not
correctly propagated is also fixed in this change.

Bonus: Additionally, the server responds with the actual (or net)
changes performed in the attach/detach API call. For e.g. if a user
already has policy A applied, and a call to attach policies A and B is
performed, the server will respond that B was attached successfully.
 2023-06-21 22:44:50 -07:00
Harshavardhana 9af6c6ceef
under rebalance look for expired versions v/s remaining versions (#17482) 
A continuation of PR #17479 for rebalance behavior must
also match the decommission behavior.

Fixes bug where rebalance would ignore rebalancing object
versions after one of the version returned "ObjectNotFound"
 2023-06-21 13:23:20 -07:00
Praveen raj Mani b94ab07c2f
Honor global root CAs for kafka audit tls (#17481) 
honor global root CAs for kafka audit tls
 2023-06-21 10:50:40 -07:00
Harshavardhana 7605d07bb2
add support for bucket level request count per API (#17468) 
New metrics added to calculate API request count
per bucket, per API.  Captures errors, including
4xx, 5xx HTTP status codes separately.
 2023-06-21 09:41:59 -07:00
Harshavardhana ccc5801112
always look for expired versions v/s remaining versions (#17479) 
while decommissioning it can so happen that the non-current
versions are all expired but there is a DEL marker as the
latest version.

For such objects, we should not decommission them instead
calculate the remaining versions and if the remaining versions
is one and that version is a DEL marker consider such
an object not to be scheduled for decommissioning.
 2023-06-21 08:49:28 -07:00
Praveen raj Mani 7c72b25ef0
Add an option to make bucket notifications synchronous (#17406) 
With the current asynchronous behaviour in sending notification events
to the targets, we can't provide guaranteed delivery as the systems
might go for restarts.

For such event-driven use-cases, we can provide an option to enable
synchronous events where the APIs wait until the event is successfully
sent or persisted.

This commit adds 'MINIO_API_SYNC_EVENTS' env which when set to 'on'
will enable sending/persisting events to targets synchronously.
 2023-06-20 17:38:59 -07:00
Harshavardhana 02c2ec3027
skip onlineDisks with parity mismatch (#17478) 2023-06-20 13:18:24 -07:00
Harshavardhana 65c31fab12
fix: do not crash rebalance code instead set the object layer (#17465) 
fixes #17421
 2023-06-20 09:28:23 -07:00
jiuker b6b68be052
fix: replication check for duplicate endpoints detection with wrong route (#17474) 2023-06-20 09:27:54 -07:00
Harshavardhana 15911c85f6
safely ignore out of band deletions while decommissioning (#17473) 2023-06-20 08:31:42 -07:00
Aditya Manthramurthy 5a1612fe32
Bump up madmin-go and pkg deps (#17469) 2023-06-19 17:53:08 -07:00
Harshavardhana 1443b5927a
allow quorum fileInfo to pick same parityBlocks (#17454) 
Bonus: allow replication to proceed for 503 errors such as
with error code SlowDownRead
 2023-06-18 18:20:15 -07:00
Anis Eleuch 35ef35b5c1
fix a integer divide by zero crash during rebalance (#17455) 
A state is updated with a delete marker, which does not have parity or
data blocks defined, which can cause the integer divide by zero panics.

This commit fixes to avoid panics.
 2023-06-18 11:14:53 -07:00
Harshavardhana 6806537eb3
event args list for fanOut notification must be sized same (#17450) 
without this fan-out API can crash if client cancels
the on-going request.
 2023-06-18 07:09:20 -07:00
Harshavardhana 64de61d15d
fallback on etags if they match when mtime is not same (#17424) 
on "unversioned" buckets there are situations
when successive concurrent I/O can lead to
an inconsistent state() with mtime while the
etag might be the same for the object on disk.

in such a scenario it is possible for us to
allow reading of the object since etag matches
and if etag matches we are guaranteed that we
have enough copies the object will be readable
and same.

This PR allows fallback in such scenarios.
 2023-06-17 19:18:20 -07:00
Poorna c4d0c49a5f
ensure metadata updates go to same pool where version exists (#17451) 
This PR also returns the replication status in 
proxy calls and defers replication attempt if 
HEAD on object version returned a error different
from NoSuchKey
 2023-06-17 07:30:53 -07:00
Harshavardhana 47a48b6832
do not save any metadata from the headers in tar extract (#17436) 
only preserve the same storage-class as incoming
request other than that rest of them must be
deduced.
 2023-06-15 17:44:07 -07:00
Anis Eleuch a2aed12dcd
decom: Fix a typo in routing decommissioning requests (#17435) 
A specific node should do the decommissioning task, however routing the
start decommissioning to that node was not working properly.

Co-authored-by: Anis Elleuch <anis@min.io>
 2023-06-15 14:54:29 -07:00
Anis Eleuch d8e6e76e89
site-repl: Better error msg when setting sync in a local cluster (#17407) 2023-06-15 12:44:22 -07:00
Harshavardhana ad4e511026
do not save plain-text ETag when encryption is requested (#17427) 
fixes an issue under bucket replication could cause
ETags for replicated SSE-S3 single part PUT objects,
to fail as we would attempt a decryption while listing,
or stat() operation.
 2023-06-15 12:43:26 -07:00
Klaus Post 4a562d6732
fix: fanout error response - error must be string for marshaling (#17433) 
Uses https://github.com/minio/minio-go/pull/1839
 2023-06-15 09:21:53 -07:00
Poorna a9082e4f79
site replication: cancel ongoing op properly (#17428) 2023-06-15 08:05:08 -07:00
jiuker 0474791cf8
fix: set time format right (#17402) 2023-06-14 07:49:13 -07:00
Harshavardhana f32efd5429
more compliance related fixes (#17408) 
- lifecycle must return InvalidArgument for rule errors
- do not return `null` versionId in HTTP header
- reject mixed SSE uploads with correct error message
 2023-06-13 13:52:33 -07:00
jiuker 22c247a988
fix: preserve multiple values for query params (#17392) 2023-06-13 11:38:46 -07:00
Shubhendu 35d71682f6
fix: do not allow removal of inbuilt policies unless they are already persisted (#17264) 
Dont allow removal of inbuilt policies such as `readwrite, readonly, writeonly and diagnostics`
 2023-06-13 11:06:17 -07:00
drivebyer 3d6b88a60e
fix: syscall to record time on non-linux (#17383) 2023-06-13 11:04:50 -07:00
Harshavardhana 26a0803388
various compliance related fixes (#17401) 
- getObjectTagging to be allowed for anonymous policies
- return correct errors for invalid retention period
- return sorted list of tags for an object
- putObjectTagging must return 200 OK not 204 OK
- return 409 ErrObjectLockConfigurationNotAllowed for existing buckets
 2023-06-12 13:22:07 -07:00
Anis Eleuch ae95384dd8
Revert "heal: Update object parity with the latest configured SC (#17187)" (#17404) 2023-06-12 11:54:51 -07:00
Anis Eleuch 0f0dcf0c5e
tar: Avoid storing snowball extraction header in extract objects (#17389) 2023-06-12 09:42:06 -07:00
Klaus Post 6f2406b0b6
fix: protect ReplicationStats against concurrent map iteration and write crash (#17403) 2023-06-12 09:17:11 -07:00
Anis Eleuch bb24346e04
listen: Only error out if not able to bind any interface (#17353) 2023-06-12 09:09:28 -07:00
Harshavardhana be45ffd8a4
return 204 status code for DeleteBucketTagging (#17400) 2023-06-11 20:49:02 -07:00
Poorna Krishnamoorthy f986b0c493 replication: perform bucket resync in parallel (#16707) 
Default number of parallel resync operations for a bucket to 10
to speed up resync.
 2023-06-11 16:09:55 -07:00
Harshavardhana c9e87f0548
service accounts are allowed to have no expiration (#17397) 2023-06-11 10:34:59 -07:00
Harshavardhana 43468f4d47
return InvalidRequest when no parts are provided (#17395) 2023-06-10 21:59:51 -07:00
Harshavardhana b829e80ecb
do not disable root for invalid API config values (#17386) 2023-06-08 15:50:06 -07:00
Klaus Post 6e38d0f3ab
Add more bootstrap info in debug mode (#17362) 2023-06-08 08:39:47 -07:00
Anis Eleuch 38342b1df5
decom: Parallelize decommissining (#17364) 2023-06-07 14:27:51 -07:00
Harshavardhana 49ce85ee3d
allow prefix/markers to have '/' in the beginning to throw an empty (#17373) 2023-06-07 11:25:26 -07:00
Harshavardhana 442c50ff00
remove delimiter if not set by client, also fetchOwner is optional (#17366) 2023-06-06 21:31:47 -07:00
Harshavardhana d1448adbda
use slices package and remove some helpers (#17342) 2023-06-06 10:12:52 -07:00
jiuker 5a21b1f353
fix: Delete dir failed when .DS_Store in it (#17352) 2023-06-06 10:12:06 -07:00
Harshavardhana 2f9e2147f5
allow quota enforcement to rely on older values (#17351) 
PUT calls cannot afford to have large latency build-ups due
to contentious usage.json, or worse letting them fail with
some unexpected error, this can happen when this file is
concurrently being updated via scanner or it is being
healed during a disk replacement heal.

However, these are fairly quick in theory, stressed clusters
can quickly show visible latency this can add up leading to
invalid errors returned during PUT.

It is perhaps okay for us to relax this error return requirement
instead, make sure that we log that we are proceeding to take in
the requests while the quota is using an older value for the quota
enforcement. These things will reconcile themselves eventually,
via scanner making sure to overwrite the usage.json.

Bonus: make sure that storage-rest-client sets ExpectTimeouts to
be 'true', such that DiskInfo() call with contextTimeout does
not prematurely disconnect the servers leading to a longer
healthCheck, back-off routine. This can easily pile up while also
causing active callers to disconnect, leading to quorum loss.

DiskInfo is actively used in the PUT, Multipart call path for
upgrading parity when disks are down, it in-turn shouldn't cause
more disks to go down.
 2023-06-05 16:56:35 -07:00
Harshavardhana 75c6fc4f02
only allow decryption of etag for only sse-s3 (#17335) 2023-06-05 13:08:51 -07:00
Anis Eleuch 1436858347
log: Add a log when saving pool.bin fails (#17338) 
Co-authored-by: Anis Elleuch <anis@min.io>
 2023-06-04 14:20:21 -07:00
Kaan Kabalak 0649aca219
Add expiration to ListServiceAccounts function (#17249) 2023-06-02 16:17:26 -07:00
Harshavardhana b210ea79bc
do not save MTime in newMultipartUpload() to avoid side-affects (#17340) 2023-06-02 14:38:09 -07:00
Poorna 68f80b5fe7
replication: ignore retention mode validation for replica (#17332) 2023-06-01 18:53:12 -07:00
Poorna e95825a42e
replication: use latest object info for metrics update (#17333) 2023-06-01 18:52:55 -07:00
Anis Eleuch 931712dc46
fix: converting 'server closed idle connection' to errDiskNotFound (#17330) 2023-06-01 15:40:28 -07:00
Harshavardhana 54e544e03e
allow lookup()/head() operations on Veeam SOS objects (#17331) 2023-06-01 15:26:26 -07:00
Poorna f86b9abf32
site removal: update site config and reload targets after update (#17327) 2023-06-01 10:19:56 -07:00
Anis Eleuch 9ef7eda33a
heal: Avoid objects created after the heal disk start time (#17323) 2023-05-31 13:10:45 -07:00
Klaus Post c9e26401fa
Fix GetObject encrypted etag (#17302) 
Co-authored-by: Harshavardhana <harsha@minio.io>
 2023-05-31 13:10:25 -07:00
jiuker 14f6ac9222
fix: fail large content in DeleteMultipleObjects() early (#17321) 2023-05-31 10:58:14 -07:00
drivebyer b8474295af
fix: time() returned function not being called as expected in globalSync() (#17319) 2023-05-31 09:40:23 -07:00
Shireesh Anjal 817e85a3e0
fix: proxy not set on subnet logger webhook sometimes (#17320) 2023-05-31 08:09:09 -07:00
Harshavardhana 1cd7f1e38d
fix: cleanup empty multipart folders upon stale upload cleanup (#17312) 2023-05-30 09:56:50 -07:00
jiuker 043fd8b536
fix: on windows use FindClose close handler (#17306) 2023-05-30 02:15:57 -07:00
Harshavardhana 394690dcfb
check for upto 50%+ data disks to be offline (#17294) 2023-05-26 22:56:19 -07:00
Klaus Post c839b64f6a
fix: compressed+encrypted block overhead (#17289) 2023-05-26 10:57:07 -07:00
Anis Eleuch 6425fec366
s3: Add x-minio-error-code header for S3 HEAD requests (#17283) 2023-05-26 10:13:18 -07:00
Harshavardhana d5059840ef
fix: for delete marked objects choose appropriate parity (#17287) 2023-05-26 09:57:44 -07:00
Aditya Manthramurthy 65cba212e8
Remove older policy attach behavior for LDAP (#17240) 2023-05-26 06:31:24 -07:00
Aditya Manthramurthy 7a69c9c75a
Update builtin policy entities command (#17241) 2023-05-25 22:31:05 -07:00
Harshavardhana 5cd9dcb844
rebalance 'null' delete markers properly (#17282) 2023-05-25 16:12:53 -07:00
Anis Eleuch 54c5c88fe6
Add number of offline disks in quorum errors (#16822) 2023-05-25 09:39:06 -07:00
Harshavardhana 9b5829c16e
avoid decommissioning DEL markers with single versions (#17274) 2023-05-25 09:18:49 -07:00
jiuker d749aaab69
fix: ignore existing target status when adding new targets (#17250) 2023-05-24 22:57:37 -07:00
Krishnan Parthasarathi 62df731006
Add updatedAt for GetBucketLifecycleConfig (#17271) 2023-05-24 22:52:39 -07:00
Harshavardhana d0a0eb9738
support fan-out objects via PostUpload() (#17233) 2023-05-24 22:51:07 -07:00
Klaus Post 66156b8230
Stricter partNumber checks (#17270) 
Fixes #17269
 2023-05-24 08:00:47 -07:00
Klaus Post 5677f73794
Add PostObject Checksum (#17244) 2023-05-23 07:58:33 -07:00
Harshavardhana ef54200db7
offline drives more than 50% of total drives return error (#17252) 2023-05-23 07:57:57 -07:00
Krishnan Parthasarathi 3e128c116e
Add lifecycle event source to audit log tags (#17248) 2023-05-22 15:28:56 -07:00
Harshavardhana fc03be7891
simplify bucket metadata lookups for versioning/object locking (#17253) 2023-05-22 12:05:14 -07:00
Poorna 2920b0fc6d
allow specification of path/virtual style bucket lookup in batch replication (#17201) 2023-05-21 15:16:31 -07:00
Anis Eleuch a30a55f3b1
Add object parity in listing V2M and listing versions M (#17238) 2023-05-19 09:42:45 -07:00
Praveen raj Mani ecfb18b26a
Freeze the s3 APIs until the notification sub-system initializes completely (#17182) 2023-05-19 08:44:48 -07:00
jiuker 7d433f16c4
before return make globalScannerMetrics.incTime call (#17230) 2023-05-18 13:45:05 -07:00
Aditya Manthramurthy 9d96b18df0
Add "name" and "description" params to service acc (#17172) 2023-05-17 17:05:36 -07:00
drivebyer ad2ab6eb3e
fix: Give accurate cap to slice (#17224) 2023-05-17 15:14:09 -07:00
jiuker f037c9b286
Protecting the read index is not out of bounds (#17226) 2023-05-17 12:09:41 -07:00
Harshavardhana f7d29b4a53
cleanup of multipart per disk must cleanup itself only (#17223) 2023-05-17 01:45:58 -07:00
Harshavardhana 06557fe8be
allow decommissioned pools to be removed while others are finishing (#17221) 2023-05-16 16:00:57 -07:00
Poorna 2131046427
replication: fix audit log reporting (#17222) 2023-05-16 15:35:08 -07:00
Klaus Post aaf1abc993
simplify HardLimitReader by using LimitReader for internal usage (#17218) 2023-05-16 13:14:37 -07:00
jiuker 413549bcf5
fix: loadStatsFromDisk() should return nil for configNotFound (#17217) 2023-05-16 12:23:38 -07:00
jiuker 9a799065b3
fix: make slice cap of right size (#17192) 2023-05-16 08:10:07 -07:00
jiuker fd2959fa3a
fix: workers.New err must be returned (#17208) 2023-05-16 08:08:00 -07:00
jiuker 15bec32bb4
fix: tier handlers must write error only once (#17205) 2023-05-15 23:56:52 -07:00
Anis Eleuch e2b7a08c10
heal: Update object parity with the latest configured SC (#17187) 2023-05-15 21:32:13 -07:00
Harshavardhana ef2fc0f99e
fix: reduce using memory and temporary files. (#17206) 2023-05-15 14:08:54 -07:00
Harshavardhana d063596430
fix: veeam SOS API 'system.xml' strings (#17202) 2023-05-15 12:06:42 -07:00
jiuker bd2dc6c670
fix: in healing tracker printTo when err (#17207) 2023-05-15 10:14:48 -07:00
Harshavardhana b62791617c
fix: notify systemd as soon as we wait on the OS signal (#17199) 2023-05-12 16:42:17 -07:00
Poorna e07c2ab868
Use hash.NewLimitReader for internal multipart calls (#17191) 2023-05-12 11:19:08 -07:00
jiuker 203755793c
fix: in printEndpointError count error once per init() (#17193) 2023-05-12 10:41:54 -07:00
Anis Eleuch 883c98e26f
fix: remove objects when there are skipped versions due to ILM in decom (#17198) 2023-05-12 10:37:38 -07:00
Harshavardhana f5a20a5d06
allow nodes offline in k8s setups when expanding pools (#17183) 2023-05-11 17:41:33 -07:00
Poorna ef7177ebbd
disallow bucket replication setup with site replication (#17189) 2023-05-11 15:48:40 -07:00
Harshavardhana 3637aad36e
do not count ILM expired objects and other skipped objects (#17184) 2023-05-11 13:35:16 -07:00
Aditya Manthramurthy 77db9686fb
Update console to v0.27.0 (#17188) 2023-05-11 12:18:17 -07:00
Shireesh Anjal c326e5a34e
Add metrics for webhook endpoint stats (#17179) 2023-05-11 11:24:37 -07:00
jiuker c23c982593
xmlDecoder err use ErrMalformedXML when PutBucketACLHandler (#17185) 2023-05-11 11:11:15 -07:00
Shireesh Anjal a3d666356c
fix: error in capturing XFS error config in health report (#17176) 2023-05-10 15:20:48 -07:00
jiuker 3cdbc2f414
add validationErr to validateConfig When DeleteIdentityProviderCfg (#17173) 2023-05-10 09:37:30 -07:00
Harshavardhana b92cdea578
fix: start using pkg/workers to spawn parallel workers (#17170) 2023-05-09 16:37:31 -07:00
jiuker 5e629a99af
fix: for profiling duration parsing error reply use ErrInvalidRequest (#17169) 2023-05-09 14:27:49 -07:00
Harshavardhana a7f266c907
allow JWT parsing on large session policy based tokens (#17167) 2023-05-09 00:53:08 -07:00
Praveen raj Mani 57acacd5a7
Support persistent queue store for loggers (#17121) 2023-05-08 21:20:31 -07:00
jiuker 629503ff73
add Err to BucketExists when NoSuchBucket (#17155) 2023-05-08 07:51:59 -07:00
jiuker e3a070e3de
put *msgp.Reader back to pool (#17156) 2023-05-08 07:51:39 -07:00
Poorna c5c1426262
Validate if replication config being added is self referential (#17142) 2023-05-06 13:35:43 -07:00
Klaus Post 76913a9fd5
Signed trailers for signature v4 (#16484) 2023-05-05 19:53:12 -07:00
Harshavardhana 5569acd95c
disallow EC:0 if not set during server startup (#17141) 2023-05-04 14:44:30 -07:00
Harshavardhana 1d0211d395
allow deletes on directory objects to perform permanent deletes (#17132) 2023-05-04 14:43:52 -07:00
Anis Eleuch 06cd0a636e
Avoid calling KES Status when peers ping each other (#17140) 2023-05-04 11:28:33 -07:00
Klaus Post 7f7b489a3d
snowball: use latest time when mtime is missing (#17133) 2023-05-04 07:29:33 -07:00
Klaus Post bb6f4d7633
Remove redundant checkFormatJSON logging (#17134) 2023-05-04 07:28:37 -07:00
Alex 6e24dff26a
Added MINIO_BROWSER_LOGIN_ANIMATION env support for WebUI console (#17123) 
Signed-off-by: Benjamin Perez <benjamin@bexsoft.net>
 2023-05-03 15:32:50 -07:00
Harshavardhana e372e4e592
add nodeName to the log while taking drive offline (#17124) 2023-05-03 15:05:45 -07:00
Harshavardhana 9571b0825e
add configurable VRF interface and user-timeout (#17108) 2023-05-03 14:12:25 -07:00
Poorna 90e2cc3d4c
Add audit logging of site replication multipart proxying (#17122) 2023-05-03 11:19:45 -07:00
Harshavardhana 9112ca4e29
change ttfb_distribution metrics to histogramMetric (#17115) 2023-05-03 07:31:00 -07:00
Poorna ec84bad882
batch replication now supports arbitrary S3 targets (#17113) 2023-05-02 22:52:35 -07:00
Harshavardhana b53376a3a4
change directory objects to never create new versions (#17109) 2023-05-02 16:09:33 -07:00
Krishnan Parthasarathi 0ec722bc54
Add tags to NewerNoncurrentVersions audit event (#17110) 2023-05-02 12:56:33 -07:00
Praveen raj Mani 1704abaf6b
fix: store notification events immediately for persistent queues (#17112) 2023-05-02 07:53:13 -07:00
WGH ab34f0065c
Support systemd notify protocol (#17062) 2023-05-01 23:15:08 -07:00
Klaus Post e8c0a50862
optimization use small blocks up to 64KB (#17107) 2023-05-01 09:47:49 -07:00
Harshavardhana 02d8f3cdc8
fix: remove active healing on .minio.sys/ during startup (#17072) 2023-04-29 02:05:28 -07:00
Harshavardhana 7ae69accc0
allow root user to be disabled via config settings (#17089) 2023-04-28 12:24:14 -07:00
Klaus Post 7fad0c8b41
Remove checksums from HTTP range request, add part checksums (#17105) 2023-04-28 08:26:32 -07:00
Anis Eleuch d5aff735be
info: Add drives per set and sets count per pool information (#17100) 2023-04-27 15:24:03 -07:00
Poorna 98c26df53e
fix: allow past retention headers to be copied in batch replication (#17095) 2023-04-27 13:43:18 -07:00
Anis Eleuch 2448a9e047
grafana: Remove minio_s3_requests_errors_total metric (#17094) 2023-04-27 10:55:30 -07:00
Krishnan Parthasarathi e7cac8acef
Add tags to auditLogLifecycle (#17081) 2023-04-26 17:49:00 -07:00
Harshavardhana 6105997299
remove unnecessary log when listing resume fails (#17086) 2023-04-26 14:53:25 -07:00
Harshavardhana 8c874884fc
fix: do not copy context in DiskInfo cache (#17085) 2023-04-26 12:13:54 -07:00
Aditya Manthramurthy ebfe81e5fd
Fix put bucket policy error code (#17084) 2023-04-26 11:21:27 -07:00
Praveen raj Mani 72802a5972
Use 'minio/pkg/sync/errgroup' and 'minio/pkg/workers' (#17069) 2023-04-25 22:57:40 -07:00
Harshavardhana b1f3935c5b
allow ListObjects() when a prefix is an object (#17074) 2023-04-25 22:41:54 -07:00
Harshavardhana dbd53af369
fix: initialize reverse proxy forwarder with right public certs (#17080) 2023-04-25 15:50:32 -07:00
Harshavardhana b09fe0e50e
fix: DeleteBucket for peers() must recreate bucket upon errors (#17079) 2023-04-25 14:16:35 -07:00
Krishnan Parthasarathi fae9000304
heal: Pick maximally occuring modTime in quorum (#17071) 2023-04-25 10:13:57 -07:00
Harshavardhana 8fd07bcd51
simplify sort.Sort by using sort.Slice (#17066) 2023-04-24 13:28:18 -07:00
Anis Eleuch 6addc7a35d
server-info: Return initializing state properly (#17070) 2023-04-24 09:10:02 -07:00
Harshavardhana 477230c82e
avoid attempting to migrate old configs (#17004) 2023-04-21 13:56:08 -07:00
Harshavardhana d1737199ed
fix: delete DNS upon success, update failure message (#17059) 2023-04-21 12:12:31 -07:00
Harshavardhana 84f31ed45d
simplify MRF, converge it to regular healing (#17026) 2023-04-19 07:47:42 -07:00
jiuker 8a81e317d6
verify maxPartID in object options helpers (#17015) 2023-04-18 22:34:30 -07:00
Anis Eleuch 224d9a752f
fix: the race in healing tracker code (#17048) 2023-04-18 14:49:56 -07:00
Anis Eleuch 0db34e4b85
Listen bucket events to send empty events with new line (#17037) 2023-04-18 08:11:30 -07:00
Klaus Post f66625be67
Snowball: Extract headers for metadata (#17042) 2023-04-17 12:16:54 -07:00
Harshavardhana 6825bd7e75
fix: inlined objects don't need to honor long locks (#17039) 2023-04-17 12:16:37 -07:00
Klaus Post 839b9c9271
Reduce allocations in Walkdir (#17036) 2023-04-15 10:25:25 -07:00
Harshavardhana dd9ed85e22
implement support for FTP/SFTP server (#16952) 2023-04-15 07:34:02 -07:00
jiuker 6c1410f7f5
fix: Type of rejection for FIFO quota input (#17016) 2023-04-15 01:22:18 -07:00
Krishnan Parthasarathi f92450d8b3
commonParity should pick readable FileInfo (#17032) 2023-04-14 16:23:28 -07:00
Klaus Post c133979b8e
Add part count to checksum (#17035) 2023-04-14 09:44:45 -07:00
Poorna a9269cee29
heal: avoid logging version not found (#17031) 2023-04-13 19:45:52 -07:00
Klaus Post 958a480e53
fix: lambda function expiration when cred.Expiration is set (#17029) 2023-04-13 08:10:57 -07:00
Anis Eleuch a42650c065
Add minio_bucket_usage_version_total metric to Prometheus (#17023) 2023-04-12 20:08:07 -07:00
Harshavardhana bdad3730f7
fix: do not error out if the local bucket is missing (#17025) 2023-04-12 15:44:16 -07:00
Harshavardhana a5835cecbf
fix: regression in counting total requests (#17024) 2023-04-12 14:37:19 -07:00
Poorna cd6dec49c0
Add trace support for ilm activity (#16993) 2023-04-11 19:22:32 -07:00
Krishnan Parthasarathi 6877578bbc
Update minio_node_bucket_scans_finished metrics (#17006) 2023-04-11 19:21:34 -07:00
Harshavardhana f3682b6149
allow writes to pools with inconsistent xl.meta (#17008) 2023-04-11 11:17:46 -07:00
ferhat elmas 056ca0c68e
refactor: reuse open file in storage interface (#16970) 2023-04-09 23:09:28 -07:00
Krishnan Parthasarathi 25f7a8e406
Indicate RenameData is called by healObject (#16997) 2023-04-09 10:25:37 -07:00
Anis Eleuch 1f1c267b6c
Add used inodes to disk info (#16994) 2023-04-07 07:52:14 -07:00
Anis Eleuch fc94ea1ced
trace: Fix <unknown> func name of requests rejected by max clients (#16977) 2023-04-07 07:51:12 -07:00
jiuker 67d2cf8f30
add Err to Get getRemoteTargetClient. (#16982) 2023-04-07 07:50:46 -07:00
Harshavardhana 09a25ea7b7 lint: fix some lint issues on files 2023-04-06 22:42:10 -07:00
Klaus Post 260a63ca73
os/windows: do not return errFileNotFound in readDir's (#16988) 2023-04-06 22:28:58 -07:00
Shubhendu d3f70ea340
Enable audit log for global handlers (#16964) 
Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
 2023-04-06 21:03:39 -07:00
Anis Eleuch 91b6fe1af3
trace: Bootstrap to show the correct source line number (#16989) 2023-04-06 17:51:53 -07:00
Klaus Post 9803f68522
snowball: Restrict zstd window size (#16987) 2023-04-06 17:47:38 -07:00
Shubhendu 4c204707fd
Correct to remove `null` version while ILM rule application (#16971) 
Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
Co-authored-by: Harshavardhana <harsha@minio.io>
 2023-04-06 14:10:01 -07:00
Harshavardhana c06e0bfef9
set correct `Host:` value for replication event notification (#16984) 2023-04-06 10:20:53 -07:00
Klaus Post 8625a9dbb3
Make listing metadata permissions stricter (#16974) 2023-04-06 07:52:35 -07:00
jiuker 0320ac43cb
simplify bucketInfo return in GetBucketInfo peer call (#16983) 2023-04-06 01:30:50 -07:00
Anis Eleuch 111c7d4026
assumeRole return the correct http code for auth errors (#16967) 2023-04-05 22:19:31 -07:00
Klaus Post 62c3df0ca3
fix: directory listing on Go 1.20 windows (#16976) 2023-04-05 14:36:49 -07:00
Poorna 3158f2d12e
Add support for batch key rotation (#16844) 2023-04-04 10:56:54 -07:00
jiuker 6e359c586e
fix: close chan before return in scanner usage updates (#16960) 2023-04-04 10:51:05 -07:00
Poorna 699a24f7e5
batch: validate versioning on src/tgt buckets (#16955) 2023-04-04 10:50:11 -07:00
jiuker 5fa3665074
add isSysErrNotDir to openFileNoSync (#16958) 2023-04-04 08:00:08 -07:00
Harshavardhana 3b7781835e
add lock metrics per node (#16943) 2023-04-03 21:23:24 -07:00
Shubhendu 5fe1b46bfd
Enabled to send audit log while version deletion (#16954) 
Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
 2023-04-03 11:58:04 -07:00
Poorna 407c9ddcbf
feat: add batch replicate from remote MinIO to local cluster (#16922) 2023-03-31 10:48:36 -07:00
Anis Eleuch d90d0c8931
Use one http response recorder per external http call (#16938) 2023-03-31 09:37:29 -07:00
Harshavardhana 216a471bbb
on quorum DeleteObject() errors attempt an MRF (#16932) 2023-03-31 08:15:41 -07:00
jiuker a7b7860e0e
fix: potential data conflicts save site-resync metadata (#16926) 2023-03-30 20:59:45 -07:00
Klaus Post d703daa480
Add metadata extension to ListObjectVersions (#16930) 2023-03-30 12:20:42 -07:00
Poorna dc8fdcb9c9
fix: error checking in DeleteBucket (#16929) 2023-03-30 11:54:08 -07:00
Harshavardhana 7a6c4e438e
allow more workers for ILM expiration (#16924) 2023-03-30 10:47:15 -07:00
jiuker c468b4e2a8
fix: avoid out of slice index (#16925) 2023-03-30 08:03:48 -07:00
jiuker b04956a676
fix: put *msgp.Reader back in pool (#16927) 2023-03-30 08:03:12 -07:00
Aditya Manthramurthy 518f6e4d39
fix: missing return after error response (#16920) 2023-03-29 16:21:13 -07:00
Allan Roger Reid 483b226cc1
fix: avoid logging when object/version not found in replication (#16919) 2023-03-29 15:02:45 -07:00
Harshavardhana 8e02660a0d
update all our deps (#16899) 2023-03-28 03:45:24 -07:00
jiuker 16feef2a2c
fix: time.Parse RFC3339Nano (#16892) 2023-03-27 11:51:54 -07:00
jiuker 66ff17e452
fix: Avoid multiple write responses (#16894) 2023-03-27 09:15:23 -07:00
Harshavardhana 4c5edacae2
ignore operation timedout errors (#16891) 2023-03-26 03:16:51 -07:00
Anis Eleuch 8b4d0255b7
Set Console global Root CAs early to trust provided certs (#16890) 2023-03-25 09:58:38 -07:00
Poorna 74040b457b
Allow setting sync mode for site replication (#16876) 2023-03-24 14:41:23 -07:00
Klaus Post 5e3bfd2148
Add user tags when listing with metadata (#16883) 2023-03-23 10:27:19 -07:00
Klaus Post 8b0ab6ead6
Revert "Make localLocker lock attempts cancellable (#16510)" (#16884) 2023-03-23 10:26:21 -07:00
Shubhendu b1b0aadabf
Revert query parameter `src` from diag upload if callhome enabled (#16881) 2023-03-23 00:24:58 -07:00
Harshavardhana ac7d9c449a
add missing expiration information from 'sts info' (#16878) 2023-03-22 16:47:02 -07:00
Anis Eleuch 1346561b9d
return quorum error instead of insufficient storage error (#16874) 2023-03-22 16:22:37 -07:00
Minio Trusted 4bc52897b2 Update yaml files to latest version RELEASE.2023-03-22T06-36-24Z 2023-03-22 21:16:15 +00:00
Krishnan Parthasarathi 6017b63a06
top-locks: Group by lock request ID (#16860) 2023-03-21 18:35:29 -07:00
Harshavardhana 0448728228
fix: add deadline conns and dnsCache for remote transports (#16865) 2023-03-21 08:49:20 -07:00
Harshavardhana 12047702f5
fix: tweak the maintenance=true to satisfy baremetal first (#16864) 2023-03-21 08:48:38 -07:00
Harshavardhana fb1492f531
check for quorum errors for DeleteBucket() (#16859) 2023-03-20 23:38:06 -07:00
Aditya Manthramurthy 05444a0f6a
Use the official pub key to always verify binary (#16857) 2023-03-20 13:16:18 -07:00
Harshavardhana b3c54ec81e
reject object names with '\' on windows (#16856) 2023-03-20 13:16:00 -07:00
Harshavardhana 6c11dbffd5
add crash protection from backend modifications (#16846) 2023-03-20 09:08:42 -07:00
Harshavardhana 3b5dbf9046
allow bootstrapping to validate internode tokens (#16853) 2023-03-20 01:40:24 -07:00
Aditya Manthramurthy 09c733677a
Add test for fixed post policy exploit (#16855) 2023-03-20 01:06:45 -07:00
Harshavardhana 8d6558b236
fix: convert '\' to '/' on windows (#16852) 2023-03-20 00:35:25 -07:00
Aditya Manthramurthy 67f4ba154a
fix: post policy request security bypass (#16849) 2023-03-19 21:15:20 -07:00
Poorna 440ad20c1d
Add support for batch job cancellation (#16843) 2023-03-17 23:42:43 -07:00
Krishnan Parthasarathi 31fba6f434
Save bootstrap trace events in a circular buffer (#16823) 2023-03-17 16:01:03 -07:00
Harshavardhana 280442e533
reduce 250ms to 50ms retry looking for metacache block (#16795) 2023-03-17 14:44:01 -07:00
Shubhendu 850a945a18
Added query parameter `src` to diag upload if callhome enabled (#16837) 
Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
 2023-03-17 08:30:16 -07:00
Harshavardhana 46f9049fb4
simplify error responses for KMS (#16793) 2023-03-16 11:59:42 -07:00
Aditya Manthramurthy 58266c9e2c
Add enable flag for LDAP IDP config (#16805) 2023-03-16 11:58:59 -07:00
Poorna d1e775313d
support decommissioning of tiered objects (#16751) 2023-03-16 07:48:05 -07:00
Harshavardhana e700be8cd6
fix: return appropriate Location header for MakeBucket() (#16820) 2023-03-15 13:40:40 -07:00
Harshavardhana e0f4dd6027
remove unncessary logs from WalkDir(), PutObject() (#16818) 2023-03-15 11:52:23 -07:00
Harshavardhana de02eca467
restore rotating root credentials properly (#16812) 2023-03-15 08:07:42 -07:00
Harshavardhana c7f7e67a10
Do not allow adding root user to IAM subsystem (#16803) 2023-03-13 12:46:17 -07:00
Klaus Post 628042e65e
tests: Protect globalLocalDrives against races (#16800) 2023-03-13 06:04:20 -07:00
Aditya Manthramurthy 6305b206e1
fix: site-repl should heal STS with virtual parent (#16792) 2023-03-10 16:21:51 -08:00
Klaus Post d85da9236e
Add Object Version count histogram (#16739) 2023-03-10 08:53:59 -08:00
Anis Elleuch 5c087bdcad
fix: a cosmetic error reporting with a lock timeout (#16788) 2023-03-09 15:16:13 -08:00
Klaus Post a547bf517d
Remove locks on usage cache (#16786) 2023-03-09 15:15:46 -08:00
Harshavardhana b984bf8d1a
allow expiration of all versions during Listing() (#16757) 2023-03-09 15:15:30 -08:00
Poorna fb6ab1cca2
fix: allow replication of 'null' delete markers (#16773) 2023-03-08 07:03:29 -08:00
Krishnan Parthasarathi 56c57e2c53
tier-stats: Avoid repeated logs (#16774) 2023-03-07 16:43:05 -08:00
Poorna a6057c35cc
Avoid peer notification when peer is offline, tune retries (#16737) 2023-03-07 08:13:28 -08:00
Harshavardhana 901887e6bf
feat: add lambda transformation functions target (#16507) 2023-03-07 08:12:41 -08:00
Poorna ee54643004
Avoid unnecessary replication heal attempts (#16769) 2023-03-07 07:43:38 -08:00
Harshavardhana 0a17acdb34
return error if policy changes on disabled groups (#16766) 2023-03-06 10:46:24 -08:00
Harshavardhana 72e5212842
fix: handle syscall.EROFS also for osIsPermission() (#16765) 2023-03-06 08:56:29 -08:00
ferhat elmas 714283fae2
cleanup ignored static analysis (#16767) 2023-03-06 08:56:10 -08:00
ferhat elmas 3423028713
cleanup Go linter settings (#16736) 2023-03-04 20:57:35 -08:00
jiuker 9d062b37d7
return underlying error with BackendDown{} error (#16738) 2023-03-03 23:56:53 -08:00
jiuker f350953a19
calculate disk cache usage percent accurately (#16740) 2023-03-02 20:32:22 -08:00
Daniel Valdivia 958bba5b42
Attach creds, owner and region to madmin calls (#16658) 
Signed-off-by: Daniel Valdivia <18384552+dvaldivia@users.noreply.github.com>
 2023-03-02 14:35:24 -08:00
Poorna 0f2b95b497
fix a data race in IAM loading (#16742) 2023-03-02 14:32:54 -08:00
Klaus Post d07089ceac
Fix scanner deadlock on lost global lock (#16726) 2023-02-28 21:34:45 -08:00
Harshavardhana 0ff931dc76
fix: allow CORS to work by default (#16713) 2023-02-27 10:10:45 -08:00
Praveen raj Mani 4d708cebe9
Support adding service accounts with expiration (#16430) 
Co-authored-by: Harshavardhana <harsha@minio.io>
 2023-02-27 10:10:22 -08:00
dorman 4d7c8e3bb8
Remove redundant log (#16710) 
Co-authored-by: z30001483 <zekaifeng2@huawei.com>
 2023-02-27 09:59:47 -08:00
Aditya Manthramurthy 8cde38404d
Add metrics for custom auth plugin (#16701) 2023-02-27 09:55:18 -08:00
Krishnan Parthasarathi fe7bf6cbbc
Support tier-add if tier backend not empty (#16715) 2023-02-27 09:26:26 -08:00
Shubhendu 8b4eb2304b
Set logger webhook proxy on subnet proxy change (#16665) 
Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
 2023-02-27 08:35:36 -08:00
Harshavardhana ae029191a3
liveness returns "busy" if queued requests > available capacity (#16719) 2023-02-27 08:34:52 -08:00
Harshavardhana bfedea9bad
fix: disk healing should honor the right pool/set index (#16712) 2023-02-27 04:55:32 -08:00
Aditya Manthramurthy 7777d3b43a
Remove globalSTSTLSConfig (#16709) 2023-02-26 23:37:00 -08:00
Aditya Manthramurthy 9ed4fc9687
Remove globalOpenIDConfig (#16708) 2023-02-25 21:01:37 -08:00
jiuker b49b39e99d
fix: errNoSuchPolicy should use errors.Is (#16656) 2023-02-25 00:01:37 -08:00
jiuker cd3a2de5a3
add isValidLocation to common parseLocation (#16690) 2023-02-25 08:09:20 +05:30
jiuker 6e8960ccdd
fix: delete globalProfiler should lock (#16697) 2023-02-25 08:07:44 +05:30
Aditya Manthramurthy e05f3d5d84
Remove globalLDAPConfig (#16706) 2023-02-25 08:07:22 +05:30
Anis Elleuch 94c6cb1323
tests: Add test for S3 API error codes (#16705) 2023-02-25 08:06:29 +05:30
Klaus Post 9acf1024e4
Remove bloom filter (#16682) 
Removes the bloom filter since it has so limited usability, often gets saturated anyway and adds a bunch of complexity to the scanner.

Also removes a tiny bit of CPU by each write operation.
 2023-02-24 09:03:31 +05:30
Harshavardhana a0f06eac2a
add Veeam SOS API first implementation (#16688) 2023-02-22 19:54:57 +05:30
jiuker 83fe1a2732
log: add more info about BROWSER_URL (#16678) 2023-02-22 19:54:05 +05:30
Harshavardhana 5c98223c89
add correct HostId instead of deploymentId for error responses (#16686) 2023-02-22 15:41:09 +05:30
jiuker 663a0b7783
save correct bucketInfo on it's indexes (#16685) 2023-02-22 14:08:34 +05:30
Anis Elleuch 6efe4d1df6
perf: Only remove generated data when no bucket name specified (#16610) 2023-02-21 21:21:40 -08:00
Daniel Valdivia fb17f97cf3
move audit and logger message structure to minio/pkg (#16655) 
Signed-off-by: Daniel Valdivia <18384552+dvaldivia@users.noreply.github.com>
 2023-02-21 21:21:17 -08:00
Poorna 9202c6e26a
Reload bucket targets when replication config is refreshed (#16684) 2023-02-21 21:18:49 -08:00
Poorna 59a5456091
fix: STS error translation to API error (#16683) 2023-02-22 09:57:48 +05:30
Allan Roger Reid 8bfe972bab
Set meaningful message from minio with env variable KMS_SECRET_KEY (#16584) 2023-02-22 07:13:01 +05:30
Klaus Post fd6622458b
Add detailed scanner trace output and notifications (#16668) 2023-02-21 09:33:33 -08:00
Poorna 8a08861dd9
fix: healing of replication config for endpoint changes (#16648) 2023-02-20 16:06:13 +05:30
Harshavardhana b66d7dc708
add missing x-amz-id-2 to event notification date (#16646) 2023-02-20 15:41:47 +05:30
Harshavardhana b94733ab31
avoid locks when unnecessary in SiteReplicationMetaInfo() (#16650) 2023-02-18 05:35:22 -08:00
Klaus Post 84bb7d05a9
fix: healing deadlocks and ordering (#16643) 2023-02-17 23:22:43 +05:30
Harshavardhana 98a84d88e2
fix: trim and ignore './' directories (#16642) 2023-02-17 07:15:03 -08:00
jiuker e470268c7c
fix: a possible closer leak in SelectObjectHandler (#16598) 2023-02-17 01:44:40 -08:00
jiuker 3a6cd4f73d
fix: sync.pool won't Put reader back (#16638) 2023-02-17 01:42:43 -08:00
Harshavardhana 6ea150fd68
fix: avoid printing certain errors under few locations (#16631) 2023-02-17 01:40:31 -08:00
Anis Elleuch a7188bc9d0
fix: evaluate BypassGov policy action in deletion correctly (#16635) 2023-02-17 07:53:34 +05:30
Krishnan Parthasarathi a1dd08f2e6
Include tier name in MinIO/S3 target user-agent (#16630) 2023-02-15 22:09:46 -08:00
Krishnan Parthasarathi d136ac0596
Don't close transition task channel on server exit (#16627) 2023-02-15 22:09:25 -08:00
Poorna c33a237067
fix: under site replication disallow remote target modification (#16628) 2023-02-15 20:22:13 -08:00
Poorna eb7d3da994
Fix site replication status reporting of quota (#16626) 2023-02-16 08:08:35 +05:30
Harshavardhana 37134e42d4
ignore io.EOF, io.ErrUnexpectedEOF on xl.meta reads in WalkDir() (#16625) 2023-02-15 07:12:48 -08:00
Harshavardhana 0c1f8b4e0f
add user-agent for all minio.Client usage (#16619) 2023-02-14 13:19:30 -08:00
Anis Elleuch 857674c3a0
heal: Do not mark buckets as done when there is no online disks (#16621) 2023-02-14 12:50:13 -08:00
Harshavardhana 15a75bd79b
ignore preconditionFailed error in batch replication (#16615) 2023-02-14 07:22:08 -08:00
Andreas Auernhammer 74887c7372
kms: add support for KES API keys and switch to KES Go SDK (#16617) 
Signed-off-by: Andreas Auernhammer <hi@aead.dev>
 2023-02-14 07:19:20 -08:00
Harshavardhana 31188e9327
add parallel workers in batch replication (#16609) 2023-02-13 12:07:58 -08:00
Harshavardhana ee6d96eb46
Periodically remove stale buckets from in-memory (#16597) 2023-02-13 08:09:52 -08:00
Anis Elleuch d6d01067a0
fix: allow global leader lock context merge to be canceled (#16603) 
Global leader lock was first designated to only acquired once
until the node is killed. However, currently, the code acquires
it repeatedly during the lifetime of the server, now there is a
goroutine leak.
 2023-02-13 01:26:38 -08:00
jiuker a15b6f21b8
remove incorrect use of WaitGroup (#16596) 2023-02-12 20:59:45 -08:00
Anis Elleuch 689179bf18
ServerInfo: return per erasure set information (#16583) 2023-02-11 18:31:56 +05:30
Klaus Post d0f4cc89a5
Re-add Veeam Listing workaround (#16593) 2023-02-10 10:48:39 -08:00
Harshavardhana d65debb6bc
fix: comply with RFC6750 UserInfo endpoint requirements (#16592) 2023-02-10 22:20:25 +05:30
Harshavardhana 72daccd468
fix: scanner in healing cycle must use actual size (#16589) 2023-02-10 06:53:03 -08:00
Harshavardhana b363400587
fix: username replacements for aws:username must use parentUser (#16591) 2023-02-10 06:52:31 -08:00
jiuker b9f5f9ba3f
fix: aclHandlers convert XML parse error to relevant client error (#16587) 2023-02-10 02:59:10 -08:00
Anis Elleuch c8ffa59d28
Periodically refresh buckets metadata from the backend disks (#16561) 
fixes #16553
 2023-02-09 10:29:20 -08:00
Poorna 52aeebebea
Fix site replication meta info call to be non-blocking (#16526) 
Co-authored-by: Harshavardhana <harsha@minio.io>
 2023-02-08 21:16:53 -08:00
Harshavardhana 71f02adfca Revert "Print golang http errors in MinIO log format (#16465)" 
This reverts commit 1fd7946dce.
 2023-02-09 09:27:27 +05:30
Krishnan Parthasarathi 9de26531e4
tiering: UpdateWorkers may be called before Init (#16573) 2023-02-08 19:13:34 -08:00
Anis Elleuch b1d98febfd
New disk healing goes through the healing workers (#16568) 2023-02-08 09:25:29 -08:00
jiuker 1828fb212a
fix: avoid goroutine leak after timeouts in PeerMetrics (#16569) 2023-02-08 09:11:16 -08:00
jiuker be92046dfd
Anonymous pools in health info output are double counted (#16566) 2023-02-08 06:58:50 -08:00
Krishnan Parthasarathi 990fc415f7
Ensure safety of transitionState at startup (#16563) 2023-02-07 23:11:42 -08:00
Harshavardhana d8daabae9b
make site replication healing safer (#16560) 2023-02-07 21:44:42 -08:00
Harshavardhana 84fe4fd156
fix: multiObjectDelete by passing versionId for authorization (#16562) 2023-02-08 08:01:00 +05:30
Harshavardhana 747d475e76
initialize subsystems that are not dependent on buckets first (#16559) 2023-02-07 12:46:47 -08:00
Anis Elleuch 095b518802
Show a better error msg when internal data encryption key is incorrect (#16549) 2023-02-07 05:22:54 -08:00
Harshavardhana 0319ae756a
fix: pass proper username (simple) string as expected (#16555) 2023-02-07 03:43:08 -08:00
Harshavardhana 11c7ecb5cf
support if-match/if-none-match with s3 uploads (#16551) 2023-02-06 18:58:29 -08:00
Anis Elleuch ffd57fde90
Convert 'server closed idle connection' to errDiskNotFound (#16548) 2023-02-06 10:42:08 -08:00
jiuker a451d1cb8d
fix: close resp.body checking for kubernetes version (#16547) 2023-02-06 10:41:41 -08:00
Harshavardhana 14cf8f1b22
upgrade deps for minio/pkg v1.6.1 to include groups conditions (#16538) 2023-02-06 09:27:29 -08:00
Harshavardhana 5996c8c4d5
feat: allow offline disks on a fresh start (#16541) 2023-02-06 09:26:09 -08:00
Harshavardhana 21885f9457
fix: liveness/readiness must return errors if KMS is unreachable (#16540) 2023-02-06 08:55:56 -08:00
Harshavardhana aa8b9572b9
remove double ENABLED help output (#16528) 2023-02-03 05:52:52 -08:00
Poorna 876e1a91b2
replication: Fix typo checking PreconditionFailed status code (#16517) 2023-02-02 19:22:02 +05:30
Anis Elleuch 2278fc8f47
Print original error when IAM load is failed in some places (#16511) 2023-02-01 17:32:22 +05:30
Harshavardhana a91f353621
support 'mc admin service restart' for windows (#16512) 2023-02-01 17:31:46 +05:30
Klaus Post cdb1b48ad9
Make localLocker lock attempts cancellable (#16510) 2023-01-31 09:41:17 -08:00
Kaan Kabalak 2d0f30f062
Fix typo in code comment (#16509) 2023-01-31 07:54:19 +05:30
Krishnan Parthasarathi cea2ca8c8e
Add restore-status header for multipart objects (#16508) 2023-01-31 07:53:45 +05:30
Klaus Post f713436dd0
Fix truncated list response on deleted replicated objects (#16504) 2023-01-30 09:13:53 -08:00
Klaus Post b923a62425
Check pool-index for invalid setups (#16501) 2023-01-30 18:33:07 +05:30
Harshavardhana 67fce4a5b3
fix: dangling delete() upon success should return 404 (#16494) 2023-01-27 12:43:45 -08:00
Poorna eaa65b7ade
fix replication healing on list to consider all versions (#16496) 2023-01-27 12:43:28 -08:00
Poorna 820d94447c
replication: fix target bucket passed on GET proxy (#16495) 2023-01-27 10:24:51 -08:00
Poorna ed20134a7b
replication: detect proxy header presence correctly (#16489) 2023-01-27 01:29:32 -08:00
Harshavardhana d19cbc81b5
fix: do not return IAM/Bucket metadata replication errors to client (#16486) 2023-01-26 11:11:54 -08:00
Anis Elleuch 1fd7946dce
Print golang http errors in MinIO log format (#16465) 2023-01-26 22:46:16 +05:30
Klaus Post 027ff0f3a8
fix: set modTime to current in snowball if archive shows empty (#16482) 2023-01-26 22:20:35 +05:30
Harshavardhana 54b561898f
fix: anonymize the x-amz-id-2 value from hostname (#16478) 2023-01-25 10:25:36 -08:00
Harshavardhana 65c104a589
add x-amz-id-2 to indicate the node that received the request (#16474) 2023-01-25 09:14:10 -08:00
Anis Elleuch 0a0416b6ea
Better error when setting up replication with a service account alias (#16472) 2023-01-25 21:50:12 +05:30
Anis Elleuch 441babdc41
Rename peer S3 prefix to avoid collision in the future (#16473) 2023-01-25 06:46:30 -08:00
Harshavardhana e64b9f6751
fix: disallow SSE-C encrypted objects on replicated buckets (#16467) 2023-01-24 15:46:33 -08:00
Florian Schwab d67a846ec4
allow restarting of decommissioning if completed, failed or canceld (#16464) 2023-01-24 07:07:59 -08:00
Poorna ca2a1c3f60
replication: clone metrics while loading metrics cache (#16462) 2023-01-24 02:10:32 -08:00
Poorna 93fbb228bf
Validate if parent user exists for service acct (#16443) 2023-01-24 08:17:18 +05:30
Anis Elleuch f37a5b6dae
Add CPU info in the check update user-agent (#16447) 2023-01-23 08:07:55 -08:00
Harshavardhana 31b0decd46
migrate to minio/mux from gorilla/mux (#16456) 2023-01-23 16:42:47 +05:30
Harshavardhana eb561e1c05
allow bootstrap platform checks to be pool specific (#16455) 2023-01-23 16:24:50 +05:30
Poorna ddad231921
replication: Avoid logging PreConditionFailed error (#16450) 2023-01-21 07:33:04 +05:30
Klaus Post 03b94f907f
fix: deleted object names for directory objects (#16448) 2023-01-20 21:16:06 +05:30
Shireesh Anjal 0f591d245d
fix: incorrect anonymization of drive endpoint (#16442) 2023-01-20 07:35:44 +05:30
Poorna 1b02e046c2
Fix bandwidth monitoring to be per remote target (#16360) 2023-01-19 18:52:16 +05:30
Harshavardhana d08e3cc895
add a way to avoid blocking queueHealTask() depending on caller (#16433) 2023-01-19 18:50:54 +05:30
Anis Elleuch d98116559b
Use async healing in PutObject call (#16431) 2023-01-19 00:54:22 -08:00
Krishnan Parthasarathi 71c95ad0d0
Signal stop-rebalance to all rebalancing pools (#16438) 2023-01-19 06:54:23 +05:30
Aditya Manthramurthy 698862ec5d
Fix transports/timeouts related regressions (#16427) 2023-01-18 10:06:38 +05:30
Harshavardhana b4ef5ff294
remove unnecessary code checking for supported features (#16423) 2023-01-17 19:37:47 +05:30
Harshavardhana 3db658e51e
use correct xml package for custom MarshalXML() (#16421) 2023-01-17 05:08:33 +05:30
Shireesh Anjal 5a9f7516d6
Add monthly license update job (#16391) 2023-01-17 05:08:15 +05:30
Anis Elleuch 3039fd4519
Optimize background heal status to use LocalStorageInfo (#16414) 2023-01-17 05:02:00 +05:30
Harshavardhana 095fc0561d
feat: allow decom of multiple pools (#16416) 2023-01-16 21:36:34 +05:30
Anis Elleuch beb1924437
Properly restart fresh disk healing when failed in some places (#16413) 2023-01-14 05:06:46 +05:30
jiuker c8e1154f1e
fix: reading from erasureDisks must be protected via read lock() (#16407) 2023-01-13 04:16:23 -08:00
Poorna b204c2dbec
fix: enforce deny on DeleteVersionAction (#16409) 2023-01-13 04:16:00 -08:00
Poorna b22b39de96
Avoid dangling deletes if disk not found (#16401) 2023-01-12 22:20:19 -08:00
Harshavardhana c242e6c391
fix: calculate common parity properly (#16406) 2023-01-13 03:28:16 +05:30
Anis Elleuch e05205756f
metrics: Add more logs when unable to read bucket usage (#16405) 2023-01-13 02:32:00 +05:30
Anis Elleuch 475a88b555
fix: error out if an object is found after a full decom (#16277) 2023-01-12 05:52:51 +05:30
Anis Elleuch 1ece3d1dfe
Add comment field to service accounts (#16380) 2023-01-10 21:57:52 +04:00
Anis Elleuch 2146ed4033
xl: Quit early when EC config is incorrect (#16390) 
Co-authored-by: Anis Elleuch <anis@min.io>
 2023-01-09 23:07:45 -08:00
Anis Elleuch ebd4388cca
s3: Return XMinioInvalidObjectName if the object contains null char (#16372) 2023-01-06 10:11:18 -08:00
Anis Elleuch 0333412148
fix: heal only once per disk per set among multiple disks (#16358) 2023-01-05 20:41:19 -08:00
Harshavardhana e0086c1be7
reduce startup delays on kubernetes (#16356) 2023-01-05 02:32:43 -08:00
Anis Elleuch 7883e55da2
Merge buckets list from different nodes in ListBuckets() call (#16357) 2023-01-04 08:53:58 -08:00
Harshavardhana a15a2556c3
converge listBuckets() as a peer call (#16346) 2023-01-03 23:39:40 -08:00
Harshavardhana f1bbb7fef5
vectorize cluster-wide calls such as bucket operations (#16313) 2023-01-03 08:16:39 -08:00
Harshavardhana 1cd8e1d8b6
remove the startup jitter before locks() (#16340) 2023-01-02 01:40:09 -08:00
jiuker 62cd918061
fix: close helmInfo file descriptor (#16319) 2023-01-01 23:26:59 -08:00
Klaus Post 6a04067514
fix: tweak read buffer size to reduce over-reading (#16338) 2023-01-01 08:14:20 -08:00
Taran Pelkey 49b3908635
fix: misplaced write response command in DetachPolicy() (#16333) 2022-12-30 20:04:03 -08:00
Harshavardhana f93183f66e
fix: a deadlock by refactoring listBuckets() under site replication (#16323) 2022-12-29 00:08:31 -08:00
Harshavardhana 2937711390
fix: DeleteObject() API with versionId under replication (#16325) 2022-12-28 22:48:33 -08:00
Anis Elleuch 27417459fb
metrics: Show healing info for all nodes (#16315) 2022-12-26 08:35:32 -08:00
Harshavardhana 5b8fe2e89a
allow locks with object affinity to spread across pools (#16312) 2022-12-23 20:55:45 -08:00
Anis Elleuch acc9c033ed
debug: Add X-Amz-Request-ID to lock/unlock calls (#16309) 2022-12-23 19:49:07 -08:00
Poorna 8528b265a9
Validate replication target update to avoid duplicate endpoints (#16311) 2022-12-23 15:44:48 -08:00
Harshavardhana b882310e2b
avoid locks for internal and invalid buckets in MakeBucket() (#16302) 2022-12-23 07:46:00 -08:00
Poorna de0b43de32
persist replication stats with leader lock (#16282) 2022-12-22 14:25:13 -08:00
jiuker 29dd7f1d68
tier verification leaks fd, that must be closed (#16296) 
Co-authored-by: Harshavardhana <harsha@minio.io>
 2022-12-22 10:35:54 -08:00
Poorna 6423e4c767
Remove site replication config if it succeeded locally (#16279) 2022-12-22 01:31:20 -08:00
Krishnan Parthasarathi 2fa35def2c
Fix DeleteObject when only free versions remain (#16289) 2022-12-21 16:24:07 -08:00
Anis Elleuch 34167c51d5
trace: Add bootstrap tracing events (#16286) 2022-12-21 15:52:29 -08:00
Harshavardhana a5f8af4efb
serialize replication stats() only when needed (#16280) 2022-12-20 00:07:53 -08:00
Harshavardhana 5a218f38a1
allow retries for transaction lock on startup (#16273) 2022-12-19 22:00:00 -08:00
Anis Elleuch e57e946206
Do not save credentials in config.json (#16275) 2022-12-19 12:27:06 -08:00
Klaus Post b4f71362e9
Avoid config migration on every startup (#16278) 2022-12-19 11:10:14 -08:00
Taran Pelkey ed37b7a9d5
Add API to fetch policy user/group associations (#16239) 2022-12-19 10:37:03 -08:00
Anis Elleuch 89db3fdb5d
Do not return an error when version disparity is detected (#16269) 2022-12-16 08:52:12 -08:00
Harshavardhana 80fc3a8a52
use newDynamicTimeoutWithOpts() when appropriate (#16266) 2022-12-15 13:11:37 -08:00
Klaus Post 988a2e8fed
Faster startup of large distributed systems with latency (#16259) 2022-12-15 08:31:21 -08:00
Harshavardhana 2433698372
fix: remove unnecessary logs for client conn errors (#16261) 2022-12-15 08:25:05 -08:00
Harshavardhana 5d7e8f79ed
fix: remove scanner healing with unnecessary logs (#16260) 2022-12-14 16:39:18 -08:00
Harshavardhana bad229e16e
fix: support event name s3:Restore:* (#16257) 2022-12-14 05:12:07 -08:00
Poorna d37e514733
Cleanup remote targets automatically on replication config removal. (#16221) 2022-12-14 03:24:06 -08:00
Harshavardhana c73ea27ed7
do not log checksum mismatch error, client received the error (#16246) 2022-12-14 01:57:40 -08:00
Krishnan Parthasarathi 0159b56717
fix: rebalance to account for object's on-disk size (#16240) 2022-12-14 00:15:14 -08:00
Aditya Manthramurthy 9e6cc847f8
Add HTTP2 config option for policy plugin (#16225) 2022-12-13 14:28:48 -08:00
Taran Pelkey 709eb283d9
Add endpoints for managing IAM policies (#15897) 
Co-authored-by: Taran <taran@minio.io>
Co-authored-by: ¨taran-p¨ <¨taran@minio.io¨>
Co-authored-by: Aditya Manthramurthy <donatello@users.noreply.github.com>
 2022-12-13 12:13:23 -08:00
Anis Elleuch 76dde82b41
Implement STS account info API (#16115) 2022-12-13 08:38:50 -08:00
Anis Elleuch 939c0100a6
log: Do not interpret verbs in object names in console output (#16233) 2022-12-13 08:27:40 -08:00
Aditya Manthramurthy 2d60bf8c50
Refactor HTTP transports (#16222) 2022-12-12 20:31:21 -08:00
Harshavardhana 37e20f6ef2
feat: allow listening specific addrs for API port (#16223) 2022-12-12 18:48:46 -08:00
Harshavardhana 2fc182d8e6
fix: iso8601TimeFormat padding issue for certain nanoseconds (#16207) 2022-12-12 10:28:30 -08:00
Shireesh Anjal a2cbeaa9e6
Use different subnet public key during dev/test (#16216) 2022-12-12 10:28:15 -08:00
Harshavardhana 444ff20bc5
do not rename multipart failed transactions back to tmp (#16204) 2022-12-12 01:40:29 -08:00
Harshavardhana 20ef5e7a6a
avoid double deletes() when no more versions (#16206) 2022-12-12 01:40:04 -08:00
Aditya Manthramurthy e06127566d
Add IAM API to attach/detach policies for LDAP (#16182) 2022-12-09 13:08:33 -08:00
Harshavardhana dfe73629a3
fix: delete marker discrepancies via DeleteObject() API (#16195) 2022-12-08 18:15:16 -08:00
Harshavardhana b03dd1af17
remove hard limit for number of buckets (#16194) 2022-12-08 12:24:03 -08:00
Harshavardhana 4bc367c490
fix: translate tier add errors properly (#16191) 2022-12-08 11:18:07 -08:00
Klaus Post 3eb2d086b2
Replace filepathx with fork (#16192) 2022-12-08 10:42:44 -08:00
Klaus Post 70986b6e6e
Add version id to healresult (#16193) 2022-12-08 07:49:10 -08:00
Klaus Post ebe395788b
feat: Encrypt s3zip file index (#16179) 2022-12-07 14:56:07 -08:00
Klaus Post 12fd6678ee
Encrypt checksums with KMS on CompleteMultipartUpload (#16177) 2022-12-07 10:18:18 -08:00
Harshavardhana 90d35b70b4
remove unnecessary logs for truncated XML inputs (#16184) 2022-12-07 08:30:52 -08:00
Javier Adriel 04ae9058ed
Populate end_session_endpoint (#16183) 2022-12-06 16:56:37 -08:00
Aditya Manthramurthy a30cfdd88f
Bump up madmin-go to v2 (#16162) 2022-12-06 13:46:50 -08:00
Anis Elleuch 1bae32dc96
xl: Delete older data-dir when replacing an existing version-id (#16176) 2022-12-06 13:43:18 -08:00
Anis Elleuch 932d2c3c62
Add X-Amz-Request-Id to internode calls (#16146) 2022-12-06 09:27:26 -08:00
jiuker 8d8d07ac5c
use readlock instead of writelock to get heal information (#16175) 2022-12-06 08:08:22 -08:00
Anis Elleuch 44735be38e
s3: Return correct error when Version is invalid in policy document (#16178) 2022-12-06 08:07:24 -08:00
Klaus Post 3fd9059b4e
opt: Only stream big data usage caches (#16168) 2022-12-05 13:01:11 -08:00
Klaus Post a713aee3d5
Run staticcheck on CI (#16170) 2022-12-05 11:18:50 -08:00
Andreas Auernhammer d882ba2cb4
kms: add support for KES enclaves (#16139) 
Signed-off-by: Andreas Auernhammer <hi@aead.dev>
 2022-12-04 02:34:24 -08:00
jiuker 6086f45d25
fix: in disk cache readCacheFileStream should closed upon return (#16138) 2022-12-04 02:28:10 -08:00
Klaus Post 98cffbce03
s3zip: Limit over-read for single file (#16161) 2022-12-02 08:53:24 -08:00
Klaus Post 1cd875de1e
Persist updated metadata (#16160) 2022-12-02 08:35:04 -08:00
Harshavardhana 5a8df7efb3
re-implement StorageInfo to be a peer call (#16155) 2022-12-01 14:31:35 -08:00
Anis Elleuch c84e2939e4
trace: Publish storage layer errors (#16153) 2022-12-01 12:10:54 -08:00
Anis Elleuch 641ab24aec
repl: resync orchestrator to use global shared lock (#16154) 2022-12-01 12:10:09 -08:00
Harshavardhana 71133105d7
re-order the top-level config keys for priority (#16150) 2022-12-01 07:50:08 -08:00
Aditya Manthramurthy 87cbd41265
feat: Allow at most one claim based OpenID IDP (#16145) 2022-11-29 15:40:49 -08:00
Klaus Post cc1d8f0057
Check for abandoned data when healing (#16122) 2022-11-28 10:20:55 -08:00
Anis Elleuch 1f1dcdce65
move HTTP recorder to an internal library (#16128) 2022-11-28 10:20:27 -08:00
Shireesh Anjal 98a67a3776
Improvements in logger and audit webhooks (#16102) 2022-11-28 08:03:26 -08:00
Poorna 63fc6ba2cd
preserve replicated ETag properly on target (#16129) 2022-11-26 14:43:32 -08:00
jiuker ce53d7f6c2
add disk.Close() in healFreshDisk to indicate idiomatic flow of code (#16124) 2022-11-26 00:26:15 -08:00
jiuker fe8eed963e
fix: wrapped error will not equal in decommissioning (#16113) 2022-11-24 08:00:42 -08:00
Shireesh Anjal 59f877fc64
fix: Timestamp not added in diagnostics report (#16114) 2022-11-23 07:11:22 -08:00
Klaus Post f96fe9773c
fix: duplicated shared prefix with custom delimiter when listing (#16111) 2022-11-22 08:51:04 -08:00
Anis Elleuch 04948b4d55
fix: checking for stale STS account under site replication (#16109) 2022-11-22 07:26:33 -08:00
Klaus Post 98ba622679
Reduce temporary file clean-up waits (#16110) 2022-11-22 07:23:36 -08:00
Harshavardhana 08103870a5
update single drive setup error message (#16098) 2022-11-18 14:47:38 -08:00
Anis Elleuch 993e586855
config: return XMinioConfigNotFound code for non existing config (#16065) 2022-11-18 10:28:14 -08:00
Harshavardhana 58ec835af0
fix: skip free version ID and marker in metadata equality (#16093) 2022-11-18 05:48:22 -08:00
Harshavardhana 6aea950d74
avoid partID lock validating uploadID exists prematurely (#16086) 2022-11-18 03:09:35 -08:00
Poorna 7198be5be9
bucket resync: persist reset id to bucket metadata (#16088) 2022-11-18 01:39:05 -08:00
Klaus Post a22b4adf4c
distribute replication ops based on names (#16083) 2022-11-17 15:20:09 -08:00
Klaus Post b7bb122be8
fix: replication auto-scaling deadlock (#16084) 2022-11-17 07:35:02 -08:00
Krishnan Parthasarathi 8441a3bf5f
fix: update metacache entry only once (#16072) 2022-11-16 11:25:00 -08:00
Harshavardhana 853c4de75a
allow changing endpoints in distributed setups (#16071) 2022-11-16 07:59:10 -08:00
jiuker 3597af789e
allow resultCh to be closed() after clusterMetaHealthInfo() (#16073) 2022-11-16 03:04:36 -08:00
Shireesh Anjal 5246e3be84
Send health diagnostics data as part of callhome (#16006) 2022-11-15 13:53:05 -08:00
Klaus Post 8a07000e58
fix: refactor getReplicationDiff for safe use (#16051) 2022-11-15 07:59:21 -08:00
Krishnan Parthasarathi 3bb82ef60d
top-locks: Include lock-held duration (#16061) 2022-11-15 07:57:52 -08:00
Harshavardhana 91f45c4aa6
avoid inconsistent versions healing when versions are large (#16066) 2022-11-14 18:35:26 -08:00
Poorna d6bc141bd1
feat: Add support for site level resync (#15753) 2022-11-14 07:16:40 -08:00
jiuker 7ac64ad24a
fix: use errors.Is for wrapped returns (#16062) 2022-11-14 07:15:46 -08:00
Harshavardhana 6d76db9d6c
improve server startup error when pools are incorrect (#16056) 2022-11-11 19:40:45 -08:00
jiuker bdcb485740
netPerfRX Reset() should use write Lock() (#16043) 2022-11-10 19:44:20 -08:00
Poorna e32b948a49
fix: parsing multipart uploadID under site replicated setup (#16048) 
continue the fix from #16034
 2022-11-10 16:17:45 -08:00
Klaus Post 5b242f1d11
Add Audit target metrics (#16044) 2022-11-10 10:20:21 -08:00
Poorna 34d28dd79f
replication: Avoid blocking on mrf save (#16045) 2022-11-10 10:20:02 -08:00
Krishnan Parthasarathi 6eef9b4a23
lifecycle: simplify Eval and HasActiveRules (#16036) 2022-11-10 07:17:45 -08:00
Aditya Manthramurthy 5f1999cc71
fix: avoid URL unsafe chars in multipart upload ID (#16034) 2022-11-09 16:41:16 -08:00
Krishnan Parthasarathi 40a2c6b882
Return remote tier as StorageClass for transitioned objects (#16035) 2022-11-09 15:57:34 -08:00
jiuker 7b7356f04c
close the reader under disk cache bitrot verification (#16024) 2022-11-09 04:20:11 -08:00
Klaus Post bbc312fce6
Add notification queue metrics (#16026) 2022-11-08 16:36:47 -08:00
Anis Elleuch 7260241511
Remove some logs caused by external apps (#16027) 2022-11-08 13:29:05 -08:00
Anis Elleuch 3b1a9b9fdf
Use the same lock for the scanner and site replication healing (#15985) 2022-11-08 08:55:55 -08:00
Harshavardhana 72afc2727a
rebalance status must return appropriate error initially (#16022) 2022-11-08 07:56:45 -08:00
Aditya Manthramurthy 76d822bf1e
Add LDAP policy entities API (#15908) 2022-11-07 14:35:09 -08:00
Klaus Post ddeca9f12a
fix: filter rest errors and logs returned (#16019) 2022-11-07 10:38:08 -08:00
Harshavardhana 1f3db03bf0
allow changing argument for path for SNSD setup (#16013) 2022-11-07 00:11:58 -08:00
Harshavardhana 944c62daf4
skip flaky tests on windows OS (#16015) 2022-11-07 00:11:21 -08:00
Harshavardhana 9547b7d0e9
add deadlineConnections on remoteTransport (#16010) 2022-11-05 11:09:21 -07:00
Klaus Post 808ecfe0f2
merge versions across sets when listing (#16003) 2022-11-04 11:33:22 -07:00
Klaus Post 2894dd4d1a
fix: hold lock while serializing replication stats (#16007) 2022-11-04 09:59:14 -07:00
jiuker fd8750e959
fix: http body must be drained in downloadBinary() (#16001) 2022-11-04 08:22:38 -07:00
Poorna 4f5d38a4b1
site replication edit: validate endpoint belongs to deployment (#16000) 2022-11-03 16:23:45 -07:00
Anis Elleuch 7e73fc2870
Implement inspect data API v2 (#15474) 
Co-authored-by: Klaus Post <klauspost@gmail.com>
 2022-11-02 13:36:38 -07:00
Harshavardhana 0d49b365ff
converge SNSD deployments into single code (#15988) 2022-11-01 16:41:01 -07:00
Anis Elleuch 7721595aa9
config: Deprecated delay/max_wait/scanner and introduce speed (#15941) 2022-11-01 08:04:07 -07:00
Harshavardhana fd6f6fc8df
cleanup stale parent multipart directories (#15980) 2022-11-01 08:00:02 -07:00
Aditya Manthramurthy 4fb47cd568
fix: update admin IDP APIs to be more RESTful (#15896) 2022-10-31 14:52:26 -07:00
Klaus Post ecc932d5dd
Clean entire tmp-old on restart (#15979) 2022-10-31 07:27:50 -07:00
Harshavardhana b57fbff7c1
ignore background healInfo in single drive setup (#15968) 2022-10-31 07:26:10 -07:00
Poorna d765b89a63
improve validation for replication resync API (#15964) 2022-10-28 23:21:33 -07:00
Harshavardhana 6e4acf0504
add a message of removal for gateway and hide the command (#15965) 2022-10-28 14:11:20 -07:00
Klaus Post 71954faa3a
mark pubsub type safe via generics (#15961) 2022-10-28 10:55:42 -07:00
Klaus Post 0f0e154315
fix: inconsistent replication delete marker timestamps (#15956) 2022-10-27 09:46:52 -07:00
Harshavardhana 136d41775f
remove numAvailableDisks check as it doesn't serve any purpose (#15954) 2022-10-27 09:05:24 -07:00
Harshavardhana ec77d28e62
make subnet subsys dynamic and simplify callhome (#15927) 2022-10-27 00:20:01 -07:00
Klaus Post 86420a1f46
Store multipart checksums (#15953) 2022-10-26 18:14:58 -07:00
Poorna 7dd8b6c8ed
ensure ILM expiry creates non null deleteMarker for versioned bucket (#15947) 2022-10-26 16:09:27 -07:00
Anis Elleuch 533c9d4fe3
fix: lockName to disallow parallel same erasure set healing (#15951) 2022-10-26 12:43:54 -07:00
Anis Elleuch a35ef155fc
return appropriate error status code in the lock handler (#15950) 2022-10-26 09:51:26 -07:00
Poorna 8dd3c41b2a
allow MakeBucket errors to be handled lazily (#15945) 
remote error is not required to be passed back to the 
client - this is mostly because we have healing that should 
eventually, catch up on this and heal the bucket.
 2022-10-25 23:32:37 -07:00
Krishnan Parthasarathi 4523da6543
feat: introduce pool-level rebalance (#15483) 2022-10-25 12:36:57 -07:00
Poorna ce8456a1a9
proxy multipart to peers via multipart uploadID (#15926) 2022-10-25 10:52:29 -07:00
Poorna 9ce1884732
reject editing bucket replication config when site replication is enabled (#15937) 2022-10-24 20:24:32 -07:00
Harshavardhana 23b329b9df
remove gateway completely (#15929) 2022-10-24 17:44:15 -07:00
Krishnan Parthasarathi 0c34e51a75
Filter out tiering metadata during CopyObject (#15936) 2022-10-24 16:32:31 -07:00
Anis Elleuch fc6c794972
Audit dangling object removal (#15933) 2022-10-24 11:35:07 -07:00
Klaus Post 86d543d0f6
Check for s3zip content offset (#15924) 2022-10-21 15:37:48 -07:00
Poorna e4e90b53c1
fix: delete-marker replication check properly (#15923) 2022-10-21 14:45:06 -07:00
Anis Elleuch 58d776daa0
Set CONSOLE_MINIO_SERVER to 127.0.0.1 by default (#15887) 2022-10-21 14:42:28 -07:00
Krishnan Parthasarathi f6b2e89109
Pass encrypted etag as is for immediate tiering (#15925) 2022-10-21 14:40:50 -07:00
Anis Elleuch ac85c2af76
lifecycle: refactor rules filtering and tagging support (#15914) 2022-10-21 10:46:53 -07:00
Shireesh Anjal 5aba2aedb3
Do not freeze s3 traffic in healthinfo api (#15912) 2022-10-21 00:34:32 -07:00
Harshavardhana a8332efa94
fix: Delete() of bucket metadata should not parse the config (#15904) 2022-10-19 17:55:09 -07:00
Aditya Manthramurthy 3dbef72dc7
fix: AccountInfo API for roleARN based accounts (#15907) 2022-10-19 17:54:41 -07:00
Aditya Manthramurthy 2d16e74f38
Add LDAP IDP Configuration APIs (#15840) 2022-10-19 11:00:10 -07:00
Anis Elleuch de5070446d
Deprecate --listeners flag (#15900) 2022-10-19 08:45:50 -07:00
Harshavardhana 374abd1e7d
add filter support for tags and metadata in batch replication (#15885) 2022-10-18 21:22:21 -07:00
Anis Elleuch 0506d9e83d
storage: Return errDiskNotFound when a peer is during shutdown (#15868) 2022-10-18 13:50:46 -07:00
Klaus Post bd3dfad8b9
Add concurrent Snowball extraction + options (#15836) 2022-10-18 13:50:21 -07:00
Harshavardhana 9fff315555
do not need to trace ignored objects (#15894) 2022-10-18 13:47:55 -07:00
Anis Elleuch 18fb86b7be
convert context.DeadlineExceed to offline disk in DiskInfo() (#15886) 2022-10-18 03:01:16 -07:00
Harshavardhana 58a8275e84
do not assume invalid buf to be non-xl.meta (#15843) 2022-10-17 09:39:21 -07:00
Aditya Manthramurthy 85fc7cea97
Pass role ARN for OIDC providers to console (#15862) 2022-10-15 12:57:03 -07:00
Harshavardhana 328d660106
support CRC32 Checksums on single drive setup (#15873) 2022-10-15 11:58:47 -07:00
Harshavardhana c68910005b
validate bucket before attempting batch replication (#15861) 2022-10-15 11:58:31 -07:00
Harshavardhana c79bcc8838 Revert "convert context.DeadlineExceed to offline disk in DiskInfo() (#15869)" 
This reverts commit 0fe58dbb34.
 2022-10-14 20:37:50 -07:00
Anis Elleuch 0fe58dbb34
convert context.DeadlineExceed to offline disk in DiskInfo() (#15869) 2022-10-14 19:32:13 -07:00
Harshavardhana 6cb2f56395 Revert "Revert "tests: Add context cancelation (#15374)"" 
This reverts commit 564a0afae1.
 2022-10-14 03:08:40 -07:00
Harshavardhana 59e33b3b21
validate setBucketTarget properly as per BucketExists() call (#15860) 2022-10-13 17:46:49 -07:00
Poorna 0e3c92c027 attempt delete marker replication after object is replicated (#15857) 
Ensure delete marker replication success, especially since the
recent optimizations to heal on HEAD, LIST and GET can force
replication attempts on delete marker before underlying object
version could have synced.
 2022-10-13 17:45:23 -07:00
Anis Elleuch db7a9b2c37
heal-info: Return the endpoint of a disk with unknown state (#15854) 2022-10-13 16:41:44 -07:00
Harshavardhana 44097faec1
support deleteMarkers and all versions in batch replication (#15858) 2022-10-13 14:42:10 -07:00
Klaus Post bf3da5081f
Omit empty checksums in responses (#15850) 2022-10-13 00:49:46 -07:00
Harshavardhana 5532982857
do not disable IsKubernetes(), IsDocker() checks with MINIO_CI_CD (#15852) 2022-10-12 23:40:48 -07:00
Anis Elleuch 783dd875f7
refactor objectQuorumFromMeta() to search for parity quorum (#15844) 2022-10-12 16:42:45 -07:00
Harshavardhana 97112c69be
fix: replication stats() to not crash under any situation (#15851) 
Co-authored-by: Daniel Valdivia <18384552+dvaldivia@users.noreply.github.com>
 2022-10-12 15:47:41 -07:00
Javier Adriel 2939000342
Add metrics, version and apis handlers (#15839) 2022-10-12 12:08:03 -07:00
Harshavardhana 41e1654f9a
remove spurious logging for object not found (#15842) 2022-10-12 04:28:21 -07:00
Harshavardhana e3cb0278ce
honor specified target prefix under batch replication (#15834) 2022-10-11 14:36:06 -07:00
Harshavardhana 0c81f1bdb3
indicate how long it took to bring the drive online (#15835) 2022-10-11 11:33:56 -07:00
Klaus Post 6220875803
Add missing server info fields (#15826) 2022-10-11 11:31:26 -07:00
Aditya Manthramurthy 64cf887b28
use LDAP config from minio/pkg to share with console (#15810) 2022-10-07 22:12:36 -07:00
Harshavardhana 927a879052
authenticate the request first for headObject() (#15820) 2022-10-07 21:45:53 -07:00
Anis Elleuch dfe0c96b87
preserve Version and DeleteMarker sort order in the list XML response (#15819) 2022-10-07 16:12:36 -07:00