Commit Graph

12055 Commits

Author SHA1 Message Date
Harshavardhana 64baedf5a4
fix: hide prefixes for Hadoop properly (#19821) 2024-05-28 15:53:15 -07:00
Minio Trusted 2f64d5f77e Update yaml files to latest version RELEASE.2024-05-28T17-19-04Z 2024-05-28 19:23:04 +00:00
Anis Eleuch f79a4ef4d0
policy: More defensive code validating svc:DurationSeconds (#19820)
This does not fix any current issue, but merging https://github.com/minio/madmin-go/pull/282
can lose the validation of the service account expiration time.

Add more defensive code for now. In the future, we should avoid doing
validation in another library.
2024-05-28 10:19:04 -07:00
Taran Pelkey 2d53854b19
Restrict access keys for users and groups to not allow '=' or ',' (#19749)
* initial commit

* Add UTF check

---------

Co-authored-by: Harshavardhana <harsha@minio.io>
2024-05-28 10:14:16 -07:00
Harshavardhana e5c83535af
chore: upgrade deps (#19819)
Signed-off-by: Harshavardhana <harsha@minio.io>
2024-05-28 02:27:44 -07:00
jiuker c904ef966e
feat: support tags for PostPolicy upload (#19816) 2024-05-27 21:44:00 -07:00
Minio Trusted 8f266e0772 Update yaml files to latest version RELEASE.2024-05-27T19-17-46Z 2024-05-27 23:52:43 +00:00
Harshavardhana e0fe7cc391
fix: information disclosure bug in preconditions GET (#19810)
precondition check was being honored before, validating
if anonymous access is allowed on the metadata of an
object, leading to metadata disclosure of the following
headers.

```
Last-Modified
Etag
x-amz-version-id
Expires:
Cache-Control:
```

although the information presented is minimal in nature,
and of opaque nature. It still simply discloses that an
object by a specific name exists or not without even having
enough permissions.
2024-05-27 12:17:46 -07:00
Harshavardhana 9d20dec56a Revert "remove dataErrs from er.deleteIfDangling code"
This reverts commit 7d75b1e758.

This fails multipart tests we need this code to handle
existing challenges, so wait for the comprehensive fix.
2024-05-26 11:13:29 -07:00
Harshavardhana 597a785253
fix: authenticate LDAP via actual DN instead of normalized DN (#19805)
fix: authenticate LDAP via actual DN instead of normalized DN

Normalized DN is only for internal representation, not for
external communication, any communication to LDAP must be
based on actual user DN. LDAP servers do not understand
normalized DN.

fixes #19757
2024-05-25 06:43:06 -07:00
Harshavardhana 7d75b1e758 remove dataErrs from er.deleteIfDangling code
avoid this until a comprehensive change is
merged such as https://github.com/minio/minio/pull/19797
2024-05-24 18:20:04 -07:00
Aditya Manthramurthy 5f78691fcf
ldap: Add user DN attributes list config param (#19758)
This change uses the updated ldap library in minio/pkg (bumped
up to v3). A new config parameter is added for LDAP configuration to
specify extra user attributes to load from the LDAP server and to store
them as additional claims for the user.

A test is added in sts_handlers.go that shows how to access the LDAP
attributes as a claim.

This is in preparation for adding SSH pubkey authentication to MinIO's SFTP
integration.
2024-05-24 16:05:23 -07:00
Shireesh Anjal a591e06ae5
Add cluster scanner metrics in metrics-v3 (#19517)
endpoint: /minio/metrics/v3/cluster/scanner
metrics:
 - bucket_scans_finished (counter)
 - bucket_scans_started (counter)
 - directories_scanned (counter)
 - last_activity_nano_seconds (gauge)
 - objects_scanned (counter)
 - versions_scanned (counter)
2024-05-24 12:29:25 -07:00
Harshavardhana 443c93c634
compute time spent in ILM properly (#19806) 2024-05-24 12:28:51 -07:00
Shireesh Anjal 5659cddc84
Add cluster config metrics in metrics-v3 (#19507)
endpoint: /minio/metrics/v3/cluster/config
metrics:
- write_quorum
- rrs_parity
- standard_parity
2024-05-24 05:50:46 -07:00
Shireesh Anjal 2a03a34bde
Upgrade madmin-go to v3.0.52 (#19798)
This will ensure that content of /proc/cmdline from each server is
captured in the health report.
2024-05-24 05:34:57 -07:00
Shubhendu 1654a9b7e6
Use point in time values for `gauge` metrics in graphs (#19690)
Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
2024-05-24 04:11:51 -07:00
Shireesh Anjal 673a521711
Change endpoint of v3 notification metrics (#19804)
from /cluster/notification to /notification
2024-05-24 04:10:24 -07:00
Harshavardhana 2e23076688
move windows runners to in-house (#19800)
GitHub CI runners for windows have gotten very slow,
moving them to our own hosted runners
2024-05-23 15:29:33 -07:00
Klaus Post b92ac55250
Add multipart combination to xl-meta (#19780)
Add combination of multiple parts.

Parts will be reconstructed and saved separately and can manually be combined to the complete object.

Parts will be named `(version_id)-(filename).(partnum).(in)complete`.
2024-05-23 09:37:31 -07:00
Shireesh Anjal 7981509cc8
Add cluster and bucket replication metrics in metrics-v3 (#19546)
endpoint: /minio/metrics/v3/cluster/replication
metrics:
- average_active_workers
- average_queued_bytes
- average_queued_count
- average_transfer_rate
- current_active_workers
- current_transfer_rate
- last_minute_queued_bytes
- last_minute_queued_count
- max_active_workers
- max_queued_bytes
- max_queued_count
- max_transfer_rate
- recent_backlog_count

endpoint: /minio/metrics/v3/api/bucket/replication
metrics:
- last_hour_failed_bytes
- last_hour_failed_count
- last_minute_failed_bytes
- last_minute_failed_count
- latency_ms
- proxied_delete_tagging_requests_total
- proxied_get_requests_failures
- proxied_get_requests_total
- proxied_get_tagging_requests_failures
- proxied_get_tagging_requests_total
- proxied_head_requests_failures
- proxied_head_requests_total
- proxied_put_tagging_requests_failures
- proxied_put_tagging_requests_total
- sent_bytes
- sent_count
- total_failed_bytes
- total_failed_count
- proxied_delete_tagging_requests_failures
2024-05-23 00:41:18 -07:00
Krishnan Parthasarathi 6d5bc045bc
Disallow ExpiredObjectAllVersions with object lock (#19792)
Relaxes restrictions on Expiration and NoncurrentVersionExpiration
placed by https://github.com/minio/minio/pull/19785.
ref: https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lock-managing.html#object-lock-managing-lifecycle

> Object lifecycle management configurations continue functioning
normally on protected objects, including placing delete markers.
However, a locked version of an object cannot be deleted by a S3
Lifecycle expiration policy. Object Lock is maintained regardless of
the object's storage class and throughout S3 Lifecycle
transitions between storage classes.
2024-05-22 18:12:48 -07:00
Harshavardhana d38e020b29
remove errant logs for disconnected remote (#19793)
Signed-off-by: Harshavardhana <harsha@minio.io>
2024-05-22 18:12:23 -07:00
Poorna 7d29030292
fix list results returned for spark max-keys=2 listing (#19791)
This PR continues fix #19725 for some unhandled cases
2024-05-22 16:16:34 -07:00
Shubhendu 7c7650b7c3
Add sufficient deadlines and countermeasures to handle hung node scenario (#19688)
Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
Signed-off-by: Harshavardhana <harsha@minio.io>
2024-05-22 16:07:14 -07:00
Harshavardhana ca80eced24
usage of deadline conn at Accept() breaks websocket (#19789)
fortunately not wired up to use, however if anyone
enables deadlines for conn then sporadically MinIO
startups fail.
2024-05-22 10:49:27 -07:00
Anis Eleuch d0e0b81d8e
Fix race get/set system/audit targest to avoid race errors (#19790) 2024-05-22 09:23:03 -07:00
jiuker 391baa1c9a
test: add reject ilm rule test case (#19788) 2024-05-22 04:26:59 -07:00
Harshavardhana ae14681c3e Revert "Fix two-way stream cancelation and pings (#19763)"
This reverts commit 4d698841f4.
2024-05-22 03:00:00 -07:00
Klaus Post 4d698841f4
Fix two-way stream cancelation and pings (#19763)
Do not log errors on oneway streams when sending ping fails. Instead, cancel the stream.

This also makes sure pings are sent when blocked on sending responses.
2024-05-22 01:25:25 -07:00
jiuker 9906b3ade9
fix: reject ilm rule when bucket LockEnabled (#19785) 2024-05-21 23:50:03 -07:00
Anis Eleuch bf1769d3e0
xl: Avoid marking a drive offline after one part read failure (#19779)
This commit will fix one rare case of a multipart object that
can be read in theory but GetObject API returned an error.

It turned out that a six years old code was marking a drive offline
when the bitrot streaming fails to read a part in a disk with any error.
This can affect reading a subsequent part, though having enough shards,
but unable to construct because one drive was marked offline earlier.

This commit will remove the drive marking offline code. It will also
close the bitrotstreaming reader before marking it as nil.
2024-05-21 07:36:21 -07:00
Harshavardhana 63e1ad9f29 fix: the user-agent for Veeam 2024-05-20 11:54:52 -07:00
Klaus Post 2c7bcee53f
Add cross-version remapped merges to xl-meta (#19765)
Adds `-xver` which can be used with `-export` and `-combine` to attempt to combine files across versions if data is suspected to be the same. Overlapping data is compared.

Bonus: Make `inspect` accept wildcards.
2024-05-19 08:31:54 -07:00
Harshavardhana 1fd90c93ff
re-use StorageAPI while loading drive formats (#19770)
Bonus: safe settings for deployment ID to avoid races
2024-05-19 01:06:49 -07:00
Poorna e947a844c9
Fix test scripts to use mc ready (#19768) 2024-05-18 11:19:01 -07:00
Poorna 4e2d39293a
Fix build script to wait for server to come up (#19767) 2024-05-17 14:43:59 -07:00
Krishnan Parthasarathi 1228d6bf1a
Return NumVersions in quorum when available (#19766)
Similar to https://github.com/minio/minio/pull/17925
2024-05-17 13:57:37 -07:00
Shireesh Anjal fc4561c64c
Start callhome immediately after enabling (#19764)
Currently, on enabling callhome (or restarting the server), the callhome
job gets scheduled. This means that one has to wait for 24hrs (the
default frequency duration) to see it in action and to figure out if it
is working as expected.

It will be a better user experience to perform the first callhome
execution immediately after enabling it (or on server start if already
enabled).

Also, generate audit event on callhome execution, setting the error
field in case the execution has failed.
2024-05-17 09:53:34 -07:00
Klaus Post 3b7747b42b
Tweak multipart uploads (#19756)
* Store ModTime in the upload ID; return it when listing instead of the current time.
* Use this ModTime to expire and skip reading the file info.
* Consistent upload sorting in listing (since it now has the ModTime).
* Exclude healing disks to avoid returning an empty list.
2024-05-17 09:40:09 -07:00
Harshavardhana e432e79324
avoid calling 'admin info' for disk, cpu, net metrics collection (#19762)
resource metrics collection was incorrectly making fan-out
liveness peer calls where it's not needed.
2024-05-17 08:15:13 -07:00
Harshavardhana 08d74819b6
handle racy updates to globalSite config (#19750)
```
==================
WARNING: DATA RACE
Read at 0x0000082be990 by goroutine 205:
  github.com/minio/minio/cmd.setCommonHeaders()

Previous write at 0x0000082be990 by main goroutine:
  github.com/minio/minio/cmd.lookupConfigs()
```
2024-05-16 16:13:47 -07:00
Poorna aa3fde1784
Add ListObjectsV2 unit test (#19753)
for PR: #19725
2024-05-15 20:40:51 -07:00
Harshavardhana 0b3eb7f218
add more deadlines and pass around context under most situations (#19752) 2024-05-15 15:19:00 -07:00
Anis Eleuch 69c9496c71
Upgrade github.com/minio/pkg/v2 and other deps (#19747) 2024-05-15 11:04:40 -07:00
Klaus Post b792b36495
Add Veeam storage class override (#19748)
Recent Veeam is very picky about storage class names. Add `_MINIO_VEEAM_FORCE_SC` env var.

It will override the storage class returned by the storage backend if it is non-standard
and we detect a Veeam client by checking the User Agent.

Applies to HeadObject/GetObject/ListObject*
2024-05-15 11:04:16 -07:00
Harshavardhana d3db7d31a3
fix: add deadlines for all synchronous REST callers (#19741)
add deadlines that can be dynamically changed via
the drive max timeout values.

Bonus: optimize "file not found" case and hung drives/network - circuit break the check and return right
away instead of waiting.
2024-05-15 09:52:29 -07:00
Shireesh Anjal c05ca63158
Fix crash on /minio/metrics/v3?list (#19745)
An unchecked map access was causing panic.
2024-05-15 09:06:35 -07:00
Klaus Post 6d3e0c7db6
Tweak one way stream ping (#19743)
Do not log errors on oneway streams when sending ping fails. Instead cancel the stream.

This also makes sure pings are sent when blocked on sending responses.

I will do a separate PR that includes this and adds pings to two-way streams as well as tests for pings.
2024-05-15 08:39:21 -07:00
Shireesh Anjal 0e59e50b39
Capture ttfb api metrics only for GetObject (#19733)
as that is the only API where the TTFB metric is beneficial, and
capturing this for all APIs exponentially increases the response size in
large clusters.
2024-05-14 23:25:13 -07:00