Commit Graph

5735 Commits

Author SHA1 Message Date
Aditya Manthramurthy cc960adbee
fix: remove policy mapping file when empty (#18828)
On a policy detach operation, if there are no policies remaining
attached to the user/group, remove the policy mapping file, instead of
leaving a file containing an empty list of policies.
2024-01-19 10:31:40 -08:00
Shubhendu 19387cafab
Use +Inf label additionally for Histogram metrics (#18807) 2024-01-18 14:51:28 -08:00
Harshavardhana 7c0673279b
capture I/O in waiting and total tokens in diskMetrics (#18819)
This is needed for the subsequent changes
in ServerUpdate(), ServerRestart() etc.
2024-01-18 11:17:43 -08:00
Anis Eleuch 7ce0d71a96
Do not log volume not empty when healing dangling buckets (#18822)
Healing dangling buckets is conservative, and it is a typical use case to
fail to remove a dangling bucket because it contains some data because
healing danging bucket code is not allowed to remove data: only healing
the dangling object is allowed to do so.
2024-01-18 10:39:27 -08:00
Harshavardhana dd2542e96c
add codespell action (#18818)
Original work here, #18474,  refixed and updated.
2024-01-17 23:03:17 -08:00
Harshavardhana 21d60eab7c
remove all older unused APIs (#18769) 2024-01-17 20:41:23 -08:00
Harshavardhana a4a74e9844 re-init the worker group to ensure errs[] slice is fresh 2024-01-17 20:33:25 -08:00
Harshavardhana 9588978028
fix: HealBucket regression for empty buckets, simplify it (#18815) 2024-01-17 15:19:09 -08:00
chienguo 8cd967803c
fix: a typo in storeDataUsageInBackend() comment (#18778) 2024-01-16 15:48:54 -08:00
Harshavardhana a0e1163fb6
reject reference format from a different deployment (#18800)
reference format is constant for any lifetime of
a minio cluster, we do not have to ever replace
it during HealFormat() as it will never change.

additionally we should simply reject reference
formats that we do not understand early on.
2024-01-16 15:13:14 -08:00
Sveinn 30bd5e2669
adding a missing return case to fix GetObjectTagging (#18793) 2024-01-15 16:11:06 -08:00
Harshavardhana 38637897ba
fix: listing SSE encrypted multipart objects (#18786)
GetActualSize() was heavily relying on o.Parts()
to be non-empty to figure out if the object is multipart or not, 
However, we have many indicators of whether an object is multipart 
or not.

Blindly assuming that o.Parts == nil is not a multipart, is an 
incorrect expectation instead, multipart must be obtained via

- Stored metadata value indicating this is a multipart encrypted object.

- Rely on <meta>-actual-size metadata to get the object's actual size.
  This value is preserved for additional reasons such as these.

- ETag != 32 length
2024-01-15 00:57:49 -08:00
Harshavardhana 993d96feef
treat all localhost endpoints as local setup with same port (#18784)
fixes #18783 and avoids user mistakes
2024-01-12 23:53:03 -08:00
Poorna b2b26d9c95
support proxying of tagging requests in replication (#18649)
support proxying of tagging requests in active-active replication

Note: even if proxying is successful, PutObjectTagging/DeleteObjectTagging
will continue to report a 404 since the object is not present locally.
2024-01-12 23:51:33 -08:00
Krishnan Parthasarathi cba3dd276b
Add more size intervals to obj size histogram (#18772)
New intervals:
[1024B, 64KiB)
[64KiB, 256KiB)
[256KiB, 512KiB)
[512KiB, 1MiB)

The new intervals helps us see object size distribution with higher
resolution for the interval [1024B, 1MiB).
2024-01-12 23:51:08 -08:00
Anis Eleuch a47fc75c26
xl: Remove wrong wording for errCorruptedFormat (#18775)
Also add errCorruptedBackend to make it easier to differentiate between
corrupted content or something else wrong in the backend drive
2024-01-12 14:48:44 -08:00
Harshavardhana e5c8794b8b
avoid disk monitoring leaks under various conditions (#18777)
- HealFormat() was leaking healthcheck goroutines for
  disks, we are only interested in enabling healthcheck
  for the newly formatted disk, not for existing disks.

- When disk is a root-disk a random disk monitor was
  leaking while we ignored the drive.

- When loading the disk for each erasure set, we were
  leaking goroutines for the prepare-storage.go disks
  which were replaced via the globalLocalDrives slice

- avoid disk monitoring utilizing health tokens that
  would cause exhaustion in the tokens, prematurely
  which were meant for incoming I/O. This is ensured
  by avoiding writing O_DIRECT aligned buffer instead
  write 2048 worth of content only as O_DSYNC, which is
  sufficient.
2024-01-12 01:48:36 -08:00
Taran Pelkey ac90a873eb
Verify that remote target bucket is on MinIO server for bucket replication (#18656) 2024-01-11 14:56:16 -08:00
jiuker c1a78224cf
fix: prevent queries from starting before initialization (#18766) 2024-01-10 15:21:52 -08:00
Harshavardhana 39f9350697
optimize readdir() open calls to be dealt with directly via 'fd' (#18762) 2024-01-10 08:48:50 -08:00
Shubhendu e31081d79d
Heal buckets at node level (#18612)
Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
2024-01-09 20:34:04 -08:00
Harshavardhana f02d282754
avoid frivolous logs for expired credentials (#18767) 2024-01-09 12:25:18 -08:00
Krishnan Parthasarathi 3a90af0bcd
Add line, col to types used in batch-expire (#18747) 2024-01-08 15:22:28 -08:00
jiuker 53ceb0791f
fix: prevent queries from starting before initialization (#18756)
Prevent queries from starting before initialization
2024-01-08 12:40:27 -08:00
jiuker 2cd98a0d21
remove outdated notes (#18755) 2024-01-08 08:04:19 -08:00
Anis Eleuch 04135fa6cd
audit: Add the drives where the dangling object is removed (#18737) 2024-01-05 14:17:24 -08:00
Harshavardhana 42dc6329e6
simplify success response for GetObjectAttributes() (#18746) 2024-01-05 12:50:07 -08:00
Sveinn 9b8ba97f9f
feat: add support for GetObjectAttributes API (#18732) 2024-01-05 10:43:06 -08:00
Anis Eleuch 7705605b5a
scanner: Add a config to disable short sleep between objects scan (#18734)
Add a hidden configuration under the scanner sub section to configure if
the scanner should sleep between two objects scan. The configuration has
only effect when there is no drive activity related to s3 requests or
healing.

By default, the code will keep the current behavior which is doing
sleep between objects.

To forcefully enable the full scan speed in idle mode, you can do this:

   `mc admin config set myminio scanner idle_speed=full`
2024-01-04 15:07:17 -08:00
Anis Eleuch 414bcb0c73
prom: Add read quorum per erasure set metric (#18736) 2024-01-04 15:05:13 -08:00
Harshavardhana f4710948c4
fix: an odd crash when deleting `null` DEL markers (#18727)
fixes #18724

A regression was introduced in #18547, that attempted
to file adding a missing `null` marker however we
should not skip returning based on versionID instead
it must be based on if we are being asked to create
a DEL marker or not.

The PR also has a side-affect for replicating `null`
marker permanent delete, as it may end up adding a
`null` marker while removing one.

This PR should address both scenarios.
2024-01-02 15:08:18 -08:00
Anis Eleuch 3f4488c589
scanner: Allow full throttle if there is no parallel disk ops (#18109) 2024-01-02 13:51:24 -08:00
Pedro Juarez 8f13c8c3bf
Support to store browser config settings (#18631)
* csp_policy
* hsts_seconds
* hsts_include_subdomains
* hsts_preload
* referrer_policy
2024-01-01 08:36:33 -08:00
Zhou Ting 31d16f6cc2
allow sha256 payload to be configurable for object perf test (#18712)
Signed-off-by: Zhou Ting <ting.z.zhou@intel.com>
2023-12-29 23:56:50 -08:00
Harshavardhana a50ea92c64
feat: introduce list_quorum="auto" to prefer quorum drives (#18084)
NOTE: This feature is not retro-active; it will not cater to previous transactions
on existing setups. 

To enable this feature, please set ` _MINIO_DRIVE_QUORUM=on` environment
variable as part of systemd service or k8s configmap. 

Once this has been enabled, you need to also set `list_quorum`. 

```
~ mc admin config set alias/ api list_quorum=auto` 
```

A new debugging tool is available to check for any missing counters.
2023-12-29 15:52:41 -08:00
Harshavardhana 5b2ced0119
re-use globalLocalDrives properly (#18721) 2023-12-29 09:30:10 -08:00
Anis Eleuch 8a0ba093dd
audit: Fix merrs and derrs object dangling message (#18714)
merrs and derrs are empty when a dangling object is deleted. Fix the bug
and adds invalid-meta data for data blocks
2023-12-27 22:27:04 -08:00
Daniel Valdivia 5fc7da345d
Upgrade Console to v0.44.0 (#18717)
Signed-off-by: Daniel Valdivia <18384552+dvaldivia@users.noreply.github.com>
2023-12-27 11:19:13 -08:00
Anis Eleuch 8bd4f6568b
server-info: Avoid initializing audit/log http/kafka targets (#18703)
This can cause unnecessary ServerInfo() call delay.
2023-12-22 10:25:08 -08:00
Harshavardhana da55499db0
fix: reject clients that do not send proper payload (#18701) 2023-12-22 01:26:17 -08:00
Anis Eleuch 22f8e39b58
tier: Allow edit of the new Azure and AWS auth params (#18690)
Allow editing for the service principal credentials from Azure
and the web identity token for AWS;

Also, more validation of input parameters.
2023-12-21 16:58:10 -08:00
Harshavardhana eba23bbac4
rename object_size -> block_size for cache subsystem (#18694) 2023-12-21 16:57:13 -08:00
Harshavardhana 4550535cbb
send proper IPv6 names avoid bracketing notation (#18699)
Following policies if present

```
       "Condition": {
         "IpAddress": {
            "aws:SourceIp": [
              "54.240.143.0/24",
               "2001:DB8:1234:5678::/64"
             ]
          }
        }
```

And client is making a request to MinIO via IPv6 can
potentially crash the server.

Workarounds are turn-off IPv6 and use only IPv4
2023-12-21 16:56:55 -08:00
Anis Eleuch 8432fd5ac2
prom: Add online and healing drives metrics per erasure set (#18700) 2023-12-21 16:56:43 -08:00
Harshavardhana 7c948adf88
allow pre-allocating buffers to reduce frequent GCs during growth (#18686)
This PR also increases per node bpool memory from 1024 entries
to 2048 entries; along with that, it also moves the byte pool
centrally instead of being per pool.
2023-12-21 08:59:38 -08:00
Krishnan Parthasarathi 56b7045c20
Export tier metrics (#18678)
minio_node_tier_ttlb_seconds - Distribution of time to last byte for streaming objects from warm tier
minio_node_tier_requests_success - Number of requests to download object from warm tier that were successful
minio_node_tier_requests_failure - Number of requests to download object from warm tier that failed
2023-12-20 20:13:40 -08:00
Poorna d55b6b9909
Fix quota config replication for SR (#18684)
Fixing regression introduced by PR #17988
2023-12-19 13:22:47 -08:00
Shireesh Anjal 7680e5f81d
Read new key license_v2 from SUBNET response (#18669)
SUBNET now has a v2 of license that is returned in the new key
`license_v2`. mc will start reading and storing the same. (The old key
`license` is deprecated but is still available in SUBNET response to
ensure that the current released version of minio doesn't break)
2023-12-18 08:21:44 -08:00
Taran Pelkey ad8a34858f
Add APIs to create and list access keys for LDAP (#18402) 2023-12-15 13:00:43 -08:00
Krishnan Parthasarathi 162eced7d2
Fix incorrect metric desc for bucketRequestsDuration (#18657) 2023-12-14 19:02:11 -08:00