a436fd513b
track client disconnections properly for all ListObjects calls ( #17804 )
...
Currently ListObjects* calls were returning 200 OK for timed-out clients,
this makes debugging via `mc admin trace` very hard.
2023-08-04 15:57:27 -07:00
533cd8d6df
fix: batch replication pull must preserve versionID ( #17805 )
...
batch replication pull must preserve versionID regardless
of destination bucket versioning configuration.
This is similar to the issue with decommissioning and rebalancing
2023-08-04 12:09:10 -07:00
cb089dcb52
error out by default beyond 10000 versions per object ( #17803 )
...
```
You've exceeded the limit on the number of versions you can create on this object
```
2023-08-04 10:40:21 -07:00
239ccc9c40
fix: crash in globalTierJournal when TierConfig is not initialized ( #17791 )
2023-08-03 14:16:15 -07:00
b762fbaf21
sts: validate if iam subsystem initialized in handlers ( #17796 )
2023-08-03 13:24:25 -07:00
0285df5a02
fix: prioritize audit_webhook and logger_webhook ENVs over the config KVS ( #17783 )
2023-08-03 02:47:07 -07:00
45fb375c41
allow healing to prefer local disks over remote ( #17788 )
2023-08-03 02:18:18 -07:00
4a4950fe41
fix: honor requested allow origin settings properly ( #17789 )
...
fixes #17778
2023-08-02 20:41:21 -07:00
1664fd8bb1
Avoid logging errors twice during transitioned objects expiration ( #17782 )
2023-08-02 09:06:03 -07:00
21cdd2bf5d
avoid overwriting metrics on success, save it in defer ( #17780 )
2023-08-01 22:19:56 -07:00
0153f96a20
add deadlines for readMetadata() in listing ( #17776 )
...
Bonus: also skip spending time looking for xl.json
- Listing()
- Delete()
2023-08-01 21:52:31 -07:00
a7a7533190
add new errors for Disks with timeouts ( #17770 )
2023-08-01 12:47:50 -07:00
311380f8cb
replication resync: fix queueing ( #17775 )
...
Assign resync of all versions of object to the same worker to avoid locking
contention. Fixes parallel resync implementation in #16707
2023-08-01 11:51:15 -07:00
b0f0e53bba
fix: make sure to correctly initialize health checks ( #17765 )
...
health checks were missing for drives replaced since
- HealFormat() would replace the drives without a health check
- disconnected drives when they reconnect via connectEndpoint()
the loop also loses health checks for local disks and merges
these into a single code.
- other than this separate cleanUp, health check variables to avoid
overloading them with similar requirements.
- also ensure that we compete via context selector for disk monitoring
such that the canceled disks don't linger around longer waiting for
the ticker to trigger.
- allow disabling active monitoring.
2023-08-01 10:54:26 -07:00
004f1e2f66
Fix trailing header signature mismatch ( #17774 )
...
Seems like clients may omit a newline at the end of the trailer chunk. Each header should end with a newline. Add that if missing.
Fixes #17662
2023-08-01 08:45:57 -07:00
2fa561f22e
do not crash on invalid metric values ( #17764 )
...
```
minio[1032735]: panic: label value "\xc0.\xc0." is not valid UTF-8
minio[1032735]: goroutine 1781101 [running]:
minio[1032735]: github.com/prometheus/client_golang/prometheus.MustNewConstMetric(...)
```
log such errors for investigation
2023-08-01 00:55:39 -07:00
81be718674
fix: optimize DiskInfo() call avoid metrics when not needed ( #17763 )
2023-07-31 15:20:48 -07:00
49a1e2f98e
update-notifier.go: misleading version age message ( #17750 )
2023-07-31 08:36:19 -07:00
684c46369c
Send events for extracted objects ( #17760 )
...
Fixes #17759
2023-07-31 08:33:51 -07:00
73edd5b8fd
introduce 'mc admin config set alias/ api odirect=on' ( #17753 )
...
change disable_odirect=off -> odirect=on to make it
easier to understand, instead of making it double
negative.
2023-07-31 00:12:53 -07:00
5e5bdf5432
capture total errors data availability and any timeout errors ( #17748 )
2023-07-29 23:26:26 -07:00
f13cfcb83e
allow disabling O_DIRECT for write ops ( #17751 )
...
on really slow systems, O_DIRECT simply kills the drives
allow for a way to disable them.
2023-07-29 15:17:56 -07:00
731e03fe5a
add ReadFileStream deadline for disk call ( #17745 )
...
timeout the reader side if hung via disk max timeout
2023-07-28 15:37:53 -07:00
7057d00a28
s3: Return invalid bucket name the first thing in all S3 calls ( #17742 )
2023-07-28 10:49:20 -07:00
114fab4c70
export cluster health as prometheus metrics ( #17741 )
2023-07-28 01:16:53 -07:00
a92cb66468
Get the signed headers in the order they were signed ( #17690 )
...
use pSignValues to get signed headers in order
2023-07-27 11:45:30 -07:00
535f97ba61
check if metadata headers/url values are equal with signed headers ( #17737 )
2023-07-27 11:44:56 -07:00
14ebd82dbd
fix: missing disk metrics when query metric api from peer ( #17738 )
2023-07-27 11:44:13 -07:00
47dcfcbdd4
introduce deadlines on READ operations ( #17724 )
2023-07-27 07:33:05 -07:00
bf3901342c
Include SuccessorModTime for FileInfo quorum ( #17732 )
2023-07-26 17:04:16 -07:00
b28bcad11b
avoid Access() calls on known bucket paths ( #17719 )
2023-07-26 11:31:40 -07:00
a7c71e4c6b
protect disk monitoring to avoid busy loop configuration ( #17723 )
2023-07-25 20:02:22 -07:00
1a42693d68
replication: limit larger uploads to a subset of workers ( #17687 )
...
Limit large uploads (> 128MiB) to a max of 10 workers, intent is to avoid
larger uploads from using all replication bandwidth, giving room for smaller
uploads to sync faster.
2023-07-25 20:02:02 -07:00
e7b60c4d65
Add slow drive timeouts to match with active disk monitoring ( #17701 )
...
allow active disk-monitoring to be configurable, and use
these add deadlines in various call layers for various
syscalls.
2023-07-25 16:58:31 -07:00
f95129894d
Use decrypted object size while computing object size summary ( #17717 )
...
Corrects an issue with encrypted versioned objects being reported under
`unversioned` bin in the object version histogram
2023-07-24 17:13:25 -07:00
c32c71c836
allow DNS cache TTL to be configurable ( #17709 )
...
this is added for now as a hidden variable
2023-07-24 15:13:35 -07:00
14e1ace552
remove serializing WalkDir() across all buckets/prefixes on SSDs ( #17707 )
...
slower drives get knocked off because they are too slow via
active monitoring, we do not need to block calls arbitrarily.
Serializing adds latencies for already slow calls, remove
it for SSDs/NVMEs
Also, add a selection with context when writing to `out <-`
channel, to avoid any potential blocks.
2023-07-24 09:30:19 -07:00
a7fb3a3853
fix: Create metrics slice when necessary in getCacheMetrics() ( #17711 )
2023-07-24 08:40:21 -07:00
2da4bd5f1a
Revert "don't error when asked for 0-based range on empty objects ( #17708 ) ( #17713 )
...
Revert "don't error when asked for 0-based range on empty objects (#17708 )"
This reverts commit 7e76d66184https://datatracker.ietf.org/doc/html/rfc7233#section-4.4 )
> The 416 (Range Not Satisfiable) status code indicates that none of the ranges in the
> request's Range header field (Section 3.1) overlap the current extent of the selected resource...
A request for "bytes=0-" is a request for the first byte of a resource. If the resource is 0-length,
the range [0,0] does not overlap the resource content and the server responds with an error.
2023-07-24 07:56:28 -07:00
7e76d66184
don't error when asked for 0-based range on empty objects ( #17708 )
...
In a reverse proxying setup, a proxy in front of MinIO may attempt to
request objects in slices for enhanced cache efficiency. Since such a
a proxy cannot have prior knowledge of how large a requested resource is,
it usually sends a header of the form:
Range: 0-$slice_size
... and, depending on the size of the resource, expects either:
- an empty response, if $resource_size == 0
- a full response, if $resource_size <= $slice_size
- a partial response, if $resource_size > $slice_size
Prior to this change, MinIO would respond 416 Range Not Satisfiable if a
client tried to request a range on an empty resource. This behavior is
technically consistent with RFC9110[1] – However, it renders sliced
reverse proxying, such as implemented in Nginx, broken in the case of
empty files. Nginx itself seems to break this convention to enable
"useful" responses in these cases, and MinIO should probably do that
too.
[1]: https://www.rfc-editor.org/rfc/rfc9110#byte.ranges
2023-07-23 00:10:03 -07:00
7764f4a8e3
return tags as part of Head/Get calls ( #17635 )
...
AWS S3 only returns the number of tag
counts, along with that we must return
the tags as well to avoid another metadata
call to the server.
2023-07-22 07:19:43 -07:00
6624f970c0
Fix spelling of 'already' across repository ( #17703 )
2023-07-21 08:45:08 -07:00
331bdc2245
fix: remove CompleteMultipartUpload() 200 OK response for blocking calls ( #17699 )
...
sending whitespace character with CompleteMultipartUpload()
with 200 OK was an AWS S3 compatible implementation detail,
and it was expected that the client SDK must look for both
successful XML as well as error XML for 200 OK.
But this is not useful anymore on MinIO, since we do not
have any large delayed coalescing of parts anymore.
2023-07-20 22:14:38 -07:00
e12ab486a2
avoid using os.Getenv for internal code, use env.Get() instead ( #17688 )
2023-07-20 07:52:49 -07:00
9eeee92d36
Add deletemarker_total metric ( #17689 )
2023-07-20 07:52:32 -07:00
756d6aa729
fix: report correct pool/set/disk indexes for offline disks ( #17695 )
2023-07-20 07:48:21 -07:00
bddd53d6d2
fix: retry listing in decommissioning if it fails perpetually ( #17682 )
2023-07-19 13:09:37 -07:00
a99cd825ab
fix: byHost realTime metrics API ( #17681 )
2023-07-18 23:50:30 -07:00
6426b74770
move bucket centric metrics to /minio/v2/metrics/bucket handlers ( #17663 )
...
users/customers do not have a reasonable number of buckets anymore,
this is why we must avoid overpopulating cluster endpoints, instead
move the bucket monitoring to a separate endpoint.
some of it's a breaking change here for a couple of metrics, but
it is imperative that we do it to improve the responsiveness of
our Prometheus cluster endpoint.
Bonus: Added new cluster metrics for usage, objects and histograms
2023-07-18 22:25:12 -07:00
4f257bf1e6
pick internode interface properly via globalLocalNodeName ( #17680 )
...
current code will not pick the right interface name
if --address or --interface is not provided.
2023-07-18 19:18:11 -07:00
0120ff93bc
admin-info: add DeleteMarkers count ( #17659 )
2023-07-18 10:49:40 -07:00
49638fa533
s3: Delete Bucket should not recreate bucket if it does not exist ( #17676 )
...
Also return Bucket Not Found error in the same use case.
2023-07-18 09:32:19 -07:00
7a3a7b19e5
Added a start script to inspect command output ( #17591 )
...
Using this script, post decrypt we should be able to bring up the
MinIO instance with same configuration.
Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
2023-07-17 14:15:28 -07:00
24e86d0c59
avoid passing around poolIdx, setIdx instead pass the relevant disks ( #17660 )
2023-07-17 09:52:05 -07:00
d118031ed6
fix: when Origin: null is set return back '*' for allow origins ( #17651 )
2023-07-15 12:15:06 -07:00
341a89c00d
return a descriptive error when loading any IAM item fails ( #17654 )
...
Sometimes IAM fails to load certain items, which could be a user,
a service account or a policy but with not enough information for
us to debug.
This commit will create a more descriptive error to make it easier to
debug in such situations.
2023-07-14 20:17:14 -07:00
df29d25e6b
return different status code for internode communication ( #17655 )
...
mc admin trace -a will be able to quickly show
401 Unauthorized header to pinpoint trivial issues
between nodes, such as wrong root
credentials and skewed time.
2023-07-14 18:34:55 -07:00
3e196fa7b3
fix: ILM newer noncurrent version limit must return correct versions ( #17652 )
...
objects/versions that are not expired via NewerNoncurrentVersions
must be properly returned to be applied under further ILM actions.
this would cause legitimately expired objects to be missed
from expiration.
2023-07-14 16:42:35 -07:00
04c792476f
fix: provide a possible slice cap for heal failed metrics items ( #17647 )
...
Signed-off-by: Wu <yang.wu@daocloud.io>
2023-07-14 11:02:45 -07:00
005a4a275a
add more bootstrap messages to provide latency ( #17650 )
...
- simplify refreshing bucket metadata, wait() to
depend on how fast the bucket metadata can load.
- simplify resync to start resync in single pass.
2023-07-14 04:00:29 -07:00
bdddf597f6
shuffle buckets randomly before being scanned ( #17644 )
...
this randomness is needed to avoid scanning
the same buckets across different erasure sets,
in the same order.
allow random buckets to be scanned instead
allowing a wider spread of ILM, replication
checks.
Additionally do not loop over twice to fill
the channel, fill the channel regardless of
having bucket new or old.
2023-07-14 02:25:40 -07:00
bb6921bf9c
Send AuditLog via new middleware fn for admin APIs ( #17632 )
...
A new middleware function is added for admin handlers, including options
for modifying certain behaviors. This admin middleware:
- sets the handler context via reflection in the request and sends AuditLog
- checks for object API availability (skipping it if a flag is passed)
- enables gzip compression (skipping it if a flag is passed)
- enables header tracing (adding body tracing if a flag is passed)
While the new function is a middleware, due to the flags used for
conditional behavior modification, which is used in each route registration
call.
To try to ensure that no regressions are introduced, the following
changes were done mechanically mostly with `sed` and regexp:
- Remove defer logger.AuditLog in admin handlers
- Replace newContext() calls with r.Context()
- Update admin routes registration calls
Bonus: remove unused NetSpeedtestHandler
Since the new adminMiddleware function checks for object layer presence
by default, we need to pass the `noObjLayerFlag` explicitly to admin
handlers that should work even when it is not available. The following
admin handlers do not require it:
- ServerInfoHandler
- StartProfilingHandler
- DownloadProfilingHandler
- ProfileHandler
- SiteReplicationDevNull
- SiteReplicationNetPerf
- TraceHandler
For these handlers adminMiddleware does not check for the object layer
presence (disabled by passing the `noObjLayerFlag`), and for all other
handlers, the pre-check ensures that the handler is not called when the
object layer is not available - the client would get a
ErrServerNotInitialized and can retry later.
This `noObjLayerFlag` is added based on existing behavior for these
handlers only.
2023-07-13 14:52:21 -07:00
4f89e5bba9
Add active disk health checks ( #17539 )
...
Add check every 2 minutes to see if a write+read operation can complete.
If disk is unresponsive for 2 minutes or returns errFaultyDisk, take it offline.
2023-07-13 11:41:55 -07:00
183428db03
fear: Implement 'mc support top net' ( #17598 )
2023-07-13 11:41:19 -07:00
fc6d873758
Use os.ReadFile instead of ioutil.ReadFile ( #17649 )
...
ioutil.ReadFile is deprecated and also doesn't work with certain kinds
of symlinks.
2023-07-13 09:07:10 -07:00
5e2f8d7a42
replication: Simplify mrf requeueing and add backlog handler ( #17171 )
...
Simplify MRF queueing and add backlog handler
- Limit re-tries to 3 to avoid repeated re-queueing. Fall offs
to be re-tried when the scanner revisits this object or upon access.
- Change MRF to have each node process only its MRF entries.
- Collect MRF backlog by the node to allow for current backlog visibility
2023-07-12 23:51:33 -07:00
9b9871cfbb
Added `endpoint` and `versions` attributes to KMS details ( #17350 )
...
Now it would list details of all KMS instances with additional
attributes `endpoint` and `version`. In the case of k8s-based
deployment the list would consist of a single entry.
Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
2023-07-12 23:50:38 -07:00
f80b6926d3
chore: fix minor issues reported via staticcheck ( #17639 )
2023-07-12 20:33:11 -07:00
6dc55fe5ed
Corrected the API name for audit logging purpose ( #17642 )
...
This would better to record the correct API name so that
any verification around audit logs to figure out if required
APIs are called required no of times, would be correct.
Here in this case of policy attached, API `AttachDetachPolicyBuiltin`
would be called with `requestPath` as `/minio/admin/v3/idp/builtin/policy/attach`
and in case of detach policy the value would be `/minio/admin/v3/idp/builtin/policy/detach`
Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
2023-07-12 15:38:49 -07:00
2d1cda2061
fix: do not os.Exit(1) while writing goroutines during shutdown ( #17640 )
...
Also shutdown poll add jitter, to verify if the shutdown
sequence can finish before 500ms, this reduces the overall
time taken during "restart" of the service.
Provides speedup for `mc admin service restart` during
active I/O, also ensures that systemd doesn't treat the
returned 'error' as a failure, certain configurations in
systemd can cause it to 'auto-restart' the process by-itself
which can interfere with `mc admin service restart`.
It can be observed how now restarting the service is
much snappier.
2023-07-12 07:18:30 -07:00
a566bcf613
treat 0-byte objects to honor same quorum as delete marker ( #17633 )
...
on unversioned buckets its possible that 0-byte objects
might lose quorum on flaky systems, allow them to be same
as DELETE markers. Since practically speak they have no
content.
2023-07-11 21:53:49 -07:00
9885a0a6af
Fix hasSpaceFor in SNSD setup ( #17630 )
...
If drive is offline or filled we divide by 0.
Fixes #17629
Bonus: Reject when any valid disk exceeds minimum inode threshold.
2023-07-11 14:29:34 -07:00
f64d62b01d
Fix style of logOnceIf calls w/unique identifiers ( #17631 )
2023-07-11 13:17:45 -07:00
82075e8e3a
use strconv variants to improve on performance per 'op' ( #17626 )
...
```
BenchmarkItoa
BenchmarkItoa-8 673628088 1.946 ns/op 0 B/op 0 allocs/op
BenchmarkFormatInt
BenchmarkFormatInt-8 592919769 2.012 ns/op 0 B/op 0 allocs/op
BenchmarkSprint
BenchmarkSprint-8 26149144 49.06 ns/op 2 B/op 1 allocs/op
BenchmarkSprintBool
BenchmarkSprintBool-8 26440180 45.92 ns/op 4 B/op 1 allocs/op
BenchmarkFormatBool
BenchmarkFormatBool-8 1000000000 0.2558 ns/op 0 B/op 0 allocs/op
```
2023-07-11 07:46:58 -07:00
5b7c83341b
move per bucket metrics to peer location ( #17627 )
2023-07-11 07:46:24 -07:00
fb49aead9b
replication: add validation API ( #17520 )
...
To check if replication is set up properly on a bucket.
2023-07-10 20:09:20 -07:00
85f5700e4e
fix: missing audit logger call for some admin APIs ( #17623 )
2023-07-10 16:59:44 -07:00
43b3c093ef
Fix: set request id in trace context properly ( #17622 )
2023-07-10 15:40:44 -07:00
bd6842d917
Further print log messages once per error ( #17618 )
2023-07-10 07:59:57 -07:00
e8c98c3246
Avoid extra GetObjectInfo call in DeleteObject API ( #17599 )
...
Optimize DeleteObject API to avoid extra
GetObjectInfo call on the replicating side.
For receiving side, it is just a regular
DeleteObject call.
Bonus: Fix a corner case where version purged is
absent on target (either due to replication not yet
complete or target version already deleted in a
one-way replication or when replication was disabled).
In such cases, mark version purge complete.
2023-07-10 07:57:56 -07:00
dfd7cca0d2
fix: allow cancel of decom only when its in progress ( #17607 )
2023-07-10 07:55:38 -07:00
f6186965c3
honor DeleteAllVersions in list(), head() calls ( #17604 )
2023-07-08 15:42:10 -07:00
28a01f0320
update missing license header in files ( #17603 )
2023-07-08 10:42:05 -07:00
6d0bc5ab1e
prometheus: Fix internode stats ( #17594 )
...
Internode calculation was done inside S3 handlers, fix it by moving it
to internode handlers.
Remove admin stats since it is not used.
2023-07-08 07:35:11 -07:00
7af78af1f0
fix: set request ID in tracing context key ( #17602 )
...
Since `addCustomerHeaders` middleware was after the `httpTracer`
middleware, the request ID was not set in the http tracing context. By
reordering these middleware functions, the request ID header becomes
available. We also avoid setting the tracing context key again in
`newContext`.
Bonus: All middleware functions are renamed with a "Middleware" suffix
to avoid confusion with http Handler functions.
2023-07-08 07:31:42 -07:00
abb1f22057
Revert "change ttfb_distribution metrics to histogramMetric ( #17115 )"
...
This reverts commit 9112ca4e29
2023-07-07 13:57:37 -07:00
f41edb23e2
add variadic delays in peer notification retries ( #17592 )
...
just adds more `jitter` in our retries to avoid
burst flooding for peer calls.
2023-07-07 07:47:38 -07:00
e20aab25ec
Check for progress before we reach the limit ( #17552 )
2023-07-07 00:13:57 -07:00
ff5988f4e0
Reduce allocations ( #17584 )
...
* Reduce allocations
* Add stringsHasPrefixFold which can compare string prefixes, while ignoring case and not allocating.
* Reuse all msgp.Readers
* Reuse metadata buffers when not reading data.
* Make type safe. Make buffer 4K instead of 8.
* Unslice
2023-07-06 16:02:08 -07:00
c47ff44f5e
fix: disable site network test if site replication is disabled ( #17579 )
2023-07-06 09:19:14 -07:00
8af0773baf
remove deprecated Content-Security-Policy ( #17580 )
...
https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Security-Policy/block-all-mixed-content
2023-07-06 09:18:38 -07:00
2dbb1cff4a
feat: support perf site replication ( #17477 )
2023-07-05 22:28:26 -07:00
6efcf9c982
Do lockless last minute latency metrics ( #17576 )
...
Collect metrics in one second and accumulate lockless before sending upstream.
2023-07-05 10:40:45 -07:00
0bc34952eb
fix: under FanOut API avoid repeated md5sum calculation ( #17572 )
...
md5sum calculation has a high CPU overhead, avoid calculating
it repeatedly for similar fanOut calls.
To fix following CPU profiler result
```
(pprof) top10
Showing nodes accounting for 678.68s, 84.67% of 801.54s total
Dropped 1072 nodes (cum <= 4.01s)
Showing top 10 nodes out of 156
flat flat% sum% cum cum%
332.54s 41.49% 41.49% 332.54s 41.49% runtime/internal/syscall.Syscall6
228.39s 28.49% 69.98% 228.39s 28.49% crypto/md5.block
48.07s 6.00% 75.98% 48.07s 6.00% runtime.memmove
28.91s 3.61% 79.59% 28.91s 3.61% github.com/minio/highwayhash.updateAVX2
8.25s 1.03% 80.61% 8.25s 1.03% runtime.futex
8.25s 1.03% 81.64% 10.81s 1.35% runtime.step
6.99s 0.87% 82.52% 22.35s 2.79% runtime.pcvalue
6.67s 0.83% 83.35% 38.90s 4.85% runtime.mallocgc
5.77s 0.72% 84.07% 32.61s 4.07% runtime.gentraceback
4.84s 0.6% 84.67% 10.49s 1.31% runtime.lock2
```
2023-07-05 03:16:05 -07:00
e37c4efc6e
fix: upon DNS refresh() failure use previous values ( #17561 )
...
DNS refresh() in-case of MinIO can safely re-use
the previous values on bare-metal setups, since
bare-metal arrangements do not change DNS in any
manner commonly.
This PR simplifies that, we only ever need DNS caching
on bare-metal setups.
- On containerized setups do not enable DNS
caching at all, as it may have adverse effects on
the overall effectiveness of k8s DNS systems.
k8s DNS systems are dynamic and expect applications
to avoid managing DNS caching themselves, instead
provide a cleaner container native caching
implementations that must be used.
- update IsDocker() detection, including podman runtime
- move to minio/dnscache fork for a simpler package
2023-07-03 12:30:51 -07:00
15fd5ce2fa
fix: A typo in per pool make/delete bucket errs calculation ( #17553 )
2023-07-03 09:47:40 -07:00
7f782983ca
fix: for FTP server driver allow implicit trust of TLS ( #17541 )
...
fixes #17535
2023-06-30 08:04:13 -07:00
9d628346eb
fix: service account list for root user ( #17547 )
...
Fixes https://github.com/minio/minio/issues/17545
2023-06-30 08:02:12 -07:00
bde533a9c7
fix: OpenID config initialization ( #17544 )
...
This is due to a regression in the handling of the enable key in OpenID
configuration.
2023-06-29 23:38:26 -07:00
aae6846413
feat: allow expiration of all versions via ILM Expiration action ( #17521 )
...
Following extension allows users to specify immediate purge of
all versions as soon as the latest version of this object has
expired.
```
<LifecycleConfiguration>
<Rule>
<ID>ClassADocRule</ID>
<Filter>
<Prefix>classA/</Prefix>
</Filter>
<Status>Enabled</Status>
<Expiration>
<Days>3650</Days>
<ExpiredObjectAllVersions>true</ExpiredObjectAllVersions>
</Expiration>
</Rule>
...
```
2023-06-28 22:12:28 -07:00
5317a0b755
fix: support LDAP settings properly in ftp/sftp ( #17536 )
...
Bonus this PR enhances and supports creating
buckets via ftp `mkdir`
fixes #17526
2023-06-28 13:15:21 -07:00
73de721a63
fix: handle copyObjectPart encryption properly ( #17530 )
...
- look for requested encryption while compressing
not just via HTTP Headers, but also via multipart
metadata
- look for SSE-S3 etag decryption not just via HTTP
Headers, but also via multipart metadata
fixes #17519
2023-06-28 09:43:50 -07:00
d2f5c3621f
fix: add additional decommission traces for ILM expired content ( #17522 )
...
current decommission traces were missing for
- Skipped ILM expired versions
- Skipped single DELETE marked version
- A success or failure in decommissioning DELETE marker
- allow additional info to be shared in DecomStatus() API
2023-06-27 11:59:40 -07:00
1818764840
fix: bug in passing Versioned field set for getHealReplicationInfo() ( #17498 )
...
Bonus: rejects prefix deletes on object-locked buckets earlier
2023-06-27 09:45:50 -07:00
d3e5e607a7
allow site-replication checks to work on non-distributed setups ( #17524 )
...
fixes #17523
2023-06-27 09:23:50 -07:00
c1943ea3af
Capture realtime metrics in health report ( #17516 )
2023-06-27 01:39:18 -07:00
87b6fb37d6
chore: pkg imported more than once ( #17444 )
2023-06-26 09:21:29 -07:00
21fbe88e1f
Print certain log messages once per error ( #17484 )
2023-06-24 20:29:13 -07:00
1f8b9b4bd5
fix: do not listAndHeal() inline with PutObject() ( #17499 )
...
there is a possibility that slow drives can actually add latency
to the overall call, leading to a large spike in latency.
this can happen if there are other parallel listObjects()
calls to the same drive, in-turn causing each other to sort
of serialize.
this potentially improves performance and makes PutObject()
also non-blocking.
2023-06-24 19:31:04 -07:00
216069d0da
Remove 'null' version ID from directory object response ( #17495 )
...
Fixes #17494
Regression from #17132
2023-06-23 13:26:00 -07:00
eefa047974
fix: keep decommission in a go-routine ( #17496 )
...
This was removed by mistake in #17491
2023-06-23 12:29:32 -07:00
d8dad5c9ea
s3: Make/Delete buckets to use error quorum per pool ( #17467 )
2023-06-23 11:48:23 -07:00
bf8a68879c
fix: Time ILM Actions for scanner info ( #17493 )
...
ILM Actions were not timed fix it.
2023-06-23 07:48:36 -07:00
f3248a4b37
Redact all secrets from config viewing APIs ( #17380 )
...
This change adds a `Secret` property to `HelpKV` to identify secrets
like passwords and auth tokens that should not be revealed by the server
in its configuration fetching APIs. Configuration reporting APIs now do
not return secrets.
2023-06-23 07:45:27 -07:00
d315d012a4
decom: during multiple pool decom preserve current pool status ( #17491 )
...
removal of completed pools must retain pool status of other
pools in draining, to resume any remaining draining operations.
2023-06-23 07:44:18 -07:00
bd9bf3693f
lambda: negative duration for presigned URL default to 1H ( #17489 )
...
fixes a bug where users created with Expiration as
timeSentinel is not rejected while generating the
presigned URL for lambda processing.
2023-06-23 00:17:24 -07:00
82ce78a17c
Fix locking in policy attach API ( #17426 )
...
For policy attach/detach API to work correctly the server should hold a
lock before reading existing policy mapping and until after writing the
updated policy mapping. This is fixed in this change.
A site replication bug, where LDAP policy attach/detach were not
correctly propagated is also fixed in this change.
Bonus: Additionally, the server responds with the actual (or net)
changes performed in the attach/detach API call. For e.g. if a user
already has policy A applied, and a call to attach policies A and B is
performed, the server will respond that B was attached successfully.
2023-06-21 22:44:50 -07:00
9af6c6ceef
under rebalance look for expired versions v/s remaining versions ( #17482 )
...
A continuation of PR #17479 for rebalance behavior must
also match the decommission behavior.
Fixes bug where rebalance would ignore rebalancing object
versions after one of the version returned "ObjectNotFound"
2023-06-21 13:23:20 -07:00
b94ab07c2f
Honor global root CAs for kafka audit tls ( #17481 )
...
honor global root CAs for kafka audit tls
2023-06-21 10:50:40 -07:00
7605d07bb2
add support for bucket level request count per API ( #17468 )
...
New metrics added to calculate API request count
per bucket, per API. Captures errors, including
4xx, 5xx HTTP status codes separately.
2023-06-21 09:41:59 -07:00
ccc5801112
always look for expired versions v/s remaining versions ( #17479 )
...
while decommissioning it can so happen that the non-current
versions are all expired but there is a DEL marker as the
latest version.
For such objects, we should not decommission them instead
calculate the remaining versions and if the remaining versions
is one and that version is a DEL marker consider such
an object not to be scheduled for decommissioning.
2023-06-21 08:49:28 -07:00
7c72b25ef0
Add an option to make bucket notifications synchronous ( #17406 )
...
With the current asynchronous behaviour in sending notification events
to the targets, we can't provide guaranteed delivery as the systems
might go for restarts.
For such event-driven use-cases, we can provide an option to enable
synchronous events where the APIs wait until the event is successfully
sent or persisted.
This commit adds 'MINIO_API_SYNC_EVENTS' env which when set to 'on'
will enable sending/persisting events to targets synchronously.
2023-06-20 17:38:59 -07:00
02c2ec3027
skip onlineDisks with parity mismatch ( #17478 )
2023-06-20 13:18:24 -07:00
65c31fab12
fix: do not crash rebalance code instead set the object layer ( #17465 )
...
fixes #17421
2023-06-20 09:28:23 -07:00
b6b68be052
fix: replication check for duplicate endpoints detection with wrong route ( #17474 )
2023-06-20 09:27:54 -07:00
15911c85f6
safely ignore out of band deletions while decommissioning ( #17473 )
2023-06-20 08:31:42 -07:00
5a1612fe32
Bump up madmin-go and pkg deps ( #17469 )
2023-06-19 17:53:08 -07:00
1443b5927a
allow quorum fileInfo to pick same parityBlocks ( #17454 )
...
Bonus: allow replication to proceed for 503 errors such as
with error code SlowDownRead
2023-06-18 18:20:15 -07:00
35ef35b5c1
fix a integer divide by zero crash during rebalance ( #17455 )
...
A state is updated with a delete marker, which does not have parity or
data blocks defined, which can cause the integer divide by zero panics.
This commit fixes to avoid panics.
2023-06-18 11:14:53 -07:00
6806537eb3
event args list for fanOut notification must be sized same ( #17450 )
...
without this fan-out API can crash if client cancels
the on-going request.
2023-06-18 07:09:20 -07:00
64de61d15d
fallback on etags if they match when mtime is not same ( #17424 )
...
on "unversioned" buckets there are situations
when successive concurrent I/O can lead to
an inconsistent state() with mtime while the
etag might be the same for the object on disk.
in such a scenario it is possible for us to
allow reading of the object since etag matches
and if etag matches we are guaranteed that we
have enough copies the object will be readable
and same.
This PR allows fallback in such scenarios.
2023-06-17 19:18:20 -07:00
c4d0c49a5f
ensure metadata updates go to same pool where version exists ( #17451 )
...
This PR also returns the replication status in
proxy calls and defers replication attempt if
HEAD on object version returned a error different
from NoSuchKey
2023-06-17 07:30:53 -07:00
47a48b6832
do not save any metadata from the headers in tar extract ( #17436 )
...
only preserve the same storage-class as incoming
request other than that rest of them must be
deduced.
2023-06-15 17:44:07 -07:00
a2aed12dcd
decom: Fix a typo in routing decommissioning requests ( #17435 )
...
A specific node should do the decommissioning task, however routing the
start decommissioning to that node was not working properly.
Co-authored-by: Anis Elleuch <anis@min.io>
2023-06-15 14:54:29 -07:00
d8e6e76e89
site-repl: Better error msg when setting sync in a local cluster ( #17407 )
2023-06-15 12:44:22 -07:00
ad4e511026
do not save plain-text ETag when encryption is requested ( #17427 )
...
fixes an issue under bucket replication could cause
ETags for replicated SSE-S3 single part PUT objects,
to fail as we would attempt a decryption while listing,
or stat() operation.
2023-06-15 12:43:26 -07:00
4a562d6732
fix: fanout error response - error must be string for marshaling ( #17433 )
...
Uses https://github.com/minio/minio-go/pull/1839
2023-06-15 09:21:53 -07:00
a9082e4f79
site replication: cancel ongoing op properly ( #17428 )
2023-06-15 08:05:08 -07:00
0474791cf8
fix: set time format right ( #17402 )
2023-06-14 07:49:13 -07:00
f32efd5429
more compliance related fixes ( #17408 )
...
- lifecycle must return InvalidArgument for rule errors
- do not return `null` versionId in HTTP header
- reject mixed SSE uploads with correct error message
2023-06-13 13:52:33 -07:00
22c247a988
fix: preserve multiple values for query params ( #17392 )
2023-06-13 11:38:46 -07:00
35d71682f6
fix: do not allow removal of inbuilt policies unless they are already persisted ( #17264 )
...
Dont allow removal of inbuilt policies such as `readwrite, readonly, writeonly and diagnostics`
2023-06-13 11:06:17 -07:00
3d6b88a60e
fix: syscall to record time on non-linux ( #17383 )
2023-06-13 11:04:50 -07:00
26a0803388
various compliance related fixes ( #17401 )
...
- getObjectTagging to be allowed for anonymous policies
- return correct errors for invalid retention period
- return sorted list of tags for an object
- putObjectTagging must return 200 OK not 204 OK
- return 409 ErrObjectLockConfigurationNotAllowed for existing buckets
2023-06-12 13:22:07 -07:00
ae95384dd8
Revert "heal: Update object parity with the latest configured SC ( #17187 )" ( #17404 )
2023-06-12 11:54:51 -07:00
0f0dcf0c5e
tar: Avoid storing snowball extraction header in extract objects ( #17389 )
2023-06-12 09:42:06 -07:00
6f2406b0b6
fix: protect ReplicationStats against concurrent map iteration and write crash ( #17403 )
2023-06-12 09:17:11 -07:00
bb24346e04
listen: Only error out if not able to bind any interface ( #17353 )
2023-06-12 09:09:28 -07:00
be45ffd8a4
return 204 status code for DeleteBucketTagging ( #17400 )
2023-06-11 20:49:02 -07:00
f986b0c493
replication: perform bucket resync in parallel ( #16707 )
...
Default number of parallel resync operations for a bucket to 10
to speed up resync.
2023-06-11 16:09:55 -07:00
c9e87f0548
service accounts are allowed to have no expiration ( #17397 )
2023-06-11 10:34:59 -07:00
43468f4d47
return InvalidRequest when no parts are provided ( #17395 )
2023-06-10 21:59:51 -07:00
b829e80ecb
do not disable root for invalid API config values ( #17386 )
2023-06-08 15:50:06 -07:00
6e38d0f3ab
Add more bootstrap info in debug mode ( #17362 )
2023-06-08 08:39:47 -07:00
38342b1df5
decom: Parallelize decommissining ( #17364 )
2023-06-07 14:27:51 -07:00
49ce85ee3d
allow prefix/markers to have '/' in the beginning to throw an empty ( #17373 )
2023-06-07 11:25:26 -07:00
442c50ff00
remove delimiter if not set by client, also fetchOwner is optional ( #17366 )
2023-06-06 21:31:47 -07:00
d1448adbda
use slices package and remove some helpers ( #17342 )
2023-06-06 10:12:52 -07:00
5a21b1f353
fix: Delete dir failed when .DS_Store in it ( #17352 )
2023-06-06 10:12:06 -07:00
2f9e2147f5
allow quota enforcement to rely on older values ( #17351 )
...
PUT calls cannot afford to have large latency build-ups due
to contentious usage.json, or worse letting them fail with
some unexpected error, this can happen when this file is
concurrently being updated via scanner or it is being
healed during a disk replacement heal.
However, these are fairly quick in theory, stressed clusters
can quickly show visible latency this can add up leading to
invalid errors returned during PUT.
It is perhaps okay for us to relax this error return requirement
instead, make sure that we log that we are proceeding to take in
the requests while the quota is using an older value for the quota
enforcement. These things will reconcile themselves eventually,
via scanner making sure to overwrite the usage.json.
Bonus: make sure that storage-rest-client sets ExpectTimeouts to
be 'true', such that DiskInfo() call with contextTimeout does
not prematurely disconnect the servers leading to a longer
healthCheck, back-off routine. This can easily pile up while also
causing active callers to disconnect, leading to quorum loss.
DiskInfo is actively used in the PUT, Multipart call path for
upgrading parity when disks are down, it in-turn shouldn't cause
more disks to go down.
2023-06-05 16:56:35 -07:00
75c6fc4f02
only allow decryption of etag for only sse-s3 ( #17335 )
2023-06-05 13:08:51 -07:00
1436858347
log: Add a log when saving pool.bin fails ( #17338 )
...
Co-authored-by: Anis Elleuch <anis@min.io>
2023-06-04 14:20:21 -07:00
0649aca219
Add expiration to ListServiceAccounts function ( #17249 )
2023-06-02 16:17:26 -07:00
b210ea79bc
do not save MTime in newMultipartUpload() to avoid side-affects ( #17340 )
2023-06-02 14:38:09 -07:00
68f80b5fe7
replication: ignore retention mode validation for replica ( #17332 )
2023-06-01 18:53:12 -07:00
e95825a42e
replication: use latest object info for metrics update ( #17333 )
2023-06-01 18:52:55 -07:00
931712dc46
fix: converting 'server closed idle connection' to errDiskNotFound ( #17330 )
2023-06-01 15:40:28 -07:00
54e544e03e
allow lookup()/head() operations on Veeam SOS objects ( #17331 )
2023-06-01 15:26:26 -07:00
f86b9abf32
site removal: update site config and reload targets after update ( #17327 )
2023-06-01 10:19:56 -07:00
9ef7eda33a
heal: Avoid objects created after the heal disk start time ( #17323 )
2023-05-31 13:10:45 -07:00
c9e26401fa
Fix GetObject encrypted etag ( #17302 )
...
Co-authored-by: Harshavardhana <harsha@minio.io>
2023-05-31 13:10:25 -07:00
14f6ac9222
fix: fail large content in DeleteMultipleObjects() early ( #17321 )
2023-05-31 10:58:14 -07:00
b8474295af
fix: time() returned function not being called as expected in globalSync() ( #17319 )
2023-05-31 09:40:23 -07:00
817e85a3e0
fix: proxy not set on subnet logger webhook sometimes ( #17320 )
2023-05-31 08:09:09 -07:00
1cd7f1e38d
fix: cleanup empty multipart folders upon stale upload cleanup ( #17312 )
2023-05-30 09:56:50 -07:00
043fd8b536
fix: on windows use FindClose close handler ( #17306 )
2023-05-30 02:15:57 -07:00
394690dcfb
check for upto 50%+ data disks to be offline ( #17294 )
2023-05-26 22:56:19 -07:00
c839b64f6a
fix: compressed+encrypted block overhead ( #17289 )
2023-05-26 10:57:07 -07:00
6425fec366
s3: Add x-minio-error-code header for S3 HEAD requests ( #17283 )
2023-05-26 10:13:18 -07:00
d5059840ef
fix: for delete marked objects choose appropriate parity ( #17287 )
2023-05-26 09:57:44 -07:00
65cba212e8
Remove older policy attach behavior for LDAP ( #17240 )
2023-05-26 06:31:24 -07:00
7a69c9c75a
Update builtin policy entities command ( #17241 )
2023-05-25 22:31:05 -07:00
5cd9dcb844
rebalance 'null' delete markers properly ( #17282 )
2023-05-25 16:12:53 -07:00
54c5c88fe6
Add number of offline disks in quorum errors ( #16822 )
2023-05-25 09:39:06 -07:00
9b5829c16e
avoid decommissioning DEL markers with single versions ( #17274 )
2023-05-25 09:18:49 -07:00
d749aaab69
fix: ignore existing target status when adding new targets ( #17250 )
2023-05-24 22:57:37 -07:00
62df731006
Add updatedAt for GetBucketLifecycleConfig ( #17271 )
2023-05-24 22:52:39 -07:00
d0a0eb9738
support fan-out objects via PostUpload() ( #17233 )
2023-05-24 22:51:07 -07:00
66156b8230
Stricter partNumber checks ( #17270 )
...
Fixes #17269
2023-05-24 08:00:47 -07:00
5677f73794
Add PostObject Checksum ( #17244 )
2023-05-23 07:58:33 -07:00
ef54200db7
offline drives more than 50% of total drives return error ( #17252 )
2023-05-23 07:57:57 -07:00
3e128c116e
Add lifecycle event source to audit log tags ( #17248 )
2023-05-22 15:28:56 -07:00
fc03be7891
simplify bucket metadata lookups for versioning/object locking ( #17253 )
2023-05-22 12:05:14 -07:00
2920b0fc6d
allow specification of path/virtual style bucket lookup in batch replication ( #17201 )
2023-05-21 15:16:31 -07:00
a30a55f3b1
Add object parity in listing V2M and listing versions M ( #17238 )
2023-05-19 09:42:45 -07:00
ecfb18b26a
Freeze the s3 APIs until the notification sub-system initializes completely ( #17182 )
2023-05-19 08:44:48 -07:00
7d433f16c4
before return make globalScannerMetrics.incTime call ( #17230 )
2023-05-18 13:45:05 -07:00
9d96b18df0
Add "name" and "description" params to service acc ( #17172 )
2023-05-17 17:05:36 -07:00
ad2ab6eb3e
fix: Give accurate cap to slice ( #17224 )
2023-05-17 15:14:09 -07:00
f037c9b286
Protecting the read index is not out of bounds ( #17226 )
2023-05-17 12:09:41 -07:00
f7d29b4a53
cleanup of multipart per disk must cleanup itself only ( #17223 )
2023-05-17 01:45:58 -07:00
06557fe8be
allow decommissioned pools to be removed while others are finishing ( #17221 )
2023-05-16 16:00:57 -07:00
2131046427
replication: fix audit log reporting ( #17222 )
2023-05-16 15:35:08 -07:00
aaf1abc993
simplify HardLimitReader by using LimitReader for internal usage ( #17218 )
2023-05-16 13:14:37 -07:00
413549bcf5
fix: loadStatsFromDisk() should return nil for configNotFound ( #17217 )
2023-05-16 12:23:38 -07:00
9a799065b3
fix: make slice cap of right size ( #17192 )
2023-05-16 08:10:07 -07:00
fd2959fa3a
fix: workers.New err must be returned ( #17208 )
2023-05-16 08:08:00 -07:00
15bec32bb4
fix: tier handlers must write error only once ( #17205 )
2023-05-15 23:56:52 -07:00
e2b7a08c10
heal: Update object parity with the latest configured SC ( #17187 )
2023-05-15 21:32:13 -07:00
ef2fc0f99e
fix: reduce using memory and temporary files. ( #17206 )
2023-05-15 14:08:54 -07:00
d063596430
fix: veeam SOS API 'system.xml' strings ( #17202 )
2023-05-15 12:06:42 -07:00
bd2dc6c670
fix: in healing tracker printTo when err ( #17207 )
2023-05-15 10:14:48 -07:00
b62791617c
fix: notify systemd as soon as we wait on the OS signal ( #17199 )
2023-05-12 16:42:17 -07:00
e07c2ab868
Use hash.NewLimitReader for internal multipart calls ( #17191 )
2023-05-12 11:19:08 -07:00
203755793c
fix: in printEndpointError count error once per init() ( #17193 )
2023-05-12 10:41:54 -07:00
883c98e26f
fix: remove objects when there are skipped versions due to ILM in decom ( #17198 )
2023-05-12 10:37:38 -07:00
f5a20a5d06
allow nodes offline in k8s setups when expanding pools ( #17183 )
2023-05-11 17:41:33 -07:00
ef7177ebbd
disallow bucket replication setup with site replication ( #17189 )
2023-05-11 15:48:40 -07:00
3637aad36e
do not count ILM expired objects and other skipped objects ( #17184 )
2023-05-11 13:35:16 -07:00
77db9686fb
Update console to v0.27.0 ( #17188 )
2023-05-11 12:18:17 -07:00
c326e5a34e
Add metrics for webhook endpoint stats ( #17179 )
2023-05-11 11:24:37 -07:00
c23c982593
xmlDecoder err use ErrMalformedXML when PutBucketACLHandler ( #17185 )
2023-05-11 11:11:15 -07:00
a3d666356c
fix: error in capturing XFS error config in health report ( #17176 )
2023-05-10 15:20:48 -07:00
3cdbc2f414
add validationErr to validateConfig When DeleteIdentityProviderCfg ( #17173 )
2023-05-10 09:37:30 -07:00
b92cdea578
fix: start using pkg/workers to spawn parallel workers ( #17170 )
2023-05-09 16:37:31 -07:00
5e629a99af
fix: for profiling duration parsing error reply use ErrInvalidRequest ( #17169 )
2023-05-09 14:27:49 -07:00
a7f266c907
allow JWT parsing on large session policy based tokens ( #17167 )
2023-05-09 00:53:08 -07:00
57acacd5a7
Support persistent queue store for loggers ( #17121 )
2023-05-08 21:20:31 -07:00
629503ff73
add Err to BucketExists when NoSuchBucket ( #17155 )
2023-05-08 07:51:59 -07:00
e3a070e3de
put *msgp.Reader back to pool ( #17156 )
2023-05-08 07:51:39 -07:00
c5c1426262
Validate if replication config being added is self referential ( #17142 )
2023-05-06 13:35:43 -07:00
76913a9fd5
Signed trailers for signature v4 ( #16484 )
2023-05-05 19:53:12 -07:00
5569acd95c
disallow EC:0 if not set during server startup ( #17141 )
2023-05-04 14:44:30 -07:00
1d0211d395
allow deletes on directory objects to perform permanent deletes ( #17132 )
2023-05-04 14:43:52 -07:00
06cd0a636e
Avoid calling KES Status when peers ping each other ( #17140 )
2023-05-04 11:28:33 -07:00
7f7b489a3d
snowball: use latest time when mtime is missing ( #17133 )
2023-05-04 07:29:33 -07:00
bb6f4d7633
Remove redundant checkFormatJSON logging ( #17134 )
2023-05-04 07:28:37 -07:00
6e24dff26a
Added MINIO_BROWSER_LOGIN_ANIMATION env support for WebUI console ( #17123 )
...
Signed-off-by: Benjamin Perez <benjamin@bexsoft.net>
2023-05-03 15:32:50 -07:00
e372e4e592
add nodeName to the log while taking drive offline ( #17124 )
2023-05-03 15:05:45 -07:00
9571b0825e
add configurable VRF interface and user-timeout ( #17108 )
2023-05-03 14:12:25 -07:00
90e2cc3d4c
Add audit logging of site replication multipart proxying ( #17122 )
2023-05-03 11:19:45 -07:00
9112ca4e29
change ttfb_distribution metrics to histogramMetric ( #17115 )
2023-05-03 07:31:00 -07:00
ec84bad882
batch replication now supports arbitrary S3 targets ( #17113 )
2023-05-02 22:52:35 -07:00
b53376a3a4
change directory objects to never create new versions ( #17109 )
2023-05-02 16:09:33 -07:00
0ec722bc54
Add tags to NewerNoncurrentVersions audit event ( #17110 )
2023-05-02 12:56:33 -07:00
1704abaf6b
fix: store notification events immediately for persistent queues ( #17112 )
2023-05-02 07:53:13 -07:00
ab34f0065c
Support systemd notify protocol ( #17062 )
2023-05-01 23:15:08 -07:00
e8c0a50862
optimization use small blocks up to 64KB ( #17107 )
2023-05-01 09:47:49 -07:00
02d8f3cdc8
fix: remove active healing on .minio.sys/ during startup ( #17072 )
2023-04-29 02:05:28 -07:00
7ae69accc0
allow root user to be disabled via config settings ( #17089 )
2023-04-28 12:24:14 -07:00
7fad0c8b41
Remove checksums from HTTP range request, add part checksums ( #17105 )
2023-04-28 08:26:32 -07:00
d5aff735be
info: Add drives per set and sets count per pool information ( #17100 )
2023-04-27 15:24:03 -07:00
98c26df53e
fix: allow past retention headers to be copied in batch replication ( #17095 )
2023-04-27 13:43:18 -07:00
2448a9e047
grafana: Remove minio_s3_requests_errors_total metric ( #17094 )
2023-04-27 10:55:30 -07:00
e7cac8acef
Add tags to auditLogLifecycle ( #17081 )
2023-04-26 17:49:00 -07:00
6105997299
remove unnecessary log when listing resume fails ( #17086 )
2023-04-26 14:53:25 -07:00
8c874884fc
fix: do not copy context in DiskInfo cache ( #17085 )
2023-04-26 12:13:54 -07:00
ebfe81e5fd
Fix put bucket policy error code ( #17084 )
2023-04-26 11:21:27 -07:00
72802a5972
Use 'minio/pkg/sync/errgroup' and 'minio/pkg/workers' ( #17069 )
2023-04-25 22:57:40 -07:00
b1f3935c5b
allow ListObjects() when a prefix is an object ( #17074 )
2023-04-25 22:41:54 -07:00
dbd53af369
fix: initialize reverse proxy forwarder with right public certs ( #17080 )
2023-04-25 15:50:32 -07:00
b09fe0e50e
fix: DeleteBucket for peers() must recreate bucket upon errors ( #17079 )
2023-04-25 14:16:35 -07:00
fae9000304
heal: Pick maximally occuring modTime in quorum ( #17071 )
2023-04-25 10:13:57 -07:00
8fd07bcd51
simplify sort.Sort by using sort.Slice ( #17066 )
2023-04-24 13:28:18 -07:00
6addc7a35d
server-info: Return initializing state properly ( #17070 )
2023-04-24 09:10:02 -07:00
477230c82e
avoid attempting to migrate old configs ( #17004 )
2023-04-21 13:56:08 -07:00
d1737199ed
fix: delete DNS upon success, update failure message ( #17059 )
2023-04-21 12:12:31 -07:00
84f31ed45d
simplify MRF, converge it to regular healing ( #17026 )
2023-04-19 07:47:42 -07:00
8a81e317d6
verify maxPartID in object options helpers ( #17015 )
2023-04-18 22:34:30 -07:00
224d9a752f
fix: the race in healing tracker code ( #17048 )
2023-04-18 14:49:56 -07:00
0db34e4b85
Listen bucket events to send empty events with new line ( #17037 )
2023-04-18 08:11:30 -07:00
f66625be67
Snowball: Extract headers for metadata ( #17042 )
2023-04-17 12:16:54 -07:00
6825bd7e75
fix: inlined objects don't need to honor long locks ( #17039 )
2023-04-17 12:16:37 -07:00
839b9c9271
Reduce allocations in Walkdir ( #17036 )
2023-04-15 10:25:25 -07:00
dd9ed85e22
implement support for FTP/SFTP server ( #16952 )
2023-04-15 07:34:02 -07:00
6c1410f7f5
fix: Type of rejection for FIFO quota input ( #17016 )
2023-04-15 01:22:18 -07:00
f92450d8b3
commonParity should pick readable FileInfo ( #17032 )
2023-04-14 16:23:28 -07:00
c133979b8e
Add part count to checksum ( #17035 )
2023-04-14 09:44:45 -07:00
a9269cee29
heal: avoid logging version not found ( #17031 )
2023-04-13 19:45:52 -07:00
958a480e53
fix: lambda function expiration when cred.Expiration is set ( #17029 )
2023-04-13 08:10:57 -07:00
a42650c065
Add minio_bucket_usage_version_total metric to Prometheus ( #17023 )
2023-04-12 20:08:07 -07:00
bdad3730f7
fix: do not error out if the local bucket is missing ( #17025 )
2023-04-12 15:44:16 -07:00
a5835cecbf
fix: regression in counting total requests ( #17024 )
2023-04-12 14:37:19 -07:00
cd6dec49c0
Add trace support for ilm activity ( #16993 )
2023-04-11 19:22:32 -07:00
6877578bbc
Update minio_node_bucket_scans_finished metrics ( #17006 )
2023-04-11 19:21:34 -07:00
f3682b6149
allow writes to pools with inconsistent xl.meta ( #17008 )
2023-04-11 11:17:46 -07:00
056ca0c68e
refactor: reuse open file in storage interface ( #16970 )
2023-04-09 23:09:28 -07:00
25f7a8e406
Indicate RenameData is called by healObject ( #16997 )
2023-04-09 10:25:37 -07:00
1f1c267b6c
Add used inodes to disk info ( #16994 )
2023-04-07 07:52:14 -07:00
fc94ea1ced
trace: Fix <unknown> func name of requests rejected by max clients ( #16977 )
2023-04-07 07:51:12 -07:00
67d2cf8f30
add Err to Get getRemoteTargetClient. ( #16982 )
2023-04-07 07:50:46 -07:00
09a25ea7b7
lint: fix some lint issues on files
2023-04-06 22:42:10 -07:00
260a63ca73
os/windows: do not return errFileNotFound in readDir's ( #16988 )
2023-04-06 22:28:58 -07:00
d3f70ea340
Enable audit log for global handlers ( #16964 )
...
Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
2023-04-06 21:03:39 -07:00
91b6fe1af3
trace: Bootstrap to show the correct source line number ( #16989 )
2023-04-06 17:51:53 -07:00
9803f68522
snowball: Restrict zstd window size ( #16987 )
2023-04-06 17:47:38 -07:00
4c204707fd
Correct to remove `null` version while ILM rule application ( #16971 )
...
Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
Co-authored-by: Harshavardhana <harsha@minio.io>
2023-04-06 14:10:01 -07:00
c06e0bfef9
set correct `Host:` value for replication event notification ( #16984 )
2023-04-06 10:20:53 -07:00
8625a9dbb3
Make listing metadata permissions stricter ( #16974 )
2023-04-06 07:52:35 -07:00
0320ac43cb
simplify bucketInfo return in GetBucketInfo peer call ( #16983 )
2023-04-06 01:30:50 -07:00
111c7d4026
assumeRole return the correct http code for auth errors ( #16967 )
2023-04-05 22:19:31 -07:00
62c3df0ca3
fix: directory listing on Go 1.20 windows ( #16976 )
2023-04-05 14:36:49 -07:00
3158f2d12e
Add support for batch key rotation ( #16844 )
2023-04-04 10:56:54 -07:00
6e359c586e
fix: close chan before return in scanner usage updates ( #16960 )
2023-04-04 10:51:05 -07:00
699a24f7e5
batch: validate versioning on src/tgt buckets ( #16955 )
2023-04-04 10:50:11 -07:00
5fa3665074
add isSysErrNotDir to openFileNoSync ( #16958 )
2023-04-04 08:00:08 -07:00
3b7781835e
add lock metrics per node ( #16943 )
2023-04-03 21:23:24 -07:00
5fe1b46bfd
Enabled to send audit log while version deletion ( #16954 )
...
Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
2023-04-03 11:58:04 -07:00
407c9ddcbf
feat: add batch replicate from remote MinIO to local cluster ( #16922 )
2023-03-31 10:48:36 -07:00
d90d0c8931
Use one http response recorder per external http call ( #16938 )
2023-03-31 09:37:29 -07:00
216a471bbb
on quorum DeleteObject() errors attempt an MRF ( #16932 )
2023-03-31 08:15:41 -07:00
a7b7860e0e
fix: potential data conflicts save site-resync metadata ( #16926 )
2023-03-30 20:59:45 -07:00
d703daa480
Add metadata extension to ListObjectVersions ( #16930 )
2023-03-30 12:20:42 -07:00
dc8fdcb9c9
fix: error checking in DeleteBucket ( #16929 )
2023-03-30 11:54:08 -07:00
7a6c4e438e
allow more workers for ILM expiration ( #16924 )
2023-03-30 10:47:15 -07:00
c468b4e2a8
fix: avoid out of slice index ( #16925 )
2023-03-30 08:03:48 -07:00
b04956a676
fix: put *msgp.Reader back in pool ( #16927 )
2023-03-30 08:03:12 -07:00
518f6e4d39
fix: missing return after error response ( #16920 )
2023-03-29 16:21:13 -07:00
483b226cc1
fix: avoid logging when object/version not found in replication ( #16919 )
2023-03-29 15:02:45 -07:00
8e02660a0d
update all our deps ( #16899 )
2023-03-28 03:45:24 -07:00
16feef2a2c
fix: time.Parse RFC3339Nano ( #16892 )
2023-03-27 11:51:54 -07:00
66ff17e452
fix: Avoid multiple write responses ( #16894 )
2023-03-27 09:15:23 -07:00
4c5edacae2
ignore operation timedout errors ( #16891 )
2023-03-26 03:16:51 -07:00
8b4d0255b7
Set Console global Root CAs early to trust provided certs ( #16890 )
2023-03-25 09:58:38 -07:00
74040b457b
Allow setting sync mode for site replication ( #16876 )
2023-03-24 14:41:23 -07:00
5e3bfd2148
Add user tags when listing with metadata ( #16883 )
2023-03-23 10:27:19 -07:00
8b0ab6ead6
Revert "Make localLocker lock attempts cancellable ( #16510 )" ( #16884 )
2023-03-23 10:26:21 -07:00
b1b0aadabf
Revert query parameter `src` from diag upload if callhome enabled ( #16881 )
2023-03-23 00:24:58 -07:00
ac7d9c449a
add missing expiration information from 'sts info' ( #16878 )
2023-03-22 16:47:02 -07:00
1346561b9d
return quorum error instead of insufficient storage error ( #16874 )
2023-03-22 16:22:37 -07:00
4bc52897b2
Update yaml files to latest version RELEASE.2023-03-22T06-36-24Z
2023-03-22 21:16:15 +00:00
6017b63a06
top-locks: Group by lock request ID ( #16860 )
2023-03-21 18:35:29 -07:00
0448728228
fix: add deadline conns and dnsCache for remote transports ( #16865 )
2023-03-21 08:49:20 -07:00
12047702f5
fix: tweak the maintenance=true to satisfy baremetal first ( #16864 )
2023-03-21 08:48:38 -07:00
fb1492f531
check for quorum errors for DeleteBucket() ( #16859 )
2023-03-20 23:38:06 -07:00
05444a0f6a
Use the official pub key to always verify binary ( #16857 )
2023-03-20 13:16:18 -07:00
b3c54ec81e
reject object names with '\' on windows ( #16856 )
2023-03-20 13:16:00 -07:00
6c11dbffd5
add crash protection from backend modifications ( #16846 )
2023-03-20 09:08:42 -07:00
3b5dbf9046
allow bootstrapping to validate internode tokens ( #16853 )
2023-03-20 01:40:24 -07:00
09c733677a
Add test for fixed post policy exploit ( #16855 )
2023-03-20 01:06:45 -07:00
8d6558b236
fix: convert '\' to '/' on windows ( #16852 )
2023-03-20 00:35:25 -07:00
67f4ba154a
fix: post policy request security bypass ( #16849 )
2023-03-19 21:15:20 -07:00
440ad20c1d
Add support for batch job cancellation ( #16843 )
2023-03-17 23:42:43 -07:00
31fba6f434
Save bootstrap trace events in a circular buffer ( #16823 )
2023-03-17 16:01:03 -07:00
280442e533
reduce 250ms to 50ms retry looking for metacache block ( #16795 )
2023-03-17 14:44:01 -07:00
850a945a18
Added query parameter `src` to diag upload if callhome enabled ( #16837 )
...
Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
2023-03-17 08:30:16 -07:00
46f9049fb4
simplify error responses for KMS ( #16793 )
2023-03-16 11:59:42 -07:00
58266c9e2c
Add enable flag for LDAP IDP config ( #16805 )
2023-03-16 11:58:59 -07:00
d1e775313d
support decommissioning of tiered objects ( #16751 )
2023-03-16 07:48:05 -07:00
e700be8cd6
fix: return appropriate Location header for MakeBucket() ( #16820 )
2023-03-15 13:40:40 -07:00
e0f4dd6027
remove unncessary logs from WalkDir(), PutObject() ( #16818 )
2023-03-15 11:52:23 -07:00
de02eca467
restore rotating root credentials properly ( #16812 )
2023-03-15 08:07:42 -07:00
c7f7e67a10
Do not allow adding root user to IAM subsystem ( #16803 )
2023-03-13 12:46:17 -07:00
628042e65e
tests: Protect globalLocalDrives against races ( #16800 )
2023-03-13 06:04:20 -07:00
6305b206e1
fix: site-repl should heal STS with virtual parent ( #16792 )
2023-03-10 16:21:51 -08:00
d85da9236e
Add Object Version count histogram ( #16739 )
2023-03-10 08:53:59 -08:00
5c087bdcad
fix: a cosmetic error reporting with a lock timeout ( #16788 )
2023-03-09 15:16:13 -08:00
a547bf517d
Remove locks on usage cache ( #16786 )
2023-03-09 15:15:46 -08:00
b984bf8d1a
allow expiration of all versions during Listing() ( #16757 )
2023-03-09 15:15:30 -08:00
fb6ab1cca2
fix: allow replication of 'null' delete markers ( #16773 )
2023-03-08 07:03:29 -08:00
56c57e2c53
tier-stats: Avoid repeated logs ( #16774 )
2023-03-07 16:43:05 -08:00
a6057c35cc
Avoid peer notification when peer is offline, tune retries ( #16737 )
2023-03-07 08:13:28 -08:00
901887e6bf
feat: add lambda transformation functions target ( #16507 )
2023-03-07 08:12:41 -08:00
ee54643004
Avoid unnecessary replication heal attempts ( #16769 )
2023-03-07 07:43:38 -08:00
0a17acdb34
return error if policy changes on disabled groups ( #16766 )
2023-03-06 10:46:24 -08:00
72e5212842
fix: handle syscall.EROFS also for osIsPermission() ( #16765 )
2023-03-06 08:56:29 -08:00
714283fae2
cleanup ignored static analysis ( #16767 )
2023-03-06 08:56:10 -08:00
3423028713
cleanup Go linter settings ( #16736 )
2023-03-04 20:57:35 -08:00
9d062b37d7
return underlying error with BackendDown{} error ( #16738 )
2023-03-03 23:56:53 -08:00
f350953a19
calculate disk cache usage percent accurately ( #16740 )
2023-03-02 20:32:22 -08:00
958bba5b42
Attach creds, owner and region to madmin calls ( #16658 )
...
Signed-off-by: Daniel Valdivia <18384552+dvaldivia@users.noreply.github.com>
2023-03-02 14:35:24 -08:00
0f2b95b497
fix a data race in IAM loading ( #16742 )
2023-03-02 14:32:54 -08:00
d07089ceac
Fix scanner deadlock on lost global lock ( #16726 )
2023-02-28 21:34:45 -08:00
0ff931dc76
fix: allow CORS to work by default ( #16713 )
2023-02-27 10:10:45 -08:00
4d708cebe9
Support adding service accounts with expiration ( #16430 )
...
Co-authored-by: Harshavardhana <harsha@minio.io>
2023-02-27 10:10:22 -08:00
4d7c8e3bb8
Remove redundant log ( #16710 )
...
Co-authored-by: z30001483 <zekaifeng2@huawei.com>
2023-02-27 09:59:47 -08:00
8cde38404d
Add metrics for custom auth plugin ( #16701 )
2023-02-27 09:55:18 -08:00
fe7bf6cbbc
Support tier-add if tier backend not empty ( #16715 )
2023-02-27 09:26:26 -08:00
8b4eb2304b
Set logger webhook proxy on subnet proxy change ( #16665 )
...
Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
2023-02-27 08:35:36 -08:00
ae029191a3
liveness returns "busy" if queued requests > available capacity ( #16719 )
2023-02-27 08:34:52 -08:00
bfedea9bad
fix: disk healing should honor the right pool/set index ( #16712 )
2023-02-27 04:55:32 -08:00
7777d3b43a
Remove globalSTSTLSConfig ( #16709 )
2023-02-26 23:37:00 -08:00
9ed4fc9687
Remove globalOpenIDConfig ( #16708 )
2023-02-25 21:01:37 -08:00
b49b39e99d
fix: errNoSuchPolicy should use errors.Is ( #16656 )
2023-02-25 00:01:37 -08:00
cd3a2de5a3
add isValidLocation to common parseLocation ( #16690 )
2023-02-25 08:09:20 +05:30
6e8960ccdd
fix: delete globalProfiler should lock ( #16697 )
2023-02-25 08:07:44 +05:30
e05f3d5d84
Remove globalLDAPConfig ( #16706 )
2023-02-25 08:07:22 +05:30
94c6cb1323
tests: Add test for S3 API error codes ( #16705 )
2023-02-25 08:06:29 +05:30
9acf1024e4
Remove bloom filter ( #16682 )
...
Removes the bloom filter since it has so limited usability, often gets saturated anyway and adds a bunch of complexity to the scanner.
Also removes a tiny bit of CPU by each write operation.
2023-02-24 09:03:31 +05:30
a0f06eac2a
add Veeam SOS API first implementation ( #16688 )
2023-02-22 19:54:57 +05:30
83fe1a2732
log: add more info about BROWSER_URL ( #16678 )
2023-02-22 19:54:05 +05:30
5c98223c89
add correct HostId instead of deploymentId for error responses ( #16686 )
2023-02-22 15:41:09 +05:30
663a0b7783
save correct bucketInfo on it's indexes ( #16685 )
2023-02-22 14:08:34 +05:30
6efe4d1df6
perf: Only remove generated data when no bucket name specified ( #16610 )
2023-02-21 21:21:40 -08:00
fb17f97cf3
move audit and logger message structure to minio/pkg ( #16655 )
...
Signed-off-by: Daniel Valdivia <18384552+dvaldivia@users.noreply.github.com>
2023-02-21 21:21:17 -08:00
9202c6e26a
Reload bucket targets when replication config is refreshed ( #16684 )
2023-02-21 21:18:49 -08:00
59a5456091
fix: STS error translation to API error ( #16683 )
2023-02-22 09:57:48 +05:30
8bfe972bab
Set meaningful message from minio with env variable KMS_SECRET_KEY ( #16584 )
2023-02-22 07:13:01 +05:30
fd6622458b
Add detailed scanner trace output and notifications ( #16668 )
2023-02-21 09:33:33 -08:00
8a08861dd9
fix: healing of replication config for endpoint changes ( #16648 )
2023-02-20 16:06:13 +05:30
Anis Eleuch
Harshavardhana
Poorna
Praveen raj Mani
Klaus Post
Sho Ce
ruspaul013
drivebyer
Krishnan Parthasarathi
flisk
Kaan Kabalak
jiuker
Shubhendu
Aditya Manthramurthy
Shireesh Anjal
guangwu
Alex
WGH
ferhat elmas
Shubhendu
Allan Roger Reid
Minio Trusted
Daniel Valdivia
dorman