MyFavMirrors/minio
1
0
Fork 0
mirror of https://github.com/minio/minio.git synced 2024-12-25 14:45:54 -05:00
Code Issues Packages Projects Releases Wiki Activity
12,016 Commits 4 Branches 506 Tags 189 MiB
ba39ed9af7
Commit Graph

771 Commits

Author SHA1 Message Date
Klaus Post
74253e1ddc
Fix BackendInfo() race (#18305) 
`GetParityForSC` has a value receiver, so Config is copied before the lock is obtained.

Make it pointer receiver.

Fixes:

```
WARNING: DATA RACE
Read at 0x0000079cdd10 by goroutine 190:
  github.com/minio/minio/cmd.(*erasureServerPools).BackendInfo()
      github.com/minio/minio/cmd/erasure-server-pool.go:579 +0x6f
  github.com/minio/minio/cmd.(*erasureServerPools).LocalStorageInfo()
      github.com/minio/minio/cmd/erasure-server-pool.go:614 +0x3c6
  github.com/minio/minio/cmd.(*peerRESTServer).LocalStorageInfoHandler()
      github.com/minio/minio/cmd/peer-rest-server.go:347 +0x4ea
  github.com/minio/minio/cmd.(*peerRESTServer).LocalStorageInfoHandler-fm()
...

WARNING: DATA RACE
Read at 0x0000079cdd10 by goroutine 190:
  github.com/minio/minio/cmd.(*erasureServerPools).BackendInfo()
      github.com/minio/minio/cmd/erasure-server-pool.go:579 +0x6f
  github.com/minio/minio/cmd.(*erasureServerPools).LocalStorageInfo()
      github.com/minio/minio/cmd/erasure-server-pool.go:614 +0x3c6
  github.com/minio/minio/cmd.(*peerRESTServer).LocalStorageInfoHandler()
      github.com/minio/minio/cmd/peer-rest-server.go:347 +0x4ea
  github.com/minio/minio/cmd.(*peerRESTServer).LocalStorageInfoHandler-fm()
```
 2023-10-24 08:15:41 -07:00
Krishnan Parthasarathi
8cd80fec8c
Add unit test for lifecycle.FilterRules (#18284) 2023-10-19 21:33:28 -07:00
Klaus Post
e37508fb8f
fix: linter errors in Windows specific code (#18276) 2023-10-18 11:08:15 -07:00
Krishnan Parthasarathi
557df666fd
Don't skip rules with ExpiredObjectDeleteMarker (#18256) 2023-10-16 22:46:46 -07:00
Aditya Manthramurthy
28a2d1eb3d
Allow OpenID ARN resource ID to start with a - (#18255) 2023-10-16 13:50:51 -07:00
Klaus Post
128256e3ab
Add event counters (#18232) 
Export metric for global events sent and skipped for the lifetime of the server.
 2023-10-12 15:39:22 -07:00
Shubhendu
5b9656374c
Error if target went offline (#18221) 
If target went offline while MinIO was down, error once
while trying to send message. If target goes offline during
MinIO server running, it already comes through ping() call
and errors out if target offline.

Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
 2023-10-12 06:13:57 -07:00
Harshavardhana
6829ae5b13
completely remove drive caching layer from gateway days (#18217) 
This has already been deprecated for close to a year now.
 2023-10-11 21:18:17 -07:00
Praveen raj Mani
c27d0583d4
Send kafka notification messages in batches when queue_dir is enabled (#18164) 
Fixes #18124
 2023-10-07 08:07:38 -07:00
Sveinn
603437e70f
Fix startup formatting (#18156) 
Percentages in root user names are used for formatting.

Before:
```
S3-API: http://192.168.50.21:9000  http://172.31.96.1:9000  http://127.0.0.1:9000
RootUser: "U4B6Zi!b75DXSPm%!!(MISSING)a(MISSING)vZb"
RootPass: "Q4#Q6y8G%!P(MISSING)x#npP4dudUobU#NBcGB7RMKV4ajYb"

Console: http://192.168.50.21:51915 http://172.31.96.1:51915 http://127.0.0.1:51915
RootUser: "U4B6Zi!b75DXSPm%!!(MISSING)a(MISSING)vZb"
RootPass: "Q4#Q6y8G%!P(MISSING)x#npP4dudUobU#NBcGB7RMKV4ajYb"

Command-line: https://min.io/docs/minio/linux/reference/minio-mc.html#quickstart
FORMAT: %117s MESSAGE: $ mc alias set myminio http://192.168.50.21:9000 "U4B6Zi!b75DXSPm%avZb" "Q4#Q6y8G%%Px#npP4dudUobU#NBcGB7RMKV4ajYb"
   $ mc alias set myminio http://192.168.50.21:9000 "U4B6Zi!b75DXSPm%!a(MISSING)vZb" "Q4#Q6y8G%Px#npP4dudUobU#NBcGB7RMKV4ajYb"
```

After:

```
Status:         1 Online, 0 Offline.
S3-API: http://192.168.50.21:9000  http://172.31.96.1:9000  http://127.0.0.1:9000
RootUser: "U4B6Zi!b75DXSPm%avZb"
RootPass: "Q4#Q6y8G%%Px#npP4dudUobU#NBcGB7RMKV4ajYb"

Console: http://192.168.50.21:52421 http://172.31.96.1:52421 http://127.0.0.1:52421
RootUser: "U4B6Zi!b75DXSPm%avZb"
RootPass: "Q4#Q6y8G%%Px#npP4dudUobU#NBcGB7RMKV4ajYb"

Command-line: https://min.io/docs/minio/linux/reference/minio-mc.html#quickstart
   $ mc alias set myminio http://192.168.50.21:9000 "U4B6Zi!b75DXSPm%avZb" "Q4#Q6y8G%%Px#npP4dudUobU#NBcGB7RMKV4ajYb"
```

No need for special Windows case. `mc` works just fine.
 2023-10-02 07:39:47 -06:00
Shubhendu
10d5dd3a67
fix: a regression with audit log sending (#18112) 
Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
 2023-09-26 12:23:02 -07:00
Harshavardhana
d9f1df01eb
return an error in CopyAligned upon premature EOF (#18110) 
add a unit-test to capture this corner case
 2023-09-26 11:20:06 -07:00
Anis Eleuch
4eeb48f8e0
Return cached online/offline status for audit/http loggers (#18083) 
To avoid having delays in prometheus scrape and in 'mc admin info' command.
 2023-09-21 16:58:24 -07:00
Harshavardhana
1472875670
fix: failed messages counting in audit_http metrics (#18075) 
all retries must not be counted as failed messages,
a failed message is a single counter not for all
retries, this PR fixes this.

Also we do not need to retry 10-times, instead we should
retry at max 3 times with some jitter to deliver the
messages.
 2023-09-21 11:24:56 -07:00
Harshavardhana
2add57cfed
apply healing per object at 1024 cycles (#18050) 
- we already have MRF for most recent failures
- we trigger healing during HEAD/GET operation

These are enough, also change the default max wait
from 5sec to 1sec for default scanner speed.
 2023-09-19 09:24:22 -07:00
jiuker
9947c01c8e
feat: SSE-KMS use uuid instead of read all data to md5. (#17958) 2023-09-18 10:00:54 -07:00
Anis Eleuch
419e5baf16
fix: webhook notify endpoint with standard ports (#18016) 2023-09-14 20:10:44 -07:00
Alex
dc48cd841a
Added MINIO_PROMETHEUS_AUTH_TOKEN env support (#18028) 
Signed-off-by: Benjamin Perez <benjamin@bexsoft.net>
 2023-09-14 17:28:21 -07:00
Aditya Manthramurthy
cbc0ef459b
Fix policy package import name (#18031) 
We do not need to rename the import of minio/pkg/v2/policy as iampolicy
any more.
 2023-09-14 14:50:16 -07:00
Harshavardhana
32890342ce
introduce MINIO_BROWSER_REDIRECT env to enable/disable auto-redirect (#18025) 2023-09-13 18:43:57 -07:00
Anis Eleuch
41de53996b
heal: calculate the number of workers based on NRRequests (#17945) 2023-09-11 14:48:54 -07:00
Harshavardhana
9878031cfd
fix: change DISK_ to DRIVE_ for some drive related envs (#18005) 2023-09-11 12:19:22 -07:00
Harshavardhana
e3fbcaeb72
allow scanner key cycle to be empty (#18001) 
configs from 2020 server throws an
error due to deprecation of the keys
however an attempt is made to parse
them, we should have chosen existing
defaults - this PR fixes that.
 2023-09-09 08:53:32 -07:00
Anis Eleuch
b9269151a4
fix: drive rotational calculation status for partitions (#17986) 
Fix drive rotational calculation status

If a MinIO drive path is mounted to a partition and not a real disk,
getting the rotational status would fail because Linux does not expose
that status to partition; In other words,
/sys/block/drive-partition-name/queue/rotational does not exist;

To fix the issue, the code will search for the rotational status of the
disk that hosts the partition, and this can be calculated from the
real path of /sys/class/block/<drive-partition-name>
 2023-09-06 12:37:57 -07:00
Harshavardhana
5b114b43f7
refactor bandwidth throttling for replication target (#17980) 
This refactor is to allow using the bandwidth throttling
for other purposes.
 2023-09-05 20:21:59 -07:00
Aditya Manthramurthy
1c99fb106c
Update to minio/pkg/v2 (#17967) 2023-09-04 12:57:37 -07:00
Anis Eleuch
6a8d8f34a5
kafka: Do not require key when sending a message (#17962) 
Keys are helpful to ensure the strict ordering of messages, however currently the
code uses a random request id for every log, hence using the request-id
as a Kafka key is not serve any purpose;

This commit removes the usage of the key, to also fix the audit issue from
internal subsystem that does not have a request ID.
 2023-09-01 08:37:22 -07:00
Harshavardhana
0d1fbef751
fix: a possible crash in event target Close() (#17948) 
these are possible crashes when the configured
target is still in init() state and never finished
- however a delete config was initiated.
 2023-08-30 07:27:45 -07:00
Harshavardhana
07b1281046 add queue_dir to help message for logger/audit targets 2023-08-29 16:07:35 -07:00
Harshavardhana
adb8be069e
tune-kafka targets to ensure timeout triggers on hung brokers (#17898) 
hung brokers can cause slowness to the entire system
when many callers are hung, leading to large goroutine
build-up.
 2023-08-22 20:26:35 -07:00
Harshavardhana
3a0125fa1f
remove unexpected logging from peer calls (#17888) 
also make sure RequestID is set for system logs
 2023-08-21 14:25:24 -07:00
Daniel Valdivia
328cb0a076
Pass environment variable to control session length to console (#17885) 
Signed-off-by: Daniel Valdivia <18384552+dvaldivia@users.noreply.github.com>
 2023-08-21 11:55:43 -07:00
jiuker
fa2a8d7209
fix: drain the req.body into io.Discard correctly (#17881) 2023-08-21 01:09:07 -07:00
Andreas Auernhammer
8f8f8854f0
update minio/kes-go dep to v0.2.0 (#17850) 
This commit updates the minio/kes-go dependency
to v0.2.0 and updates the existing code to work
with the new KES APIs.

The `SetPolicy` handler got removed since it
may not get implemented by KES at all and could
not have been used in the past since stateless KES
is read-only w.r.t. policies and identities.

Signed-off-by: Andreas Auernhammer <hi@aead.dev>
 2023-08-19 07:37:53 -07:00
Harshavardhana
bc7c0d8624
if object is a delete marker it must skip tags filter in ILM (#17861) 2023-08-18 09:36:23 -07:00
Harshavardhana
11dfc817f3
do not log client canceled events (#17838) 2023-08-17 14:53:43 -07:00
Harshavardhana
8a9b886011
update grafana dashboard with disk -> drive rename (#17857) 2023-08-15 16:04:20 -07:00
Klaus Post
406ea4f281
Fix distributed listing not able to resume (#17855) 
Two fields in lifecycles made GOB encoding consistently fail with `gob: type lifecycle.Prefix has no exported fields`.

This meant that in distributed systems listings would never be able to continue and would restart on every call.

Fix issues and be sure to log these errors at least once per bucket. We may see some connectivity errors here, but we shouldn't hide them.
 2023-08-15 07:45:25 -07:00
Harshavardhana
c45bc32d98
skip disks under scanning when healing disks (#17822) 
Bonus:

- avoid calling DiskInfo() calls when missing blocks
  instead heal the object using MRF operation.

- change the max_sleep to 250ms beyond that we will
  not stop healing.
 2023-08-09 12:51:47 -07:00
Praveen raj Mani
0285df5a02
fix: prioritize audit_webhook and logger_webhook ENVs over the config KVS (#17783) 2023-08-03 02:47:07 -07:00
Harshavardhana
81be718674
fix: optimize DiskInfo() call avoid metrics when not needed (#17763) 2023-07-31 15:20:48 -07:00
Harshavardhana
73edd5b8fd
introduce 'mc admin config set alias/ api odirect=on' (#17753) 
change disable_odirect=off -> odirect=on to make it
easier to understand, instead of making it double
negative.
 2023-07-31 00:12:53 -07:00
Harshavardhana
f13cfcb83e
allow disabling O_DIRECT for write ops (#17751) 
on really slow systems, O_DIRECT simply kills the drives
allow for a way to disable them.
 2023-07-29 15:17:56 -07:00
Anis Eleuch
9c0e8cd15b
logger: Avoid slow calls in http logger Send() function (#17747) 
Send() is synchronous and can affect the latency of S3 requests when the
logger buffer is full.

Avoid checking if the HTTP target is online or not and increase the
workers anyway since the buffer is already full.

Also, avoid logs flooding when the audit target is down.
 2023-07-29 12:49:18 -07:00
Harshavardhana
731e03fe5a
add ReadFileStream deadline for disk call (#17745) 
timeout the reader side if hung via disk max timeout
 2023-07-28 15:37:53 -07:00
jiuker
f9d029c8fa
fix: prevCertificate save not correct in loop (#17744) 
fix certs save not correct

Co-authored-by: guozhi.li <guozhi.li@daocloud.io>
 2023-07-28 12:57:49 -07:00
Harshavardhana
114fab4c70
export cluster health as prometheus metrics (#17741) 2023-07-28 01:16:53 -07:00
Harshavardhana
e7b60c4d65
Add slow drive timeouts to match with active disk monitoring (#17701) 
allow active disk-monitoring to be configurable, and use
these add deadlines in various call layers for various
syscalls.
 2023-07-25 16:58:31 -07:00
Harshavardhana
14e1ace552
remove serializing WalkDir() across all buckets/prefixes on SSDs (#17707) 
slower drives get knocked off because they are too slow via 
active monitoring, we do not need to block calls arbitrarily.

Serializing adds latencies for already slow calls, remove
it for SSDs/NVMEs

Also, add a selection with context when writing to `out <-`
channel, to avoid any potential blocks.
 2023-07-24 09:30:19 -07:00
Harshavardhana
e12ab486a2
avoid using os.Getenv for internal code, use env.Get() instead (#17688) 2023-07-20 07:52:49 -07:00
drivebyer
9b5c2c386a
fix: return error when requested interface has no stats available (#17666) 2023-07-17 01:14:01 -07:00
jiuker
d118031ed6
fix: when Origin: null is set return back '*' for allow origins (#17651) 2023-07-15 12:15:06 -07:00
Shireesh Anjal
bb63375f1b
Do not consider subnet api key as secret (#17643) 
As it is required by mc and console to communicate with subnet
 2023-07-13 12:24:47 -07:00
jiuker
183428db03
fear: Implement 'mc support top net' (#17598) 2023-07-13 11:41:19 -07:00
Shubhendu
9b9871cfbb
Added endpoint and versions attributes to KMS details (#17350) 
Now it would list details of all KMS instances with additional
attributes `endpoint` and `version`. In the case of k8s-based
deployment the list would consist of a single entry.

Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
 2023-07-12 23:50:38 -07:00
Harshavardhana
2d1cda2061
fix: do not os.Exit(1) while writing goroutines during shutdown (#17640) 
Also shutdown poll add jitter, to verify if the shutdown
sequence can finish before 500ms, this reduces the overall
time taken during "restart" of the service.

Provides speedup for `mc admin service restart` during
active I/O, also ensures that systemd doesn't treat the
returned 'error' as a failure, certain configurations in
systemd can cause it to 'auto-restart' the process by-itself
which can interfere with `mc admin service restart`.

It can be observed how now restarting the service is
much snappier.
 2023-07-12 07:18:30 -07:00
Poorna
fb49aead9b
replication: add validation API (#17520) 
To check if replication is set up properly on a bucket.
 2023-07-10 20:09:20 -07:00
Harshavardhana
f6186965c3
honor DeleteAllVersions in list(), head() calls (#17604) 2023-07-08 15:42:10 -07:00
Harshavardhana
28a01f0320
update missing license header in files (#17603) 2023-07-08 10:42:05 -07:00
Klaus Post
45a717a142
Avoid per request URL parsing (#17593) 
Every request does a `url.Parse(c.url.String())` to clone a URL.

The host will also be static, so we rewrite that on creation.
 2023-07-07 22:07:30 -07:00
Harshavardhana
0bc34952eb
fix: under FanOut API avoid repeated md5sum calculation (#17572) 
md5sum calculation has a high CPU overhead, avoid calculating
it repeatedly for similar fanOut calls.

To fix following CPU profiler result
```
(pprof) top10
Showing nodes accounting for 678.68s, 84.67% of 801.54s total
Dropped 1072 nodes (cum <= 4.01s)
Showing top 10 nodes out of 156
      flat  flat%   sum%        cum   cum%
   332.54s 41.49% 41.49%    332.54s 41.49%  runtime/internal/syscall.Syscall6
   228.39s 28.49% 69.98%    228.39s 28.49%  crypto/md5.block
    48.07s  6.00% 75.98%     48.07s  6.00%  runtime.memmove
    28.91s  3.61% 79.59%     28.91s  3.61%  github.com/minio/highwayhash.updateAVX2
     8.25s  1.03% 80.61%      8.25s  1.03%  runtime.futex
     8.25s  1.03% 81.64%     10.81s  1.35%  runtime.step
     6.99s  0.87% 82.52%     22.35s  2.79%  runtime.pcvalue
     6.67s  0.83% 83.35%     38.90s  4.85%  runtime.mallocgc
     5.77s  0.72% 84.07%     32.61s  4.07%  runtime.gentraceback
     4.84s   0.6% 84.67%     10.49s  1.31%  runtime.lock2
```
 2023-07-05 03:16:05 -07:00
Harshavardhana
e37c4efc6e
fix: upon DNS refresh() failure use previous values (#17561) 
DNS refresh() in-case of MinIO can safely re-use
the previous values on bare-metal setups, since
bare-metal arrangements do not change DNS in any 
manner commonly.

This PR simplifies that, we only ever need DNS caching
on bare-metal setups.

- On containerized setups do not enable DNS
  caching at all, as it may have adverse effects on
  the overall effectiveness of k8s DNS systems.

  k8s DNS systems are dynamic and expect applications
  to avoid managing DNS caching themselves, instead
  provide a cleaner container native caching
  implementations that must be used.

- update IsDocker() detection, including podman runtime

- move to minio/dnscache fork for a simpler package
 2023-07-03 12:30:51 -07:00
Harshavardhana
22f5bc643c
fix: honor older scanner settings only if newer has not changed (#17564) 2023-07-03 12:28:36 -07:00
Harshavardhana
7f782983ca
fix: for FTP server driver allow implicit trust of TLS (#17541) 
fixes #17535
 2023-06-30 08:04:13 -07:00
Aditya Manthramurthy
bde533a9c7
fix: OpenID config initialization (#17544) 
This is due to a regression in the handling of the enable key in OpenID
configuration.
 2023-06-29 23:38:26 -07:00
Harshavardhana
aae6846413
feat: allow expiration of all versions via ILM Expiration action (#17521) 
Following extension allows users to specify immediate purge of
all versions as soon as the latest version of this object has
expired.

```
<LifecycleConfiguration>
    <Rule>
        <ID>ClassADocRule</ID>
        <Filter>
           <Prefix>classA/</Prefix>
        </Filter>
        <Status>Enabled</Status>
        <Expiration>
             <Days>3650</Days>
	     <ExpiredObjectAllVersions>true</ExpiredObjectAllVersions>
        </Expiration>
    </Rule>
    ...
```
 2023-06-28 22:12:28 -07:00
guangwu
87b6fb37d6
chore: pkg imported more than once (#17444) 2023-06-26 09:21:29 -07:00
Aditya Manthramurthy
f3248a4b37
Redact all secrets from config viewing APIs (#17380) 
This change adds a `Secret` property to `HelpKV` to identify secrets
like passwords and auth tokens that should not be revealed by the server
in its configuration fetching APIs. Configuration reporting APIs now do
not return secrets.
 2023-06-23 07:45:27 -07:00
Harshavardhana
74759b05a5
make sure to set relevant config entries correctly (#17485) 
Bonus: also allow skipping keys properly.
 2023-06-22 10:04:02 -07:00
Praveen raj Mani
7c72b25ef0
Add an option to make bucket notifications synchronous (#17406) 
With the current asynchronous behaviour in sending notification events
to the targets, we can't provide guaranteed delivery as the systems
might go for restarts.

For such event-driven use-cases, we can provide an option to enable
synchronous events where the APIs wait until the event is successfully
sent or persisted.

This commit adds 'MINIO_API_SYNC_EVENTS' env which when set to 'on'
will enable sending/persisting events to targets synchronously.
 2023-06-20 17:38:59 -07:00
Aditya Manthramurthy
5a1612fe32
Bump up madmin-go and pkg deps (#17469) 2023-06-19 17:53:08 -07:00
Harshavardhana
22b7c8cd8a
upgrade pkg and dperf to latest packages (#17448) 
- dperf improvements in benchmarking read and write tests
- upgrade mimedb to use latest content-types
 2023-06-17 07:31:36 -07:00
jiuker
0474791cf8
fix: set time format right (#17402) 2023-06-14 07:49:13 -07:00
Harshavardhana
f32efd5429
more compliance related fixes (#17408) 
- lifecycle must return InvalidArgument for rule errors
- do not return `null` versionId in HTTP header
- reject mixed SSE uploads with correct error message
 2023-06-13 13:52:33 -07:00
Anis Eleuch
0f0dcf0c5e
tar: Avoid storing snowball extraction header in extract objects (#17389) 2023-06-12 09:42:06 -07:00
Anis Eleuch
bb24346e04
listen: Only error out if not able to bind any interface (#17353) 2023-06-12 09:09:28 -07:00
Harshavardhana
b829e80ecb
do not disable root for invalid API config values (#17386) 2023-06-08 15:50:06 -07:00
Klaus Post
6e38d0f3ab
Add more bootstrap info in debug mode (#17362) 2023-06-08 08:39:47 -07:00
Harshavardhana
dbd4c2425e
fix: kafka broker pings must not be greater than 1sec (#17376) 2023-06-07 11:47:00 -07:00
Anis Eleuch
eba378e4a1
vrf: Fix testing for loopback coming from the address (#17372) 2023-06-07 09:53:05 -07:00
Harshavardhana
2f9e2147f5
allow quota enforcement to rely on older values (#17351) 
PUT calls cannot afford to have large latency build-ups due
to contentious usage.json, or worse letting them fail with
some unexpected error, this can happen when this file is
concurrently being updated via scanner or it is being
healed during a disk replacement heal.

However, these are fairly quick in theory, stressed clusters
can quickly show visible latency this can add up leading to
invalid errors returned during PUT.

It is perhaps okay for us to relax this error return requirement
instead, make sure that we log that we are proceeding to take in
the requests while the quota is using an older value for the quota
enforcement. These things will reconcile themselves eventually,
via scanner making sure to overwrite the usage.json.

Bonus: make sure that storage-rest-client sets ExpectTimeouts to
be 'true', such that DiskInfo() call with contextTimeout does
not prematurely disconnect the servers leading to a longer
healthCheck, back-off routine. This can easily pile up while also
causing active callers to disconnect, leading to quorum loss.

DiskInfo is actively used in the PUT, Multipart call path for
upgrading parity when disks are down, it in-turn shouldn't cause
more disks to go down.
 2023-06-05 16:56:35 -07:00
jiuker
8030e12ba5
fix: expMovingAvg is too small when startTime is zero (#17346) 2023-06-03 13:41:51 -07:00
jiuker
fb5ce3b87a
record err time when remote node is offline (#17262) 2023-05-30 10:07:26 -07:00
Klaus Post
6fe028b7c5
Revert s3 select simdjson reuse (#17310) 2023-05-30 10:02:22 -07:00
Anis Eleuch
54c5c88fe6
Add number of offline disks in quorum errors (#16822) 2023-05-25 09:39:06 -07:00
jiuker
443250d135
fix: for Target isActive use net.Dial instead (#17251) 2023-05-25 09:24:11 -07:00
jiuker
d749aaab69
fix: ignore existing target status when adding new targets (#17250) 2023-05-24 22:57:37 -07:00
Krishnan Parthasarathi
62df731006
Add updatedAt for GetBucketLifecycleConfig (#17271) 2023-05-24 22:52:39 -07:00
Harshavardhana
d0a0eb9738
support fan-out objects via PostUpload() (#17233) 2023-05-24 22:51:07 -07:00
Klaus Post
5677f73794
Add PostObject Checksum (#17244) 2023-05-23 07:58:33 -07:00
Krishnan Parthasarathi
55a3310446
logger-http: Don't retry after a succesful send (#17266) 2023-05-22 14:53:18 -07:00
Harshavardhana
fc03be7891
simplify bucket metadata lookups for versioning/object locking (#17253) 2023-05-22 12:05:14 -07:00
jiuker
b1b00a5055
fix: Avoid Income globalStats twice upon error (#17263) 2023-05-22 07:42:27 -07:00
jiuker
41fa8fa2d2
fix: increment counter when entry be skipped (#17237) 2023-05-19 08:36:52 -07:00
jiuker
e94e6adf91
fix: return proper error if OIDC Discoverydoc fails to respond (#17242) 2023-05-19 02:13:33 -07:00
Klaus Post
b06d7bf834
fix: leaking connections in JSON SQL with limited return (#17239) 2023-05-18 11:26:46 -07:00
Aditya Manthramurthy
9d96b18df0
Add "name" and "description" params to service acc (#17172) 2023-05-17 17:05:36 -07:00
Praveen raj Mani
85912985b6
Check for only network errors in audit webhook for reachability (#17228) 2023-05-17 11:10:33 -07:00
Klaus Post
aaf1abc993
simplify HardLimitReader by using LimitReader for internal usage (#17218) 2023-05-16 13:14:37 -07:00
Harshavardhana
ef2fc0f99e
fix: reduce using memory and temporary files. (#17206) 2023-05-15 14:08:54 -07:00
Anis Eleuch
684399433b
lock: Retry locking with an increasing random interval (#17200) 2023-05-13 08:42:21 -07:00
Poorna
e07c2ab868
Use hash.NewLimitReader for internal multipart calls (#17191) 2023-05-12 11:19:08 -07:00
Klaus Post
99c4ffa34f
fix: avoid audit log race protection deadlocks (#17168) 2023-05-09 08:11:32 -07:00
Harshavardhana
a7f266c907
allow JWT parsing on large session policy based tokens (#17167) 2023-05-09 00:53:08 -07:00
Praveen raj Mani
57acacd5a7
Support persistent queue store for loggers (#17121) 2023-05-08 21:20:31 -07:00
Klaus Post
76913a9fd5
Signed trailers for signature v4 (#16484) 2023-05-05 19:53:12 -07:00
Harshavardhana
5569acd95c
disallow EC:0 if not set during server startup (#17141) 2023-05-04 14:44:30 -07:00
Alex
6e24dff26a
Added MINIO_BROWSER_LOGIN_ANIMATION env support for WebUI console (#17123) 
Signed-off-by: Benjamin Perez <benjamin@bexsoft.net>
 2023-05-03 15:32:50 -07:00
Harshavardhana
9571b0825e
add configurable VRF interface and user-timeout (#17108) 2023-05-03 14:12:25 -07:00
Krishnan Parthasarathi
0ec722bc54
Add tags to NewerNoncurrentVersions audit event (#17110) 2023-05-02 12:56:33 -07:00
Anis Eleuch
4640b13c66
Use expontential backoff algo for internode reconnections (#17052) 2023-05-02 12:35:52 -07:00
Praveen raj Mani
1704abaf6b
fix: store notification events immediately for persistent queues (#17112) 2023-05-02 07:53:13 -07:00
Klaus Post
e8c0a50862
optimization use small blocks up to 64KB (#17107) 2023-05-01 09:47:49 -07:00
Harshavardhana
7ae69accc0
allow root user to be disabled via config settings (#17089) 2023-04-28 12:24:14 -07:00
Klaus Post
7fad0c8b41
Remove checksums from HTTP range request, add part checksums (#17105) 2023-04-28 08:26:32 -07:00
jiuker
6e27264c6b
update cleanupRoutine comment (#17102) 2023-04-28 01:11:51 -07:00
Anis Eleuch
5c83c9724f
audit: Add request path and host to audit event (#17099) 2023-04-27 22:18:24 -07:00
jiuker
b28d391a22
fix: add correct worker count before startHTTPLogger() (#17091) 2023-04-27 10:51:16 -07:00
jiuker
c8b92f6067
protect wg.Done from being called twice (#17075) 2023-04-27 07:55:36 -07:00
Anis Eleuch
31b5acc245
tcp: Increase user timeout to 10 minutes (#17087) 2023-04-26 17:48:31 -07:00
Anis Eleuch
0b7ca094e4
Remove Expect 100-continue in internode communications (#17061) 2023-04-26 09:33:45 -07:00
Praveen raj Mani
72802a5972
Use 'minio/pkg/sync/errgroup' and 'minio/pkg/workers' (#17069) 2023-04-25 22:57:40 -07:00
Harshavardhana
8fd07bcd51
simplify sort.Sort by using sort.Slice (#17066) 2023-04-24 13:28:18 -07:00
Harshavardhana
8a9b9832fd
add Dial timeout for Kafka broker pings (#17044) 2023-04-17 15:45:01 -07:00
jiuker
e96c88e914
fix: DeleteBucketThrottle must delete ARN (#17034) 2023-04-15 02:14:26 -07:00
Klaus Post
c133979b8e
Add part count to checksum (#17035) 2023-04-14 09:44:45 -07:00
Harshavardhana
a5835cecbf
fix: regression in counting total requests (#17024) 2023-04-12 14:37:19 -07:00
Poorna
d350654aee
config: fix duplication of replication priority key (#17014) 2023-04-11 19:22:10 -07:00
Harshavardhana
09a25ea7b7 lint: fix some lint issues on files 2023-04-06 22:42:10 -07:00
Harshavardhana
47b7469a60
add buffer pool for proxy forwarder (#16942) 2023-04-06 15:54:12 -07:00
Praveen raj Mani
51f7f9aaa3
Generalize the event store using go generics (#16910) 2023-04-04 10:52:24 -07:00
Anis Eleuch
d90d0c8931
Use one http response recorder per external http call (#16938) 2023-03-31 09:37:29 -07:00
Anis Eleuch
c259a8ea38
Set tcp user timeout to clean sockets with data in the buffer (#16887) 2023-03-24 08:10:58 -07:00
mstmdev
2d51e42305
Remove the redundant conditional in the validateParity function (#16866) 2023-03-23 14:06:22 -07:00
Anis Eleuch
1346561b9d
return quorum error instead of insufficient storage error (#16874) 2023-03-22 16:22:37 -07:00
Klaus Post
11d04279c8
Add lazy init of audit logger (#16842) 2023-03-21 10:50:40 -07:00
Harshavardhana
3b5dbf9046
allow bootstrapping to validate internode tokens (#16853) 2023-03-20 01:40:24 -07:00
Aditya Manthramurthy
09c733677a
Add test for fixed post policy exploit (#16855) 2023-03-20 01:06:45 -07:00
Harshavardhana
46f9049fb4
simplify error responses for KMS (#16793) 2023-03-16 11:59:42 -07:00
Aditya Manthramurthy
58266c9e2c
Add enable flag for LDAP IDP config (#16805) 2023-03-16 11:58:59 -07:00
Nitish Tiwari
50dbd2cacc
Update audit log flow to use new headers with unit (#16797) 2023-03-13 22:50:19 -07:00
Harshavardhana
b984bf8d1a
allow expiration of all versions during Listing() (#16757) 2023-03-09 15:15:30 -08:00
Harshavardhana
901887e6bf
feat: add lambda transformation functions target (#16507) 2023-03-07 08:12:41 -08:00
ferhat elmas
714283fae2
cleanup ignored static analysis (#16767) 2023-03-06 08:56:10 -08:00
ferhat elmas
3423028713
cleanup Go linter settings (#16736) 2023-03-04 20:57:35 -08:00
Harshavardhana
0ff931dc76
fix: allow CORS to work by default (#16713) 2023-02-27 10:10:45 -08:00
Praveen raj Mani
4d708cebe9
Support adding service accounts with expiration (#16430) 
Co-authored-by: Harshavardhana <harsha@minio.io>
 2023-02-27 10:10:22 -08:00
Aditya Manthramurthy
8cde38404d
Add metrics for custom auth plugin (#16701) 2023-02-27 09:55:18 -08:00
Shubhendu
8b4eb2304b
Set logger webhook proxy on subnet proxy change (#16665) 
Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
 2023-02-27 08:35:36 -08:00
Anis Elleuch
8da0f4c5bb
Better error message when TLS certs do not have proper permissions (#16703) 2023-02-24 06:34:55 -08:00
Harshavardhana
b21d3f9b82
event target registration failures must be returned (#16700) 2023-02-23 21:59:14 +05:30
Harshavardhana
5c98223c89
add correct HostId instead of deploymentId for error responses (#16686) 2023-02-22 15:41:09 +05:30
Daniel Valdivia
fb17f97cf3
move audit and logger message structure to minio/pkg (#16655) 
Signed-off-by: Daniel Valdivia <18384552+dvaldivia@users.noreply.github.com>
 2023-02-21 21:21:17 -08:00
Shubhendu
6b65ba1551
Added attribute proxy for mc admin config set ALIAS logger_webhook (#16657) 
Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
 2023-02-21 21:19:46 -08:00
Allan Roger Reid
8bfe972bab
Set meaningful message from minio with env variable KMS_SECRET_KEY (#16584) 2023-02-22 07:13:01 +05:30
Klaus Post
fd6622458b
Add detailed scanner trace output and notifications (#16668) 2023-02-21 09:33:33 -08:00
jiuker
e470268c7c
fix: a possible closer leak in SelectObjectHandler (#16598) 2023-02-17 01:44:40 -08:00
Harshavardhana
6ea150fd68
fix: avoid printing certain errors under few locations (#16631) 2023-02-17 01:40:31 -08:00
Harshavardhana
e1e9ddd4a4
use kes.Status() for Status() call (#16629) 2023-02-16 22:12:24 +05:30
Andreas Auernhammer
74887c7372
kms: add support for KES API keys and switch to KES Go SDK (#16617) 
Signed-off-by: Andreas Auernhammer <hi@aead.dev>
 2023-02-14 07:19:20 -08:00
Harshavardhana
31188e9327
add parallel workers in batch replication (#16609) 2023-02-13 12:07:58 -08:00
Harshavardhana
d65debb6bc
fix: comply with RFC6750 UserInfo endpoint requirements (#16592) 2023-02-10 22:20:25 +05:30
Anis Elleuch
c8ffa59d28
Periodically refresh buckets metadata from the backend disks (#16561) 
fixes #16553
 2023-02-09 10:29:20 -08:00
Anis Elleuch
fadc46b906
Add the access key and parent user in the audit log (#16572) 2023-02-08 11:05:26 -08:00
Anis Elleuch
095b518802
Show a better error msg when internal data encryption key is incorrect (#16549) 2023-02-07 05:22:54 -08:00
Harshavardhana
14cf8f1b22
upgrade deps for minio/pkg v1.6.1 to include groups conditions (#16538) 2023-02-06 09:27:29 -08:00
Harshavardhana
5996c8c4d5
feat: allow offline disks on a fresh start (#16541) 2023-02-06 09:26:09 -08:00
Harshavardhana
aa8b9572b9
remove double ENABLED help output (#16528) 2023-02-03 05:52:52 -08:00
Klaus Post
0b7989aa4b
Fix Kafka initialization crash (#16523) 2023-02-02 19:21:19 +05:30
Harshavardhana
d19cbc81b5
fix: do not return IAM/Bucket metadata replication errors to client (#16486) 2023-01-26 11:11:54 -08:00
Harshavardhana
65c104a589
add x-amz-id-2 to indicate the node that received the request (#16474) 2023-01-25 09:14:10 -08:00
Harshavardhana
3683673fb0
add missing gorilla/mux migration, update credits (#16461) 2023-01-23 08:46:37 -08:00
Poorna
1b02e046c2
Fix bandwidth monitoring to be per remote target (#16360) 2023-01-19 18:52:16 +05:30
Aditya Manthramurthy
698862ec5d
Fix transports/timeouts related regressions (#16427) 2023-01-18 10:06:38 +05:30
Harshavardhana
3db658e51e
use correct xml package for custom MarshalXML() (#16421) 2023-01-17 05:08:33 +05:30
Allan Roger Reid
9815dac48f
fix: allow bind on ipv6 loopback failures (#16388) 2023-01-11 08:47:39 +05:30
Anis Elleuch
1ece3d1dfe
Add comment field to service accounts (#16380) 2023-01-10 21:57:52 +04:00
Anis Elleuch
2146ed4033
xl: Quit early when EC config is incorrect (#16390) 
Co-authored-by: Anis Elleuch <anis@min.io>
 2023-01-09 23:07:45 -08:00
Anis Elleuch
1c85652cff
lint: Fix in darwin environment (#16368) 2023-01-05 10:12:01 -08:00
Klaus Post
6a04067514
fix: tweak read buffer size to reduce over-reading (#16338) 2023-01-01 08:14:20 -08:00
Harshavardhana
75faef888e
disable builds for go1.18 (#16332) 2022-12-30 11:37:07 -08:00
Harshavardhana
b67d97b1ba
add missing fields in audit logs for non-compressed handlers (#16328) 2022-12-30 10:20:19 -08:00
Anis Elleuch
acc9c033ed
debug: Add X-Amz-Request-ID to lock/unlock calls (#16309) 2022-12-23 19:49:07 -08:00
Anis Elleuch
e57e946206
Do not save credentials in config.json (#16275) 2022-12-19 12:27:06 -08:00
Harshavardhana
80fc3a8a52
use newDynamicTimeoutWithOpts() when appropriate (#16266) 2022-12-15 13:11:37 -08:00
Klaus Post
988a2e8fed
Faster startup of large distributed systems with latency (#16259) 2022-12-15 08:31:21 -08:00
Harshavardhana
bad229e16e
fix: support event name s3:Restore:* (#16257) 2022-12-14 05:12:07 -08:00
Harshavardhana
c73ea27ed7
do not log checksum mismatch error, client received the error (#16246) 2022-12-14 01:57:40 -08:00
Aditya Manthramurthy
9e6cc847f8
Add HTTP2 config option for policy plugin (#16225) 2022-12-13 14:28:48 -08:00
Anis Elleuch
939c0100a6
log: Do not interpret verbs in object names in console output (#16233) 2022-12-13 08:27:40 -08:00
Aditya Manthramurthy
2d60bf8c50
Refactor HTTP transports (#16222) 2022-12-12 20:31:21 -08:00
Aditya Manthramurthy
a469e6768d
Add LDAP DNS SRV record lookup support (#16201) 2022-12-12 11:27:27 -08:00
Harshavardhana
2fc182d8e6
fix: iso8601TimeFormat padding issue for certain nanoseconds (#16207) 2022-12-12 10:28:30 -08:00
Aditya Manthramurthy
e06127566d
Add IAM API to attach/detach policies for LDAP (#16182) 2022-12-09 13:08:33 -08:00
jiuker
8edc2faaa9
reuse sha256 in config GetSettings (#16188) 2022-12-08 03:03:24 -08:00
Javier Adriel
04ae9058ed
Populate end_session_endpoint (#16183) 2022-12-06 16:56:37 -08:00
Aditya Manthramurthy
a30cfdd88f
Bump up madmin-go to v2 (#16162) 2022-12-06 13:46:50 -08:00
Anis Elleuch
932d2c3c62
Add X-Amz-Request-Id to internode calls (#16146) 2022-12-06 09:27:26 -08:00
Harshavardhana
419f351df3
avoid logging gzipped body in trace output (#16172) 2022-12-05 13:21:27 -08:00
Klaus Post
a713aee3d5
Run staticcheck on CI (#16170) 2022-12-05 11:18:50 -08:00
Andreas Auernhammer
d882ba2cb4
kms: add support for KES enclaves (#16139) 
Signed-off-by: Andreas Auernhammer <hi@aead.dev>
 2022-12-04 02:34:24 -08:00
Harshavardhana
9b35c72349
fix: a crash in KMS cert reload function (#16158) 2022-12-02 11:19:05 -08:00
Aditya Manthramurthy
87cbd41265
feat: Allow at most one claim based OpenID IDP (#16145) 2022-11-29 15:40:49 -08:00
Harshavardhana
be92cf5959
change dependency from amqp -> amqp091 (RabbitMQ) official (#16142) 2022-11-28 16:05:06 -08:00
Anis Elleuch
1f1dcdce65
move HTTP recorder to an internal library (#16128) 2022-11-28 10:20:27 -08:00
Shireesh Anjal
98a67a3776
Improvements in logger and audit webhooks (#16102) 2022-11-28 08:03:26 -08:00
Andreas Auernhammer
9b1e70e4f9
kms: fix possible deadlock due to nested RLock calls. (#16136) 
Signed-off-by: Andreas Auernhammer <hi@aead.dev>
 2022-11-28 07:31:07 -08:00
Harshavardhana
09d4f8cd0f
avoid serializing decryptKey() every 15mins (#16135) 
if the certs are the same in an environment where the 
cert files are symlinks (e.g Kubernetes), then we resort
to reloading certs every 15mins - we can avoid reload
of the kes client instance. Ensure that the price to pay 
for contending with the lock must happen when necessary.
 2022-11-28 01:14:33 -08:00
Anis Elleuch
97eb7dbf5f
notify: Return detailed err msg when connecting to target fails (#16118) 2022-11-24 07:59:19 -08:00
Harshavardhana
08103870a5
update single drive setup error message (#16098) 2022-11-18 14:47:38 -08:00
Anis Elleuch
993e586855
config: return XMinioConfigNotFound code for non existing config (#16065) 2022-11-18 10:28:14 -08:00
Shireesh Anjal
5246e3be84
Send health diagnostics data as part of callhome (#16006) 2022-11-15 13:53:05 -08:00
Harshavardhana
962d1f1a71
choose default values upon incorrect storage_class value (#16058) 2022-11-12 10:18:21 -08:00
Harshavardhana
6d76db9d6c
improve server startup error when pools are incorrect (#16056) 2022-11-11 19:40:45 -08:00
Ray
66239f30ce
configuring the nats target to reconnect forever (#16050) 2022-11-11 12:42:41 -08:00
jiuker
bf89f79694
save deploymentID to avoid mutating request entry in Audit (#16053) 2022-11-11 12:42:15 -08:00
Klaus Post
5b242f1d11
Add Audit target metrics (#16044) 2022-11-10 10:20:21 -08:00
Krishnan Parthasarathi
6eef9b4a23
lifecycle: simplify Eval and HasActiveRules (#16036) 2022-11-10 07:17:45 -08:00
Krishnan Parthasarathi
7ba281728f
ilm: fix x-amz-expiration header evaluation (#16029) 2022-11-09 04:20:34 -08:00
Klaus Post
bbc312fce6
Add notification queue metrics (#16026) 2022-11-08 16:36:47 -08:00
Harshavardhana
1b0dfb0f58
remove printing map() checksums (#16028) 2022-11-08 13:29:24 -08:00
Aditya Manthramurthy
76d822bf1e
Add LDAP policy entities API (#15908) 2022-11-07 14:35:09 -08:00
Klaus Post
ddeca9f12a
fix: filter rest errors and logs returned (#16019) 2022-11-07 10:38:08 -08:00
Harshavardhana
21251d8c22
initialize streaming events without lazy initialization (#16016) 2022-11-07 08:01:24 -08:00
Harshavardhana
944c62daf4
skip flaky tests on windows OS (#16015) 2022-11-07 00:11:21 -08:00
Harshavardhana
9547b7d0e9
add deadlineConnections on remoteTransport (#16010) 2022-11-05 11:09:21 -07:00
Harshavardhana
76c4ea7682
force all internal MinIO operations to be under UTC (#16009) 2022-11-04 16:44:38 -07:00
Harshavardhana
7be65f66b8
support HS256 series of JWT signature for OpenID connect (#15993) 2022-11-03 16:41:53 -07:00
Anis Elleuch
7721595aa9
config: Deprecated delay/max_wait/scanner and introduce speed (#15941) 2022-11-01 08:04:07 -07:00
Harshavardhana
4892a766a8
do not panic if webhook returns an error (#15970) 2022-10-30 16:45:53 -07:00
Poorna
d765b89a63
improve validation for replication resync API (#15964) 2022-10-28 23:21:33 -07:00
Klaus Post
71954faa3a
mark pubsub type safe via generics (#15961) 2022-10-28 10:55:42 -07:00
Shireesh Anjal
6d22e74d11
mark SUBNET config keys as sensitive info (#15962) 2022-10-28 10:54:44 -07:00
Harshavardhana
ec77d28e62
make subnet subsys dynamic and simplify callhome (#15927) 2022-10-27 00:20:01 -07:00
Harshavardhana
23b329b9df
remove gateway completely (#15929) 2022-10-24 17:44:15 -07:00
Poorna
e4e90b53c1
fix: delete-marker replication check properly (#15923) 2022-10-21 14:45:06 -07:00
Anis Elleuch
ac85c2af76
lifecycle: refactor rules filtering and tagging support (#15914) 2022-10-21 10:46:53 -07:00
Aditya Manthramurthy
2d16e74f38
Add LDAP IDP Configuration APIs (#15840) 2022-10-19 11:00:10 -07:00
Klaus Post
bd3dfad8b9
Add concurrent Snowball extraction + options (#15836) 2022-10-18 13:50:21 -07:00
Aditya Manthramurthy
85fc7cea97
Pass role ARN for OIDC providers to console (#15862) 2022-10-15 12:57:03 -07:00
Harshavardhana
328d660106
support CRC32 Checksums on single drive setup (#15873) 2022-10-15 11:58:47 -07:00
Harshavardhana
59e33b3b21
validate setBucketTarget properly as per BucketExists() call (#15860) 2022-10-13 17:46:49 -07:00
Poorna
0e3c92c027 attempt delete marker replication after object is replicated (#15857) 
Ensure delete marker replication success, especially since the
recent optimizations to heal on HEAD, LIST and GET can force
replication attempts on delete marker before underlying object
version could have synced.
 2022-10-13 17:45:23 -07:00
hellivan
b0b573052a
fix: unrecognized openid config parameters (#15847) 2022-10-12 12:19:44 -07:00
Javier Adriel
2939000342
Add metrics, version and apis handlers (#15839) 2022-10-12 12:08:03 -07:00
Anis Elleuch
afd4279cd8
lock tests: Initialize different DRWMutex for each lock (#15833) 2022-10-10 15:14:32 -07:00
Harshavardhana
3c4ef4338f
marshal retention XML in expected format (#15821) 2022-10-08 13:11:00 -07:00
Aditya Manthramurthy
64cf887b28
use LDAP config from minio/pkg to share with console (#15810) 2022-10-07 22:12:36 -07:00
Krishnan Parthasarathi
6d6a731d6d
Handle overlapping and conflicting ILM rules (#15812) 2022-10-07 14:36:23 -07:00
Poorna
b6b26dba87
fix: GetObjectRetention to parse in ISO8601 time format (#15809) 2022-10-06 13:53:56 -07:00