Harshavardhana
91d8bddbd1
use sendfile/splice implementation to perform DMA ( #18411 )
...
sendfile implementation to perform DMA on all platforms
Go stdlib already supports sendfile/splice implementations
for
- Linux
- Windows
- *BSD
- Solaris
Along with this change however O_DIRECT for reads() must be
removed as well since we need to use sendfile() implementation
The main reason to add O_DIRECT for reads was to reduce the
chances of page-cache causing OOMs for MinIO, however it would
seem that avoiding buffer copies from user-space to kernel space
this issue is not a problem anymore.
There is no Go based memory allocation required, and neither
the page-cache is referenced back to MinIO. This page-
cache reference is fully owned by kernel at this point, this
essentially should solve the problem of page-cache build up.
With this now we also support SG - when NIC supports Scatter/Gather
https://en.wikipedia.org/wiki/Gather/scatter_(vector_addressing)
2023-11-10 10:10:14 -08:00
Harshavardhana
754f7a8a39
replace io.Discard usage to fix some NUMA copy() latencies ( #18394 )
...
replace io.Discard usage to fix NUMA copy() latencies
On NUMA systems copying from 8K buffer allocated via
io.Discard leads to large latency build-up for every
```
copy(new8kbuf, largebuf)
```
can in-cur upto 1ms worth of latencies on NUMA systems
due to memory sharding across NUMA nodes.
2023-11-06 14:26:08 -08:00
Adrian Najera
06f59ad631
fix: expiration time for share link when using OpenID ( #18297 )
2023-10-30 10:21:34 -07:00
jiuker
dbc2368a7b
fix: parse the subsys env error ( #18319 )
2023-10-26 08:12:57 -07:00
Klaus Post
74253e1ddc
Fix BackendInfo() race ( #18305 )
...
`GetParityForSC` has a value receiver, so Config is copied before the lock is obtained.
Make it pointer receiver.
Fixes:
```
WARNING: DATA RACE
Read at 0x0000079cdd10 by goroutine 190:
github.com/minio/minio/cmd.(*erasureServerPools).BackendInfo()
github.com/minio/minio/cmd/erasure-server-pool.go:579 +0x6f
github.com/minio/minio/cmd.(*erasureServerPools).LocalStorageInfo()
github.com/minio/minio/cmd/erasure-server-pool.go:614 +0x3c6
github.com/minio/minio/cmd.(*peerRESTServer).LocalStorageInfoHandler()
github.com/minio/minio/cmd/peer-rest-server.go:347 +0x4ea
github.com/minio/minio/cmd.(*peerRESTServer).LocalStorageInfoHandler-fm()
...
WARNING: DATA RACE
Read at 0x0000079cdd10 by goroutine 190:
github.com/minio/minio/cmd.(*erasureServerPools).BackendInfo()
github.com/minio/minio/cmd/erasure-server-pool.go:579 +0x6f
github.com/minio/minio/cmd.(*erasureServerPools).LocalStorageInfo()
github.com/minio/minio/cmd/erasure-server-pool.go:614 +0x3c6
github.com/minio/minio/cmd.(*peerRESTServer).LocalStorageInfoHandler()
github.com/minio/minio/cmd/peer-rest-server.go:347 +0x4ea
github.com/minio/minio/cmd.(*peerRESTServer).LocalStorageInfoHandler-fm()
```
2023-10-24 08:15:41 -07:00
Harshavardhana
6829ae5b13
completely remove drive caching layer from gateway days ( #18217 )
...
This has already been deprecated for close to a year now.
2023-10-11 21:18:17 -07:00
Praveen raj Mani
c27d0583d4
Send kafka notification messages in batches when queue_dir is enabled ( #18164 )
...
Fixes #18124
2023-10-07 08:07:38 -07:00
Harshavardhana
2add57cfed
apply healing per object at 1024 cycles ( #18050 )
...
- we already have MRF for most recent failures
- we trigger healing during HEAD/GET operation
These are enough, also change the default max wait
from 5sec to 1sec for default scanner speed.
2023-09-19 09:24:22 -07:00
Alex
dc48cd841a
Added MINIO_PROMETHEUS_AUTH_TOKEN env support ( #18028 )
...
Signed-off-by: Benjamin Perez <benjamin@bexsoft.net>
2023-09-14 17:28:21 -07:00
Aditya Manthramurthy
cbc0ef459b
Fix policy package import name ( #18031 )
...
We do not need to rename the import of minio/pkg/v2/policy as iampolicy
any more.
2023-09-14 14:50:16 -07:00
Harshavardhana
32890342ce
introduce MINIO_BROWSER_REDIRECT env to enable/disable auto-redirect ( #18025 )
2023-09-13 18:43:57 -07:00
Anis Eleuch
41de53996b
heal: calculate the number of workers based on NRRequests ( #17945 )
2023-09-11 14:48:54 -07:00
Harshavardhana
9878031cfd
fix: change DISK_ to DRIVE_ for some drive related envs ( #18005 )
2023-09-11 12:19:22 -07:00
Harshavardhana
e3fbcaeb72
allow scanner key cycle to be empty ( #18001 )
...
configs from 2020 server throws an
error due to deprecation of the keys
however an attempt is made to parse
them, we should have chosen existing
defaults - this PR fixes that.
2023-09-09 08:53:32 -07:00
Aditya Manthramurthy
1c99fb106c
Update to minio/pkg/v2 ( #17967 )
2023-09-04 12:57:37 -07:00
Daniel Valdivia
328cb0a076
Pass environment variable to control session length to console ( #17885 )
...
Signed-off-by: Daniel Valdivia <18384552+dvaldivia@users.noreply.github.com>
2023-08-21 11:55:43 -07:00
Harshavardhana
8a9b886011
update grafana dashboard with disk -> drive rename ( #17857 )
2023-08-15 16:04:20 -07:00
Harshavardhana
c45bc32d98
skip disks under scanning when healing disks ( #17822 )
...
Bonus:
- avoid calling DiskInfo() calls when missing blocks
instead heal the object using MRF operation.
- change the max_sleep to 250ms beyond that we will
not stop healing.
2023-08-09 12:51:47 -07:00
Harshavardhana
73edd5b8fd
introduce 'mc admin config set alias/ api odirect=on' ( #17753 )
...
change disable_odirect=off -> odirect=on to make it
easier to understand, instead of making it double
negative.
2023-07-31 00:12:53 -07:00
Harshavardhana
f13cfcb83e
allow disabling O_DIRECT for write ops ( #17751 )
...
on really slow systems, O_DIRECT simply kills the drives
allow for a way to disable them.
2023-07-29 15:17:56 -07:00
jiuker
d118031ed6
fix: when Origin: null is set return back '*' for allow origins ( #17651 )
2023-07-15 12:15:06 -07:00
Shireesh Anjal
bb63375f1b
Do not consider subnet api key as secret ( #17643 )
...
As it is required by mc and console to communicate with subnet
2023-07-13 12:24:47 -07:00
Harshavardhana
22f5bc643c
fix: honor older scanner settings only if newer has not changed ( #17564 )
2023-07-03 12:28:36 -07:00
Aditya Manthramurthy
bde533a9c7
fix: OpenID config initialization ( #17544 )
...
This is due to a regression in the handling of the enable key in OpenID
configuration.
2023-06-29 23:38:26 -07:00
guangwu
87b6fb37d6
chore: pkg imported more than once ( #17444 )
2023-06-26 09:21:29 -07:00
Aditya Manthramurthy
f3248a4b37
Redact all secrets from config viewing APIs ( #17380 )
...
This change adds a `Secret` property to `HelpKV` to identify secrets
like passwords and auth tokens that should not be revealed by the server
in its configuration fetching APIs. Configuration reporting APIs now do
not return secrets.
2023-06-23 07:45:27 -07:00
Harshavardhana
74759b05a5
make sure to set relevant config entries correctly ( #17485 )
...
Bonus: also allow skipping keys properly.
2023-06-22 10:04:02 -07:00
Praveen raj Mani
7c72b25ef0
Add an option to make bucket notifications synchronous ( #17406 )
...
With the current asynchronous behaviour in sending notification events
to the targets, we can't provide guaranteed delivery as the systems
might go for restarts.
For such event-driven use-cases, we can provide an option to enable
synchronous events where the APIs wait until the event is successfully
sent or persisted.
This commit adds 'MINIO_API_SYNC_EVENTS' env which when set to 'on'
will enable sending/persisting events to targets synchronously.
2023-06-20 17:38:59 -07:00
Aditya Manthramurthy
5a1612fe32
Bump up madmin-go and pkg deps ( #17469 )
2023-06-19 17:53:08 -07:00
Harshavardhana
b829e80ecb
do not disable root for invalid API config values ( #17386 )
2023-06-08 15:50:06 -07:00
jiuker
d749aaab69
fix: ignore existing target status when adding new targets ( #17250 )
2023-05-24 22:57:37 -07:00
jiuker
e94e6adf91
fix: return proper error if OIDC Discoverydoc fails to respond ( #17242 )
2023-05-19 02:13:33 -07:00
Harshavardhana
5569acd95c
disallow EC:0 if not set during server startup ( #17141 )
2023-05-04 14:44:30 -07:00
Alex
6e24dff26a
Added MINIO_BROWSER_LOGIN_ANIMATION env support for WebUI console ( #17123 )
...
Signed-off-by: Benjamin Perez <benjamin@bexsoft.net>
2023-05-03 15:32:50 -07:00
Harshavardhana
7ae69accc0
allow root user to be disabled via config settings ( #17089 )
2023-04-28 12:24:14 -07:00
Poorna
d350654aee
config: fix duplication of replication priority key ( #17014 )
2023-04-11 19:22:10 -07:00
mstmdev
2d51e42305
Remove the redundant conditional in the validateParity function ( #16866 )
2023-03-23 14:06:22 -07:00
Aditya Manthramurthy
09c733677a
Add test for fixed post policy exploit ( #16855 )
2023-03-20 01:06:45 -07:00
Aditya Manthramurthy
58266c9e2c
Add enable flag for LDAP IDP config ( #16805 )
2023-03-16 11:58:59 -07:00
Harshavardhana
901887e6bf
feat: add lambda transformation functions target ( #16507 )
2023-03-07 08:12:41 -08:00
ferhat elmas
714283fae2
cleanup ignored static analysis ( #16767 )
2023-03-06 08:56:10 -08:00
Harshavardhana
0ff931dc76
fix: allow CORS to work by default ( #16713 )
2023-02-27 10:10:45 -08:00
Aditya Manthramurthy
8cde38404d
Add metrics for custom auth plugin ( #16701 )
2023-02-27 09:55:18 -08:00
Shubhendu
8b4eb2304b
Set logger webhook proxy on subnet proxy change ( #16665 )
...
Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
2023-02-27 08:35:36 -08:00
Anis Elleuch
8da0f4c5bb
Better error message when TLS certs do not have proper permissions ( #16703 )
2023-02-24 06:34:55 -08:00
Allan Roger Reid
8bfe972bab
Set meaningful message from minio with env variable KMS_SECRET_KEY ( #16584 )
2023-02-22 07:13:01 +05:30
Andreas Auernhammer
74887c7372
kms: add support for KES API keys and switch to KES Go SDK ( #16617 )
...
Signed-off-by: Andreas Auernhammer <hi@aead.dev>
2023-02-14 07:19:20 -08:00
Harshavardhana
d65debb6bc
fix: comply with RFC6750 UserInfo endpoint requirements ( #16592 )
2023-02-10 22:20:25 +05:30
Anis Elleuch
c8ffa59d28
Periodically refresh buckets metadata from the backend disks ( #16561 )
...
fixes #16553
2023-02-09 10:29:20 -08:00
Anis Elleuch
095b518802
Show a better error msg when internal data encryption key is incorrect ( #16549 )
2023-02-07 05:22:54 -08:00
Harshavardhana
5996c8c4d5
feat: allow offline disks on a fresh start ( #16541 )
2023-02-06 09:26:09 -08:00
Harshavardhana
aa8b9572b9
remove double ENABLED help output ( #16528 )
2023-02-03 05:52:52 -08:00
Anis Elleuch
2146ed4033
xl: Quit early when EC config is incorrect ( #16390 )
...
Co-authored-by: Anis Elleuch <anis@min.io>
2023-01-09 23:07:45 -08:00
Anis Elleuch
e57e946206
Do not save credentials in config.json ( #16275 )
2022-12-19 12:27:06 -08:00
Aditya Manthramurthy
9e6cc847f8
Add HTTP2 config option for policy plugin ( #16225 )
2022-12-13 14:28:48 -08:00
Aditya Manthramurthy
a469e6768d
Add LDAP DNS SRV record lookup support ( #16201 )
2022-12-12 11:27:27 -08:00
Aditya Manthramurthy
e06127566d
Add IAM API to attach/detach policies for LDAP ( #16182 )
2022-12-09 13:08:33 -08:00
jiuker
8edc2faaa9
reuse sha256 in config GetSettings ( #16188 )
2022-12-08 03:03:24 -08:00
Javier Adriel
04ae9058ed
Populate end_session_endpoint ( #16183 )
2022-12-06 16:56:37 -08:00
Aditya Manthramurthy
a30cfdd88f
Bump up madmin-go to v2 ( #16162 )
2022-12-06 13:46:50 -08:00
Klaus Post
a713aee3d5
Run staticcheck on CI ( #16170 )
2022-12-05 11:18:50 -08:00
Andreas Auernhammer
d882ba2cb4
kms: add support for KES enclaves ( #16139 )
...
Signed-off-by: Andreas Auernhammer <hi@aead.dev>
2022-12-04 02:34:24 -08:00
Aditya Manthramurthy
87cbd41265
feat: Allow at most one claim based OpenID IDP ( #16145 )
2022-11-29 15:40:49 -08:00
Anis Elleuch
97eb7dbf5f
notify: Return detailed err msg when connecting to target fails ( #16118 )
2022-11-24 07:59:19 -08:00
Harshavardhana
08103870a5
update single drive setup error message ( #16098 )
2022-11-18 14:47:38 -08:00
Anis Elleuch
993e586855
config: return XMinioConfigNotFound code for non existing config ( #16065 )
2022-11-18 10:28:14 -08:00
Shireesh Anjal
5246e3be84
Send health diagnostics data as part of callhome ( #16006 )
2022-11-15 13:53:05 -08:00
Harshavardhana
962d1f1a71
choose default values upon incorrect storage_class value ( #16058 )
2022-11-12 10:18:21 -08:00
Harshavardhana
6d76db9d6c
improve server startup error when pools are incorrect ( #16056 )
2022-11-11 19:40:45 -08:00
Aditya Manthramurthy
76d822bf1e
Add LDAP policy entities API ( #15908 )
2022-11-07 14:35:09 -08:00
Harshavardhana
7be65f66b8
support HS256 series of JWT signature for OpenID connect ( #15993 )
2022-11-03 16:41:53 -07:00
Anis Elleuch
7721595aa9
config: Deprecated delay/max_wait/scanner and introduce speed ( #15941 )
2022-11-01 08:04:07 -07:00
Shireesh Anjal
6d22e74d11
mark SUBNET config keys as sensitive info ( #15962 )
2022-10-28 10:54:44 -07:00
Harshavardhana
ec77d28e62
make subnet subsys dynamic and simplify callhome ( #15927 )
2022-10-27 00:20:01 -07:00
Harshavardhana
23b329b9df
remove gateway completely ( #15929 )
2022-10-24 17:44:15 -07:00
Aditya Manthramurthy
2d16e74f38
Add LDAP IDP Configuration APIs ( #15840 )
2022-10-19 11:00:10 -07:00
Aditya Manthramurthy
85fc7cea97
Pass role ARN for OIDC providers to console ( #15862 )
2022-10-15 12:57:03 -07:00
hellivan
b0b573052a
fix: unrecognized openid config parameters ( #15847 )
2022-10-12 12:19:44 -07:00
Aditya Manthramurthy
64cf887b28
use LDAP config from minio/pkg to share with console ( #15810 )
2022-10-07 22:12:36 -07:00
Daryl White
d44f3526dc
Update links to documentation site ( #15750 )
2022-09-28 21:28:45 -07:00
Anis Elleuch
86bb48792c
non-blocking initialization of bucket target notifications ( #15571 )
2022-09-27 17:23:28 -07:00
Harshavardhana
94dbb4a427
fix: generalize SC config and also skip healing sub-sys under SD ( #15757 )
2022-09-26 09:04:54 -07:00
Poorna
8ea6fb368d
Add auto configuration of replication workers ( #15636 )
2022-09-24 16:20:28 -07:00
Klaus Post
ff12080ff5
Remove deprecated io/ioutil ( #15707 )
2022-09-19 11:05:16 -07:00
Abirdcfly
d4e0f13bb3
chore: remove duplicate word in comments ( #15607 )
...
Signed-off-by: Abirdcfly <fp544037857@gmail.com>
Signed-off-by: Abirdcfly <fp544037857@gmail.com>
2022-08-30 08:26:43 -07:00
Harshavardhana
433b6fa8fe
upgrade golang-lint to the latest ( #15600 )
2022-08-26 12:52:29 -07:00
Aditya Manthramurthy
18dffb26e7
Allow querying a single target in config get API ( #15587 )
2022-08-25 00:17:05 -07:00
Aditya Manthramurthy
c54c13831a
fix: error types in internal config package ( #15559 )
2022-08-20 15:11:49 -07:00
Aditya Manthramurthy
21831b3fe2
fix: cleanup config KV parsing using madmin helpers ( #15552 )
2022-08-18 15:55:17 -07:00
Aditya Manthramurthy
701a82642b
Ensure subnet+callhome configs are single sub-sys ( #15542 )
2022-08-16 17:46:46 -07:00
Aditya Manthramurthy
ee2afcf70b
Move some config constants to madmin ( #15529 )
2022-08-13 14:50:16 -07:00
Aditya Manthramurthy
9986e103cf
Fix env var output in config get/export APIs ( #15528 )
...
Fix a bug where env vars are not output when the config for the
subsystem is specified solely via env vars.
2022-08-13 10:39:01 -07:00
Harshavardhana
a406bb0288
restrict number of disks used for scanning buckets upto GOMAXPROCS ( #15492 )
...
control scanner parallelism to avoid higher CPU
usage on nodes that have more drives but an old CPU.
2022-08-08 16:16:44 -07:00
Harshavardhana
1823ab6808
LDAP/OpenID must be initialized IAM Init() ( #15491 )
...
This allows for LDAP/OpenID to be non-blocking,
allowing for unreachable Identity targets to be
initialized in IAM.
2022-08-08 16:16:27 -07:00
Aditya Manthramurthy
3d94c38ec4
Add env variables to configuration APIs output ( #15465 )
...
Config export and config get APIs now include environment
variables set on the server
2022-08-04 22:21:52 -07:00
ebozduman
b57e7321e7
Replaces 'disk'=>'drive' visible to end user ( #15464 )
2022-08-04 16:10:08 -07:00
Harshavardhana
fd349103e8
fix: allow P-384/P-512 constant time implementation ( #15445 )
...
since go1.18.x P-384/P-512 are now constant time
implementations, enable them.
2022-08-01 09:27:16 -07:00
Anis Elleuch
10b49eb4fb
Fix resetting a config with a non default target name ( #15448 )
...
mc admin config reset <alias> notify_webhook:something was not working
properly.
The reason is that GetSubSys() was not calculating the target
name properly because it is quitting early when the number of config
inputs ('notify_webhook:something' in this case) is equal to 1.
This commit will make the code calculates always calculate the target
name if found.
2022-08-01 07:52:23 -07:00
Taran Pelkey
e83930333b
Allow DelKVS to delete specific sub-system fields. ( #15354 )
2022-07-22 14:48:23 -07:00
Harshavardhana
b0d70a0e5e
support additional claim info in Auditing STS calls ( #15381 )
...
Bonus: Adds a missing AuditLog from AssumeRoleWithCertificate API
Fixes #9529
2022-07-22 11:12:03 -07:00
