Commit Graph

322 Commits

Author SHA1 Message Date
Poorna b6b26dba87
fix: GetObjectRetention to parse in ISO8601 time format (#15809) 2022-10-06 13:53:56 -07:00
Harshavardhana 2a13cc28f2 feat: implement support batch replication (#15554) 2022-10-05 23:00:43 -07:00
Javier Adriel 3307aa1260
Implement KMS handlers (#15737) 2022-10-04 10:05:09 -07:00
Harshavardhana f696a221af
allow tagging policy condition for GetObject (#15777) 2022-10-02 12:29:29 -07:00
Anis Elleuch 6287e8c571
fix: race when accessing REST TCP dial values (#15770) 2022-09-29 09:27:58 -07:00
Anis Elleuch f69a98ce49
fix: loading Audit kafka configuration loading (#15766) 2022-09-29 08:35:08 -07:00
Daryl White d44f3526dc
Update links to documentation site (#15750) 2022-09-28 21:28:45 -07:00
Harshavardhana 41b633f5ea
support tagging based policy conditions (#15763) 2022-09-28 11:25:46 -07:00
Anis Elleuch 86bb48792c
non-blocking initialization of bucket target notifications (#15571) 2022-09-27 17:23:28 -07:00
Harshavardhana 94dbb4a427
fix: generalize SC config and also skip healing sub-sys under SD (#15757) 2022-09-26 09:04:54 -07:00
Anis Elleuch 048a46ec2a
Add RPC tcp timeout/errs and AVG duration to prometheus (#15747) 2022-09-26 09:04:26 -07:00
Harshavardhana 877bd95fa3
remove unused package internal/smart (#15758) 2022-09-25 08:44:53 -07:00
Poorna 8ea6fb368d
Add auto configuration of replication workers (#15636) 2022-09-24 16:20:28 -07:00
Poorna 5fd5ddea23
relax retention date validation on replication target (#15752) 2022-09-23 21:19:03 -07:00
Anis Elleuch 20c89ebbb3
freeze before exit when _MINIO_DEBUG_NO_EXIT is defined (#15709)
this is to ensure keep k8s pods running, when they reach a "crashloop" stage
2022-09-22 11:57:27 -07:00
Krishnan Parthasarathi 6f56ba80b3
lifecycle: Assign unique id to rules with empty id (#15731) 2022-09-22 10:51:54 -07:00
Klaus Post ff12080ff5
Remove deprecated io/ioutil (#15707) 2022-09-19 11:05:16 -07:00
Javier Adriel 0b6175b742
Implement KMS methods for keys, policies and identities (#15673) 2022-09-19 11:04:40 -07:00
Klaus Post c22f3ca7a8
fix: S3 Select CSV -> JSON with variable field count (#15677)
When there are fewer fields than expected, output fewer fields.
2022-09-12 17:00:59 -07:00
Anis Elleuch 4a92134235
prometheus: track errors during REST read/write calls (#15678)
minio_inter_node_traffic_errors_total currently does not track
requests body write/read errors of internode REST communications.

This commit fixes this by wrapping resp.Body.
2022-09-12 12:40:51 -07:00
Poorna 6b9fd256e1
Persist in-memory replication stats to disk (#15594)
to avoid relying on scanner-calculated replication metrics.
This will improve the accuracy of the replication stats reported.

This PR also adds on to #15556 by handing replication
traffic that could not be queued by available workers to the 
MRF queue so that entries in `PENDING` status are healed faster.
2022-09-12 12:40:02 -07:00
Harshavardhana 228c6686f8
allow non-standards fallback for all http.TimeFormats (#15662)
fixes #15645
2022-09-07 07:24:54 -07:00
Krishnan Parthasarathi 5ce1448049
audit: account for response headers separately (#15610) 2022-09-01 12:51:04 -07:00
Klaus Post 8e4a45ec41
fix: encrypt checksums in metadata (#15620) 2022-08-31 08:13:23 -07:00
Abirdcfly d4e0f13bb3
chore: remove duplicate word in comments (#15607)
Signed-off-by: Abirdcfly <fp544037857@gmail.com>

Signed-off-by: Abirdcfly <fp544037857@gmail.com>
2022-08-30 08:26:43 -07:00
Anis Elleuch 1f28a3bb80
Avoid messages from go test output (#15601)
A lot of warning messages are printed in CI/CD failures generated by go
test. Avoid that by requiring at least Error level for logging when
doing go test.
2022-08-30 08:23:40 -07:00
Klaus Post a9f1ad7924
Add extended checksum support (#15433) 2022-08-29 16:57:16 -07:00
Harshavardhana 97376f6e8f
improve performance for inlined data (#15603)
inlined data often is bigger than the allowed
O_DIRECT alignment, so potentially we can write
'xl.meta' without O_DSYNC instead we can rely on
O_DIRECT + fdatasync() instead.

This PR allows O_DIRECT on inlined data that
would gain the benefits of performing O_DIRECT,
eventually performing an fdatasync() at the end.

Performance boost can be observed here for small
objects < 128KiB. The performance boost is mainly
seen on HDD, and marginal on NVMe setups.
2022-08-29 11:19:29 -07:00
Harshavardhana 433b6fa8fe
upgrade golang-lint to the latest (#15600) 2022-08-26 12:52:29 -07:00
Krishnan Parthasarathi 99fbfe2421
Add concurrency to healing objects on a fresh disk (#15575) 2022-08-25 13:07:15 -07:00
Aditya Manthramurthy 18dffb26e7
Allow querying a single target in config get API (#15587) 2022-08-25 00:17:05 -07:00
Aditya Manthramurthy c54c13831a
fix: error types in internal config package (#15559) 2022-08-20 15:11:49 -07:00
Harshavardhana ae4ee95d25
change default lock retry interval to 50ms (#15560)
competing calls on the same object on versioned bucket
mutating calls on the same object may unexpected have
higher delays.

This can be reproduced with a replicated bucket
overwriting the same object writes, deletes repeatedly.

For longer locks like scanner keep the 1sec interval
2022-08-19 16:21:05 -07:00
Aditya Manthramurthy 21831b3fe2
fix: cleanup config KV parsing using madmin helpers (#15552) 2022-08-18 15:55:17 -07:00
Aditya Manthramurthy 701a82642b
Ensure subnet+callhome configs are single sub-sys (#15542) 2022-08-16 17:46:46 -07:00
Harshavardhana 48640b1de2
fix: trim arn:aws:kms from incoming SSE aws-kms-key-id (#15540) 2022-08-16 11:28:30 -07:00
Anis Elleuch 5682685c80
Introduce disk io stats metrics (#15512) 2022-08-16 07:13:49 -07:00
Aditya Manthramurthy ee2afcf70b
Move some config constants to madmin (#15529) 2022-08-13 14:50:16 -07:00
Aditya Manthramurthy 9986e103cf
Fix env var output in config get/export APIs (#15528)
Fix a bug where env vars are not output when the config for the
subsystem is specified solely via env vars.
2022-08-13 10:39:01 -07:00
Shireesh Anjal 316c492842
Upgrade madmin-go to latest version (v1.4.15) (#15510) 2022-08-10 07:36:13 -07:00
Harshavardhana a406bb0288
restrict number of disks used for scanning buckets upto GOMAXPROCS (#15492)
control scanner parallelism to avoid higher CPU
usage on nodes that have more drives but an old CPU.
2022-08-08 16:16:44 -07:00
Harshavardhana 1823ab6808
LDAP/OpenID must be initialized IAM Init() (#15491)
This allows for LDAP/OpenID to be non-blocking,
allowing for unreachable Identity targets to be
initialized in IAM.
2022-08-08 16:16:27 -07:00
Aditya Manthramurthy 3d94c38ec4
Add env variables to configuration APIs output (#15465)
Config export and config get APIs now include environment 
variables set on the server
2022-08-04 22:21:52 -07:00
ebozduman b57e7321e7
Replaces 'disk'=>'drive' visible to end user (#15464) 2022-08-04 16:10:08 -07:00
Andreas Auernhammer d774a3309b
kes: automatically reload KES client certificate (#15450)
This commit adds support for automatically reloading
the MinIO client certificate for authentication to KES.

The client certificate will now be reloaded:
 - when the private key / certificate file changes
 - when a SIGHUP signal is received
 - every 15 minutes

Fixes #14869

Signed-off-by: Andreas Auernhammer <hi@aead.dev>
2022-08-02 16:58:09 -07:00
Harshavardhana 026b87e39b
do not crash for unwrapErrs return `nil` (#15456)
fixes #15454
2022-08-02 15:10:11 -07:00
Harshavardhana fd349103e8
fix: allow P-384/P-512 constant time implementation (#15445)
since go1.18.x P-384/P-512 are now constant time
implementations, enable them.
2022-08-01 09:27:16 -07:00
Anis Elleuch 10b49eb4fb
Fix resetting a config with a non default target name (#15448)
mc admin config reset <alias> notify_webhook:something was not working
properly.

The reason is that GetSubSys() was not calculating the target
name properly because it is quitting early when the number of config
inputs ('notify_webhook:something' in this case) is equal to 1.

This commit will make the code calculates always calculate the target
name if found.
2022-08-01 07:52:23 -07:00
Anis Elleuch 3856d078d2
fix: set 20000 as maximum parallel event calls (#15435)
This is needed to avoid consuming a lot of goroutines when a target is
very slow or there is a bug in a target library.
2022-07-30 12:12:33 -07:00
Harshavardhana 5e763b71dc
use logger.LogOnce to reduce printing disconnection logs (#15408)
fixes #15334

- re-use net/url parsed value for http.Request{}
- remove gosimple, structcheck and unusued due to https://github.com/golangci/golangci-lint/issues/2649
- unwrapErrs upto leafErr to ensure that we store exactly the correct errors
2022-07-27 09:44:59 -07:00