Commit Graph

9612 Commits

Author SHA1 Message Date
Harshavardhana
e83e947ca3
debug/s3-verify: simplify the tool to use lower memory footprint (#15110) 2022-06-20 10:45:35 -07:00
Anis Elleuch
73733a8fb9
heal: Report correctly in multip-pools setup (#15117)
`mc admin heal -r <alias>` in a multi setup pools returns incorrectly
grey objects. The reason is that erasure-server-pools.HealObject() runs
HealObject in all pools and returns the result of the first nil
error. However, in the lower erasureObject level, HealObject() returns
nil if an object does not exist + missing error in each disk of the object
in that pool, therefore confusing mc.

Make erasureObject.HealObject() to return not found error in the lower
level, so at least erasureServerPools will know what pools to ignore.
2022-06-20 08:07:45 -07:00
daniel-bogusz95
ce6c23a360 docs: some grammatical, typo fixes
includes #15104, #15105, #15106, #15107
2022-06-19 15:35:51 -07:00
Daniel Valdivia
99d8e6a30f
Update Console to v0.19.0 (#15109)
Signed-off-by: Daniel Valdivia <18384552+dvaldivia@users.noreply.github.com>
2022-06-18 18:02:17 -07:00
Poorna
2fa1d8ac48
Add import/export APIs to migrate bucket metadata (#14929) 2022-06-18 06:55:39 -07:00
Minio Trusted
ca7e425ce8 update minio-go to v7.0.29
fixes a client GetObject() leak when the caller
has canceled the context.
2022-06-17 22:15:43 -07:00
Poorna
8b9a19eef1
fix: typo in site replication version healing (#15103) 2022-06-17 16:43:24 -07:00
Aditya Manthramurthy
7f629df4d5
Add generic function to retrieve config value with metadata (#15083)
`config.ResolveConfigParam` returns the value of a configuration for any
subsystem based on checking env, config store, and default value. Also returns info
about which config source returned the value.

This is useful to return info about config params overridden via env in the user
APIs. Currently implemented only for OpenID subsystem, but will be extended for
others subsequently.
2022-06-17 11:39:21 -07:00
Anis Elleuch
98ddc3596c
Avoid CompleteMultipart freeze with unexpected network issue (#15102)
If sending a white space during a long S3 handler call fails,
the whitespace goroutine forgets to return a result to the caller.
Therefore, the complete multipart handler will be blocked.

Remember to send the header written result to the caller 
or/and close the channel.
2022-06-17 10:41:25 -07:00
Harshavardhana
5d23be6242
fix: ignore printing io.EOF during WalkDir() on concurrently modified objects (#15100)
fix: ignore print io.EOF during WalkDir() on concurrently modified objects
2022-06-17 08:23:47 -07:00
Daniel Jakots
d15d3a524b
Update gopsutil to v3.22.5 (#15098) 2022-06-16 22:01:39 -07:00
Minio Trusted
1e1d9acb1b Update yaml files to latest version RELEASE.2022-06-17T02-00-35Z 2022-06-17 02:56:57 +00:00
Poorna
55ee94bed0
initialize site replication subsys after loading metadata (#15099) 2022-06-16 19:00:35 -07:00
Harshavardhana
d228d29944
update '-v' flag behavior to include copyRight and license (#15097)
```
~ minio -v
minio version DEVELOPMENT.2022-06-16T20-40-14Z (commit-id=e083228e2a06bfdcd006fee28d449cd2b47c542a)
Runtime: go1.18.3 linux/amd64
Copyright (c) 2015-2022 MinIO, Inc.
Licence AGPLv3 <https://www.gnu.org/licenses/agpl-3.0.html>
```
2022-06-16 16:10:48 -07:00
Harshavardhana
013cc66d8e
add dataErrs for healing debug log (#15092) 2022-06-16 09:42:45 -07:00
Harshavardhana
c7ed6eee5e
fix: background local test also via channel (#15086)
current implementation for `standalone` setups
was blocking the `perf drive`.

Bonus: remove all old unused complicated code.
2022-06-15 14:51:42 -07:00
Harshavardhana
8082d1fed6
add bucket level S3 received/sent bytes (#15084)
adds bucket level metrics for bytes received and sent bytes on all S3 API calls.
2022-06-14 15:14:24 -07:00
Harshavardhana
d2a10dbe69
fix: simplify healthcheck code to freeze calls only once (#15082)
- currently subnet health check was freezing and calling
  locks at multiple locations, avoid them.

- throw errors if first attempt itself fails with no results
2022-06-14 11:22:07 -07:00
Anis Elleuch
14645142db
erasure-sd: Evaluate versioning Prefix in multi-delete objects (#15081)
Erasure SD DeleteObjects() is only inheriting bucket versioning status
from the handler layer.

Add the missing versioning prefix evaluation for each object that will
deleted.
2022-06-14 10:05:12 -07:00
Minio Trusted
f34b2ef90b update dashboard Data Usage Growth as time series 2022-06-13 22:05:36 -07:00
George Costea
ce894665a8
examples: support configuration of a session policy file (#15078) 2022-06-13 15:36:58 -07:00
Anis Elleuch
0d00f3a55b
kms: initialize after cli parsing (#15076)
KMS depends on the --certs-dir flag. 

Ensure KMS is initialized after loading the flag.
2022-06-13 13:06:13 -07:00
Minio Trusted
48ff373ff7 fix: 'mc support perf drive' crash fix when read returns < 1s 2022-06-13 11:24:37 -07:00
Anis Elleuch
e9efee0e64
debug: Close object after check (#15077) 2022-06-13 07:21:04 -07:00
Minio Trusted
4b3e7aee0b Update yaml files to latest version RELEASE.2022-06-11T19-55-32Z 2022-06-11 21:04:23 +00:00
Anis Elleuch
dd53b287f2
sts: Avoid printing all STS errors (#15065)
Limit printing STS errors to 

- STS internal error
- STS not initialized
- STS upstream error
2022-06-11 12:55:32 -07:00
Anis Elleuch
21526efe51
Update dperf to 0.4.1 (#15071) 2022-06-11 09:39:50 -07:00
Harshavardhana
7413045f0e
fix: add missing minio_s3_requests_total (#15070)
PR #15052 caused a regression, add the missing metrics back.

Bonus:

- internode information should be only for distributed setups 
- update the dashboard to include 4xx and 5xx error panels.
2022-06-11 00:50:31 -07:00
Harshavardhana
d76c508566
debug: verify diff on latest objects on source and target buckets (#15069) 2022-06-10 16:56:51 -07:00
Minio Trusted
8fb46de5e4 Update yaml files to latest version RELEASE.2022-06-10T16-59-15Z 2022-06-10 20:12:04 +00:00
Harshavardhana
af1944f28d
support reading systemctl config automatically on baremetal setups (#15066)
this allows for customers to use `mc admin service restart`
directly even when performing RPM, DEB upgrades. Upon such 'restart'
after upgrade MinIO will re-read the /etc/default/minio for any
newer environment variables.

As long as `MINIO_CONFIG_ENV_FILE=/etc/default/minio` is set, this
is honored.
2022-06-10 09:59:15 -07:00
Harshavardhana
214ea14f29
fix: for frozen calls return if client disconnects (#15062) 2022-06-09 05:06:47 -07:00
Anis Elleuch
5fb420c703
prometheus: Add S3 4xx and 5xx S3 monitoring (#15052)
Currently minio_s3_requests_errors_total covers 4xx and 
5xx S3 responses which can be confusing when s3 applications 
sent a lot of HEAD requests with obvious 404 responses or 
when the replication is enabled.

Add 
- minio_s3_requests_4xx_errors_total
- minio_s3_requests_5xx_errors_total

to help users monitor 4xx and 5xx HTTP status codes separately.
2022-06-08 11:22:34 -07:00
Harshavardhana
2420f6c000
fix: make metrics endpoint responsive by reducing the chatter (#15055)
peerOnlineCounter was making NxN calls to many peers, this
can be really long and tedious if there are random servers
that are going down.

Instead we should calculate online peers from the point of
view of "self" and return those online and offline appropriately
by performing a healthcheck.
2022-06-08 02:43:13 -07:00
Harshavardhana
b0d7332a0c
healthcheck cluster endpoint should honor write/readQuorum per pool (#15053) 2022-06-07 19:08:21 -07:00
Daniel Valdivia
f71b56a5d0
Bump Console v0.18.1 (#15051)
Signed-off-by: Daniel Valdivia <18384552+dvaldivia@users.noreply.github.com>
2022-06-07 12:19:38 -07:00
Harshavardhana
d55efc791f
relax O_DIRECT in single drive mode if unsupported (#15045) 2022-06-07 06:44:01 -07:00
Minio Trusted
f63645546d update minimum goroutine threshold on dashboard 2022-06-06 22:13:54 -07:00
Kaan Kabalak
e2dd3e3587
Include the entirety of vendor folder in .gitignore (#15046)
The 'go mod vendor' command generates a directory called 
'vendor' in the main module's root directory, which includes 
the required packages to support builds. Therefore, we can 
include the 'vendor' directory in .gitignore completely, 
regardless of any file extension.
2022-06-06 20:47:51 -07:00
Minio Trusted
27ab780317 Update yaml files to latest version RELEASE.2022-06-07T00-33-41Z 2022-06-07 01:06:59 +00:00
Minio Trusted
e2d4d097e7 do not print errors upon 'nil' err 2022-06-06 17:33:41 -07:00
Minio Trusted
ac8cb6ba0d Update yaml files to latest version RELEASE.2022-06-06T23-14-52Z 2022-06-06 23:47:31 +00:00
Shireesh Anjal
4ce81fd07f
Add periodic callhome functionality (#14918)
* Add periodic callhome functionality

Periodically (every 24hrs by default), fetch callhome information and
upload it to SUBNET.

New config keys under the `callhome` subsystem:

enable - Set to `on` for enabling callhome. Default `off`
frequency - Interval between callhome cycles. Default `24h`

* Improvements based on review comments

- Update `enableCallhome` safely
- Rename pctx to ctx
- Block during execution of callhome
- Store parsed proxy URL in global subnet config
- Store callhome URL(s) in constants
- Use existing global transport
- Pass auth token to subnetPostReq
- Use `config.EnableOn` instead of `"on"`

* Use atomic package instead of lock

* Use uber atomic package

* Use `Cancel` instead of `cancel`

Co-authored-by: Harshavardhana <harsha@minio.io>

Co-authored-by: Harshavardhana <harsha@minio.io>
Co-authored-by: Aditya Manthramurthy <donatello@users.noreply.github.com>
2022-06-06 16:14:52 -07:00
Harshavardhana
df9eeb7f8f
fix: do not log concurrently when multiple disks return errors (#15044)
since the values inside 'context' are mutated internally by
logger, make sure to log serially upon errors not concurrently.
2022-06-06 15:15:11 -07:00
Harshavardhana
31c4fdbf79
fix: resyncing 'null' version on pre-existing content (#15043)
PR #15041 fixed replicating 'null' version however
due to a regression from #14994 caused the target
versions for these 'null' versioned objects to have
different 'versions', this may cause confusion with
bi-directional replication and cause double replication.

This PR fixes this properly by making sure we replicate
the correct versions on the objects.
2022-06-06 15:14:56 -07:00
Harshavardhana
48e367ff7d
reject resync start on misconfigured replication rules (#15041)
we expect resync to start on buckets with replication
rule ExistingObjects enabled, if not we reject such
calls.
2022-06-06 02:54:39 -07:00
Anis Elleuch
fd02492cb7
avoid limits on the number of parallel trace/bucket notifications listeners (#14799)
Simplifies overall limits on the incoming listeners for notifications.

Fixes #14566
2022-06-05 14:29:12 -07:00
Aditya Manthramurthy
addfa35d93
Add FIPS build to CI and add README.fips.md (#15038) 2022-06-04 18:25:37 -07:00
Harshavardhana
5afdc56796
allow single drive mode to run on root disk (#15037)
for practical reasons, allow root disk based installs for single drive mode.
2022-06-03 12:53:42 -07:00
Harshavardhana
fb1c333a83 update latest dperf v0.4.0 2022-06-03 11:13:20 -07:00