Commit Graph

9596 Commits

Author SHA1 Message Date
Harshavardhana
8082d1fed6
add bucket level S3 received/sent bytes (#15084)
adds bucket level metrics for bytes received and sent bytes on all S3 API calls.
2022-06-14 15:14:24 -07:00
Harshavardhana
d2a10dbe69
fix: simplify healthcheck code to freeze calls only once (#15082)
- currently subnet health check was freezing and calling
  locks at multiple locations, avoid them.

- throw errors if first attempt itself fails with no results
2022-06-14 11:22:07 -07:00
Anis Elleuch
14645142db
erasure-sd: Evaluate versioning Prefix in multi-delete objects (#15081)
Erasure SD DeleteObjects() is only inheriting bucket versioning status
from the handler layer.

Add the missing versioning prefix evaluation for each object that will
deleted.
2022-06-14 10:05:12 -07:00
Minio Trusted
f34b2ef90b update dashboard Data Usage Growth as time series 2022-06-13 22:05:36 -07:00
George Costea
ce894665a8
examples: support configuration of a session policy file (#15078) 2022-06-13 15:36:58 -07:00
Anis Elleuch
0d00f3a55b
kms: initialize after cli parsing (#15076)
KMS depends on the --certs-dir flag. 

Ensure KMS is initialized after loading the flag.
2022-06-13 13:06:13 -07:00
Minio Trusted
48ff373ff7 fix: 'mc support perf drive' crash fix when read returns < 1s 2022-06-13 11:24:37 -07:00
Anis Elleuch
e9efee0e64
debug: Close object after check (#15077) 2022-06-13 07:21:04 -07:00
Minio Trusted
4b3e7aee0b Update yaml files to latest version RELEASE.2022-06-11T19-55-32Z 2022-06-11 21:04:23 +00:00
Anis Elleuch
dd53b287f2
sts: Avoid printing all STS errors (#15065)
Limit printing STS errors to 

- STS internal error
- STS not initialized
- STS upstream error
2022-06-11 12:55:32 -07:00
Anis Elleuch
21526efe51
Update dperf to 0.4.1 (#15071) 2022-06-11 09:39:50 -07:00
Harshavardhana
7413045f0e
fix: add missing minio_s3_requests_total (#15070)
PR #15052 caused a regression, add the missing metrics back.

Bonus:

- internode information should be only for distributed setups 
- update the dashboard to include 4xx and 5xx error panels.
2022-06-11 00:50:31 -07:00
Harshavardhana
d76c508566
debug: verify diff on latest objects on source and target buckets (#15069) 2022-06-10 16:56:51 -07:00
Minio Trusted
8fb46de5e4 Update yaml files to latest version RELEASE.2022-06-10T16-59-15Z 2022-06-10 20:12:04 +00:00
Harshavardhana
af1944f28d
support reading systemctl config automatically on baremetal setups (#15066)
this allows for customers to use `mc admin service restart`
directly even when performing RPM, DEB upgrades. Upon such 'restart'
after upgrade MinIO will re-read the /etc/default/minio for any
newer environment variables.

As long as `MINIO_CONFIG_ENV_FILE=/etc/default/minio` is set, this
is honored.
2022-06-10 09:59:15 -07:00
Harshavardhana
214ea14f29
fix: for frozen calls return if client disconnects (#15062) 2022-06-09 05:06:47 -07:00
Anis Elleuch
5fb420c703
prometheus: Add S3 4xx and 5xx S3 monitoring (#15052)
Currently minio_s3_requests_errors_total covers 4xx and 
5xx S3 responses which can be confusing when s3 applications 
sent a lot of HEAD requests with obvious 404 responses or 
when the replication is enabled.

Add 
- minio_s3_requests_4xx_errors_total
- minio_s3_requests_5xx_errors_total

to help users monitor 4xx and 5xx HTTP status codes separately.
2022-06-08 11:22:34 -07:00
Harshavardhana
2420f6c000
fix: make metrics endpoint responsive by reducing the chatter (#15055)
peerOnlineCounter was making NxN calls to many peers, this
can be really long and tedious if there are random servers
that are going down.

Instead we should calculate online peers from the point of
view of "self" and return those online and offline appropriately
by performing a healthcheck.
2022-06-08 02:43:13 -07:00
Harshavardhana
b0d7332a0c
healthcheck cluster endpoint should honor write/readQuorum per pool (#15053) 2022-06-07 19:08:21 -07:00
Daniel Valdivia
f71b56a5d0
Bump Console v0.18.1 (#15051)
Signed-off-by: Daniel Valdivia <18384552+dvaldivia@users.noreply.github.com>
2022-06-07 12:19:38 -07:00
Harshavardhana
d55efc791f
relax O_DIRECT in single drive mode if unsupported (#15045) 2022-06-07 06:44:01 -07:00
Minio Trusted
f63645546d update minimum goroutine threshold on dashboard 2022-06-06 22:13:54 -07:00
Kaan Kabalak
e2dd3e3587
Include the entirety of vendor folder in .gitignore (#15046)
The 'go mod vendor' command generates a directory called 
'vendor' in the main module's root directory, which includes 
the required packages to support builds. Therefore, we can 
include the 'vendor' directory in .gitignore completely, 
regardless of any file extension.
2022-06-06 20:47:51 -07:00
Minio Trusted
27ab780317 Update yaml files to latest version RELEASE.2022-06-07T00-33-41Z 2022-06-07 01:06:59 +00:00
Minio Trusted
e2d4d097e7 do not print errors upon 'nil' err 2022-06-06 17:33:41 -07:00
Minio Trusted
ac8cb6ba0d Update yaml files to latest version RELEASE.2022-06-06T23-14-52Z 2022-06-06 23:47:31 +00:00
Shireesh Anjal
4ce81fd07f
Add periodic callhome functionality (#14918)
* Add periodic callhome functionality

Periodically (every 24hrs by default), fetch callhome information and
upload it to SUBNET.

New config keys under the `callhome` subsystem:

enable - Set to `on` for enabling callhome. Default `off`
frequency - Interval between callhome cycles. Default `24h`

* Improvements based on review comments

- Update `enableCallhome` safely
- Rename pctx to ctx
- Block during execution of callhome
- Store parsed proxy URL in global subnet config
- Store callhome URL(s) in constants
- Use existing global transport
- Pass auth token to subnetPostReq
- Use `config.EnableOn` instead of `"on"`

* Use atomic package instead of lock

* Use uber atomic package

* Use `Cancel` instead of `cancel`

Co-authored-by: Harshavardhana <harsha@minio.io>

Co-authored-by: Harshavardhana <harsha@minio.io>
Co-authored-by: Aditya Manthramurthy <donatello@users.noreply.github.com>
2022-06-06 16:14:52 -07:00
Harshavardhana
df9eeb7f8f
fix: do not log concurrently when multiple disks return errors (#15044)
since the values inside 'context' are mutated internally by
logger, make sure to log serially upon errors not concurrently.
2022-06-06 15:15:11 -07:00
Harshavardhana
31c4fdbf79
fix: resyncing 'null' version on pre-existing content (#15043)
PR #15041 fixed replicating 'null' version however
due to a regression from #14994 caused the target
versions for these 'null' versioned objects to have
different 'versions', this may cause confusion with
bi-directional replication and cause double replication.

This PR fixes this properly by making sure we replicate
the correct versions on the objects.
2022-06-06 15:14:56 -07:00
Harshavardhana
48e367ff7d
reject resync start on misconfigured replication rules (#15041)
we expect resync to start on buckets with replication
rule ExistingObjects enabled, if not we reject such
calls.
2022-06-06 02:54:39 -07:00
Anis Elleuch
fd02492cb7
avoid limits on the number of parallel trace/bucket notifications listeners (#14799)
Simplifies overall limits on the incoming listeners for notifications.

Fixes #14566
2022-06-05 14:29:12 -07:00
Aditya Manthramurthy
addfa35d93
Add FIPS build to CI and add README.fips.md (#15038) 2022-06-04 18:25:37 -07:00
Harshavardhana
5afdc56796
allow single drive mode to run on root disk (#15037)
for practical reasons, allow root disk based installs for single drive mode.
2022-06-03 12:53:42 -07:00
Harshavardhana
fb1c333a83 update latest dperf v0.4.0 2022-06-03 11:13:20 -07:00
Harshavardhana
c3e1da8e04
honor canceled context and do not leak on mergeChannels (#15034)
mergeEntryChannels has the potential to perpetually
wait on the results channel, context might be closed
and we did not honor the caller context canceling.
2022-06-03 05:59:02 -07:00
Anis Elleuch
20a753e2e5
Fix a possible service freeze after perf object (#15036)
The S3 service can be frozen indefinitely if a client or mc asks for object
perf API but quits early or has some networking issues. The reason is
that partialWrite() can block indefinitely.

This commit makes partialWrite() listens to context cancellation as well. It
also renames deadlinedCtx to healthCtx since it covers handler context
cancellation and not only not only the speedtest deadline.
2022-06-03 05:58:45 -07:00
Minio Trusted
3a398775fb Update yaml files to latest version RELEASE.2022-06-03T01-40-53Z 2022-06-03 02:36:56 +00:00
Aditya Manthramurthy
61a7434379
Update --version option behavior (#15032)
- Add git commit ID
- Add go version
2022-06-02 18:40:53 -07:00
Aditya Manthramurthy
09f5e29327
Bump up console to v0.18.0 (#15031) 2022-06-02 17:34:37 -07:00
Poorna
29edb4ccfe
fix: site replication bucket heal to not panic if replication config is missing (#15025) 2022-06-02 12:34:03 -07:00
Minio Trusted
197d6fb644 Update yaml files to latest version RELEASE.2022-06-02T16-16-26Z 2022-06-02 17:46:32 +00:00
Anis Elleuch
d4e565e595
Add defensive check for one stream message size (#15029)
In a streaming response, the client knows the size of a streamed
message but never checks the message size. Add the check to error 
out if the response message is truncated.
2022-06-02 09:16:26 -07:00
Minio Trusted
1fce2b180f Update yaml files to latest version RELEASE.2022-06-02T02-11-04Z 2022-06-02 02:42:14 +00:00
Aditya Manthramurthy
be6ccd129d
fix: typo in FIPS sha256 (#15024) 2022-06-01 19:11:04 -07:00
Klaus Post
f7cecf0945
Make isIndexedMetaV2 return errors (#15012)
Indexed streams would be decoded by the legacy loader if there 
was an error loading it. Return an error when the stream is indexed 
and it cannot be loaded.

Fixes "unknown minor metadata version" on corrupted xl.meta files and 
returns an actual error.
2022-05-31 19:06:57 -07:00
Harshavardhana
7b2198f7e5
handle IPv6 sourceIPs properly (#15005) 2022-05-31 06:04:12 -07:00
Harshavardhana
52221db7ef
fix: for unexpected errors in reading versioning config panic (#14994)
We need to make sure if we cannot read bucket metadata
for some reason, and bucket metadata is not missing and
returning corrupted information we should panic such
handlers to disallow I/O to protect the overall state
on the system.

In-case of such corruption we have a mechanism now
to force recreate the metadata on the bucket, using
`x-minio-force-create` header with `PUT /bucket` API
call.

Additionally fix the versioning config updated state
to be set properly for the site replication healing
to trigger correctly.
2022-05-31 02:57:57 -07:00
Harshavardhana
befbf48563 fix: s3-check-md5 to not panic for incomplete md5 2022-05-30 20:58:42 -07:00
Anis Elleuch
56a61bab56
test: Add GetObjectNInfo test with some outdated disks (#15004)
Add a test reading an object which has some old data in some outdated
disks, in a versioned and non-versioned bucket.
2022-05-30 17:52:59 -07:00
Harshavardhana
d480022711
fix: invalidate outdated disks appropriately during readAllXL (#15002)
readAllXL would return inlined data for outdated disks
causing "read" to return incorrect content to the client,

this PR fixes this behavior by making sure we skip such
outdated disks appropriately based on the latest ModTime
on the disk.
2022-05-30 12:43:54 -07:00