MyFavMirrors/minio
1
0
Fork 0
mirror of https://github.com/minio/minio.git synced 2024-12-25 22:55:54 -05:00
Code Issues Packages Projects Releases Wiki Activity
10,302 Commits 4 Branches 506 Tags 189 MiB
430669cfad
Commit Graph

532 Commits

Author SHA1 Message Date
Daniel Jakots
d15d3a524b
Update gopsutil to v3.22.5 (#15098) 2022-06-16 22:01:39 -07:00
George Costea
ce894665a8
examples: support configuration of a session policy file (#15078) 2022-06-13 15:36:58 -07:00
Minio Trusted
48ff373ff7 fix: 'mc support perf drive' crash fix when read returns < 1s 2022-06-13 11:24:37 -07:00
Anis Elleuch
21526efe51
Update dperf to 0.4.1 (#15071) 2022-06-11 09:39:50 -07:00
Daniel Valdivia
f71b56a5d0
Bump Console v0.18.1 (#15051) 
Signed-off-by: Daniel Valdivia <18384552+dvaldivia@users.noreply.github.com>
 2022-06-07 12:19:38 -07:00
Harshavardhana
fb1c333a83 update latest dperf v0.4.0 2022-06-03 11:13:20 -07:00
Aditya Manthramurthy
09f5e29327
Bump up console to v0.18.0 (#15031) 2022-06-02 17:34:37 -07:00
Harshavardhana
7b2198f7e5
handle IPv6 sourceIPs properly (#15005) 2022-05-31 06:04:12 -07:00
Minio Trusted
372a08be49 Update minio-go to v7.0.27 2022-05-26 05:48:41 +00:00
Minio Trusted
80fe166902 update vulnerable deps coredns, ldap/v3 2022-05-24 15:53:52 -07:00
Minio Trusted
2a3f5e1ad1 update console release to v0.17.2 2022-05-23 11:45:11 -07:00
Minio Trusted
ee28f6caaa update console v0.17.0 2022-05-19 03:47:09 -07:00
Minio Trusted
74285d50c4 update console v0.16.3 2022-05-11 19:45:51 -07:00
Harshavardhana
def75ffcfe
allow versioning config changes under site replication (#14876) 
PR #14828 introduced prefix-level exclusion of versioning
and replication - however our site replication implementation
since it defaults versioning on all buckets did not allow
changing versioning configuration once the bucket was created.

This PR changes this and ensures that such changes are honored
and also propagated/healed across sites appropriately.
 2022-05-07 18:39:40 -07:00
Shireesh Anjal
3ec1844e4a
return kubernetes info in health report (#14865) 2022-05-06 12:41:07 -07:00
Aditya Manthramurthy
a53dc1d9c8
Update console to v0.16.2 (#14857) 2022-05-03 13:33:22 -07:00
Harshavardhana
c3f689a7d9
JWKS should be parsed before usage (#14842) 
fixes #14811
 2022-04-30 15:23:53 -07:00
Harshavardhana
85f3a9f3b0 Remove Azure gateway implementation (#14418) 
refer #14331
 2022-04-29 12:51:23 -07:00
Aditya Manthramurthy
757eaeae92
Update console to v0.16.0 (#14816) 2022-04-25 18:20:24 -07:00
Minio Trusted
d2a95a04a4 update pkg v1.1.22 2022-04-25 10:33:38 -07:00
Harshavardhana
0cc993f403 Remove GCS, HDFS gateway implementations #14418 
refer #14331
 2022-04-24 10:19:17 -07:00
Poorna
3a64580663
Add support for site replication healing (#14572) 
heal bucket metadata and IAM entries for
sites participating in site replication from
the site with the most updated entry.

Co-authored-by: Harshavardhana <harsha@minio.io>
Co-authored-by: Aditya Manthramurthy <aditya@minio.io>
 2022-04-24 02:36:31 -07:00
Aditya Manthramurthy
4685b76e08
Update dperf v0.3.6 (#14773) 2022-04-19 02:40:36 -07:00
Minio Trusted
78c9109f6c update console to v0.15.14 2022-04-18 17:29:56 -07:00
Harshavardhana
53ca589c11 update deps for minio-go/v7 and jwt/v4 2022-04-15 00:50:22 -07:00
Daniel Valdivia
ca8ff8718e
Update Console v0.15.13 (#14751) 
Signed-off-by: Daniel Valdivia <18384552+dvaldivia@users.noreply.github.com>
 2022-04-14 18:35:00 -07:00
Aditya Manthramurthy
e8e48e4c4a
S3 select switch to new parquet library and reduce locking (#14731) 
- This change switches to a new parquet library
- SelectObjectContent now takes a single lock at the beginning and holds it
during the operation. Previously the operation took a lock every time the
parquet library performed a Seek on the underlying object stream.
- Add basic support for LogicalType annotations for timestamps.
 2022-04-14 06:54:47 -07:00
Shireesh Anjal
5c53620a72
Include speedtest as part of healthinfo api (#14696) 
Execute the object, drive and net speedtests as part of the healthinfo
(if requested by the client), and include their result in the response.

The options for the speedtests have been picked from the default values
used by `mc support perf` command.
 2022-04-12 13:17:44 -07:00
Aditya Manthramurthy
e162a055cc
Bump up console to v0.15.11 (#14734) 2022-04-11 23:55:35 -07:00
Aditya Manthramurthy
8be6d887e2
Bump up dperf to 0.3.5 (#14730) 2022-04-11 15:50:15 -07:00
Krishnan Parthasarathi
1a1b55e133
Add support for minio tier type (#14468) 2022-04-11 13:24:40 -07:00
Daniel Valdivia
e290c010e6
Console v0.15.10 (#14723) 
Signed-off-by: Daniel Valdivia <hola@danielvaldivia.com>
 2022-04-09 20:55:36 -07:00
Harshavardhana
91bfefcf8c move back go.mod to 1.17 2022-04-08 16:25:20 -07:00
Daniel Valdivia
255116fde7
Update Console Dependency to v0.15.9 (#14699) 
Signed-off-by: Daniel Valdivia <hola@danielvaldivia.com>
 2022-04-05 20:46:17 -07:00
Aditya Manthramurthy
892a204013
Update console to v0.15.8 (#14671) 2022-03-31 20:41:39 -07:00
Andreas Auernhammer
4d2fc530d0
add support for SSE-S3 bulk ETag decryption (#14627) 
This commit adds support for bulk ETag
decryption for SSE-S3 encrypted objects.

If KES supports a bulk decryption API, then
MinIO will check whether its policy grants
access to this API. If so, MinIO will use
a bulk API call instead of sending encrypted
ETags serially to KES.

Note that MinIO will not use the KES bulk API
if its client certificate is an admin identity.

MinIO will process object listings in batches.
A batch has a configurable size that can be set
via `MINIO_KMS_KES_BULK_API_BATCH_SIZE=N`.
It defaults to `500`.

This env. variable is experimental and may be
renamed / removed in the future.

Signed-off-by: Andreas Auernhammer <hi@aead.dev>
 2022-03-25 15:01:41 -07:00
Minio Trusted
028bc2f9be update console release to v0.15.6 2022-03-24 19:59:15 -07:00
Harshavardhana
c433572585
update go mod to go1.16 deps (#14614) 2022-03-23 17:43:44 -07:00
Minio Trusted
7f8b8a0e43 update console to v0.15.4 2022-03-20 15:35:20 -07:00
Harshavardhana
0bf80b3c89 update console v0.15.3 2022-03-16 01:19:00 -07:00
Harshavardhana
dda18c28c5 Bump github.com/nats-io/nats-server/v2 from 2.7.2 to 2.7.4 2022-03-11 15:57:45 -08:00
Harshavardhana
5a5e9b8a89 update console to v0.15.2 2022-03-11 03:08:23 -08:00
Harshavardhana
23345098ea change dperf to use standard Go io.Copy 2022-03-10 12:53:39 -08:00
Harshavardhana
decfd6108c update dperf to calculate timing for fdatasync()/close() calls as well 2022-03-09 13:47:44 -08:00
Harshavardhana
21d8298fe1 update console UI to release v0.15.1 2022-03-07 23:40:58 -08:00
Harshavardhana
92a77cc78e
update pkg v1.1.20 to reload certs in k8s always (#14470) 2022-03-04 20:34:39 -08:00
Lenin Alevski
289fcbd08c
KES dependency upgrade (#14454) 
- Updating KES dependency to v.0.18.0
- Fixing incompatibility issue when checking for errors during KES key creation

Signed-off-by: Lenin Alevski <alevsk.8772@gmail.com>
 2022-03-02 23:03:40 -08:00
Harshavardhana
5b5deee5b3 update minio/pkg to v1.1.18 2022-03-02 19:25:07 -08:00
Krishnan Parthasarathi
7dae4cb685
Update minio/pkg to v1.1.17 (#14450) 
Fix for admin policy validation of KMSCreateKey
 2022-03-02 17:06:06 -08:00
Harshavardhana
58f7e3a829 update console v0.15.0, coredns v1.9.0 2022-03-01 17:17:18 -08:00
Harshavardhana
b7c90751b0 allow drive tests to respond only drive paths 2022-02-25 18:54:46 -08:00
Klaus Post
e3f24a29fa
Upgrade simdjson & compress deps (#14411) 2022-02-25 10:48:41 -08:00
Harshavardhana
29b7164468 update console update v0.14.8 2022-02-24 14:12:01 -08:00
Harshavardhana
77210513c9 update minio/pkg, minio/madmin-go, minio/minio-go/v7 2022-02-23 14:34:47 -08:00
Harshavardhana
b29224f62f update console to v0.14.7 2022-02-16 21:32:26 -08:00
Klaus Post
b88c57ba93
Add fgprof profiles (#14321) 
https://github.com/felixge/fgprof#rocket-fgprof---the-full-go-profiler
 2022-02-16 12:00:10 -08:00
Harshavardhana
ae1b495262 update minio-go v7.0.22 2022-02-16 10:42:52 -08:00
Harshavardhana
21a0f857d3 update console to v0.14.6 2022-02-15 16:35:27 -08:00
Harshavardhana
2c0f121550 Bump github.com/nats-io/nats-streaming-server v0.21.2 -> v0.24.1 2022-02-11 15:59:58 -08:00
dependabot[bot]
9b39616c1b
Bump github.com/nats-io/nats-server/v2 from 2.3.2 to 2.7.2 (#14297) 
Bumps [github.com/nats-io/nats-server/v2](https://github.com/nats-io/nats-server) from 2.3.2 to 2.7.2.
- [Release notes](https://github.com/nats-io/nats-server/releases)
- [Changelog](https://github.com/nats-io/nats-server/blob/main/.goreleaser.yml)
- [Commits](https://github.com/nats-io/nats-server/compare/v2.3.2...v2.7.2)

---
updated-dependencies:
- dependency-name: github.com/nats-io/nats-server/v2
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
 2022-02-11 15:55:28 -08:00
Harshavardhana
ff99ef74c8 remove the replace directive for redigo 2022-02-10 21:45:40 -08:00
Harshavardhana
6990e73b11 update console to v0.14.5 2022-02-10 17:43:04 -08:00
Harshavardhana
22b7226581 update console to release v0.14.3 2022-02-04 20:40:59 -08:00
Shireesh Anjal
778cccb15d
Use madmin-go v1.3.1 (#14250) 2022-02-04 11:01:04 -08:00
Harshavardhana
88a93838de update console to latest master to fix terminal hangs 2022-02-03 22:56:49 -08:00
Harshavardhana
0855988427 update console to latest master 2022-02-03 17:36:24 -08:00
Sidhartha Mani
d7df6bc738
add support for speedtest drive (#14182) 2022-02-01 22:38:05 -08:00
Harshavardhana
77b780b8ca update console UI to v0.14.2 2022-02-01 00:06:30 -08:00
Harshavardhana
dbd05d6e82
remove FIFO bucket quota, use ILM expiration instead (#14206) 2022-01-31 11:07:04 -08:00
Poorna
38e3c7a8f7
Added filters for SiteReplicationStatus API to support new UI changes (#14177) 2022-01-28 15:37:55 -08:00
Harshavardhana
9a6e569412 update console to v0.14.1 2022-01-26 19:53:02 -08:00
Krishnan Parthasarathi
d2e5f01542
feat: maintain in-memory tier stats for the last 24hrs (#13782) 2022-01-26 14:33:10 -08:00
Harshavardhana
c8691db2b7 update console version to v0.14.0 2022-01-26 11:45:36 -08:00
Harshavardhana
67b5e0dbe8 update minio/pkg v1.1.15 2022-01-24 22:39:31 -08:00
Harshavardhana
5f36167f1a update console v0.13.3 2022-01-21 23:44:23 -08:00
Poorna
48da4aeee0
Add API for removing site(s) from site replication (#14022) 2022-01-21 08:48:21 -08:00
Harshavardhana
7f214a0e46
use dnscache resolver for resolving command line endpoints (#14135) 
this helps in caching the resolved values early on, avoids
causing further resolution for individual nodes when
object layer comes online.

this can speed up our startup time during, upgrades etc by
an order of magnitude.

additional changes in connectLoadInitFormats() and parallelize
all calls that might be potentially blocking.
 2022-01-20 13:03:15 -08:00
Harshavardhana
9d588319dd
support site replication to replicate IAM users,groups (#14128) 
- Site replication was missing replicating users,
  groups when an empty site was added.

- Add site replication for groups and users when they
  are disabled and enabled.

- Add support for replicating bucket quota config.
 2022-01-19 20:02:24 -08:00
Klaus Post
64d4da5a37
Add Put input readahead (#14084) 
When reading input for PutObject or PutObjectPart add a readahead buffer for big inputs.

This will make network reads+hashing separate run async with erasure coding and writes. This will reduce overall latency in distributed setups where the input is from upstream and writes go to other servers.

We will read at 2 buffers ahead, meaning one will always be ready/waiting and one is currently being read from.

This improves PutObject and PutObjectParts for these cases.
 2022-01-14 10:01:25 -08:00
Harshavardhana
3bd9636a5b
do not remove Sid from svcaccount policies (#14064) 
fixes #13905
 2022-01-10 14:26:26 -08:00
Aditya Manthramurthy
2d31d92271
Fix redigo dep to latest unretracted version (#14043) 
To avoid error message like:

```
go: warning: github.com/gomodule/redigo@v2.0.0+incompatible: retracted by module author: Old development version not maintained or published.
go: to switch to the latest unretracted version, run:
	go get github.com/gomodule/redigo@latest
```
 2022-01-06 16:00:31 -08:00
Aditya Manthramurthy
1981fe2072
Add internal IDP and OIDC users support for site-replication (#14041) 
- This allows site-replication to be configured when using OpenID or the
  internal IDentity Provider.

- Internal IDP IAM users and groups will now be replicated to all members of the
  set of replicated sites.

- When using OpenID as the external identity provider, STS and service accounts
  are replicated.

- Currently this change dis-allows root service accounts from being
  replicated (TODO: discuss security implications).
 2022-01-06 15:52:43 -08:00
Harshavardhana
0d3ae3810f
make sure to comply with MQTT spec (#14037) 
- keep-alive cannot be 0 by default anymore
- client_id cannot be empty

fixes #13993
 2022-01-06 11:25:39 -08:00
Shireesh Anjal
c27110e37d
Add timeinfo to health data (#14013) 
Capture RoundtripDuration to figure out 
NTP issues in subnet health analyzer.
 2022-01-06 01:51:10 -08:00
Minio Trusted
d2b6aa9033 update console to v0.13.2 2022-01-03 23:41:07 -08:00
Minio Trusted
bb97eafa82 madmin-go v1.1.23 and pkg v1.1.11 2021-12-26 23:23:18 -08:00
Minio Trusted
51d5efee1b update go v0.13.1 2021-12-26 22:36:49 -08:00
Harshavardhana
7e3a7d7044
add healing for invalid shards by skipping the blocks (#13978) 
Built on top of #13945, now we need to simply skip the
shards and its automated.
 2021-12-23 23:01:46 -08:00
Aditya Manthramurthy
5a96cbbeaa
Fix user privilege escalation bug (#13976) 
The AddUser() API endpoint was accepting a policy field. 
This API is used to update a user's secret key and account 
status, and allows a regular user to update their own secret key. 

The policy update is also applied though does not appear to 
be used by any existing client-side functionality.

This fix changes the accepted request body type and removes 
the ability to apply policy changes as that is possible via the 
policy set API.

NOTE: Changing passwords can be disabled as a workaround
for this issue by adding an explicit "Deny" rule to disable the API
for users.
 2021-12-23 09:21:21 -08:00
Harshavardhana
691b763613 update console v0.12.9 2021-12-17 20:42:33 -08:00
Shireesh Anjal
13441ad0f8
Add IsKubernetes and IsDocker to health data (#13936) 2021-12-17 14:46:54 -08:00
Harshavardhana
da540ccf8c
update selfupdate dependency to fix signature verification (#13919) 
minisign v0.10.0 tool broke compatibility, that leads
to our library failing to parse the newer signatures.

This PR
fixes - https://github.com/minio/operator/issues/913
fixes - https://github.com/minio/minio/issues/13824

A workaround for users facing this problem is to unset

```
MINIO_UPDATE_MINISIGN_PUBKEY
```

or set it to `empty` string then signature verification
is skipped automatically.
 2021-12-15 15:05:23 -08:00
Harshavardhana
83e8da57b8 update console to release v0.12.8 2021-12-08 22:19:41 -08:00
Harshavardhana
92fdcafb66
add verification tests for ETag on replicated content (#13857) 2021-12-07 10:08:26 -08:00
Harshavardhana
f0fc77fded update CREDITS file with new deps 2021-12-03 13:24:49 -08:00
Klaus Post
f56cac6381
jwt: Parse standard claims faster (#13821) 
* Use structless/allocationless decoding for header (note "typ" isn't used)
* Create custom unmarshal code using jsonparser for StandardClaims.

Before/After:

```
BenchmarkParseJWTStandardClaims-32    	 4270724	       294.0 ns/op	     706 B/op	      16 allocs/op
BenchmarkParseJWTStandardClaims-32    	 5634847	       214.7 ns/op	     544 B/op	       9 allocs/op

BenchmarkParseJWTMapClaims-32    	 2763045	       428.6 ns/op	    1251 B/op	      29 allocs/op
BenchmarkParseJWTMapClaims-32    	 2839455	       410.9 ns/op	    1219 B/op	      26 allocs/op
```
 2021-12-03 13:19:38 -08:00
jiangfucheng
7460fb8349
fix padding error and compatible with uploaded objects (#13803) 2021-12-03 09:26:30 -08:00
Harshavardhana
f7c357ebad update console to v0.12.6 2021-12-02 17:54:29 -08:00
Harshavardhana
99d87c5ca2
fix: totalDrives reported in speedTest for multiple-pools (#13770) 
totalDrives reported in speedTest result were wrong
for multiple pools, this PR fixes this.

Bonus: add support for configurable storage-class, this
allows us to test REDUCED_REDUNDANCY to see further
maximum throughputs across the cluster.
 2021-11-29 09:05:46 -08:00
Aditya Manthramurthy
4c0f48c548
Add role ARN support for OIDC identity provider (#13651) 
- Allows setting a role policy parameter when configuring OIDC provider

- When role policy is set, the server prints a role ARN usable in STS API requests

- The given role policy is applied to STS API requests when the roleARN parameter is provided.

- Service accounts for role policy are also possible and work as expected.
 2021-11-26 19:22:40 -08:00
Harshavardhana
932500e43d update console to v0.12.5 2021-11-24 12:46:53 -08:00
Harshavardhana
91e0823ff0
allow service freeze/unfreeze on a setup (#13707) 
an active running speedTest will reject all
new S3 requests to the server, until speedTest
is complete.

this is to ensure that speedTest results are
accurate and trusted.

Co-authored-by: Klaus Post <klauspost@gmail.com>
 2021-11-23 12:02:16 -08:00
Klaus Post
142c6b11b3
Reduce JWT overhead for internode tokens (#13738) 
Since JWT tokens remain valid for up to 15 minutes, we 
don't have to regenerate tokens for every call.

Cache tokens for matching access+secret+audience 
for up to 15 seconds.

```
BenchmarkAuthenticateNode/uncached-32         	  270567	      4179 ns/op	    2961 B/op	      33 allocs/op
BenchmarkAuthenticateNode/cached-32           	 7684824	       157.5 ns/op	      48 B/op	       1 allocs/op
```

Reduces internode call allocations a great deal.
 2021-11-23 09:51:53 -08:00
Harshavardhana
1e72e9b1cd update console to v0.12.4 2021-11-20 09:49:14 -08:00
Harshavardhana
556ae07857
simplify the reader for speedtest (#13682) 
additionally count only success operations,
truncated incomplete calls don't need to be
counted.
 2021-11-19 10:41:37 -08:00
Klaus Post
faf013ec84
Improve performance on multiple versions (#13573) 
Existing:

```go
type xlMetaV2 struct {
    Versions []xlMetaV2Version `json:"Versions" msg:"Versions"`
}
```

Serialized as regular MessagePack.

```go
//msgp:tuple xlMetaV2VersionHeader
type xlMetaV2VersionHeader struct {
	VersionID [16]byte
	ModTime   int64
	Type      VersionType
	Flags     xlFlags
}
```

Serialize as streaming MessagePack, format:

```
int(headerVersion)
int(xlmetaVersion)
int(nVersions)
for each version {
    binary blob, xlMetaV2VersionHeader, serialized
    binary blob, xlMetaV2Version, serialized.
}
```

xlMetaV2VersionHeader is <= 30 bytes serialized. Deserialized struct 
can easily be reused and does not contain pointers, so efficient as a 
slice (single allocation)

This allows quickly parsing everything as slices of bytes (no copy).

Versions are always *saved* sorted by modTime, newest *first*. 
No more need to sort on load.

* Allows checking if a version exists.
* Allows reading single version without unmarshal all.
* Allows reading latest version of type without unmarshal all.
* Allows reading latest version without unmarshal of all.
* Allows checking if the latest is deleteMarker by reading first entry.
* Allows adding/updating/deleting a version with only header deserialization.
* Reduces allocations on conversion to FileInfo(s).
 2021-11-18 12:15:22 -08:00
Shireesh Anjal
7152915318
Use pointer based TLS field (#13659) 
This will help other projects like `health-analyzer` to verify that the
struct was indeed populated by the minio server, and is not
default-populated during unmarshalling of the JSON.

Signed-off-by: Shireesh Anjal <shireesh@minio.io>
 2021-11-18 09:02:33 -08:00
Harshavardhana
8378bc9958
support dynamic redirect_uri based on incoming 'host' header (#13666) 
This feature is useful in situations when console is exposed
over multiple intranent or internet entities when users are
connecting over local IP v/s going through load balancer.

Related console work was merged here

373bfbfe3f
 2021-11-16 18:40:39 -08:00
Harshavardhana
5a36179c19 update go.mod to v0.12.3 2021-11-13 09:05:28 -08:00
Aditya Manthramurthy
e8c6314770
IAM: init IAM with Init() rather than InitStore() in tests (#13643) 
- rename InitStore() to initStore() and fix tests

- Use IAMSys.Lock() only when IAMSys struct is being mutated
 2021-11-11 21:03:02 -08:00
Aditya Manthramurthy
087c1b98dc
Add tests for OpenID STS creds and add to CI (#13638) 2021-11-11 11:23:30 -08:00
Harshavardhana
f4b55ea7a7 update console to v0.12.2 2021-11-08 19:21:45 -08:00
Harshavardhana
b1754fc5ff update go.mod and CREDITS 2021-11-06 11:39:17 -07:00
Harshavardhana
520037e721
move to jwt-go v4 with correct releases (#13586) 2021-11-05 12:20:08 -07:00
Harshavardhana
df9f479d58 update console to v0.12.1 2021-11-04 16:43:59 -07:00
Aditya Manthramurthy
ecd54b4cba
Move all IAM storage functionality into iam store type (#13567) 
This reverts commit 091a7ae359.

- Ensure all actions accessing storage lock properly.

- Behavior change: policies can be deleted only when they
  are not associated with any active credentials.

Also adds fix for accidental canned policy removal that was present in the
reverted version of the change.
 2021-11-03 19:47:49 -07:00
Harshavardhana
ca2b288a4b update console to v0.12.0 2021-11-03 16:23:45 -07:00
Harshavardhana
1016fbb8f9
feat: detect starting from windows explorer (#13570) 
Windows users often click on the binary without
knowing MinIO is a command-line tool and should be
run from a terminal. Throw a message to guide them
on what to do.

Co-authored-by: Klaus Post <klauspost@gmail.com>
 2021-11-03 14:22:13 -07:00
Krishna Srinivas
58934e5881
Support live updates for clients during speedtest (#13566) 2021-11-02 15:27:03 -07:00
Harshavardhana
bb639d9f29
remove double reads delete versions (#13544) 
deleting collection of versions belonging
to same object, we can avoid re-reading
the xl.meta from the disk instead purge
all the requested versions in-memory,

the tradeoff is to allocate a map to de-dup
the versions, allow disks to be read only
once per object.

additionally reduce the data transfer between
nodes by shortening msgp data values.
 2021-11-01 10:50:07 -07:00
Harshavardhana
bd88b86919 update console to latest to fix CVE-2021-42836 2021-10-27 21:14:02 -07:00
Krishnan Parthasarathi
939fbb3c38
ilm: Make per-tier stats available via admin-tier-info (#13381) 2021-10-23 18:38:33 -07:00
Harshavardhana
9694fa8d3a update console to v0.11.0 release 2021-10-22 20:28:24 -07:00
Shireesh Anjal
ce40392803
Capture TLS info in health report (#13470) 
So that TLS related checks can be added in subnet health-analyzer
 2021-10-20 10:12:01 -07:00
Harshavardhana
d7fd396b7c update minio-go v7.0.15 2021-10-18 19:13:29 -07:00
Klaus Post
c2eb60df4a
bz2: limit max concurrent CPU (#13458) 
Ensure that bz2 decompression will never take more than 50% CPU.
 2021-10-18 08:44:36 -07:00
Klaus Post
76239fa1ae
Fix s3zip not returning data (#13442) 
We do not reliably know the length of compressed data, including headers.

Request until the end-of-stream. Results will still be properly truncated.

Fixes #13441
 2021-10-14 12:37:30 -07:00
Klaus Post
5e53f767c4
Use concurrent bz2 decompression (#13360) 
Testing with `mc sql --compression BZIP2 --csv-input "rd=\n,fh=USE,fd=;" --query="select COUNT(*) from S3Object" local2/testbucket/nyc-taxi-data-10M.csv.bz2`

Before 96.98s, after 10.79s. Uses about 70% CPU while running.
 2021-10-14 11:11:07 -07:00
Harshavardhana
129f41cee9 update dependencies for minio/console and minio/pkg 
IAM policy parser was incorrectly dropping duplicate
statements, this update brings the right fix for
these situations.
 2021-10-12 17:23:17 -07:00
Aditya Manthramurthy
3a7c79e2c7
Add new site replication feature (#13311) 
This change allows a set of MinIO sites (clusters) to be configured 
for mutual replication of all buckets (including bucket policies, tags, 
object-lock configuration and bucket encryption), IAM policies, 
LDAP service accounts and LDAP STS accounts.
 2021-10-06 16:36:31 -07:00
Harshavardhana
9b9ce1c625 update console release to v0.10.3 2021-10-05 13:34:53 -07:00
Harshavardhana
3d5750f31c
update and use rs/dnscache implementation instead of custom (#13348) 
additionally optimize for IP only setups, avoid doing
unnecessary lookups if the Dial addr is an IP.

allow support for multiple listeners on same socket,
this is mainly meant for future purposes.
 2021-10-05 10:13:04 -07:00
Klaus Post
bc6067d195
Add admin inspect Glob support (#13328) 
* Add admin Glob support

Allow returning multiple files on inspect calls.

```
λ mc admin inspect --json local2/testbucket/nyc-taxi-data-10M.csv.zst/*

...

λ unzip -l inspect.5f0643b2.zip

Archive:  inspect.5f0643b2.zip
  Length      Date    Time    Name
---------  ---------- -----   ----
        0  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/
      802  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/xl.meta
        0  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/
      802  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/xl.meta
        0  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/
      802  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/xl.meta
        0  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/
      802  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/xl.meta
---------                     -------
     3208                     8 files
```

Using fully recursive:

```
λ  mc admin inspect local2/testbucket/nyc-taxi-data-10M.csv.zst/**

...

Archive:  inspect.79c261cb.zip
  Length      Date    Time    Name
---------  ---------- -----   ----
        0  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/
        0  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.1
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.10
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.11
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.12
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.13
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.14
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.15
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.16
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.17
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.18
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.19
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.2
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.20
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.21
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.22
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.23
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.24
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.25
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.26
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.27
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.28
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.29
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.3
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.30
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.31
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.32
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.33
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.34
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.35
  3439368  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.36
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.4
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.5
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.6
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.7
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.8
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.9
      802  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/xl.meta
        0  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/
        0  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.1
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.10
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.11
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.12
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.13
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.14
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.15
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.16
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.17
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.18
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.19
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.2
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.20
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.21
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.22
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.23
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.24
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.25
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.26
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.27
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.28
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.29
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.3
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.30
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.31
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.32
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.33
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.34
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.35
  3439368  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.36
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.4
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.5
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.6
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.7
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.8
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.9
      802  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/xl.meta
        0  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/
        0  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.1
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.10
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.11
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.12
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.13
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.14
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.15
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.16
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.17
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.18
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.19
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.2
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.20
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.21
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.22
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.23
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.24
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.25
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.26
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.27
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.28
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.29
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.3
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.30
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.31
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.32
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.33
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.34
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.35
  3439368  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.36
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.4
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.5
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.6
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.7
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.8
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.9
      802  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/xl.meta
        0  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/
        0  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.1
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.10
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.11
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.12
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.13
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.14
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.15
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.16
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.17
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.18
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.19
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.2
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.20
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.21
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.22
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.23
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.24
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.25
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.26
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.27
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.28
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.29
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.3
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.30
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.31
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.32
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.33
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.34
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.35
  3439368  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.36
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.4
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.5
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.6
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.7
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.8
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.9
      802  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/xl.meta
---------                     -------
601034920                     156 files

```

Furthermore allow `inspect` to do direct decode from `mc`, for example:

```
λ mc admin inspect --json local2/testbucket/nyc-taxi-data-10M.csv.zst/*|inspect -json
Output decrypted to inspect.5f0643b2.zip
```

- Correct error, forward non-EOF errors.
- Add some extra safety. Log FNF when no files.
- Add `xl-meta` zip support.
For `xl-meta` multiple inputs output object with names as key.
Automatically switches `xl-meta` to single-line output when multiple objects.
Add double-star wildcard support to xl-meta input.

Co-authored-by: Harshavardhana <harsha@minio.io>
 2021-10-01 11:50:00 -07:00
Poorna Krishnamoorthy
7f6ed35347
Allow null versions to be replicated (#13310) 
for pre-existing objects present in a bucket
prior to enabling existing object replication.

Co-authored-by: Poorna Krishnamoorthy <poorna@minio.io>
 2021-09-28 10:26:12 -07:00
Harshavardhana
4519450363 update console release to v0.10.2 2021-09-27 19:09:03 -07:00
Harshavardhana
1984d0671b
update reedsolomon package to v1.9.13 (#13288) 2021-09-23 10:46:24 -07:00
Harshavardhana
c25b482301 update console to v0.10.1 2021-09-22 19:42:31 -07:00
Poorna Krishnamoorthy
806b10b934
fix: improve error messages returned during replication setup (#13261) 2021-09-21 13:03:20 -07:00
Aditya Manthramurthy
1fa0553c71
Remove support for elasticsearch versions < 7.x (#13260) 2021-09-21 12:57:10 -07:00
Poorna Krishnamoorthy
c4373ef290
Add support for multi site replication (#12880) 2021-09-18 13:31:35 -07:00
Poorna Krishnamoorthy
246cbe1312
update minio-go with healthcheck fixes (#13244) 2021-09-17 21:14:13 -07:00
Harshavardhana
6d42569ade
remove ListBucketsMetadata instead add them to AccountInfo() (#13241) 2021-09-17 15:02:21 -07:00
Harshavardhana
bcc6359dec support Console UI with userInfo claims for OpenID 2021-09-14 17:09:18 -07:00
Krishna Srinivas
03a2a74697
Support speedtest autotune on the server side (#13086) 2021-09-10 17:43:34 -07:00
Poorna Krishnamoorthy
9af4e7b1da
Add healthcheck back for replication targets (#13168) 
This will allow objects to relinquish read lock held during
replication earlier if the target is known to be down
without waiting for connection timeout when replication 
is attempted.
 2021-09-08 15:34:50 -07:00
Harshavardhana
198a838d00 update console tag to v0.9.8 2021-09-08 14:05:17 -07:00
Klaus Post
b2c92cdaaa
select: Add more compression formats (#13142) 
Support Zstandard, LZ4, S2, and snappy as additional 
compression formats for S3 Select.
 2021-09-06 09:09:53 -07:00
Harshavardhana
b8f5a7db33 update console release to v0.9.6 2021-09-01 23:57:19 -07:00
Harshavardhana
f422f09dff update console to latest master 2021-09-01 18:10:44 -07:00
Harshavardhana
c89aee37b9
fix: log errors for incorrect environment inputs (#13121) 
Invalid MINIO_ARGS, MINIO_ENDPOINTS would be silently
ignored when using remoteEnv style, make sure to log
errors to indicate invalid configuration.
 2021-09-01 11:34:07 -07:00