Commit Graph

7014 Commits

Author SHA1 Message Date
谭九鼎 bc285cf0dd
use https link for brew.sh (#9692) 2020-05-24 09:28:10 -07:00
谭九鼎 209a3ee7af
fix backtick markdown in chinese docs (#9691) 2020-05-24 09:27:47 -07:00
Krishna Srinivas 7d19ab9f62
readiness returns error quickly if any of the set is down (#9662)
This PR adds a new configuration parameter which allows readiness
check to respond within 10secs, this can be reduced to a lower value
if necessary using 

```
mc admin config set api ready_deadline=5s
```

 or

```
export MINIO_API_READY_DEADLINE=5s
```
2020-05-23 17:38:39 -07:00
P R 3f6d624c7b
add gateway object tagging support (#9124) 2020-05-23 11:09:35 -07:00
Harshavardhana c138272d63
reject object lock requests on existing buckets (#9684)
a regression was introduced fix it to ensure that we
do not allow object locking settings on existing buckets
without object locking
2020-05-23 10:01:01 -07:00
Harshavardhana 7dbfea1353
avoid net/http ErrorLog for consistent logging experience (#9672)
net/http exposes ErrorLog but it is log.Logger
instance not an interface which can be overridden,
because of this reason the logging is interleaved
sometimes with TLS with messages like this on the
server

```
http: TLS handshake error from 139.178.70.188:63760: EOF
```

This is bit problematic for us as we need to have
consistent logging view for allow --json or --quiet
flags.

With this PR we ensure that this format is adhered to.
2020-05-22 21:59:18 -07:00
Sidhartha Mani c121d27f31
progressively report obd results (#9639) 2020-05-22 17:56:45 -07:00
Anis Elleuch 43c19a6b82
nas: ensure loading of bucket notifications during startup (#9681) 2020-05-22 11:55:30 -07:00
Anis Elleuch 6542bc4a03
sql, csv: Cache some values between Read() calls to gain performance (#9645)
Below is the benchmark enhancement after this commit:

benchmark                                            old ns/op     new ns/op     delta
BenchmarkRead-8                                      2807          2189          -22.02%
BenchmarkReadWithFieldsPerRecord-8                   2802          2179          -22.23%
BenchmarkReadWithoutFieldsPerRecord-8                2824          2181          -22.77%
BenchmarkReadLargeFields-8                           3584          3371          -5.94%
BenchmarkReadReuseRecord-8                           2044          1480          -27.59%
BenchmarkReadReuseRecordWithFieldsPerRecord-8        2056          1483          -27.87%
BenchmarkReadReuseRecordWithoutFieldsPerRecord-8     2047          1482          -27.60%
BenchmarkReadReuseRecordLargeFields-8                2777          2594          -6.59%

benchmark                                            old allocs     new allocs     delta
BenchmarkRead-8                                      26             16             -38.46%
BenchmarkReadWithFieldsPerRecord-8                   26             16             -38.46%
BenchmarkReadWithoutFieldsPerRecord-8                26             16             -38.46%
BenchmarkReadLargeFields-8                           36             24             -33.33%
BenchmarkReadReuseRecord-8                           16             6              -62.50%
BenchmarkReadReuseRecordWithFieldsPerRecord-8        16             6              -62.50%
BenchmarkReadReuseRecordWithoutFieldsPerRecord-8     16             6              -62.50%
BenchmarkReadReuseRecordLargeFields-8                24             12             -50.00%

benchmark                                            old bytes     new bytes     delta
BenchmarkRead-8                                      672           664           -1.19%
BenchmarkReadWithFieldsPerRecord-8                   672           664           -1.19%
BenchmarkReadWithoutFieldsPerRecord-8                672           664           -1.19%
BenchmarkReadLargeFields-8                           3948          3936          -0.30%
BenchmarkReadReuseRecord-8                           32            24            -25.00%
BenchmarkReadReuseRecordWithFieldsPerRecord-8        32            24            -25.00%
BenchmarkReadReuseRecordWithoutFieldsPerRecord-8     32            24            -25.00%
BenchmarkReadReuseRecordLargeFields-8                2988          2976          -0.40%
2020-05-22 10:15:08 -07:00
Ashish Kumar Sinha bede525dc9
bypass flag: Allow object retention removal (#9677) 2020-05-22 09:38:58 -07:00
Harshavardhana e45c90060f
remove references for deprecated dockerfiles and deployment styles (#9675) 2020-05-22 08:40:59 -07:00
Klaus Post 7d79c723e5
Fix Windows memory leak (#9680)
When running a zoned setup simply uploading will run
the system out of memory very fast.

Root cause: nFileSystemNameSize is a DWORD and 
not a pointer. No idea how this didn't crash hard.

Furthermore replace poor mans utf16 -> string 
conversion to support arbitrary output.

Fixes #9630
2020-05-22 08:26:43 -07:00
darkdragon-001 5f6d6c3b70
Facilitate selective running of single CI stages (#9663)
Pass '--target core' to docker build in order to only test core.
Pass '--target frontend' to docker build in order to only test frontend.
Pass '--target gateway' to docker build in order to only test gateway.

Numerical references might be supported in the future: https://github.com/moby/buildkit/issues/1503
2020-05-21 23:10:07 -07:00
Harshavardhana d15042470e
add missing signature v2 query params (#9670) 2020-05-21 18:51:23 -07:00
poornas f1f414ca59
fix madmin SetBucketQuota API signature (#9669) 2020-05-21 14:45:12 -07:00
Anis Elleuch cdf4815a6b
Add x-amz-expiration header in some S3 responses (#9667)
x-amz-expiration is described in the S3 specification as a header which
indicates if the object in question will expire any time in the future.
2020-05-21 14:12:52 -07:00
kannappanr fade056244
filter all encryption headers in gateway (#9661)
fixes #9655
2020-05-21 11:07:50 -07:00
Harshavardhana a546047c95
keep bucket metadata fields to be consistent (#9660)
added bonus reload bucket metadata always after
a successful MakeBucket, current we were only
doing it with object locking enabled.
2020-05-21 11:03:59 -07:00
darkdragon-001 ea210319ce
fix whitespace in browser/webpack.config.js (#9665) 2020-05-21 09:21:14 -07:00
ebozduman 2896e780ae
fixes misleading assume role error msgs (#9642) 2020-05-21 09:09:18 -07:00
Nicholas Bishop eaafb23535
fix a typo in README.md (#9666) 2020-05-21 08:45:27 -07:00
Harshavardhana baa30f4289
reload bucket metadata outside the locker (#9659) 2020-05-20 14:11:13 -07:00
Andreas Auernhammer 2164984d2b
fix KMS quickstart guide layout (#9658)
This commit fixes a layout issue w.r.t. the KMS
Quickstart guide. The problem seems to be caused
by docs server not converting the markdown into html
as expected.

This commit fixes this by converting the ordered list
into subsections.
2020-05-20 13:55:54 -07:00
Harshavardhana 189c861835
fix: remove LDAP groups claim and store them on server (#9637)
Groups information shall be now stored as part of the
credential data structure, this is a more idiomatic
way to support large LDAP groups.

Avoids the complication of setups where LDAP groups
can be in the range of 150+ which may lead to excess
HTTP header size > 8KiB, to reduce such an occurrence
we shall save the group information on the server as
part of the credential data structure.

Bonus change support multiple mapped policies, across
all types of users.
2020-05-20 11:33:35 -07:00
Harshavardhana 6656fa3066
simplify further bucket configuration properly (#9650)
This PR is a continuation from #9586, now the
entire parsing logic is fully merged into
bucket metadata sub-system, simplify the
quota API further by reducing the remove
quota handler implementation.
2020-05-20 10:18:15 -07:00
Praveen raj Mani 0cc2ed04f5
humanize `timeToFirstByte` and `timeToResponse` upto nanoseconds (#9641) 2020-05-19 18:34:02 -07:00
Andreas Auernhammer b11adfa5cd
simplify the KMS guide and remove unnecessary sections (#9629)
This commit simplifies the KMS configuration guide by
adding a get started section that uses our KES play instance
at `https://play.min.io:7373`.

Further, it removes sections that we don't recommend for production
anyways (MASTER_KEY).
2020-05-19 18:33:11 -07:00
Anis Elleuch 9baeda781a
fix storage info output with unordered endpoints arguments (#9610)
Shuffling arguments that we pass to MinIO server are supported. However,
when that happens, Prometheus returns wrong information about disks usage
and online/offline status.

The commit fixes the issue by avoiding relying on xl.endpoints since
it is not ordered.
2020-05-19 14:27:20 -07:00
Harshavardhana bd032d13ff
migrate all bucket metadata into a single file (#9586)
this is a major overhaul by migrating off all
bucket metadata related configs into a single
object '.metadata.bin' this allows us for faster
bootups across 1000's of buckets and as well
as keeps the code simple enough for future
work and additions.

Additionally also fixes #9396, #9394
2020-05-19 13:53:54 -07:00
Harshavardhana 730775bb4e
update all the outdated browser deps (#9640) 2020-05-19 09:32:14 -07:00
Harshavardhana d31eaddba3
fix: avoid double body reads in SelectObject call (#9638)
Bonus fix handle encryption headers in response
properly for both notification and response to
the client.
2020-05-19 02:01:08 -07:00
poornas 3202f78f0f
Fix cache metadata update for range GET (#9636)
This was inadvertently deleting cached ranges
because HTTPRangeSpec was not being passed down

fixes #9597
2020-05-18 18:33:43 -07:00
Harshavardhana 6de410a0aa
fix: possiblity of double write lockers on same resource (#9616)
To avoid this issue with refCounter refactor the code
such that

- locker() always increases refCount upon success
- unlocker() always decrements refCount upon success
  (as a special case removes the resource if the
  refCount is zero)

By these two assumptions we are able to see that we
are never granted two write lockers in any situation.

Thanks to @vcabbage for writing a nice reproducer.
2020-05-18 17:33:35 -07:00
darkdragon-001 a3f41c7049
Improve browser README (#9628)
- Add instructions to download node dependencies
- Add instructions to allow development from IPs other than localhost
2020-05-18 11:35:57 -07:00
Klaus Post 1847f17f50
Set Deployment ID before starting handlers (#9635)
Global handler ID is added to response headers, so initialize it before the server starts.

Fixes #9634
2020-05-18 11:35:05 -07:00
Harshavardhana 1bc32215b9
enable full linter across the codebase (#9620)
enable linter using golangci-lint across
codebase to run a bunch of linters together,
we shall enable new linters as we fix more
things the codebase.

This PR fixes the first stage of this
cleanup.
2020-05-18 09:59:45 -07:00
Anis Elleuch 96009975d6
relax validation when loading lifecycle document from the backend (#9612) 2020-05-18 08:33:43 -07:00
Harshavardhana de9b391db3
fix: Disable presigned without appropriate policy (#9621)
Fixes #9590
2020-05-17 23:38:52 -07:00
kannappanr a62572fb86
Check for address flags in all positions (#9615)
Fixes #9599
2020-05-17 08:46:23 -07:00
poornas 011a2c0b78
Add docs for bucket quota feature (#9503)
This PR also adds a check to not enforce
bucket quota for server-side metadata copy
of an object onto itself.
2020-05-16 19:27:33 -07:00
Harshavardhana daf4418cbb
add 50m for windows tests (#9614) 2020-05-15 21:07:27 -07:00
Minio Trusted 0f8df3d340 Update yaml files to latest version RELEASE.2020-05-16T01-33-21Z 2020-05-16 01:40:31 +00:00
Harshavardhana 9cac385aec add comment on exported error 2020-05-15 18:17:54 -07:00
Harshavardhana 814ddc0923
add missing admin actions, enhance AccountUsageInfo (#9607) 2020-05-15 18:16:45 -07:00
Harshavardhana 247795dd36
add github workflow for windows (#9611)
bye, bye travis
2020-05-15 15:54:39 -07:00
Anis Elleuch dfadf70a7f
mint: Add more SQL tests (#9540) 2020-05-15 11:20:57 -07:00
Harshavardhana d348ec0f6c
avoid double listObjectParts calls improves performance (#9606)
this PR is to avoid double calls across multiple calls
in APIs

- CopyObjectPart
- PutObjectPart
2020-05-15 08:06:45 -07:00
Harshavardhana b730bd1396
fix: possible race in FS local lockMap (#9598) 2020-05-14 23:59:07 -07:00
Klaus Post 56e0c6adf8
Track if bloom filter is dirty (#9601)
Only save bloom filter on cycles and updates.

Fixes #9600
2020-05-14 21:46:36 -07:00
Anis Elleuch f44a960dcd
tests: Fix one multi-delete test failure in Windows CI (#9602)
There is a disparency of behavior under Linux & Windows about
the returned error when trying to rename a non existant path.

err := os.Rename("/path/does/not/exist", "/tmp/copy")

Linux:
  isSysErrNotDir(err) = false
  os.IsNotExist(err) = true

Windows:
  isSysErrNotDir(err) = true
  os.IsNotExist(err) = true

ENOTDIR in Linux is returned when the destination path
of the rename call contains a file in one of the middle
segments of the path (e.g. /tmp/file/dst, where /tmp/file
is an actual file not a directory)

However, as shown above, Windows has more scenarios when
it returns ENOTDIR. For example, when the source path contains
an inexistant directory in its path.

In that case, we want errFileNotFound returned and not
errFileAccessDenied, so this commit will add a further check to close
the disparency between Windows & Linux.
2020-05-14 18:09:30 -07:00