Commit Graph

7543 Commits

Author SHA1 Message Date
Harshavardhana 3d3beb6a9d
Add response header timeouts (#9170)
- Add conservative timeouts upto 3 minutes
  for internode communication
- Add aggressive timeouts of 30 seconds
  for gateway communication

Fixes #9105
Fixes #8732
Fixes #8881
Fixes #8376
Fixes #9028
2020-03-21 22:10:13 -07:00
poornas 27b8f18cce
Fix storage info message on startup (#9177) 2020-03-21 10:02:20 -07:00
Harshavardhana bf545dc320
migrate to new minio-go with latest changes (#9176)
- extract userTags from Get/Head request (#1249)
- fix: Context cancellation not handled (#1250)
- Check for correct http status in remove object tagging (#1248)
- simplify extracting metadata in Head/Get object (#1245)
- fix: close and remove .minio.part file on errors (#1243)
2020-03-20 17:28:36 -07:00
stefan-work f001e99fcd
create the final file with mode 0666 for multipart-uploads (#9173)
NAS gateway creates non-multipart-uploads with mode 0666.
But multipart-uploads are created with a differing mode of 0644.

Both modes should be equal! Else it leads to files with different
permissions based on its file-size. This patch solves that by
using 0666 for both cases.
2020-03-20 15:32:15 -07:00
Harshavardhana b4bfdc92cc fix: admin console logger changes to log.Info 2020-03-20 15:14:14 -07:00
Harshavardhana ae654831aa
Add madmin package context support (#9172)
This is to improve responsiveness for all
admin API operations and allowing callers
to cancel any on-going admin operations,
if they happen to be waiting too long.
2020-03-20 15:00:44 -07:00
Stephen N 1ffa983a9d
added support for SASL/SCRAM on Kafka bucket notifications. (#9168)
fixes #9167
2020-03-20 11:10:27 -07:00
Nitish Tiwari ecf1566266
Add an option to allow plaintext connection to LDAP/AD Server (#9151) 2020-03-19 19:20:51 -07:00
Minio Trusted c5b87f93dd Update yaml files to latest version RELEASE.2020-03-19T21-49-00Z 2020-03-19 21:57:16 +00:00
Harshavardhana b1a2169dcc
fix: data usage crawler env handling, usage-cache.bin location (#9163)
canonicalize the ENVs such that we can bring these ENVs 
as part of the config values, as a subsequent change.

- fix location of per bucket usage to `.minio.sys/buckets/<bucket_name>/usage-cache.bin`
- fix location of the overall usage in `json` at `.minio.sys/buckets/.usage.json`
  (avoid conflicts with a bucket named `usage.json` )
- fix location of the overall usage in `msgp` at `.minio.sys/buckets/.usage.bin`
  (avoid conflicts with a bucket named `usage.bin`
2020-03-19 09:47:47 -07:00
Harshavardhana d45a1808f2
fix: Walk() should require quorum number of disks only (#9164) 2020-03-18 20:56:07 -07:00
Anis Elleuch db2155551a
heal: Pass scan mode to HealObjects to deep scan full quorum objects (#9159)
As an optimization of the healing, HealObjects() avoid sending an
object to the background healing subsystem when the object is
present in all disks.

However, HealObjects() should have checked the scan type, if this
deep, always pass the object to the healing subsystem.
2020-03-18 17:50:00 -07:00
Harshavardhana 09d35d3b4c
fix: sts to return appropriate errors (#9161) 2020-03-18 17:25:45 -07:00
Anis Elleuch 5b9342d35c
xl: Tree walking should not quit when one disk returns empty (#9160)
Currently, a tree walking, needed to a list objects in a specific
set quits listing as long as it finds no entries in a disk, which
is wrong.

This affected background healing, because the latter is using
tree walk directly. If one object does not exist in the first
disk for example, it will be seemed like the object does not
exist at all and no healing work is needed.

This commit fixes the behavior.
2020-03-18 16:58:05 -07:00
Klaus Post 8d98662633
re-implement data usage crawler to be more efficient (#9075)
Implementation overview: 

https://gist.github.com/klauspost/1801c858d5e0df391114436fdad6987b
2020-03-18 16:19:29 -07:00
Anis Elleuch 7fdeb44372
info: Initialize boot time early so uptime will always be correct (#9154) 2020-03-17 16:37:28 -07:00
poornas 59dced8237
Print node status even in --quiet mode (#9149) 2020-03-17 15:25:00 -07:00
Anis Elleuch 496f4a7dc7
Add service account type in IAM (#9029) 2020-03-17 10:36:13 -07:00
kannappanr 8b880a246a
fix: deleteObjectTagging should 204 on success (#9150) 2020-03-16 23:21:24 -07:00
Klaus Post eeb5942b6b
fix: remote profile names and extension (#9145)
Remote profiles are not formatted correctly:

```
profile-172.31.91.126_9000-cpu.pprof
profile-172.31.91.126_9000-goroutines-before.txt
profile-172.31.91.126_9000-goroutines.txt
profiling-172.31.80.49_9000-cpu.pprof.pprof
profiling-172.31.80.49_9000-goroutines-before.txt.pprof
profiling-172.31.80.49_9000-goroutines.txt.pprof
profiling-172.31.86.101_9000-cpu.pprof.pprof
profiling-172.31.86.101_9000-goroutines-before.txt.pprof
profiling-172.31.86.101_9000-goroutines.txt.pprof
profiling-172.31.91.191_9000-cpu.pprof.pprof
profiling-172.31.91.191_9000-goroutines-before.txt.pprof
profiling-172.31.91.191_9000-goroutines.txt.pprof
```

`profiling` -> `profile`, remove extra extension.
2020-03-16 11:39:53 -07:00
yeungc 7ec904d67b fix: wording and update content of chinese docs (#9140) 2020-03-16 10:04:16 -07:00
Harshavardhana c9212819af
fix: lock maintenance should honor quorum (#9138)
The staleness of a lock should be determined by
the quorum number of entries returning stale,
this allows for situations when locks are held
when nodes are down - we don't accidentally
clear locks unintentionally when they are valid
and correct.

Also lock maintenance should be run by all servers,
not one server, stale locks need to be run outside
the requirement for holding distributed locks.

Thanks @klauspost for reproducing this issue
2020-03-15 11:55:52 -07:00
poornas 10fd53d6bb
Fix: admin config set API for notifications (#9085)
Filter out targets set via env when
validating incoming config change against
configured notification targets

Fixes #9066
2020-03-14 00:01:15 -07:00
gzur 3fea1d5e35
Align STS web-identity code snippet to documentation (minio#9114) (#9130) 2020-03-13 22:58:53 -07:00
Anis Elleuch 35ecc04223
Support configurable quote character parameter in Select (#8955) 2020-03-13 22:09:34 -07:00
Harshavardhana 3ca9f5ffa3 Update yaml files to latest version RELEASE.2020-03-14T02-21-58Z 2020-03-13 20:05:27 -07:00
Krishna Srinivas 2e9fed1a14
non-empty dirs should not be listed as objects (#9129) 2020-03-13 17:43:00 -07:00
Nitish Tiwari 6b92f3fd99
Add docker files for ARM32 and ARM64 builds (#9132) 2020-03-13 13:37:39 -07:00
Kody A Kantor 06e30b5aa1
Skip building directio on platforms that don't support Direct IO (#9059) 2020-03-12 18:57:41 -07:00
Harshavardhana 603cf2a8bb
fix: broken gzip handling with Select API (#9128)
This PR fixes a regression introduced in a1c7c9ea73
2020-03-12 15:34:11 -07:00
Harshavardhana a54cdb9587
fix: Send x-amz-mp-parts-count for multiparted objects (#9116)
Some AWS SDKs latently rely on this value some times
to calculate the right number of parts during a parallel
GetObject request, this is feature used along with
content-range - we should support this as well.
2020-03-12 12:37:27 -07:00
Andreas Auernhammer ed4bd20a7c
change ca path env. var in KMS guide (#9125)
This commit fixes the env. variable in the
KMS guide used to specify the CA certificates
for the KES server.

Before the env. variable `MINIO_KMS_KES_CAPATH` has
been used - which works in non-containerized environments
due to how MinIO merges the config file and environment
variables. In containerized environments (e.g. docker)
this does not work and trying to specify `MINIO_KMS_KES_CAPATH`
instead of `MINIO_KMS_KES_CA_PATH` eventually leads to MinIO not
trusting the certificate presented by the kes server.

See: cfd12914e1/cmd/crypto/config.go (L186)
2020-03-12 07:47:40 -07:00
Harshavardhana cfd12914e1
fix: crash in serverInfo handler when ldap is configured (#9123) 2020-03-11 23:13:32 -07:00
Klaus Post c55aeaf814
Update compression package (#9120)
Fix a potential problem on non-AMD64 platforms and very small files

https://github.com/klauspost/compress/pull/244
2020-03-11 23:02:15 -07:00
Anis Elleuch fdf65aa9b9
heal: Add info about the next background healing round (#9122)
- avoid setting last heal activity when starting self-healing

This can be confusing to users thinking that the self healing
cycle was already performed.

- add info about the next background healing round
2020-03-11 23:00:31 -07:00
Harshavardhana 69b2aacf5a
fix return proper error for OperationTimedout (#9117)
OperationTimedout error occurs when locking
timesout, trying to acquire a lock. This
error should be returned appropriately to
the client with http status "408" (request timedout)

This translation was broken, fix it.
2020-03-11 14:11:04 -07:00
Anis Elleuch 0af62d35a0
xl: Implement posix.DeletePrefixes to enhance delete perf (#9100)
Bulk delete API was using cleanupObjectsBulk() which calls posix
listing and delete API to remove objects internal files in the
backend (xl.json and parts) one by one.

Add DeletePrefixes in the storage API to remove the content
of a directory in a single call.

Also use a remove goroutine for each disk to accelerate removal.
2020-03-11 08:56:36 -07:00
Nitish Tiwari 7c32f3f554
Fix the URL for MinIO update when using custom download server (#9111)
Co-authored-by: Nitish Tiwari <nitish@minio.io>
Co-authored-by: Harshavardhana <harsha@minio.io>
2020-03-11 20:09:20 +05:30
Aditya Manthramurthy cec8cdb35e
S3Select: Handle array selection in from clause (#9076) 2020-03-10 22:34:58 -07:00
Harshavardhana 5ab9cc029d
fix: crash observed for anonymous deletes from UI (#9107) 2020-03-09 21:21:35 -07:00
Minio Trusted 667f42515a Update yaml files to latest version RELEASE.2020-03-09T18-26-53Z 2020-03-09 18:38:07 +00:00
Harshavardhana 3614cb7a8b
update minio-go library to fix couple of issues (#9099)
- Add PutObjectOptions.PartSize docs (#1239) (03/06/20) <Harshavardhana>
- list: Check EncodingType in list resp before decoding object names (#1238) (03/04/20) <Harshavardhana>
- Add Support for Legal-Hold (#1233) (03/04/20) <kannappanr>
- Add LegalHold API Support (#1226) (02/19/20) <Nitish Tiwari>
- extract userMetadata from event response (#1229) (02/18/20) <Harshavardhana>
- fix: ignore AWS elb endpoints in region extraction (#1228) (02/14/20) <Harshavardhana>
- Bucket and object name length error message fixes (#1227) (02/12/20) <Ville Skyttä>
- Add BucketEncryption apis (#1217) (02/01/20) <ebozduman>
- Set IAM endpoint to default value if unspecified (#1224) (02/01/20) <radix-aw>
- fix testListObjects functional test (#1222) (01/28/20) <poornas>
- fix: retry AccessDenied only if Region is present (#1221) (01/24/20) <Harshavardhana>
- Add new Amazon S3 endpoints (#1220) (01/23/20) <kannappanr>
2020-03-09 12:27:25 +05:30
kumy b809c84338
fix: notifications doc elaborate env values for targets (#9103) 2020-03-08 18:33:43 -07:00
kannappanr 33edb072a3
Add TopLocksAdminAction to diagsnostics canned policy (#9104) 2020-03-08 18:32:39 -07:00
Harshavardhana 6a00eb10bf
fix: allow set drive count of proper divisible values (#9101)
Currently the code assumed some orthogonal requirements
which led situations where when we have a setup where
we have let's say for example 168 drives, the final
set_drive_count chosen was 14. Indeed 168 drives are
divisible by 12 but this wasn't allowed due to an
unexpected requirement to have 12 to be a perfect modulo
of 14 which is not possible. This assumption was incorrect.

This PR fixes this old assumption properly, also adds
few tests and some negative tests as well. Improvements
are seen in error messages as well.
2020-03-08 13:30:25 -07:00
Harshavardhana 792ee48d2c
add additional logging during server formatting (#9102) 2020-03-08 12:12:07 -07:00
Minio Trusted 52873ac3a3 Update yaml files to latest version RELEASE.2020-03-06T22-23-56Z 2020-03-06 22:32:45 +00:00
Harshavardhana 88ae0f1196
Improve delete performance by reducing the number of calls (#9092)
- Remove the requirement to honor storage class for deletes
- Improve `posix.DeleteFileBulk` code to Stat the volumeDir
  only once per call, rather than for all object paths.
2020-03-06 13:44:24 -08:00
Anis Elleuch 23a0415eb7
profiling: Fix crash when enabling goroutines profiling (#9097)
This commit replaces 'goroutines' with 'goroutine' when passing it
to pprof library when activating goroutine type profiling
2020-03-06 13:22:47 -08:00
Anis Elleuch 75a0661213
data-usage: Fix the calculation of the next crawling round (#9096)
This commit fixes a simple typo miscalculated the waiting time
until the next round of data crawling to compute the data usage.
2020-03-06 11:34:12 -08:00