Commit Graph

757 Commits

Author SHA1 Message Date
Harshavardhana
610dbe3479 config: Do not migrate config file if not needed. (#4264)
Also improve the error message returned by `pkg/quick`.

Fixes #4233
2017-05-06 10:16:59 -07:00
Harshavardhana
76f4f20609 fs: Migrate object metadata to objects directory. (#4195)
Fixes #3352
2017-05-05 08:49:09 -07:00
Harshavardhana
99ddd35343 docs: use IEC format such as iB everywhere. (#4247) 2017-05-05 08:28:08 -07:00
Remco Verhoef
01e9adc4b3 Implement anonymous uploads, fixes #4250 (#4259) 2017-05-04 20:03:56 -07:00
Harshavardhana
df027a8f51 Webhook endpoints can fail, we must start the server. (#4255)
This PR fixes a regression introduced in #4060
2017-05-04 13:43:54 -07:00
Krishnan Parthasarathi
02910725c5 Make gateway help for s3/azure similar (#4249) 2017-05-04 10:38:48 -07:00
Remco Verhoef
069cf9e8aa Use s3.amazonaws.com as default endpoint, fixes #4240 (#4242) 2017-05-03 22:41:03 -07:00
Aditya Manthramurthy
2121b78ea7 Fix bug in JSON representation of object properties (#4238)
Introduced in #4003
2017-05-03 20:10:00 -07:00
Remco Verhoef
5016649f47 Add s3 backend to help, fixes #4219 (#4221)
* Add s3 backend to help, fixes #4219

* Add samples for Gateway usage with S3
2017-05-03 17:55:30 -07:00
Bala FA
2b78444056 fix: ignore TLS handshake error. (#4227)
Fixes #4200
2017-05-03 03:23:15 -07:00
Karthic Rao
9b58a669e5 tests: Fix rare test crash (#4175)
Fix rare test crash by improving the randomness logic.
2017-05-02 23:54:22 -07:00
Krishna Srinivas
e5b2e25caf gateway-s3: vendor-update minio-go (#4220) 2017-05-02 18:46:39 -07:00
Krishna Srinivas
4aa65910e5 gateway: Restore bucket policy functionality for Azure (#4209) 2017-05-02 12:27:25 -07:00
Harshavardhana
8b272a3163 config: Improve config migrate messaging. (#4216)
Previous message

```
Migration from version ‘17’ to ‘18’ completed successfully.
```

For example didn't provide any meaningful insights.

This PR attempts to improve this message as below

```
Configuration file '/home/harsha/.minio/config.json' migrated from version '17' to '18' successfully.
```

Fixes #4199
2017-05-02 11:43:27 -07:00
Harshavardhana
f0b5c0ec7c windows: Support all REPARSE_POINT attrib files properly. (#4203)
This change adopts the upstream fix in this regard at
https://go-review.googlesource.com/#/c/41834/ for Minio's
purposes.

Go's current os.Stat() lacks support for lot of strange
windows files such as

 - share symlinks on SMB2
 - symlinks on docker nanoserver
 - de-duplicated files on NTFS de-duplicated volume.

This PR attempts to incorporate the change mentioned here

   https://blogs.msdn.microsoft.com/oldnewthing/20100212-00/?p=14963/

The article suggests to use Windows I/O manager to
dereference the symbolic link.

Fixes #4122
2017-05-02 02:35:27 -07:00
Remco Verhoef
44d53c9c67 cleanup and fix comments (#4212) 2017-05-01 14:44:31 -07:00
Krishna Srinivas
6cf6828a4c gateway: Rename gateway files to have "gateway-" prefix (#4207) 2017-05-01 10:32:18 -07:00
Krishna Srinivas
01f04c717e gateway: reject bad path segments in URL (#4202) 2017-04-28 17:17:18 -07:00
Krishna Srinivas
0d32b22359 gateway: Fix help message for gateway (#4201) 2017-04-28 16:42:16 -07:00
Anis Elleuch
d36dd80a8a cors: Set Access-Control-Allow-Credentials to true (#4185)
This allow browsers to send credentials with preflighted requests.
2017-04-27 12:40:22 -07:00
Remco Verhoef
3a539ce660 Implement gateway S3 support (#3940) 2017-04-27 11:26:00 -07:00
Harshavardhana
57c5c75611 web: Simplify and converge common functions in web/obj API. (#4179)
RemoveObject() in webAPI currently re-implements some part
of the code to remove objects combine them for simplicity
and code convergence.
2017-04-26 23:27:48 -07:00
Bala FA
cf1fc45142 Improve duration humanization. (#4071) 2017-04-26 03:38:35 -07:00
Harshavardhana
48aa2ac392 server: Validate path for bad components in a handler. (#4170) 2017-04-24 18:13:46 -07:00
Frank
0d1e2ab509 Remove hardcoded min and max limit for erasure coding (#4157) 2017-04-24 10:00:33 -07:00
Anis Elleuch
83abad0b37 admin: ServerInfo() returns info for each node (#4150)
ServerInfo() will gather information from all nodes before returning
it back to the client.
2017-04-21 07:15:53 -07:00
Harshavardhana
df346753e1 api: Fix registering of s3 endpoint peers properly (#4159)
We need to have local peer initialized properly
for listen bucket to work, current code did initialize
properly but the resulting code was initializing
peer on a wrong target v/s what listen bucket expected
it to be.

This regression came in de204a0a52

Fixes #4158
2017-04-20 15:28:29 -07:00
Harshavardhana
5a3c5aec31 server/mux: Fix serverMux to set deadlines based on UTC time. (#4146)
Avoid using `time.Now()` instead rely on UTC time
for the final deadline, this is to be consistent with
all our internal functions.

Reduce the default read timeout to 15 seconds
in lieu with a newly discovered issue
   - https://github.com/minio/minio/issues/4139

Additionally also change the Read() conn wrapper
to set deadline only upon successful Reads().
2017-04-19 13:16:06 -07:00
Aditya Manthramurthy
a4305742e8 Add key for Kafka messages (fixes #4143) (#4151) 2017-04-19 11:26:35 -07:00
Harshavardhana
640ebb2f79 lock: Fix missing formatting directives while printing. (#4147)
Current log prints in this form

```
ERRO[8150] Lock maintenance failed to remove entry for write
lock (should never happen)%!!(MISSING)(EXTRA ....
```

Fix this by using proper formatting directive.
2017-04-19 10:37:56 -07:00
Harshavardhana
f4dac979a2 server: Fix message when corrupted or unsupported format is found. (#4142)
Refer https://github.com/minio/minio/issues/4140

This is a fix to provide a little more elaborate message.
2017-04-18 10:35:17 -07:00
Krishnan Parthasarathi
3032f0f505 Remove duration field from lock instrumentation (#4111)
Duration for which a lock was held can be computed from the `Since`
field of `OpsLockState`. It is the difference between current time and
time at which the namespace lock was held. This change avoids
superfluous instrumentation.
2017-04-15 11:40:01 -07:00
Harshavardhana
7765081db7 cache: Increasing caching GC percent from 20 to 50. (#4041)
Previous value was set to avoid large cache value build
up but we can clearly see this can cause lots of GC
pauses which can lead to significant drop in performance.

Change this value to 50% and decrease the value to 25%
once the 75% cache size is used. To have a larger
window for GC pauses.

Another change is to only allow caching if a server has
more than 24GB of RAM instead of 8GB.
2017-04-15 02:16:49 -07:00
Anis Elleuch
14f0047295 fs: Remove fs meta lock when PutObject() fails (#4114)
Removing the fs meta lock file when PutObject() encounters any error
during its execution, such as upload getting permatuerly cancelled
by the client.
2017-04-14 12:06:24 -07:00
Krishna Srinivas
e6b2253da9 gateway: Fix help message for custom Azure Blob Storage endpoint. (#4113) 2017-04-14 11:02:43 -07:00
Krishnan Parthasarathi
ca64b86112 Return possible states a heal operation (#4045) 2017-04-14 10:28:35 -07:00
Karthic Rao
5f065e2a96 server: Fix CI build complaints (#4119)
- Ineffassign fixes.
- Spell check correction.
2017-04-14 08:00:04 -07:00
Harshavardhana
a7afa469e2 xl: Add stat calls to keep track of ignored errors. (#4117)
Such that in a situation where all errors were
ignored we need to reduce the errors using
readQuorum to get a consistent error value.

Without this change errors generated will
never be consistent with for an expected scenario.

For example in a 6 disk setup 1 disk is missing
and 5 do not have the volume (testbucket)

Without this change Stat() would result in different
errors depending on which disk died. Can cause
confusion to S3 client application.

This change addresses need to track type of
errors we ignored and bring readQuorum to
choose the maximally occuring as the value
of truth.
2017-04-14 01:46:16 -07:00
Bala FA
d103d5fb7c server: Error out if loopback addr is used for Distributed Erasure (#4105) 2017-04-12 20:27:24 -07:00
Harshavardhana
6683247080 tests: Fix the sopradic test failure in TestListObjectPartsDiskNotFound (#4107)
getBucketInfo() should keep track errors ignored,
such that in a situation where all errors were
ignored we need to reduce the errors using readQuorum
to get a consistent error value.

This is the problem we see with DiskNotFound test
disks are randomly removed.

Fixes #4095
2017-04-12 15:38:35 -07:00
Anis Elleuch
e4bd882f11 handlers: Ignore malformatted datetime type header (#4097)
Ignore headers, such as If-Modified-Since, If-Unmodified-Since, etc.. when they
are received with a format other than HTTP date.
2017-04-12 12:34:57 -07:00
Harshavardhana
952c618441 server: Fix a regression in printing startup banner. (#4100)
Octect based sorting was lost in the previous commit

de204a0a52

This PR fixes a regression - fixes #4099
2017-04-12 09:22:35 -07:00
Krishna Srinivas
c5249c35d3 gateway: Support for custom endpoint. (#4086) 2017-04-11 17:44:26 -07:00
Bala FA
de204a0a52 Add extensive endpoints validation (#4019) 2017-04-11 15:44:27 -07:00
Harshavardhana
1b1b9e4801 lock/rpc: change rpcPath to be called serviceEndpoint. (#4088)
This is a cleanup to ensure proper naming.
2017-04-11 10:25:21 -07:00
Karthic Rao
929a13f33f Fix for writes from Apache Spark. (#4074)
- Due to usage of amazon SDK, spark expects md5sum of empty string to be
  returned when it does PUT on a directory.
- The fix returns md5sum of a empty string for the above mentioned case.
- This fixes the issue of Apache Spark not being able to write into Minio.
2017-04-10 19:51:23 -07:00
Krishna Srinivas
a4209c10ea signature-v4: Use sha256("") for calculating canonical request (#4064) 2017-04-10 09:58:08 -07:00
Harshavardhana
b927523223 server: Introduce a new env MINIO_REGION. (#4078)
This is implemented to be able to override region
through command line just like how access and
secret keys are provided.
2017-04-09 10:44:10 -07:00
Aditya Manthramurthy
604417baf4 Allow cluster to start when only n/2 servers are up (#4066)
Fixes #3234.

Relaxes the quorum requirement to start the object layer, and skips
quick-healing at start-up (as no write quorum is present).
2017-04-09 00:28:27 -07:00
Harshavardhana
0497d5c342 api: SourceInfo should be populated in GET/HEAD notification. (#4073)
Refer https://github.com/minio/mc/issues/2073
2017-04-08 01:39:20 -07:00