Commit Graph

898 Commits

Author SHA1 Message Date
Brendan Ashworth
bccc386994 fs: drop Stat() call from fsDeleteFile,deleteFile (#4744)
This commit makes fsDeleteFile() simply call deleteFile() after calling
the relevant path length checking functions. This DRYs the code base.

This commit removes the Stat() call from deleteFile(). This improves
performance and removes any possibility of a race condition.

This additionally adds tests and a benchmark for said function. The
results aren't very consistent, although I'd expect this commit to make
it faster.
2017-08-03 20:04:28 -07:00
ebozduman
0f401b67ad Removes max limit requirement on accessKey and secretKey length (#4730) 2017-08-03 20:03:37 -07:00
Brendan Ashworth
ec5293ce29 jwt,browser: allow short-expiry tokens for GETs (#4684)
This commit fixes a potential security issue, whereby a full-access
token to the server would be available in the GET URL of a download
request. This fixes that issue by introducing short-expiry tokens, which
are only valid for one minute, and are regenerated for every download
request.

This commit specifically introduces the short-lived tokens, adds tests
for the tokens, adds an RPC call for generating a token given a
full-access token, updates the browser to use the new tokens for
requests where the token is passed as a GET parameter, and adds some
tests with the new temporary tokens.

Refs: https://github.com/minio/minio/pull/4673
2017-07-24 12:46:37 -07:00
Harshavardhana
4785555d34 api: Upon bucket delete remove in-memory state properly. (#4716)
This PR fixes the issue of cleaning up in-memory state
properly. Without this PR we can lead to security
situations where new bucket would inherit wrong
permissions on bucket and expose objects erroneously.

Fixes #4714
2017-07-23 19:35:18 -07:00
A. Elleuch
b918a6592f gcs: Better parsing of address flag (#4709) 2017-07-20 16:39:11 -07:00
Krishna Srinivas
eb787d8613 gateway-gcs: remove files older than 2 weeks in minio.sys.temp (#4599).
Rename ##minio## to {minio}.
2017-07-20 15:36:48 -07:00
Harshavardhana
f8bd9cfd83 rpc: Do not use read/write deadlines for rpc connections. (#4647)
Fixes #4626
2017-07-18 09:30:46 -07:00
Harshavardhana
bc73a1a1cb gcs: Save partNumber as part of backend format. (#4666)
Fixes #4637
2017-07-13 23:20:16 -07:00
Krishna Srinivas
ce7c9c651d gateway-azure: Return right error when Part size is > 100MB (#4652) 2017-07-12 16:42:14 -07:00
Bala FA
c3dd7c1f6c Refactor HTTP server to address bugs (#4636)
* Refactor HTTP server to address bugs
* Remove unnecessary goroutine to start multiple TCP listeners.
* HTTP server waits for shutdown to maximum of Server.ShutdownTimeout
  than per serverShutdownPoll.
* Handles new connection errors properly.
* Handles read and write timeout properly.
* Handles error on start of HTTP server properly by exiting minio
  process.

Fixes #4494 #4476 & fixed review comments
2017-07-12 16:33:21 -07:00
Harshavardhana
2d23cd4f39 gcs: Fetch port as GlobalString(). (#4657)
Currently we were looking for `address` flag
under local flags. This PR fixes #4656
2017-07-11 18:06:26 -07:00
Harshavardhana
ce7af3aae1 gcs: Fix writer/reader go-routine leaks and code re-use (#4651)
This PR serves to fix following things in GCS gateway.

- fixes leaks in object reader and writer, not getting closed
  under certain situations. This led to go-routine leaks.

- apparent confusing issue in case of complete multipart upload,
  where it is currently possible for an entirely different
  object name to concatenate parts of a different object name
  if you happen to know the upload-id and parts of the object.
  This is a very rare scenario but it is possible.

- succint usage of certain parts of code base and re-use.
2017-07-11 09:25:19 -07:00
Krishna Srinivas
1b92c5136b Append "-1" to etag when it is not MD5 (#4641)
* gateway-azure: append "-1" to ETag so that clients do not interpret it as MD5. fixes #4537. Added unit tests.
2017-07-10 18:21:12 -07:00
Harshavardhana
cc8a8cb877 posix: Check for min disk space and inodes (#4618)
This is needed such that we don't start or
allow writing to a posix disk which doesn't
have minimum total disk space available.

One part fix for #4617
2017-07-10 18:14:48 -07:00
Harshavardhana
f5ce685aa1 Remove dead unused errs and constants. (#4627) 2017-07-07 14:31:42 -07:00
Krishna Srinivas
c83055500d fs: Fail CompleteMultipartUpload if partSize < 5M unless it is last part (#4642)
fixes #4625
2017-07-07 08:41:29 -07:00
Andreas Auernhammer
b0fbddc051 fix confusing code for http.Header handling (#4623)
Fixed header-to-metadat extraction. The extractMetadataFromHeader function should return an error if the http.Header contains a non-canonicalized key. The reason is that the keys can be manually set (through a map access) which can lead to ugly bugs.
Also fixed header-to-metadata extraction. Return a InternalError if a non-canonicalized key is found in a http.Header. Also log the error.
2017-07-05 16:56:10 -07:00
Harshavardhana
4e0c08e9c5 ListenBucketNotification should set proper MIME type. (#4621)
This is needed to avoid proxies buffering the connection
this is also a HTTP standard way to handle this situation
where server is sending back events in asynchronously.

For more details read https://goo.gl/RCML9f

Fixes - https://github.com/minio/minio-go/issues/731
2017-07-03 19:59:41 -07:00
Nitish Tiwari
e91e9e8a38 GCS ListObjectV2 honours continuationToken (#4608) 2017-06-29 11:19:55 -07:00
Bala FA
53e7fdc847 create subcomposed objects if total parts are > 32 (#4593) 2017-06-27 22:27:05 -07:00
Aditya Manthramurthy
1af331c05c Remove unnecessary newline at beginning of server output (#4600) 2017-06-27 19:46:58 -07:00
ebozduman
0e5b9c7fe4 Adds 'gcs not ready for production' msg (#4604) 2017-06-27 19:44:47 -07:00
A. Elleuch
c88dca984d web: Encode path in presigned GET urls (#4596)
When the browser asks for a GET presigned url, this latter is not
encoded and can be confusing when the user copies-pastes it somewhere,
especially when the path contains a space.
2017-06-25 18:39:14 -07:00
Harshavardhana
1054f9cbf0 fix: Remove adverb from erasure coded startup message. (#4594) 2017-06-25 18:38:55 -07:00
Nitish Tiwari
7bd1f44491 Add support for helm package info in useragent (#4592) 2017-06-24 13:17:28 -07:00
A. Elleuch
eaa41e4086 gcs: Check if the given project id argument exists (#4583)
Using GCS resource manager API, check if the provided
project id is already created and associated to the current
user account.
2017-06-23 22:10:29 -07:00
Krishna Srinivas
6b70f429ed gateway/azure: Parse error responses for anonymous requests (#4543)
fixes #4481
2017-06-23 22:07:46 -07:00
Krishna Srinivas
0a6e9a1834 gateway-gcs: cleanup minio.sys.temp before deleting the bucket (#4582)
fixes #4560
fixes #4569
2017-06-23 17:57:25 -07:00
Nitish Tiwari
15b65a8342 Added AnonListObjectsV2 support to GCS (#4584) 2017-06-23 17:35:45 -07:00
Harshavardhana
8b7df7da37 api: No need to set x-amz-bucket-region if region is empty. (#4586) 2017-06-23 16:05:40 -07:00
Krishnan Parthasarathi
237c8af5ef Improve GCS gateway example instruction (#4587) 2017-06-23 13:13:26 -07:00
Krishna Srinivas
ff036c171f gateway-gcs: double quotes should be striped from ETag (#4590) 2017-06-23 12:19:10 -07:00
Harshavardhana
a3b085300d gcs: Add missing AnonListObjectsV2 2017-06-22 12:09:13 -07:00
Harshavardhana
b90cefdb88 Merge remote-tracking branch 'origin/feature-gcs' 2017-06-22 11:52:12 -07:00
Frank Wessels
46897b1100 Name return values to prevent the need (and unnecessary code bloat) (#4576)
This is done to explicitly instantiate objects for every return statement.
2017-06-21 19:53:09 -07:00
Harshavardhana
cec8b238f3 sign: StreamingSign should use region from client. (#4577)
This is a fix to make streaming signature to behave
the same as regular signature and presigned signature.

Fixes https://github.com/minio/minio-go/issues/718
2017-06-21 11:30:34 -07:00
Krishna Srinivas
13ab8e17e2 gateway-gcs: use minio.sys.temp/multipart/v1 as url base (#4562) 2017-06-21 10:27:44 -07:00
Krishnan Parthasarathi
146bc3e638 Add MINIO_REGION to server help message (#4558)
* Add e.g for setting MINIO_REGION env variable
* Add MINIO_REGION to region table
2017-06-20 15:02:18 -07:00
Harshavardhana
0543d45fb3 fix: OwnerID in response should be 64 character in length. (#4554)
Rather than sending a custom "minio" string, we can
change this to `sha256('arn:aws:iam::minio:user/admin')`.

Fixes #4553
2017-06-20 15:01:13 -07:00
Krishnan Parthasarathi
fe426944ea Fix GCS help message (#4570) 2017-06-20 14:25:16 -07:00
Aditya Manthramurthy
c1a6ca0c33 Fix spelling of function name to startLockMaintenance (#4561) 2017-06-20 12:10:02 -07:00
Harshavardhana
5a78266821 gateway/gcs: Complete minio browser support for gcs. (#4552)
Fixes #4460
2017-06-19 19:45:13 -07:00
Krishna Srinivas
3928c1e14c gateway/gcs: Change in multipart backend format (#4455) 2017-06-17 16:00:41 -07:00
Harshavardhana
a86c2e2ce1 xl/fs: Return InvalidPart{} error for part ETag mismatch. (#4541)
Fixes #4539
2017-06-17 11:20:39 -07:00
Harshavardhana
e99244be02 xl: prepare storage should Abort properly. (#4542)
Current state-machine didn't honor a situation
which can arise when there is a combination of

 - formatted
 - unformatted
 - corrupted

disks - this combination invariably goes into a
mode where all servers are waiting perpetually
forever thinking we will get quorum in future.

At this point there is a distant possibility of
ever getting a quorum since we don't even have
quorum number of disks offline.

We should exit and print a proper message per disk
to indicate what went wrong and what was detected
by the server.

Refer #4477
2017-06-17 11:20:12 -07:00
Nitish Tiwari
58833711e0 Added ListObjectsV2 and ListObjectsV2 Anon support to Gateway S3 and Azure. (#4547) 2017-06-16 22:17:00 -07:00
Harshavardhana
f99f218999 Add support for reading and saving config on Gateway. (#4463)
This is also a first step towards supporting bucket
notification for gateway.
2017-06-16 16:01:41 -07:00
Krishnan Parthasarathi
4fb5fc72d7 GCS gateway allows apps to supply their own marker (#4495)
Most s3 compatible apps use object keys returned in listing as
marker. This change allows this behaviour with gateway-gcs too.
2017-06-16 15:02:07 -07:00
Remco Verhoef
d86973dcca Allow bucket creation in different regions, closes #4287 and #4241
* I needed to remove the region check from PutBucketHandler
2017-06-16 15:02:07 -07:00
Krishnan Parthasarathi
8085ba4494 Filter out internal object prefix during listing (#4435)
We use ZZZZ-Minio/ prefix internally in our GCS gateway which should be
filtered out in the response to ListObjects.
2017-06-16 15:02:07 -07:00