Commit Graph

79 Commits

Author SHA1 Message Date
Krishnan Parthasarathi
1126410e62 Implement ListMultipartUploads, ListObjectParts for GCS gateway (#6377)
ListMultipartUploads implementation is meant for docker-registry
use-case only. It lists only the first upload with a prefix matching
the object being uploaded.
2018-09-04 13:11:33 -07:00
Harshavardhana
4487f70f08 Revert all GetObjectNInfo related PRs (#6398)
* Revert "Encrypted reader wrapped in NewGetObjectReader should be closed (#6383)"

This reverts commit 53a0bbeb5b.

* Revert "Change SelectAPI to use new GetObjectNInfo API (#6373)"

This reverts commit 5b05df215a.

* Revert "Implement GetObjectNInfo object layer call (#6290)"

This reverts commit e6d740ce09.
2018-08-31 13:10:12 -07:00
Aditya Manthramurthy
e6d740ce09 Implement GetObjectNInfo object layer call (#6290)
This combines calling GetObjectInfo and GetObject while returning a
io.ReadCloser for the object's body. This allows the two operations to
be under a single lock, fixing a race between getting object info and
reading the object body.
2018-08-27 15:28:23 +05:30
wd256
ff29aed05d gcs: Translate S3 user-defined metadata prefix to/from GCS custom metadata prefix (#6270) 2018-08-14 11:53:39 -07:00
wd256
3ec4738955 gcs: Use Pager to iterate results in ListObjectsV1/V2 (#6162)
Fixes #6052
2018-07-18 21:49:16 +05:30
Krishnan Parthasarathi
a1ef90be52 gcs: Limit number of objects listed to max-keys (#6133) 2018-07-13 10:27:26 -07:00
Mike Scarlett
c310cbbe89 Update comments regarding GCS component count (#6131) 2018-07-06 17:07:11 -07:00
wd256
25f9b0bc3b Handle ListObjectsV2 start-after parameter in ObjectLayer (#6078) 2018-07-01 09:52:45 +05:30
Harshavardhana
92a6676a2f Avoid unnecessary logging for policy not found errors (#6104) 2018-06-29 06:30:10 -07:00
Annanay Agarwal
78abe5234e Add functionality to make cache limit configurable (#5991) 2018-06-25 10:24:12 -07:00
Harshavardhana
71c4ff9d10 Fix tabs/indents in help message (#5962) 2018-05-21 11:11:57 -07:00
Krishna Srinivas
9816264eed Support for ListObjectsV1 style marker for Azure gateway (#5856)
fixes #4948
2018-04-27 16:08:46 -07:00
Bala FA
0d52126023 Enhance policy handling to support SSE and WORM (#5790)
- remove old bucket policy handling
- add new policy handling
- add new policy handling unit tests

This patch brings support to bucket policy to have more control not
limiting to anonymous.  Bucket owner controls to allow/deny any rest
API.

For example server side encryption can be controlled by allowing
PUT/GET objects with encryptions including bucket owner.
2018-04-24 15:53:30 -07:00
wd256
846f3e8f59 Disable "chunked" uploading by the GCS client for objects smaller than the chunk size. (#5835)
By disabling chunked uploading when the object size is less than the chunk size,
memory is not allocated unnecessarily.
2018-04-19 16:15:14 -07:00
wd256
9f31da5d57 Fix PutObject/CopyObject with metadata for GCS gateway (#5828)
Make sure to apply standard headers such as Content-Type,
Content-Disposition and Content-Language to the correct
GCS object attributes during object upload and copy operations.

Fixes: #5800
2018-04-18 21:40:48 +05:30
Harshavardhana
41c31d10ca
Enable s3,gcs,manta,oss,b2 gateways as production ready (#5795) 2018-04-09 16:33:48 -07:00
kannappanr
f8a3fd0c2a
Create logger package and rename errorIf to LogIf (#5678)
Removing message from error logging
Replace errors.Trace with LogIf
2018-04-05 15:04:40 -07:00
poornas
98315b8e60 Update disk cache documentation (#5743) 2018-04-02 10:25:08 +05:30
Nitish Tiwari
328076f773
Fix help message to be uniform across env vars (#5735) 2018-03-30 01:59:21 +05:30
poornas
a3e806ed61 Add disk based edge caching support. (#5182)
This PR adds disk based edge caching support for minio server.

Cache settings can be configured in config.json to take list of disk drives,
cache expiry in days and file patterns to exclude from cache or via environment
variables MINIO_CACHE_DRIVES, MINIO_CACHE_EXCLUDE and MINIO_CACHE_EXPIRY

Design assumes that Atime support is enabled and the list of cache drives is
fixed.
 - Objects are cached on both GET and PUT/POST operations.
 - Expiry is used as hint to evict older entries from cache, or if 80% of cache
   capacity is filled.
 - When object storage backend is down, GET, LIST and HEAD operations fetch
   object seamlessly from cache.

Current Limitations
 - Bucket policies are not cached, so anonymous operations are not supported in
   offline mode.
 - Objects are distributed using deterministic hashing among list of cache
   drives specified.If one or more drives go offline, or cache drive
   configuration is altered - performance could degrade to linear lookup.

Fixes #4026
2018-03-28 14:14:06 -07:00
Harshavardhana
35e64573fa Add missing env doc for MINIO_DOMAIN (#5705) 2018-03-26 09:11:39 -07:00
kannappanr
bdb1a90720 Add context support for gateway b2, manta, gcs 2018-03-15 16:28:25 -07:00
Harshavardhana
4af89543cf Update minio-go dependencies to latest 5.0.0 release (#5640)
With following changes

- Add SSE and refactor encryption API (#942) <Andreas Auernhammer>
- add copyObject test changing metadata and preserving etag (#944) <Harshavardhana>
- Add SSE-C tests for multipart, copy, get range operations (#941) <Harshavardhana>
- Removing conditional check for notificationInfoCh in api-notication (#940) <Matthew Magaldi>
- Honor prefix parameter in ListBucketPolicies API (#929) <kannappanr>
- test for empty objects uploaded with SSE-C headers (#927) <kannappanr>
- Encryption headers should also be set during initMultipart (#930) <Harshavardhana>
- Add support for Content-Language metadata header (#928) <kannappanr>
- Fix check for duplicate notification configuration entries (#917) <kannappanr>
- allow OS to cleanup sockets in TIME_WAIT (#925) <Harshavardhana>
- Sign V2: Fix signature calculation in virtual host style (#921) <A. Elleuch>
- bucket policy: Support json string in Principal field (#919) <A. Elleuch>
- Fix copyobject failure for empty files (#918) <kannappanr>
- Add new constructor NewWithOptions to SDK (#915) <poornas>
- Support redirect headers to sign again with new Host header. (#829) <Harshavardhana>
- Fail in PutObject if invalid user metadata is passed <Harshavadhana>
- PutObjectOptions Header: Don't include invalid header <Isaac Hess>
- increase max retry count to 10 (#913) <poornas>
- Add new regions for Paris and China west. (#905) <Harshavardhana>
- fix s3signer to use req.Host header (#899) <Bartłomiej Nogaś>
2018-03-14 19:38:29 +05:30
Harshavardhana
0ea54c9858 Change CopyObject{Part} to single srcInfo argument (#5553)
Refactor such that metadata and etag are
combined to a single argument `srcInfo`.

This is a precursor change for #5544 making
it easier for us to provide encryption/decryption
functions.
2018-02-21 14:18:47 +05:30
Harshavardhana
b2b5056163 gateway/gcs: Remove unused storageEndpoint (#5556) 2018-02-20 15:07:31 +05:30
poornas
4f73fd9487 Unify gateway and object layer. (#5487)
* Unify gateway and object layer. Bring bucket policies into
object layer.
2018-02-09 15:19:30 -08:00
poornas
0bb6247056 Move nslocking from s3 layer to object layer (#5382)
Fixes #5350
2018-01-13 10:04:52 +05:30
Harshavardhana
eb7c690ea9 Support in-place upgrades of new minio binary and releases. (#4961)
This PR allows 'minio update' to not only shows update banner
but also allows for in-place upgrades.

Updates are done safely by validating the downloaded
sha256 of the binary.

Fixes #4781
2017-12-15 12:33:42 -08:00
Harshavardhana
eb2894233c Convert gateways into respective packages (#5200)
- Make azure gateway a package
- Make b2 gateway a package
- Make gcs gateway a package
- Make s3 gateway a package
- Make sia gateway a package
2017-12-05 17:58:09 -08:00