Commit Graph

21 Commits

Author SHA1 Message Date
Harshavardhana
a98a7fb1ad Implement XL layer - preliminary work. 2016-04-25 12:47:31 -07:00
karthic rao
6e372f83b4 Tests: object api multipart tests and bug fixes. 2016-04-25 10:39:28 -07:00
Harshavardhana
e9fba04b36 logging: Enable logging across storage fs layer. (#1367)
Adds log.Debugf at all the layer - fixes #1074
2016-04-24 00:36:00 -07:00
Harshavardhana
d63d17012d tests: Add API suite tests back for object api. (#1352) 2016-04-21 23:40:01 -07:00
Harshavardhana
4cf73caf02 api: Add diskInfo as part of StatVol and ListVols. (#1349)
It is the bucket and volumes which needs to have this
value rather than the DiskInfo API itself. Eventually
this can be extended to show disk usage per
Buckets/Volumes whenever we have that functionality.

For now since buckets/volumes are thinly provisioned
this is the right approach.
2016-04-21 20:07:47 -07:00
Harshavardhana
e0f8fed011 object: handle Error responses and handle errDiskFull. (#1331) 2016-04-19 02:42:10 -07:00
Harshavardhana
b47d722d8e fs: Fix filtering out valid paths from previous #1321 fix (#1323)
Fixes #1324
2016-04-17 12:00:23 -07:00
Harshavardhana
33633fd15d fs: Filter out valid paths and volnames. (#1321) 2016-04-16 19:33:29 -07:00
Harshavardhana
be002ac01e fs/object: Fix issues from review comments. 2016-04-16 17:57:14 -07:00
Harshavardhana
8457af5708 fs: Add proper volume and path validation. 2016-04-16 16:25:53 -07:00
Harshavardhana
30b0b4deba storage/server/client: Enable storage server, enable client storage. 2016-04-16 16:25:53 -07:00
Krishna Srinivas
3c48537f20 refactor: refactor code to separate fs into object-layer and fs layer. (#1305) 2016-04-16 16:25:53 -07:00
Bala FA
bea6f33b08 backend/fs: remove timer channel from scanMultipartDir() (#1310)
Previously scanMultipartDir() returns object info channel and timer
channel where timer channel is used to check whether object info
channel is alive or not.  This causes a race condition that timeout
may occur while object info channel in use.

This patch fixes the issue by removing timer channel and uses object
info channel directly where each object info has End bool field
indicates whether received object info is end or not.
2016-04-11 19:00:08 -07:00
Harshavardhana
33cd910d3a backend/fs: More cleanup and start using checkBuckeArg. (#1306)
backend/fs: More cleanup and start using checkBucketArg.
2016-04-08 17:13:16 -07:00
Harshavardhana
3fcc60de91 Move the files and rename some functions.
- Rename dir.go as 'fs-multipart-dir.go'
- Move the push/pop to fs-multipart.go and rename them as save/lookup.
- Rename objectInfo instances in fs-multipart as multipartObjInfo.
2016-04-05 12:26:19 -07:00
Harshavardhana
9632c94e7a Fix list objects test and remove all the old unnecessary files.
- Fix tests for new changes.
- Change Golang err as 'e' for the time being, before we bring in
  probe removal change.
- Remove old structs and temporary files.
2016-04-05 12:07:19 -07:00
Bala.FA
083e4e9479 backend/fs: Refactor multipart upload
This patch modifies multipart upload related functions as below

* New multipart upload call creates file
  EXPORT_DIR/.minio/BUCKET/PATH/TO/OBJECT/UPLOAD_ID.uploadid

* Put object part call creates file
  EXPORT_DIR/.minio/BUCKET/PATH/TO/OBJECT/UPLOAD_ID.PART_NUMBER.MD5SUM_STRING

* Abort multipart call removes all files matching
  EXPORT_DIR/.minio/BUCKET/PATH/TO/OBJECT/UPLOAD_ID.*

* Complete multipart call does
  1. creates a staging file
  EXPORT_DIR/.minio/BUCKET/PATH/TO/OBJECT/UPLOAD_ID.complete.TEMP_NAME
  then renames to
  EXPORT_DIR/.minio/BUCKET/PATH/TO/OBJECT/UPLOAD_ID.complete
  2. rename staging file
  EXPORT_DIR/.minio/BUCKET/PATH/TO/OBJECT/UPLOAD_ID.complete
  to EXPORT_DIR/BUCKET/PATH/TO/OBJECT
2016-04-05 22:22:29 +05:30
Harshavardhana
c69fdf0cf2 listObjects: Cleanup and naming conventions.
- Marker should be escaped outside in handlers.

- Delimiter should be handled outside in handlers.

- Add missing comments and change the function names.

- Handle case of 'maxKeys' when its set to '0', its a valid

  case and should be treated as such.
2016-04-04 19:55:07 -07:00
Krishna Srinivas
85ab1df5a8 listObjects: do not do stat during readdir()
* listObjects: improve response time by not doing stat during readDir() operation.

* listObjects: Add windows support.

* listObjects: Readdir() in batches to conserve memory. Add solaris build.

* listObjects: cleanup code.
2016-04-04 17:27:55 -07:00
Harshavardhana
0479d4976b objectAPI: Fix object API interface, remove unnecessary structs.
ObjectAPI changes.
```
ListObjects(bucket, prefix, marker, delimiter string, maxKeys int) (ListObjectsInfo, *probe.Error)
ListMultipartUploads(bucket, objectPrefix, keyMarker, uploadIDMarker, delimiter string, maxUploads int) (ListMultipartsInfo, *probe.Error)
ListObjectParts(bucket, object, uploadID string, partNumberMarker, maxParts int) (ListPartsInfo, *probe.Error)
CompleteMultipartUpload(bucket string, object string, uploadID string, parts []completePart) (ObjectInfo, *probe.Error)
```
2016-04-03 15:25:01 -07:00
Harshavardhana
efc80343e3 fs: Break fs package to top-level and introduce ObjectAPI interface.
ObjectAPI interface brings in changes needed for XL ObjectAPI layer.

The new interface for any ObjectAPI layer is as below

```
// ObjectAPI interface.
type ObjectAPI interface {
        // Bucket resource API.
        DeleteBucket(bucket string) *probe.Error
        ListBuckets() ([]BucketInfo, *probe.Error)
        MakeBucket(bucket string) *probe.Error
        GetBucketInfo(bucket string) (BucketInfo, *probe.Error)

        // Bucket query API.
        ListObjects(bucket, prefix, marker, delimiter string, maxKeys int) (ListObjectsResult, *probe.Error)
        ListMultipartUploads(bucket string, resources BucketMultipartResourcesMetadata) (BucketMultipartResourcesMetadata, *probe.Error)

        // Object resource API.
        GetObject(bucket, object string, startOffset int64) (io.ReadCloser, *probe.Error)
        GetObjectInfo(bucket, object string) (ObjectInfo, *probe.Error)
        PutObject(bucket string, object string, size int64, data io.Reader, metadata map[string]string) (ObjectInfo, *probe.Error)
        DeleteObject(bucket, object string) *probe.Error

        // Object query API.
        NewMultipartUpload(bucket, object string) (string, *probe.Error)
        PutObjectPart(bucket, object, uploadID string, partID int, size int64, data io.Reader, md5Hex string) (string, *probe.Error)
        ListObjectParts(bucket, object string, resources ObjectResourcesMetadata) (ObjectResourcesMetadata, *probe.Error)
        CompleteMultipartUpload(bucket string, object string, uploadID string, parts []CompletePart) (ObjectInfo, *probe.Error)
        AbortMultipartUpload(bucket, object, uploadID string) *probe.Error
}
```
2016-04-01 15:58:39 -07:00