Manta is an Object Storage by [Joyent](https://www.joyent.com/)
This PR adds initial support for Manta. It is intended as non-production
ready so that feedback can be obtained.
This PR allows 'minio update' to not only shows update banner
but also allows for in-place upgrades.
Updates are done safely by validating the downloaded
sha256 of the binary.
Fixes#4781
- Adds a metadata argument to the CopyObjectPart API to facilitate
implementing encryption for copying APIs too.
- Update vendored minio-go - this version implements the
CopyObjectPart client API for use with the S3 gateway.
Fixes#4885
This change adds server-side-encryption support for HEAD, GET and PUT
operations. This PR only addresses single-part PUTs and GETs without
HTTP ranges.
Further this change adds the concept of reserved object metadata which is required
to make encrypted objects tamper-proof and provide API compatibility to AWS S3.
This PR adds the following reserved metadata entries:
- X-Minio-Internal-Server-Side-Encryption-Iv ('guarantees' tamper-proof property)
- X-Minio-Internal-Server-Side-Encryption-Kdf (makes Key-MAC computation negotiable in future)
- X-Minio-Internal-Server-Side-Encryption-Key-Mac (provides AWS S3 API compatibility)
The prefix `X-Minio_Internal` specifies an internal metadata entry which must not
send to clients. All client requests containing a metadata key starting with `X-Minio-Internal`
must also rejected. This is implemented by a generic-handler.
This PR implements SSE-C separated from client-side-encryption (CSE). This cannot decrypt
server-side-encrypted objects on the client-side. However, clients can encrypted the same object
with CSE and SSE-C.
This PR does not address:
- SSE-C Copy and Copy part
- SSE-C GET with HTTP ranges
- SSE-C multipart PUT
- SSE-C Gateway
Each point must be addressed in a separate PR.
Added to vendor dir:
- x/crypto/chacha20poly1305
- x/crypto/poly1305
- github.com/minio/sio
The reedsolomon library now avoids allocations during reconstruction.
This change exploits that to reduce memory allocs and GC preasure during
healing and reading.
- Region handling can now use region endpoints directly.
- All uploads are streaming no more large buffer needed.
- Major API overhaul for CopyObject(dst, src)
- Fixes bugs present in existing code for copying
- metadata replace directive CopyObject
- PutObjectPart doesn't require md5Sum and sha256
NOTE: This doesn't validate that minio will work fine
on these platforms and is tested. Since we do not validate
on these architectures this is to be treated as just a build fix.
Fixes#4858
This change provides new implementations of the XL backend operations:
- create file
- read file
- heal file
Further this change adds table based tests for all three operations.
This affects also the bitrot algorithm integration. Algorithms are now
integrated in an idiomatic way (like crypto.Hash).
Fixes#4696Fixes#4649Fixes#4359
* Prevent unnecessary verification of parity blocks while reading erasure
coded file.
* Update klauspost/reedsolomon and just only reconstruct data blocks while
reading (prevent unnecessary parity block reconstruction)
* Remove Verification of (all) reconstructed Data and Parity blocks since
in our case we are protected by bit rot protection. And even if the
verification would fail (essentially impossible) there is no way to
definitively say whether the data is still correct or not, so this call
make no sense for our use case.
This updates dependency for
- AWS S3 backend.
- pkg/madmin
```
- Relax isValidBucketName to allow reading existing buckets. (#708) (3 minutes ago) <Harshavardhana>
- For GCS the size limit of S3 is not useful. (#711) (3 days ago) <Harshavardhana>
- s3utils: Support AWS S3 US GovCloud endpoint. (#701) (3 days ago) <Harshavardhana>
- api: Always strip 80/443 port from host (#709) (3 days ago) <Anis Elleuch>
- Redact signature strings properly. (#706) (4 days ago) <Harshavardhana>
- api: Single putObject can use temporary file always. (#703) (6 days ago) <Harshavardhana>
- Spelling fix (#704) (7 days ago) <Jacob Taylor>
- api/encrypt: Get() on encrypted object should be a reader. (#699) (2 weeks ago) <Harshavardhana>
- get: Fix reading an object if its size is unknown (#694) (3 weeks ago) <Anis Elleuch>
- fixes#696 by updating the examples for put-encrypted-object and get-encrypted-object (#697) (3 weeks ago) <Tejay Cardon>
- fix InvalidAccessKeyId error according to amazon documentation (#692) (4 weeks ago) <samkevich>
- Add AWS S3 SSE-C example. (#689) (4 weeks ago) <Harshavardhana>
- According to RFC7232 Etag should be in quotes for If-Match. (#688) (5 weeks ago) <Harshavardhana>
- api: getReaderSize() should honor seeked file descriptors. (#681) (5 weeks ago) <Harshavardhana>
- tests: Use bytes.Repeat() when generating big data (#683) (5 weeks ago) <Anis Elleuch>
- api: Failed call retry with region only when http.StatusBadRequest. (#678) (5 weeks ago) <Harshavardhana>
- api: Add NewWithCredentials() (#646) (5 weeks ago) <Harshavardhana>
```
This implementation is similar to AMQP notifications:
* Notifications are published on a single topic as a JSON feed
* Topic is configurable, as is the QoS. Uses the paho.mqtt.golang
library for the mqtt connection, and supports connections over tcp
and websockets, with optional secure tls support.
* Additionally the minio server configuration has been bumped up
so mqtt configuration can be added.
* Configuration migration code is added with tests.
MQTT is an ISO standard M2M/IoT messaging protocol and was
originally designed for applications for limited bandwidth
networks. Today it's use is growing in the IoT space.
As a new configuration parameter is added, configuration version is
bumped up from 14 to 15.
The MySQL target's behaviour is identical to the PostgreSQL: rows are
deleted from the MySQL table on delete-object events, and are
created/updated on create/over-write events.
* Rename GenericArgs to AuthRPCArgs
* Rename GenericReply to AuthRPCReply
* Remove authConfig.loginMethod and add authConfig.ServiceName
* Rename loginServer to AuthRPCServer
* Rename RPCLoginArgs to LoginRPCArgs
* Rename RPCLoginReply to LoginRPCReply
* Version and RequestTime are added to LoginRPCArgs and verified by
server side, not client side.
* Fix data race in lockMaintainence loop.
This patch brings in changes from miniobrowser repo.
- Bucket policy UI and functionality fixes by @krishnasrinivas
- Bucket policy implementation by @balamurugana
- UI changes and new functionality changing password etc. @rushenn
- UI and new functionality for sharing URLs, deleting files
@rushenn and @krishnasrinivas.
- Other misc fixes by @vadmeste @brendanashworth
Ref #3229
After review with @abperiasamy we decided to remove all the unnecessary options
- MINIO_BROWSER (Implemented as a security feature but now deemed obsolete
since even if blocking access to MINIO_BROWSER, s3 API port is open)
- MINIO_CACHE_EXPIRY (Defaults to 72h)
- MINIO_MAXCONN (No one used this option and we don't test this)
- MINIO_ENABLE_FSMETA (Enable FSMETA all the time)
Remove --ignore-disks option - this option was implemented when XL layer
would initialize the backend disks and heal them automatically to disallow
XL accidentally using the root partition itself this option was introduced.
This behavior has been changed XL no longer automatically initializes
`format.json` a HEAL is controlled activity, so ignore-disks is not
useful anymore. This change also addresses the problems of our documentation
going forward and keeps things simple. This patch brings in reduction of
options and defaulting them to a valid known inputs. This patch also
serves as a guideline of limiting many ways to do the same thing.
* The user is required to specify a table name and database connection
information in the configuration file.
* INSERTs and DELETEs are done via prepared statements for speed.
* Assumes a table structure, and requires PostgreSQL 9.5 or above due to
the use of UPSERT.
* Creates the table if it does not exist with the given table name using
a query like:
CREATE TABLE myminio (
key varchar PRIMARY KEY,
value JSONB
);
* Vendors some required libraries.
- Using gjson for constructing xlMetaV1{} in realXLMeta.
- Test for parsing constructing xlMetaV1{} using gjson.
- Changes made since benchmarks showed 30-40% improvement in speed.
- Follow up comments in issue https://github.com/minio/minio/issues/2208
for more details.
- gjson parsing of parts from xl.json for listParts.
- gjson parsing of statInfo from xl.json for getObjectInfo.
- Vendorizing gjson dependency.
This change initializes rpc servers associated with disks that are
local. It makes object layer initialization on demand, namely on the
first request to the object layer.
Also adds lock RPC service vendorized minio/dsync
Current master has a regression 'mc policy <policy-type> alias/bucket/prefix'
does not work anymore, due to the way new minio-go changes do json marshalling.
This led to a regression on server side when a ``prefix`` is provided
policy is rejected as malformed from th server which is not the case with
AWS S3.
This patch uses the new ``minio-go/pkg/set`` package to address the
unmarshalling problems.
Fixes#2503
* Implement basic S3 notifications through queues
Supports multiple queues and three basic queue types:
1. NilQueue -- messages don't get sent anywhere
2. LogQueue -- messages get logged
3. AmqpQueue -- messages are sent to an AMQP queue
* api: Implement bucket notification.
Supports two different queue types
- AMQP
- ElasticSearch.
* Add support for redis
Some environments might disable access to `/dev/tty`, fall
back to '80' in such scenarios.
Move to 'cheggaaa/pb' package for better cross platform
support on fetching terminal width.
Fixes#1891
For files less than 'dataBlocks', erasure encoding would fail
with short data due to a bug in the implementation itself.
Relax the error return, even a single byte can be properly
erasure coded without issues.
Fixes#1413
This patch modifies multipart upload related functions as below
* New multipart upload call creates file
EXPORT_DIR/.minio/BUCKET/PATH/TO/OBJECT/UPLOAD_ID.uploadid
* Put object part call creates file
EXPORT_DIR/.minio/BUCKET/PATH/TO/OBJECT/UPLOAD_ID.PART_NUMBER.MD5SUM_STRING
* Abort multipart call removes all files matching
EXPORT_DIR/.minio/BUCKET/PATH/TO/OBJECT/UPLOAD_ID.*
* Complete multipart call does
1. creates a staging file
EXPORT_DIR/.minio/BUCKET/PATH/TO/OBJECT/UPLOAD_ID.complete.TEMP_NAME
then renames to
EXPORT_DIR/.minio/BUCKET/PATH/TO/OBJECT/UPLOAD_ID.complete
2. rename staging file
EXPORT_DIR/.minio/BUCKET/PATH/TO/OBJECT/UPLOAD_ID.complete
to EXPORT_DIR/BUCKET/PATH/TO/OBJECT
Paul Stack
Timon Wong
Harshavardhana
Aditya Manthramurthy
Frank Wessels
Andreas Auernhammer
Nitish Tiwari
Nitish Tiwari
Remco
splinter98
Krishna Srinivas
Bala FA
Anis Elleuch
Krishnan Parthasarathi
Karthic Rao
Mateusz Gajewski
Krishnan Parthasarathi
karthic rao
Harshavardhana
Krishna Srinivas