1
0
mirror of https://github.com/minio/minio.git synced 2025-01-27 06:33:18 -05:00
Commit Graph

28 Commits

Author SHA1 Message Date
Anis Elleuch
1f92fc3fc0
Always check for root disks unless MINIO_CI_CD is set ()
The current code considers a pool with all root disks to be as part
of a testing environment even if there are other pools with mounted
disks. This will result to illegitimate writing in root disks.

Fix this by simplifing the logic: require MINIO_CI_CD in order to skip
root disk check.
2022-02-13 15:42:07 -08:00
Harshavardhana
2c6983a2f1
fix: use consistent ports in verify-healing ()
also use unique directories in setup testing.
2021-12-02 12:40:48 -08:00
Aditya Manthramurthy
25b5904b84
Enable sanity tests for internal IDP ()
Co-authored-by: Harshavardhana <harsha@minio.io>
2021-10-18 09:31:55 -07:00
ebozduman
b154581b65
fix: partially defined cred env vars cause "minio gateway s3" to fail ()
Both credential env vars not needed to start s3 gateway
2021-06-10 22:28:09 -07:00
Krishnan Parthasarathi
c829e3a13b Support for remote tier management ()
With this change, MinIO's ILM supports transitioning objects to a remote tier.
This change includes support for Azure Blob Storage, AWS S3 compatible object
storage incl. MinIO and Google Cloud Storage as remote tier storage backends.

Some new additions include:

 - Admin APIs remote tier configuration management

 - Simple journal to track remote objects to be 'collected'
   This is used by object API handlers which 'mutate' object versions by
   overwriting/replacing content (Put/CopyObject) or removing the version
   itself (e.g DeleteObjectVersion).

 - Rework of previous ILM transition to fit the new model
   In the new model, a storage class (a.k.a remote tier) is defined by the
   'remote' object storage type (one of s3, azure, GCS), bucket name and a
   prefix.

* Fixed bugs, review comments, and more unit-tests

- Leverage inline small object feature
- Migrate legacy objects to the latest object format before transitioning
- Fix restore to particular version if specified
- Extend SharedDataDirCount to handle transitioned and restored objects
- Restore-object should accept version-id for version-suspended bucket ()
- Check if remote tier creds have sufficient permissions
- Bonus minor fixes to existing error messages

Co-authored-by: Poorna Krishnamoorthy <poorna@minio.io>
Co-authored-by: Krishna Srinivas <krishna@minio.io>
Signed-off-by: Harshavardhana <harsha@minio.io>
2021-04-23 11:58:53 -07:00
Harshavardhana
60b0f2324e
storage write call path optimizations ()
- write in o_dsync instead of o_direct for smaller
  objects to avoid unaligned double Write() situations
  that may arise for smaller objects < 128KiB
- avoid fallocate() as its not useful since we do not
  use Append() semantics anymore, fallocate is not useful
  for streaming I/O we can save on a syscall
- createFile() doesn't need to validate `bucket` name
  with a Lstat() call since createFile() is only used
  to write at `minioTmpBucket`
- use io.Copy() when writing unAligned writes to allow
  usage of ReadFrom() from *os.File providing zero
  buffer writes().
2021-03-17 09:38:38 -07:00
Harshavardhana
cb0eaeaad8
feat: migrate to ROOT_USER/PASSWORD from ACCESS/SECRET_KEY () 2021-01-05 10:22:57 -08:00
Harshavardhana
4550ac6fff
fix: refactor locks to apply them uniquely per node ()
This refactor is done for few reasons below

- to avoid deadlocks in scenarios when number
  of nodes are smaller < actual erasure stripe
  count where in N participating local lockers
  can lead to deadlocks across systems.

- avoids expiry routines to run 1000 of separate
  network operations and routes per disk where
  as each of them are still accessing one single
  local entity.

- it is ideal to have since globalLockServer
  per instance.

- In a 32node deployment however, each server
  group is still concentrated towards the
  same set of lockers that partipicate during
  the write/read phase, unlike previous minio/dsync
  implementation - this potentially avoids send
  32 requests instead we will still send at max
  requests of unique nodes participating in a
  write/read phase.

- reduces overall chattiness on smaller setups.
2020-12-10 07:28:37 -08:00
Harshavardhana
9c53cc1b83
fix: heal multiple buckets in bulk ()
makes server startup, orders of magnitude
faster with large number of buckets
2020-12-05 13:00:44 -08:00
Harshavardhana
23e8390997
fix: Allow Walk to honor load balanced drives () 2020-10-01 20:24:34 -07:00
Harshavardhana
48919de301
fix: for defer'ed deleteObject use internal context () 2020-09-11 06:39:19 -07:00
Harshavardhana
4c9de098b0
heal buckets during init and make sure to wait on quorum ()
heal buckets properly during expansion, and make sure
to wait for the quorum properly such that healing can
be retried.
2020-05-06 14:25:05 -07:00
Harshavardhana
2ab8d5e47f Enable build verification with race () 2019-12-02 15:54:26 -08:00
Harshavardhana
5d65428b29
Handle localhost distributed setups properly ()
Fixes an issue reported by @klauspost and @vadmeste

This PR also allows users to expand their clusters
from single node XL deployment to distributed mode.
2019-11-26 11:42:10 -08:00
Harshavardhana
720442b1a2
Add lock expiry handler to expire state locks () 2019-11-25 16:39:43 -08:00
Harshavardhana
4e9de58675 Avoid pointer based copy, instead use Clone() ()
This PR adds functional test to test expanded
cluster syntax.
2019-11-21 17:54:51 +05:30
Harshavardhana
94e5cb7576
Migrate to go1.13 to avail all new features ()
Read more https://blog.golang.org/go1.13
2019-09-08 16:44:15 -07:00
Harshavardhana
b83413b167 Use GOPROXY to speed up builds ()
Read more here https://proxy.golang.org proposal 
for go1.13
2019-07-30 22:27:11 +05:30
kannappanr
5ecac91a55
Replace Minio refs in docs with MinIO and links () 2019-04-09 11:39:42 -07:00
Harshavardhana
313a3a286a Migrate to go1.12 to simplify our cmd/http package ()
Simplify the cmd/http package overall by removing
custom plain text v/s tls connection detection, by
migrating to go1.12 and choose minimum version
to be go1.12

Also remove all the vendored deps, since they
are not useful anymore.
2019-04-02 18:28:39 -07:00
Sidhartha Mani
6bc0de2a75 add go modules file and start running go 1.11 style builds () 2019-03-19 13:50:58 -07:00
Harshavardhana
bebaff269c Support IPv6 in minio command line ()
Fixes 
2018-12-14 13:07:46 +05:30
Harshavardhana
0e02328c98 Migrate config.json from config-dir to backend ()
This PR is the first set of changes to move the config
to the backend, the changes use the existing `config.json`
allows it to be migrated such that we can save it in on
backend disks.

In future releases, we will slowly migrate out of the
current architecture.

Fixes 
2018-08-15 10:11:47 +05:30
Harshavardhana
fb96779a8a Add large bucket support for erasure coded backend ()
This PR implements an object layer which
combines input erasure sets of XL layers
into a unified namespace.

This object layer extends the existing
erasure coded implementation, it is assumed
in this design that providing > 16 disks is
a static configuration as well i.e if you started
the setup with 32 disks with 4 sets 8 disks per
pack then you would need to provide 4 sets always.

Some design details and restrictions:

- Objects are distributed using consistent ordering
  to a unique erasure coded layer.
- Each pack has its own dsync so locks are synchronized
  properly at pack (erasure layer).
- Each pack still has a maximum of 16 disks
  requirement, you can start with multiple
  such sets statically.
- Static sets set of disks and cannot be
  changed, there is no elastic expansion allowed.
- Static sets set of disks and cannot be
  changed, there is no elastic removal allowed.
- ListObjects() across sets can be noticeably
  slower since List happens on all servers,
  and is merged at this sets layer.

Fixes 
Fixes 
Fixes 
Fixes 
Fixes 
Fixes 
Fixes 
Fixes 
Fixes 
Fixes 
Fixes 
2018-02-15 17:45:57 -08:00
Harshavardhana
3d0dced23c Remove go1.9 specific code for windows ()
Following fix https://go-review.googlesource.com/#/c/41834/ has
been merged upstream and released with go1.9.
2017-10-13 15:31:15 +05:30
Bala FA
670e3e0c8b verify: add minio gateway s3 setup for verification tests () 2017-10-04 11:16:39 -07:00
Bala FA
6d7df1d1cb Use mc functional tests for verifying build () 2017-09-20 13:22:05 -07:00
Bala FA
1bb3a03099 build: add verify check on make test ()
This patch adds basic tests for FS/XL/Distribute setups.
2017-09-12 16:56:33 -07:00