Commit Graph

88 Commits

Author SHA1 Message Date
Aditya Manthramurthy
1c99fb106c
Update to minio/pkg/v2 (#17967) 2023-09-04 12:57:37 -07:00
Harshavardhana
c45bc32d98
skip disks under scanning when healing disks (#17822)
Bonus:

- avoid calling DiskInfo() calls when missing blocks
  instead heal the object using MRF operation.

- change the max_sleep to 250ms beyond that we will
  not stop healing.
2023-08-09 12:51:47 -07:00
Aditya Manthramurthy
5a1612fe32
Bump up madmin-go and pkg deps (#17469) 2023-06-19 17:53:08 -07:00
Anis Eleuch
9ef7eda33a
heal: Avoid objects created after the heal disk start time (#17323) 2023-05-31 13:10:45 -07:00
Harshavardhana
84f31ed45d
simplify MRF, converge it to regular healing (#17026) 2023-04-19 07:47:42 -07:00
Anis Eleuch
224d9a752f
fix: the race in healing tracker code (#17048) 2023-04-18 14:49:56 -07:00
Poorna
a9269cee29
heal: avoid logging version not found (#17031) 2023-04-13 19:45:52 -07:00
Harshavardhana
bfedea9bad
fix: disk healing should honor the right pool/set index (#16712) 2023-02-27 04:55:32 -08:00
Klaus Post
84bb7d05a9
fix: healing deadlocks and ordering (#16643) 2023-02-17 23:22:43 +05:30
Anis Elleuch
857674c3a0
heal: Do not mark buckets as done when there is no online disks (#16621) 2023-02-14 12:50:13 -08:00
Anis Elleuch
b1d98febfd
New disk healing goes through the healing workers (#16568) 2023-02-08 09:25:29 -08:00
Harshavardhana
d08e3cc895
add a way to avoid blocking queueHealTask() depending on caller (#16433) 2023-01-19 18:50:54 +05:30
Anis Elleuch
d98116559b
Use async healing in PutObject call (#16431) 2023-01-19 00:54:22 -08:00
Anis Elleuch
3039fd4519
Optimize background heal status to use LocalStorageInfo (#16414) 2023-01-17 05:02:00 +05:30
Aditya Manthramurthy
a30cfdd88f
Bump up madmin-go to v2 (#16162) 2022-12-06 13:46:50 -08:00
Harshavardhana
5a8df7efb3
re-implement StorageInfo to be a peer call (#16155) 2022-12-01 14:31:35 -08:00
Klaus Post
cc1d8f0057
Check for abandoned data when healing (#16122) 2022-11-28 10:20:55 -08:00
Krishnan Parthasarathi
96bfa77856
serialize updates to healing tracker (#15647)
When healing is parallelized by setting the ` _MINIO_HEAL_WORKERS` 
environment variable, multiple goroutines may race while updating the disk's 
healing tracker. This change serializes only these concurrent updates using a
channel. Note, the healing tracker is still not concurrency safe in other contexts.
2022-09-07 08:47:21 -07:00
Krishnan Parthasarathi
99fbfe2421
Add concurrency to healing objects on a fresh disk (#15575) 2022-08-25 13:07:15 -07:00
ebozduman
b57e7321e7
Replaces 'disk'=>'drive' visible to end user (#15464) 2022-08-04 16:10:08 -07:00
Harshavardhana
ae92521310
remove unnecessary nAgreed value in partial() func (#15242) 2022-07-07 13:45:34 -07:00
Anis Elleuch
42e2fd35d8
heal: Include dir markers when healing a fresh disk (#15158)
Directories markers are not healed when healing a new fresh disk. A
a proper fix would be moving object names encoding/decoding to erasure
object level but it is too late now since the object to set distribution is
calculated at a higher level.
2022-06-23 06:47:33 -07:00
Anis Elleuch
b3eda248a3
Parallelize new disks healing of different erasure sets (#15112)
- Always reformat all disks when a new disk is detected, this will
  ensure new uploads to be written in new fresh disks
- Always heal all buckets first when an erasure set started to be healed
- Use a lock to prevent two disks belonging to different nodes but in
  the same erasure set to be healed in parallel
- Heal different sets in parallel

Bonus:
- Avoid logging errUnformattedDisk when a new fresh disk is inserted but
  not detected by healing mechanism yet (10 seconds lag)
2022-06-21 07:53:55 -07:00
Harshavardhana
f1abb92f0c
feat: Single drive XL implementation (#14970)
Main motivation is move towards a common backend format
for all different types of modes in MinIO, allowing for
a simpler code and predictable behavior across all features.

This PR also brings features such as versioning, replication,
transitioning to single drive setups.
2022-05-30 10:58:37 -07:00
Anis Elleuch
16431d222c
heal: Enable periodic bitrot scan configuration (#14464) 2022-04-07 08:10:40 -07:00
Harshavardhana
0e3bafcc54
improve logs, fix banner formatting (#14456) 2022-03-03 13:21:16 -08:00
Harshavardhana
7ee2d1c339
fix: when healing log path when we give up (#14079) 2022-01-10 21:22:17 -08:00
Harshavardhana
f527c708f2
run gofumpt cleanup across code-base (#14015) 2022-01-02 09:15:06 -08:00
Harshavardhana
5f7e6d03ff
copy bucket slice to avoid skipping .minio.sys/buckets (#13912)
healing was skipping `.minio.sys/buckets` path so
essentially not healing `.usage.json` - fix this
by making a copy of `buckets` slice.
2021-12-15 09:18:09 -08:00
Harshavardhana
17fd71164c
retry disk replacement healing if listing fails (#13689)
listing can fail and it is allowed to be retried,
instead of returning right away return an error at
the end - heal the rest of the buckets and objects,
and when we are retrying skip the buckets that
are already marked done by using the tracked buckets.

fixes #12972
2021-11-19 08:46:47 -08:00
jiangfucheng
e1755275a0
resume heal from previous object instead of bucket after server restart (#13581) 2021-11-05 13:10:41 -07:00
Harshavardhana
a19e3bc9d9
add more dangling heal related tests (#13140)
also make sure that HealObject() never returns
'ObjectNotFound' or 'VersionNotFound' errors,
as those are meaningless and not useful for the
caller.
2021-09-02 20:56:13 -07:00
Harshavardhana
ed16ce9b73
add healing workers support to parallelize healing (#13081)
Faster healing as well as making healing more
responsive for faster scanner times.

also fixes a bug introduced in #13079, newly replaced
disks were not healing automatically.
2021-08-26 20:32:58 -07:00
Harshavardhana
c11a2ac396
refactor healing to remove certain structs (#13079)
- remove sourceCh usage from healing
  we already have tasks and resp channel

- use read locks to lookup globalHealConfig

- fix healing resolver to pick candidates quickly
  that need healing, without this resolver was
  unexpectedly skipping.
2021-08-26 14:06:04 -07:00
Harshavardhana
0559f46bbb
fix: make healObject() make non-blocking (#13071)
healObject() should be non-blocking to ensure
that scanner is not blocked for a long time,
this adversely affects performance of the scanner
and also affects the way usage is updated
subsequently.

This PR allows for a non-blocking behavior for
healing, dropping operations that cannot be queued
anymore.
2021-08-25 17:46:20 -07:00
Harshavardhana
85dfb4351c
fix: allow an entire set to be dropped (#13060)
proceed to heal the cluster when all the
drives in a set have failed, this is extremely
rare occurrence but even if it happens we allow
the cluster to be functional.
2021-08-24 12:43:57 -07:00
Anis Elleuch
7fb9301c03
heal: Return parity for storage classes in heal info API (#13038)
`mc admin heal` command will show servers/disks tolerance, for that
purpose, you need to know the number of parity disks for each storage
class.

Parity is always the same in all pools.
2021-08-23 08:50:35 -07:00
Anis Elleuch
39874b77ed
mrf: Avoid rare data race and more simplification (#12791)
This change avoids a rare data race and simplify the function that
returns MRF last activity information.
2021-07-26 08:00:59 -07:00
AlexHuang2021
df2871de53
fix: return error when listing fails to retry healing (#12765) 2021-07-22 12:14:44 -07:00
Anis Elleuch
b0b4696a64
heal: Add MRF metrics to background heal API response (#12398)
This commit gathers MRF metrics from 
all nodes in a cluster and return it to the caller. This will show information about the 
number of objects in the MRF queues 
waiting to be healed.
2021-07-15 22:32:06 -07:00
Harshavardhana
b4bf82c751
do not heal "backend-encrypted" out-of-band with migration (#12556)
backend-encrypted doesn't need to be explicitly healed anymore
since this file is deleted upon upgrade and migration to the
KMS based encrypted config/IAM credentials.
2021-06-23 12:09:10 -07:00
Harshavardhana
1f262daf6f
rename all remaining packages to internal/ (#12418)
This is to ensure that there are no projects
that try to import `minio/minio/pkg` into
their own repo. Any such common packages should
go to `https://github.com/minio/pkg`
2021-06-01 14:59:40 -07:00
Harshavardhana
81d5688d56
move the dependency to minio/pkg for common libraries (#12397) 2021-05-28 15:17:01 -07:00
Harshavardhana
1aa5858543
move madmin to github.com/minio/madmin-go (#12239) 2021-05-06 08:52:02 -07:00
Harshavardhana
069432566f update license change for MinIO
Signed-off-by: Harshavardhana <harsha@minio.io>
2021-04-23 11:58:53 -07:00
Harshavardhana
78e867e145
ignore healing .trash, .metacache amd .multipart paths (#11725) 2021-03-07 09:38:31 -08:00
Harshavardhana
d971061305
use listPathRaw for HealObjects() instead of expensive WalkVersions() (#11675) 2021-03-06 09:25:48 -08:00
Klaus Post
fa9cf1251b
Imporve healing and reporting (#11312)
* Provide information on *actively* healing, buckets healed/queued, objects healed/failed.
* Add concurrent healing of multiple sets (typically on startup).
* Add bucket level resume, so restarts will only heal non-healed buckets.
* Print summary after healing a disk is done.
2021-03-04 14:36:23 -08:00
Harshavardhana
9171d6ef65
rename all references from crawl -> scanner (#11621) 2021-02-26 15:11:42 -08:00
Anis Elleuch
f28b063091
heal: Use healDeleteDangling global const in self healing (#11579)
A small fix, use healDeleteDangling constant instead of 'true' in the
self-healing code.
2021-02-18 15:16:20 -08:00