Commit Graph

82 Commits

Author SHA1 Message Date
Harshavardhana a6e0ec4e6f
Add support converting non-inlined to inlined (#15444)
This is a feature to allow for inode compaction on
large clusters that use a lot of small files spread
across a large heirarchy.
2022-08-02 23:10:22 -07:00
Naveen d46cf50760
chore(deps): Included dependency review (#14958)
> Dependency Review GitHub Action in your repository to enforce dependency
> reviews on your pull requests.
> The action scans for vulnerable versions of dependencies introduced by package version
> changes in pull requests,
> and warns you about the associated security vulnerabilities.
> This gives you better visibility of what's changing in a pull request,
> and helps prevent vulnerabilities from being added to your repository.

https://docs.github.com/en/code-security/supply-chain-security/understanding-your-software-supply-chain/about-dependency-review#dependency-review-enforcement

Signed-off-by: Naveen <172697+naveensrinivasan@users.noreply.github.com>
2022-07-27 20:53:26 -07:00
Harshavardhana 4c6498d726
move all CI/CD to go1.18 (#15401) 2022-07-25 15:27:20 -07:00
Harshavardhana b6eb8dff64
Add decommission compression+encryption enabled tests (#15322)
update compression environment variables to follow
the expected sub-system style, however support fallback
mode.
2022-07-17 08:43:14 -07:00
Anis Elleuch 1cfa2e04bc
Add a github workflow test for root disk detection (#15267)
Use losetup to create fake disks, start a MinIO cluster, umount 
one disk, and fails if the mount point directory will have format.json
recreated. It should fail because the mount point directory will belong
to the root disk after unmount.
2022-07-13 16:29:44 -07:00
Harshavardhana b311abed31
decom IAM, Bucket metadata properly (#15220)
Current code incorrectly passed the
config asset object name while decommissioning,
make sure that we pass the right object name
to be hashed on the newer set of pools.

This PR fixes situations after a successful
decommission, the users and policies might go
missing due to wrong hashed set.
2022-07-04 14:02:54 -07:00
Aditya Manthramurthy addfa35d93
Add FIPS build to CI and add README.fips.md (#15038) 2022-06-04 18:25:37 -07:00
Aditya Manthramurthy 4629abd5a2
Add tests for Access Management Plugin (#14909) 2022-05-12 15:24:19 -07:00
Aditya Manthramurthy 0e502899a8
Add support for multiple OpenID providers with role policies (#14223)
- When using multiple providers, claim-based providers are not allowed. All
providers must use role policies.

- Update markdown config to allow `details` HTML element
2022-04-28 18:27:09 -07:00
Harshavardhana 507f993075
attempt to real resolve when there is a quorum failure on reads (#14613) 2022-04-20 12:49:05 -07:00
Harshavardhana 2a6a40e93b
enable go1.18.x builds (#14746) 2022-04-13 14:21:55 -07:00
Naveen 879de20edf
Set permissions for GitHub actions (#14693)
- Included permissions for the action. https://github.com/ossf/scorecard/blob/main/docs/checks.md#token-permissions

https://docs.github.com/en/actions/using-jobs/assigning-permissions-to-jobs

[Keeping your GitHub Actions and workflows secure Part 1: Preventing pwn requests](https://securitylab.github.com/research/github-actions-preventing-pwn-requests/)

> Restrict the GitHub token permissions only to the required ones; this way, even if the attackers will succeed in compromising your workflow, they won’t be able to do much.

https://www.legitsecurity.com/blog/github-privilege-escalation-vulnerability

Signed-off-by: naveensrinivasan <172697+naveensrinivasan@users.noreply.github.com>
2022-04-11 02:45:59 -07:00
Naveen c547a4d835
Pin actions to a full length commit SHA (#14590)
- Pinned actions by SHA https://github.com/ossf/scorecard/blob/main/docs/checks.md#pinned-dependencies
- Included permissions for the action. https://github.com/ossf/scorecard/blob/main/docs/checks.md#token-permissions

https://docs.github.com/en/actions/security-guides/security-hardening-for-github-actions#using-third-party-actions

Also, dependabot supports upgrades based on SHA.
2022-03-31 10:12:53 -07:00
Sergey Zhuk 3970204009
ci: Check for new go-version. Bump setup-go to v3 (#14598) 2022-03-25 08:56:04 -07:00
Andreas Auernhammer 9dbfd84c5b
CI: use `MINIO_KMS_SECRET_KEY` when verify healing (#14511)
This commit replaces the KMS / KES environment
variables with `MINIO_KMS_SECRET_KEY` when testing
healing on CI.

This change is necessary since KES `0.18.0` introduced
some API breaking changes and the healing tests run
a test (`verify-3604`) that requires an older MinIO
version (e.g. `2021-11-24T23-19-33Z`) which is not
able to parse a KES error as expected.

This commit allows the KES instance at `https://play.min.io:7373`
to get updated to newer versions.

Signed-off-by: Andreas Auernhammer <hi@aead.dev>
2022-03-09 10:48:29 -08:00
Harshavardhana f088e8960b
docs: turn-on more markdown rules and fix them (#14301) 2022-02-14 08:50:42 -08:00
Harshavardhana e3e0532613
cleanup markdown docs across multiple files (#14296)
enable markdown-linter
2022-02-11 16:51:25 -08:00
Harshavardhana 57fad9148c lock all issues and PRs from last 365 days 2022-01-29 19:27:43 -08:00
Eco e2a9949b16
Slight tweaks to SUBNET portion of the template (#14205) 2022-01-28 16:04:16 -08:00
Harshavardhana a8ba71edef update lock-threads to log output 2022-01-25 20:28:43 -08:00
Harshavardhana 171cedf0f0 change to do-not-close 2022-01-23 20:36:59 -08:00
Harshavardhana 27d8ef14f8
add github lock threads action (#14155) 2022-01-23 09:10:53 -08:00
Anis Elleuch 1278b0ec73
Add Subnet link in the issue template for urgency clause (#14131) 2022-01-20 09:37:40 -08:00
Harshavardhana b7c5e45fff
heal: isObjectDangling should return false when it cannot decide (#14053)
In a multi-pool setup when disks are coming up, or in a single pool
setup let's say with 100's of erasure sets with a slow network.

It's possible when healing is attempted on `.minio.sys/config`
folder, it can lead to healing unexpectedly deleting some policy
files as dangling due to a mistake in understanding when `isObjectDangling`
is considered to be 'true'.

This issue happened in commit 30135eed86
when we assumed the validMeta with empty ErasureInfo is considered
to be fully dangling. This implementation issue gets exposed when
the server is starting up.

This is most easily seen with multiple-pool setups because of the
disconnected fashion pools that come up. The decision to purge the
object as dangling is taken incorrectly prior to the correct state
being achieved on each pool, when the corresponding drive let's say
returns 'errDiskNotFound', a 'delete' is triggered. At this point,
the 'drive' comes online because this is part of the startup sequence
as drives can come online lazily.

This kind of situation exists because we allow (totalDisks/2) number
of drives to be online when the server is being restarted.

Implementation made an incorrect assumption here leading to policies
getting deleted.

Added tests to capture the implementation requirements.
2022-01-07 19:11:54 -08:00
Aditya Manthramurthy 0a224654c2
fix: progagation of service accounts for site replication (#14054)
- Only non-root-owned service accounts are replicated for now.
- Add integration tests for OIDC with site replication
2022-01-07 17:41:43 -08:00
Harshavardhana 0e3037631f
skip inconsistent shards if possible (#13945)
data shards were wrong due to a healing bug
reported in #13803 mainly with unaligned object
sizes.

This PR is an attempt to automatically avoid
these shards, with available information about
the `xl.meta` and actually disk mtime.
2021-12-21 10:08:26 -08:00
Harshavardhana e82a5c5c54
fix: site replication issues and add tests (#13861)
- deleting policies was deleting all LDAP
  user mapping, this was a regression introduced
  in #13567

- deleting of policies is properly sent across
  all sites.

- remove unexpected errors instead embed the real
  errors as part of the 500 error response.
2021-12-08 11:50:15 -08:00
Harshavardhana 92fdcafb66
add verification tests for ETag on replicated content (#13857) 2021-12-07 10:08:26 -08:00
Harshavardhana 4f3290309e Revert "disable CI/CD for draft PRs (#13784)"
This reverts commit 5a22f2cf0b.
2021-11-30 09:22:17 -08:00
Krishnan Parthasarathi 5a22f2cf0b
disable CI/CD for draft PRs (#13784) 2021-11-29 23:35:07 -08:00
Harshavardhana 91e0823ff0
allow service freeze/unfreeze on a setup (#13707)
an active running speedTest will reject all
new S3 requests to the server, until speedTest
is complete.

this is to ensure that speedTest results are
accurate and trusted.

Co-authored-by: Klaus Post <klauspost@gmail.com>
2021-11-23 12:02:16 -08:00
Harshavardhana c791de0e1e
re-implement pickValidInfo dataDir, move to quorum calculation (#13681)
dataDir loosely based on maxima is incorrect and does not
work in all situations such as disks in the following order

- xl.json migration to xl.meta there may be partial xl.json's
  leftover if some disks are not yet connected when the disk
  is yet to come up, since xl.json mtime and xl.meta is
  same the dataDir maxima doesn't work properly leading to
  quorum issues.

- its also possible that XLV1 might be true among the disks
  available, make sure to keep FileInfo based on common quorum
  and skip unexpected disks with the older data format.

Also, this PR tests upgrade from older to a newer release if the 
data is readable and matches the checksum.

NOTE: this is just initial work we can build on top of this to do further tests.
2021-11-21 10:41:30 -08:00
Aditya Manthramurthy 1e2fac054c
Add caching to CI jobs (#13712)
- Seems to be improving times for shorter jobs at least.

- Remove Go 1.16.x tests for IAM and replication
2021-11-19 16:18:23 -08:00
Aditya Manthramurthy 087c1b98dc
Add tests for OpenID STS creds and add to CI (#13638) 2021-11-11 11:23:30 -08:00
Harshavardhana 5acc8c0134
add multi-site replication tests (#13631) 2021-11-10 18:18:09 -08:00
Aditya Manthramurthy 1946922de3
Add CI for etcd IAM backend (#13614)
Runs when ETCD_SERVER env var is set
2021-11-09 09:25:13 -08:00
Harshavardhana 12e6907512
apply spelling checks for US locale (#13599) 2021-11-07 01:22:59 -08:00
Aditya Manthramurthy 01b9ff54d9
Add LDAP STS tests and workflow for CI (#13576)
Runs LDAP tests with openldap container on GH Actions
2021-11-04 08:16:30 -07:00
Aditya Manthramurthy 79a58e275c
fix: race in delete user functionality (#13547)
- The race happens with a goroutine that refreshes IAM cache data from storage.
- It could lead to deleted users re-appearing as valid live credentials.
- This change also causes CI to run tests without a race flag (in addition to
running it with).
2021-11-01 15:03:07 -07:00
Aditya Manthramurthy 900e584514
CI: Cancel in-progress jobs when a PR is updated (#13552)
- This should lead to faster results as jobs will be queued for shorter periods
when PRs are updated.

- Current behavior is that previously running CI jobs for an updated PR run to
completion needlessly, and cause new CI jobs to be queued.

Ref: https://docs.github.com/en/actions/learn-github-actions/workflow-syntax-for-github-actions#concurrency
2021-11-01 13:42:48 -07:00
Anis Elleuch f1cab828ee
fix: New disks healing should pick unformatted disks as well (#13054)
A recent regression caused new disks not being re-formatted. In the old
code, a disk needed be 'online' to be chosen to be formatted but the
disk has to be already formatted for XL storage IsOnline() function to
return true.

It is enough to check if XL storage is nil or not if we want to avoid
formatting root disks.

Co-authored-by: Anis Elleuch <anis@min.io>
2021-08-24 07:40:56 -07:00
Harshavardhana a75412440f remove flaky healing test 2021-08-23 08:01:36 -07:00
Harshavardhana 202d0b64eb
fix: enable go1.17 github ci/cd (#12997) 2021-08-18 18:35:22 -07:00
Harshavardhana 27e07e80bc
fix: accountInfo() error FS mode (#12734) 2021-07-17 01:17:35 -07:00
Harshavardhana e1870c7b7c
build things separately in separate jobs (#12533) 2021-06-18 12:08:33 -07:00
Harshavardhana cdeccb5510
feat: Deprecate embedded browser and import console (#12460)
This feature also changes the default port where
the browser is running, now the port has moved
to 9001 and it can be configured with

```
--console-address ":9001"
```
2021-06-17 20:27:04 -07:00
Harshavardhana 764721e2c6
add root_disk threshold detection (#12259)
as there is no automatic way to detect if there
is a root disk mounted on / or /var for the container
environments due to how the root disk information
is masked inside overlay root inside container.

this PR brings an environment variable to set
root disk size threshold manually to detect the
root disks in such situations.
2021-05-08 15:40:29 -07:00
Harshavardhana f7a87b30bf Revert "deprecate embedded browser (#12163)"
This reverts commit 736d8cbac4.

Bring contrib files for older contributions
2021-04-30 08:50:39 -07:00
Harshavardhana b3c8a1864f
fix: optimize ListBuckets for anonymous users (#12182)
anonymous users are never allowed to listBuckets(),
we do not need to further validate the policy, we can
simply reject if credentials are empty.
2021-04-28 21:37:02 -07:00
Harshavardhana 736d8cbac4
deprecate embedded browser (#12163)
https://github.com/minio/console takes over the functionality for the
future object browser development

Signed-off-by: Harshavardhana <harsha@minio.io>
2021-04-27 10:52:12 -07:00