Commit Graph

289 Commits

Author SHA1 Message Date
Klaus Post
4972735507
Fix lint issues from v1.62.0 upgrade (#20633)
* Fix lint issues from v1.62.0 upgrade

* Fix xlMetaV2TrimData version checks.
2024-11-11 06:51:43 -08:00
Klaus Post
8d42f37e4b
Fix msgUnPath crash (#20614)
These are needed checks for the functions to be un-crashable with any input 
given to `msgUnPath` (tested with fuzzing).

Both conditions would result in a crash, which prevents that. Some 
additional upstream checks are needed.

Fixes #20610
2024-11-05 04:37:59 -08:00
Harshavardhana
1615920f48 fix typos reported in CI/CD 2024-11-04 11:06:02 -08:00
Harshavardhana
7f1e1713ab
use absolute path for binary checksum verification (#20487) 2024-09-26 08:03:08 -07:00
Ramon de Klein
3d152015eb
Use MinIO console v1.7.1 (#20465) 2024-09-20 18:18:54 -07:00
Harshavardhana
c28a4beeb7
multipart support etag and pre-read small objects (#20423) 2024-09-12 05:24:04 -07:00
Praveen raj Mani
261111e728
Kafka notify: support batched commits for queue store (#20377)
The items will be saved per target batch and will
be committed to the queue store when the batch is full

Also, periodically commit the batched items to the queue store
based on configured commit_timeout; default is 30s;

Bonus: compress queue store multi writes
2024-09-06 16:06:30 -07:00
jiuker
241be9709c
fix: jwt error overrwriten by nil public key (#20387) 2024-09-05 19:46:36 -07:00
Harshavardhana
c2e318dd40
remove mincache EOS related feature from upstream (#20375) 2024-09-03 11:23:41 -07:00
Harshavardhana
a5702f978e
remove requests deadline, instead just reject the requests (#20272)
Additionally set

 - x-ratelimit-limit
 - x-ratelimit-remaining

To indicate the request rates.
2024-08-16 01:43:49 -07:00
Klaus Post
3ffeabdfcb
Fix govet+staticcheck issues (#20263)
This is better: https://github.com/golang/go/issues/60529
2024-08-14 10:11:51 -07:00
Andreas Auernhammer
14876a4df1
ldap: use custom TLS cipher suites (#20221)
This commit replaces the LDAP client TLS config and
adds a custom list of TLS cipher suites which support
RSA key exchange (RSA kex).

Some LDAP server connections experience a significant slowdown
when these cipher suites are not available. The Go TLS stack
disables them by default. (Can be enabled via GODEBUG=tlsrsakex=1).

fixes https://github.com/minio/minio/issues/20214

With a custom list of TLS ciphers, Go can pick the TLS RSA key-exchange
cipher. Ref:
```
	if c.CipherSuites != nil {
		return c.CipherSuites
	}
	if tlsrsakex.Value() == "1" {
		return defaultCipherSuitesWithRSAKex
	}
```
Ref: https://cs.opensource.google/go/go/+/refs/tags/go1.22.5:src/crypto/tls/common.go;l=1017

Signed-off-by: Andreas Auernhammer <github@aead.dev>
2024-08-07 05:59:47 -07:00
Shubhendu
f944a42886
Removed user and group details from logs (#20072)
Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
2024-07-14 11:12:07 -07:00
Poorna
989c318a28
replication: make large workers configurable (#20077)
This PR also improves throttling by reducing tokens requested
from rate limiter based on available tokens to avoid exceeding
throttle wait deadlines
2024-07-12 07:57:31 -07:00
Taran Pelkey
f5d2fbc84c
Add DecodeDN and QuickNormalizeDN functions to LDAP config (#20076) 2024-07-11 18:04:53 -07:00
Harshavardhana
a8c6465f22
hide some deprecated fields from 'get' output (#20069)
also update wording on `subnet license="" api_key=""`
2024-07-10 13:16:44 -07:00
Taran Pelkey
6c6f0987dc
Add groups to policy entities (#20052)
* Add groups to policy entities

* update comment

---------

Co-authored-by: Harshavardhana <harsha@minio.io>
2024-07-10 11:41:49 -07:00
Austin Chang
5f64658faa
clarify error message for root user credential (#20043)
Signed-off-by: Austin Chang <austin880625@gmail.com>
2024-07-10 09:57:01 -07:00
Anis Eleuch
b433bf14ba
Add typos check to Makefile (#20051) 2024-07-08 14:39:49 -07:00
Harshavardhana
f365a98029
fix: hot-reloading STS credential policy documents (#20012)
* fix: hot-reloading STS credential policy documents
* Support Role ARNs hot load policies (#28)

---------

Co-authored-by: Anis Eleuch <vadmeste@users.noreply.github.com>
2024-06-28 16:17:22 -07:00
Taran Pelkey
168ae81b1f
Fix error when validating DN that is not under base DN (#19971) 2024-06-21 23:35:35 -07:00
Pedro Juarez
70078eab10
Fix browser UI animation (#19966)
Browse UI is not showing the animation because the default 
content-security-policy do not trust the file https://unpkg.com/detect-gpu@5.0.38/dist/benchmarks/d-apple.json 
the GPU library needs to identify if the web browser can play it.
2024-06-20 17:58:58 -07:00
Harshavardhana
8f93e81afb
change service account embedded policy size limit (#19840)
Bonus: trim-off all the unnecessary spaces to allow
for real 2048 characters in policies for STS handlers
and re-use the code in all STS handlers.
2024-05-30 11:10:41 -07:00
Harshavardhana
aad50579ba
fix: wire up ILM sub-system properly for help (#19836) 2024-05-30 01:14:58 -07:00
Harshavardhana
597a785253
fix: authenticate LDAP via actual DN instead of normalized DN (#19805)
fix: authenticate LDAP via actual DN instead of normalized DN

Normalized DN is only for internal representation, not for
external communication, any communication to LDAP must be
based on actual user DN. LDAP servers do not understand
normalized DN.

fixes #19757
2024-05-25 06:43:06 -07:00
Aditya Manthramurthy
5f78691fcf
ldap: Add user DN attributes list config param (#19758)
This change uses the updated ldap library in minio/pkg (bumped
up to v3). A new config parameter is added for LDAP configuration to
specify extra user attributes to load from the LDAP server and to store
them as additional claims for the user.

A test is added in sts_handlers.go that shows how to access the LDAP
attributes as a claim.

This is in preparation for adding SSH pubkey authentication to MinIO's SFTP
integration.
2024-05-24 16:05:23 -07:00
Shubhendu
7c7650b7c3
Add sufficient deadlines and countermeasures to handle hung node scenario (#19688)
Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
Signed-off-by: Harshavardhana <harsha@minio.io>
2024-05-22 16:07:14 -07:00
Harshavardhana
08d74819b6
handle racy updates to globalSite config (#19750)
```
==================
WARNING: DATA RACE
Read at 0x0000082be990 by goroutine 205:
  github.com/minio/minio/cmd.setCommonHeaders()

Previous write at 0x0000082be990 by main goroutine:
  github.com/minio/minio/cmd.lookupConfigs()
```
2024-05-16 16:13:47 -07:00
Harshavardhana
d3db7d31a3
fix: add deadlines for all synchronous REST callers (#19741)
add deadlines that can be dynamically changed via
the drive max timeout values.

Bonus: optimize "file not found" case and hung drives/network - circuit break the check and return right
away instead of waiting.
2024-05-15 09:52:29 -07:00
jiuker
01bfc78535
Optimization: reuse hashedSecret when LookupConfig (#19724) 2024-05-12 22:52:27 -07:00
Andreas Auernhammer
8b660e18f2
kms: add support for MinKMS and remove some unused/broken code (#19368)
This commit adds support for MinKMS. Now, there are three KMS
implementations in `internal/kms`: Builtin, MinIO KES and MinIO KMS.

Adding another KMS integration required some cleanup. In particular:
 - Various KMS APIs that haven't been and are not used have been
   removed. A lot of the code was broken anyway.
 - Metrics are now monitored by the `kms.KMS` itself. For basic
   metrics this is simpler than collecting metrics for external
   servers. In particular, each KES server returns its own metrics
   and no cluster-level view.
 - The builtin KMS now uses the same en/decryption implemented by
   MinKMS and KES. It still supports decryption of the previous
   ciphertext format. It's backwards compatible.
 - Data encryption keys now include a master key version since MinKMS
   supports multiple versions (~4 billion in total and 10000 concurrent)
   per key name.

Signed-off-by: Andreas Auernhammer <github@aead.dev>
2024-05-07 16:55:37 -07:00
Harshavardhana
1526e7ece3
extend server config.yaml to support per pool set drive count (#19663)
This is to support deployments migrating from a multi-pooled
wider stripe to lower stripe. MINIO_STORAGE_CLASS_STANDARD
is still expected to be same for all pools. So you can satisfy
adding custom drive count based pools by adjusting the storage
class value.

```
version: v2
address: ':9000'
rootUser: 'minioadmin'
rootPassword: 'minioadmin'
console-address: ':9001'
pools: # Specify the nodes and drives with pools
  -
    args:
        - 'node{11...14}.example.net/data{1...4}'
  -
    args:
        - 'node{15...18}.example.net/data{1...4}'
  -
    args:
        - 'node{19...22}.example.net/data{1...4}'
  -
    args:
        - 'node{23...34}.example.net/data{1...10}'
    set-drive-count: 6
```
2024-05-03 08:54:03 -07:00
Aditya Manthramurthy
0c855638de
fix: LDAP init. issue when LDAP server is down (#19619)
At server startup, LDAP configuration is validated against the LDAP
server. If the LDAP server is down at that point, we need to cleanly
disable LDAP configuration. Previously, LDAP would remain configured but
error out in strange ways because initialization did not complete
without errors.
2024-04-25 14:28:16 -07:00
Aditya Manthramurthy
62c3cdee75
fix: IAM LDAP access key import bug (#19608)
When importing access keys (i.e. service accounts) for LDAP accounts,
we are requiring groups to exist under one of the configured group base
DNs. This is not correct. This change fixes this by only checking for
existence and storing the normalized form of the group DN - we do not
return an error if the group is not under a base DN.

Test is updated to illustrate an import failure that would happen
without this change.
2024-04-25 08:50:16 -07:00
Aditya Manthramurthy
ae46ce9937
ldap: Normalize DNs when importing (#19528)
This is a change to IAM export/import functionality. For LDAP enabled
setups, it performs additional validations:

- for policy mappings on LDAP users and groups, it ensures that the
corresponding user or group DN exists and if so uses a normalized form
of these DNs for storage

- for access keys (service accounts), it updates (i.e. validates
existence and normalizes) the internally stored parent user DN and group
DNs.

This allows for a migration path for setups in which LDAP mappings have
been stored in previous versions of the server, where the name of the
mapping file stored on drives is not in a normalized form.

An administrator needs to execute:

`mc admin iam export ALIAS`

followed by

`mc admin iam import ALIAS /path/to/export/file`

The validations are more strict and returns errors when multiple
mappings are found for the same user/group DN. This is to ensure the
mappings stored by the server are unambiguous and to reduce the
potential for confusion.

Bonus **bug fix**: IAM export of access keys (service accounts) did not
export key name, description and expiration. This is fixed in this
change too.
2024-04-18 08:15:02 -07:00
Harshavardhana
0c31e61343
allow protection from invalid config values (#19460)
we have had numerous reports on some config
values not having default values, causing
features misbehaving and not having default
values set properly.

This PR tries to address all these concerns
once and for all.

Each new sub-system that gets added

- must check for invalid keys
- must have default values set
- must not "return err" when being saved into
  a global state() instead collate as part of
  other subsystem errors allow other sub-systems
  to independently initialize.
2024-04-10 18:10:30 -07:00
Aditya Manthramurthy
c9e9a8e2b9
fix: ldap: use validated base DNs (#19406)
This fixes a regression from #19358 which prevents policy mappings
created in the latest release from being displayed in policy entity
listing APIs.

This is due to the possibility that the base DNs in the LDAP config are
not in a normalized form and #19358 introduced normalized of mapping
keys (user DNs and group DNs). When listing, we check if the policy
mappings are on entities that parse as valid DNs that are descendants of
the base DNs in the config.

Test added that demonstrates a failure without this fix.
2024-04-04 11:36:18 -07:00
Anis Eleuch
95bf4a57b6
logging: Add subsystem to log API (#19002)
Create new code paths for multiple subsystems in the code. This will
make maintaing this easier later.

Also introduce bugLogIf() for errors that should not happen in the first
place.
2024-04-04 05:04:40 -07:00
jiuker
3d86ae12bc
feat: support EdDSA/Ed25519 for oss (#19397) 2024-04-02 16:02:35 -07:00
Harshavardhana
1c99597a06
update() inlineBlock settings properly in storageClass config (#19382) 2024-03-29 08:07:06 -07:00
Aditya Manthramurthy
7e45d84ace
ldap: improve normalization of DN values (#19358)
Instead of relying on user input values, we use the DN value returned by
the LDAP server.

This handles cases like when a mapping is set on a DN value
`uid=svc.algorithm,OU=swengg,DC=min,DC=io` with a user input value (with
unicode variation) of `uid=svc﹒algorithm,OU=swengg,DC=min,DC=io`. The
LDAP server on lookup of this DN returns the normalized value where the
unicode dot character `SMALL FULL STOP` (in the user input), gets
replaced with regular full stop.
2024-03-27 23:45:26 -07:00
Harshavardhana
3e38fa54a5
set max versions to be IntMax to avoid premature failures (#19360)
let users/customers set relevant values make default value
to be non-applicable.
2024-03-27 18:08:07 -07:00
Harshavardhana
0a56dbde2f
allow configuring inline shard size value (#19336) 2024-03-26 15:06:19 -07:00
jiuker
d7fb6fddf6
feat: add user specific redis auth (#19285) 2024-03-18 21:37:54 -07:00
Harshavardhana
d4aac7cd72
add deprecated expiry_workers to be ignored (#19289)
avoids error during upgrades such as
```
API: SYSTEM()
Time: 19:19:22 UTC 03/18/2024
DeploymentID: 24e4b574-b28d-4e94-9bfa-03c363a600c2
Error: Invalid api configuration: found invalid keys (expiry_workers=100 ) for 'api' sub-system, use 'mc admin config reset myminio api' to fix invalid keys (*fmt.wrapError)
      11: internal/logger/logger.go:260:logger.LogIf()
...
```
2024-03-18 15:25:32 -07:00
Harshavardhana
93fb7d62d8
allow dynamically changing max_object_versions per object (#19265) 2024-03-14 18:07:19 -07:00
Harshavardhana
ce1c640ce0
feat: allow retaining parity SLA to be configurable (#19260)
at scale customers might start with failed drives,
causing skew in the overall usage ratio per EC set.

make this configurable such that customers can turn
this off as needed depending on how comfortable they
are.
2024-03-14 03:38:33 -07:00
Krishnan Parthasarathi
b69bcdcdc4
Fix ilm config at startup (#19189)
Remove api.expiration_workers config setting which was inadvertently left behind. Per review comment 

https://github.com/minio/minio/pull/18926, expiration_workers can be configured via ilm.expiration_workers.
2024-03-04 18:50:24 -08:00
Krishnan Parthasarathi
a7577da768
Improve expiration of tiered objects (#18926)
- Use a shared worker pool for all ILM expiry tasks
- Free version cleanup executes in a separate goroutine
- Add a free version only if removing the remote object fails
- Add ILM expiry metrics to the node namespace
- Move tier journal tasks to expiryState
- Remove unused on-disk journal for tiered objects pending deletion
- Distribute expiry tasks across workers such that the expiry of versions of
  the same object serialized
- Ability to resize worker pool without server restart
- Make scaling down of expiryState workers' concurrency safe; Thanks
  @klauspost
- Add error logs when expiryState and transition state are not
  initialized (yet)
* metrics: Add missed tier journal entry tasks
* Initialize the ILM worker pool after the object layer
2024-03-01 21:11:03 -08:00
jiuker
0aae0180fb
feat: add userCredentials for nats (#19139) 2024-02-27 10:11:55 -08:00