Commit Graph

198 Commits

Author SHA1 Message Date
Harshavardhana
dd2542e96c add codespell action (#18818)
Original work here, #18474,  refixed and updated.
2024-01-17 23:03:17 -08:00
Anis Eleuch
7705605b5a scanner: Add a config to disable short sleep between objects scan (#18734)
Add a hidden configuration under the scanner sub section to configure if
the scanner should sleep between two objects scan. The configuration has
only effect when there is no drive activity related to s3 requests or
healing.

By default, the code will keep the current behavior which is doing
sleep between objects.

To forcefully enable the full scan speed in idle mode, you can do this:

   `mc admin config set myminio scanner idle_speed=full`
2024-01-04 15:07:17 -08:00
Pedro Juarez
8f13c8c3bf Support to store browser config settings (#18631)
* csp_policy
* hsts_seconds
* hsts_include_subdomains
* hsts_preload
* referrer_policy
2024-01-01 08:36:33 -08:00
Harshavardhana
eba23bbac4 rename object_size -> block_size for cache subsystem (#18694) 2023-12-21 16:57:13 -08:00
Krishnan Parthasarathi
a50f26b7f5 Implement batch-expiration for objects (#17946)
Based on an initial PR from -
https://github.com/minio/minio/pull/17792

But fully completes it with newer finalized YAML spec.
2023-12-02 02:51:33 -08:00
jiuker
be02333529 feat: drive sub-sys to max timeout reload (#18501) 2023-11-27 09:15:06 -08:00
Shireesh Anjal
11dc723324 Pass SUBNET URL to console (#18503)
When minio runs with MINIO_CI_CD=on, it is expected to communicate
with the locally running SUBNET. This is happening in the case of MinIO
via call home functionality. However, the subnet-related functionality inside the
console continues to talk to the SUBNET production URL. Because of this,
the console cannot be tested with a locally running SUBNET.

Set the env variable CONSOLE_SUBNET_URL correctly in such cases. 
(The console already has code to use the value of this variable
as the subnet URL)
2023-11-24 09:59:35 -08:00
Harshavardhana
fba883839d feat: bring new HDD related performance enhancements (#18239)
Optionally allows customers to enable 

- Enable an external cache to catch GET/HEAD responses 
- Enable skipping disks that are slow to respond in GET/HEAD 
  when we have already achieved a quorum
2023-11-22 13:46:17 -08:00
Harshavardhana
6829ae5b13 completely remove drive caching layer from gateway days (#18217)
This has already been deprecated for close to a year now.
2023-10-11 21:18:17 -07:00
Aditya Manthramurthy
1c99fb106c Update to minio/pkg/v2 (#17967) 2023-09-04 12:57:37 -07:00
Harshavardhana
af564b8ba0 allow bootstrap to capture time-spent for each initializers (#17900) 2023-08-23 03:07:06 -07:00
Praveen raj Mani
0285df5a02 fix: prioritize audit_webhook and logger_webhook ENVs over the config KVS (#17783) 2023-08-03 02:47:07 -07:00
Praveen raj Mani
b94ab07c2f Honor global root CAs for kafka audit tls (#17481)
honor global root CAs for kafka audit tls
2023-06-21 10:50:40 -07:00
Aditya Manthramurthy
5a1612fe32 Bump up madmin-go and pkg deps (#17469) 2023-06-19 17:53:08 -07:00
jiuker
d749aaab69 fix: ignore existing target status when adding new targets (#17250) 2023-05-24 22:57:37 -07:00
Praveen raj Mani
57acacd5a7 Support persistent queue store for loggers (#17121) 2023-05-08 21:20:31 -07:00
Harshavardhana
901887e6bf feat: add lambda transformation functions target (#16507) 2023-03-07 08:12:41 -08:00
Aditya Manthramurthy
7777d3b43a Remove globalSTSTLSConfig (#16709) 2023-02-26 23:37:00 -08:00
Harshavardhana
aa8b9572b9 remove double ENABLED help output (#16528) 2023-02-03 05:52:52 -08:00
Harshavardhana
b4ef5ff294 remove unnecessary code checking for supported features (#16423) 2023-01-17 19:37:47 +05:30
Anis Elleuch
34167c51d5 trace: Add bootstrap tracing events (#16286) 2022-12-21 15:52:29 -08:00
Anis Elleuch
e57e946206 Do not save credentials in config.json (#16275) 2022-12-19 12:27:06 -08:00
Klaus Post
b4f71362e9 Avoid config migration on every startup (#16278) 2022-12-19 11:10:14 -08:00
Aditya Manthramurthy
9e6cc847f8 Add HTTP2 config option for policy plugin (#16225) 2022-12-13 14:28:48 -08:00
Aditya Manthramurthy
2d60bf8c50 Refactor HTTP transports (#16222) 2022-12-12 20:31:21 -08:00
Aditya Manthramurthy
a30cfdd88f Bump up madmin-go to v2 (#16162) 2022-12-06 13:46:50 -08:00
Harshavardhana
71133105d7 re-order the top-level config keys for priority (#16150) 2022-12-01 07:50:08 -08:00
Shireesh Anjal
98a67a3776 Improvements in logger and audit webhooks (#16102) 2022-11-28 08:03:26 -08:00
Harshavardhana
ec77d28e62 make subnet subsys dynamic and simplify callhome (#15927) 2022-10-27 00:20:01 -07:00
Harshavardhana
23b329b9df remove gateway completely (#15929) 2022-10-24 17:44:15 -07:00
Aditya Manthramurthy
64cf887b28 use LDAP config from minio/pkg to share with console (#15810) 2022-10-07 22:12:36 -07:00
Anis Elleuch
86bb48792c non-blocking initialization of bucket target notifications (#15571) 2022-09-27 17:23:28 -07:00
Harshavardhana
94dbb4a427 fix: generalize SC config and also skip healing sub-sys under SD (#15757) 2022-09-26 09:04:54 -07:00
Harshavardhana
9d6fddcfdf persist the non-default creds in config (#15711) 2022-09-21 16:14:47 -07:00
Aditya Manthramurthy
afbb63a197 Factor out external event notification funcs (#15574)
This change moves external event notification functionality into
`event-notification.go`. This simplifies notification related code.
2022-08-24 06:42:36 -07:00
Harshavardhana
1823ab6808 LDAP/OpenID must be initialized IAM Init() (#15491)
This allows for LDAP/OpenID to be non-blocking,
allowing for unreachable Identity targets to be
initialized in IAM.
2022-08-08 16:16:27 -07:00
Harshavardhana
7b9b7cef11 add license banner for GNU AGPLv3 (#15178)
Bonus: rewrite subnet re-use of Transport
2022-06-27 03:58:25 -07:00
Aditya Manthramurthy
7f629df4d5 Add generic function to retrieve config value with metadata (#15083)
`config.ResolveConfigParam` returns the value of a configuration for any
subsystem based on checking env, config store, and default value. Also returns info
about which config source returned the value.

This is useful to return info about config params overridden via env in the user
APIs. Currently implemented only for OpenID subsystem, but will be extended for
others subsequently.
2022-06-17 11:39:21 -07:00
Shireesh Anjal
4ce81fd07f Add periodic callhome functionality (#14918)
* Add periodic callhome functionality

Periodically (every 24hrs by default), fetch callhome information and
upload it to SUBNET.

New config keys under the `callhome` subsystem:

enable - Set to `on` for enabling callhome. Default `off`
frequency - Interval between callhome cycles. Default `24h`

* Improvements based on review comments

- Update `enableCallhome` safely
- Rename pctx to ctx
- Block during execution of callhome
- Store parsed proxy URL in global subnet config
- Store callhome URL(s) in constants
- Use existing global transport
- Pass auth token to subnetPostReq
- Use `config.EnableOn` instead of `"on"`

* Use atomic package instead of lock

* Use uber atomic package

* Use `Cancel` instead of `cancel`

Co-authored-by: Harshavardhana <harsha@minio.io>

Co-authored-by: Harshavardhana <harsha@minio.io>
Co-authored-by: Aditya Manthramurthy <donatello@users.noreply.github.com>
2022-06-06 16:14:52 -07:00
Aditya Manthramurthy
464b9d7c80 Add support for Identity Management Plugin (#14913)
- Adds an STS API `AssumeRoleWithCustomToken` that can be used to 
  authenticate via the Id. Mgmt. Plugin.
- Adds a sample identity manager plugin implementation
- Add doc for plugin and STS API
- Add an example program using go SDK for AssumeRoleWithCustomToken
2022-05-26 17:58:09 -07:00
Harshavardhana
fd46a1c3b3 fix: some races when accessing ldap/openid config globally (#14978) 2022-05-25 18:32:53 -07:00
Harshavardhana
9341201132 logger lock should be more granular (#14901)
This PR simplifies few things by splitting
the locks between audit, logger targets to
avoid potential contention between them.

any failures inside audit/logger HTTP
targets must only log to console instead
of other targets to avoid cyclical dependency.

avoids unneeded atomic variables instead
uses RWLock to differentiate a more common
read phase v/s lock phase.
2022-05-12 07:20:58 -07:00
Aditya Manthramurthy
83071a3459 Add support for Access Management Plugin (#14875)
- This change renames the OPA integration as Access Management Plugin - there is
nothing specific to OPA in the integration, it is just a webhook.

- OPA configuration is automatically migrated to Access Management Plugin and
OPA specific configuration is marked as deprecated.

- OPA doc is updated and moved.
2022-05-10 17:14:55 -07:00
Aditya Manthramurthy
2b7e75e079 Add OPA doc and remove deprecation marking (#14863) 2022-05-04 23:53:42 -07:00
Aditya Manthramurthy
0e502899a8 Add support for multiple OpenID providers with role policies (#14223)
- When using multiple providers, claim-based providers are not allowed. All
providers must use role policies.

- Update markdown config to allow `details` HTML element
2022-04-28 18:27:09 -07:00
Klaus Post
96adfaebe1 Make storage class config dynamic (#14791)
Updating the storage class is already thread safe, so we can do this safely.
2022-04-21 12:07:33 -07:00
Harshavardhana
0e3bafcc54 improve logs, fix banner formatting (#14456) 2022-03-03 13:21:16 -08:00
Shireesh Anjal
3934700a08 Make audit webhook and kafka config dynamic (#14390) 2022-02-24 09:05:33 -08:00
Shireesh Anjal
94d37d05e5 Apply dynamic config at sub-system level (#14369)
Currently, when applying any dynamic config, the system reloads and
re-applies the config of all the dynamic sub-systems.

This PR refactors the code in such a way that changing config of a given
dynamic sub-system will work on only that sub-system.
2022-02-22 10:59:28 -08:00
Shireesh Anjal
28f188e3ef Make logger webhook config dynamic (#14289)
It should not be required to restart the 
server after setting the logger webhook config.
2022-02-17 11:11:15 -08:00