Commit Graph

79 Commits

Author SHA1 Message Date
Harshavardhana
533cd8d6df
fix: batch replication pull must preserve versionID (#17805)
batch replication pull must preserve versionID regardless
of destination bucket versioning configuration.

This is similar to the issue with decommissioning and rebalancing
2023-08-04 12:09:10 -07:00
Aditya Manthramurthy
bb6921bf9c
Send AuditLog via new middleware fn for admin APIs (#17632)
A new middleware function is added for admin handlers, including options
for modifying certain behaviors. This admin middleware:

- sets the handler context via reflection in the request and sends AuditLog
- checks for object API availability (skipping it if a flag is passed)
- enables gzip compression (skipping it if a flag is passed)
- enables header tracing (adding body tracing if a flag is passed)

While the new function is a middleware, due to the flags used for
conditional behavior modification, which is used in each route registration
call.

To try to ensure that no regressions are introduced, the following
changes were done mechanically mostly with `sed` and regexp:

- Remove defer logger.AuditLog in admin handlers
- Replace newContext() calls with r.Context()
- Update admin routes registration calls

Bonus: remove unused NetSpeedtestHandler

Since the new adminMiddleware function checks for object layer presence
by default, we need to pass the `noObjLayerFlag` explicitly to admin
handlers that should work even when it is not available. The following
admin handlers do not require it:

- ServerInfoHandler
- StartProfilingHandler
- DownloadProfilingHandler
- ProfileHandler
- SiteReplicationDevNull
- SiteReplicationNetPerf
- TraceHandler

For these handlers adminMiddleware does not check for the object layer
presence (disabled by passing the `noObjLayerFlag`), and for all other
handlers, the pre-check ensures that the handler is not called when the
object layer is not available - the client would get a
ErrServerNotInitialized and can retry later.

This `noObjLayerFlag` is added based on existing behavior for these
handlers only.
2023-07-13 14:52:21 -07:00
Klaus Post
ff5988f4e0
Reduce allocations (#17584)
* Reduce allocations

* Add stringsHasPrefixFold which can compare string prefixes, while ignoring case and not allocating.
* Reuse all msgp.Readers
* Reuse metadata buffers when not reading data.

* Make type safe. Make buffer 4K instead of 8.

* Unslice
2023-07-06 16:02:08 -07:00
guangwu
87b6fb37d6
chore: pkg imported more than once (#17444) 2023-06-26 09:21:29 -07:00
Aditya Manthramurthy
5a1612fe32
Bump up madmin-go and pkg deps (#17469) 2023-06-19 17:53:08 -07:00
Poorna
2920b0fc6d
allow specification of path/virtual style bucket lookup in batch replication (#17201) 2023-05-21 15:16:31 -07:00
Klaus Post
aaf1abc993
simplify HardLimitReader by using LimitReader for internal usage (#17218) 2023-05-16 13:14:37 -07:00
Poorna
e07c2ab868
Use hash.NewLimitReader for internal multipart calls (#17191) 2023-05-12 11:19:08 -07:00
Poorna
ec84bad882
batch replication now supports arbitrary S3 targets (#17113) 2023-05-02 22:52:35 -07:00
Poorna
98c26df53e
fix: allow past retention headers to be copied in batch replication (#17095) 2023-04-27 13:43:18 -07:00
Praveen raj Mani
72802a5972
Use 'minio/pkg/sync/errgroup' and 'minio/pkg/workers' (#17069) 2023-04-25 22:57:40 -07:00
Harshavardhana
6825bd7e75
fix: inlined objects don't need to honor long locks (#17039) 2023-04-17 12:16:37 -07:00
Poorna
3158f2d12e
Add support for batch key rotation (#16844) 2023-04-04 10:56:54 -07:00
Poorna
699a24f7e5
batch: validate versioning on src/tgt buckets (#16955) 2023-04-04 10:50:11 -07:00
Poorna
407c9ddcbf
feat: add batch replicate from remote MinIO to local cluster (#16922) 2023-03-31 10:48:36 -07:00
Harshavardhana
6c11dbffd5
add crash protection from backend modifications (#16846) 2023-03-20 09:08:42 -07:00
Poorna
440ad20c1d
Add support for batch job cancellation (#16843) 2023-03-17 23:42:43 -07:00
Harshavardhana
0c1f8b4e0f
add user-agent for all minio.Client usage (#16619) 2023-02-14 13:19:30 -08:00
Harshavardhana
15a75bd79b
ignore preconditionFailed error in batch replication (#16615) 2023-02-14 07:22:08 -08:00
Harshavardhana
31188e9327
add parallel workers in batch replication (#16609) 2023-02-13 12:07:58 -08:00
jiuker
a15b6f21b8
remove incorrect use of WaitGroup (#16596) 2023-02-12 20:59:45 -08:00
Aditya Manthramurthy
a30cfdd88f
Bump up madmin-go to v2 (#16162) 2022-12-06 13:46:50 -08:00
Harshavardhana
374abd1e7d
add filter support for tags and metadata in batch replication (#15885) 2022-10-18 21:22:21 -07:00
Harshavardhana
9fff315555
do not need to trace ignored objects (#15894) 2022-10-18 13:47:55 -07:00
Harshavardhana
c68910005b
validate bucket before attempting batch replication (#15861) 2022-10-15 11:58:31 -07:00
Harshavardhana
44097faec1
support deleteMarkers and all versions in batch replication (#15858) 2022-10-13 14:42:10 -07:00
Harshavardhana
97112c69be
fix: replication stats() to not crash under any situation (#15851)
Co-authored-by: Daniel Valdivia <18384552+dvaldivia@users.noreply.github.com>
2022-10-12 15:47:41 -07:00
Harshavardhana
e3cb0278ce
honor specified target prefix under batch replication (#15834) 2022-10-11 14:36:06 -07:00
Harshavardhana
2a13cc28f2 feat: implement support batch replication (#15554) 2022-10-05 23:00:43 -07:00