Queue output items and reuse them.
Remove the unneeded type system in sql and just use the Go type system.
In best case this is more than an order of magnitude speedup:
```
BenchmarkSelectAll_1M-12 1 1841049400 ns/op 274299728 B/op 4198522 allocs/op
BenchmarkSelectAll_1M-12 14 84833400 ns/op 169228346 B/op 3146541 allocs/op
```
It looks like from implementation point of view fastjson
parser pool doesn't behave the same way as expected
when dealing many `xl.json` from multiple disks.
The fastjson parser pool usage ends up returning incorrect
xl.json entries for checksums, with references pointing
to older entries. This led to the subtle bug where checksum
info is duplicated from a previous xl.json read of a different
file from different disk.
This commit adds an admin API route and handler for
requesting status information about a KMS key.
Therefore, the client specifies the KMS key ID (when
empty / not set the server takes the currently configured
default key-ID) and the server tries to perform a dummy encryption,
re-wrap and decryption operation. If all three succeed we know that
the server can access the KMS and has permissions to generate, re-wrap
and decrypt data keys (policy is set correctly).
The change now is to ensure that we take custom URL as
well for updating the deployment, this is required for
hotfix deliveries for certain deployments - other than
the community release.
This commit changes the previous work d65a2c6725
with newer set of requirements.
Also deprecates PeerUptime()
Add API to set policy mapping for a user or group
Contains a breaking Admin APIs change.
- Also enforce all applicable policies
- Removes the previous /set-user-policy API
Bump up peerRESTVersion
Add get user info API to show groups of a user
Without explicit conversion to UTC() from Unix
time the zone information is lost, this leads
to XML marshallers marshaling the time into
a wrong format.
This PR fixes the compatibility issue with AWS STS
API by keeping Expiration format close to ISO8601
or RFC3339
Fixes#8041
This change adds admin APIs and IAM subsystem APIs to:
- add or remove members to a group (group addition and deletion is
implicit on add and remove)
- enable/disable a group
- list and fetch group info
When checking if federation is necessary, the code compares
the SRV record stored in etcd against the list of endpoints
that the MinIO server is exposing. If there is an intersection
in this list the request is forwarded.
The SRV record includes both the host and the port, but the
intersection check previously only looked at the IP address. This
would prevent federation from working in situations where the endpoint
IP is the same for multiple MinIO servers. Some examples of where this
can occur are:
- running mulitiple copies of MinIO on the same host
- using multiple MinIO servers behind a NAT with port-forwarding
Golang proactively prints this error
`http: proxy error: context canceled`
when a request arrived to the current deployment and
redirected to another deployment in a federated setup.
Since this error can confuse users, this commit will
just hide it.
Allow renaming/editing a notification config. By replying with
a successful GetBucketNotification response, without checking
for any missing config ARN in targetList.
Fixes#7650
Related to #7982, this PR refactors the code
such that we validate the OPA or JWKS in a
common place.
This is also a refactor which is already done
in the new config migration change. Attempt
to avoid any network I/O during Unmarshal of
JSON from disk, instead do it later when
updating the in-memory data structure.
Problem: MinIO incorrectly appends DNS SRV records of buckets that have a prefix match with a given bucket. E.g bucket1 would incorrectly get bucket's DNS records too.
Solution: This fix ensures that we only add SRV records that match the key exactly
This PR is based off @sinhaashish's PR for object lifecycle
management, which includes support only for,
- Expiration of object
- Filter using object prefix (_not_ object tags)
N B the code for actual expiration of objects will be included in a
subsequent PR.
The SQL parser as it stands right now ignores alias for aggregate
result, e.g. `SELECT COUNT(*) AS thing FROM s3object` doesn't actually
return record like `{"thing": 42}`, it returns a record like `{"_1": 42}`.
Column alias for aggregate result is supported in AWS's S3 Select, so
this commit fixes that by respecting the `expr.As` in the expression.
Also improve test for S3 select
On top of testing a simple `SELECT` query, we want to test a few more
"advanced" queries (e.g. aggregation).
Convert existing tests into table driven tests[1], and add the new test
cases with "advanced" queries into them.
[1] - https://github.com/golang/go/wiki/TableDrivenTests
This API returns the information related to the self healing routine.
For the moment, it returns:
- The total number of objects that are scanned
- The last time when an item was scanned
This PR adds support for adding session policies
for further restrictions on STS credentials, useful
in situations when applications want to generate
creds for multiple interested parties with different
set of policy restrictions.
This session policy is not mandatory, but optional.
Fixes#7732
This PR fixes a security issue where an IAM user based
on his policy is granted more privileges than restricted
by the users IAM policy.
This is due to an issue of prefix based Matcher() function
which was incorrectly matching prefix based on resource
prefixes instead of exact match.
With these changes we are now able to peak performances
for all Write() operations across disks HDD and NVMe.
Also adds readahead for disk reads, which also increases
performance for reads by 3x.
This patch includes the following changes in event store interface
- Removes memory store. We will not persist events in memory anymore, if `queueDir` is not set.
- Orders the events before replaying to the broker.
common prefixes in bucket name if already created
are disallowed when etcd is configured due to the
prefix matching issue. Make sure that when we look
for bucket we are only interested in exact bucket
name not the prefix.
Healing scan used to read all objects parts to check for bitrot
checksum. This commit will add a quicker way of healing scan
by only checking if parts are actually present in disks or not.
- Also, switch to jstream to generate internal record representation
from CSV/JSON readers
- This fixes a bug in which JSON output objects have their keys
reversed from the order they are specified in the Select columns.
- Also includes a fix for tests.
Currently windows support was relying on Symlink as
a way to detect a drive, this doesn't work in latest
Windows 2016, fix this to use a proper mechanism by
using win32 APIs.
Additionally also add support for detecting bind mounts
on Linux.
- The events will be persisted in queueStore if `queueDir` is set.
- Else, if queueDir is not set events persist in memory.
The events are replayed back when the mqtt broker is back online.
