MyFavMirrors
/
minio
mirror of https://github.com/minio/minio.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
minio/mint
History
Harshavardhana e5d378931d
fix: delimiter based listing was broken without marker (#11136) 
with missing nextMarker with delimiter based listing,
top level prefixes beyond 4500 or max-keys value
wouldn't be sent back for client to ask for the next
batch.

reproduced at a customer deployment, create prefixes
as shown below

```
for year in $(seq 2017 2020)
do
    for month in {01..12}
    do for day in {01..31}
       do
           mc -q cp file myminio/testbucket/dir/day_id=$year-$month-$day/;
       done
    done
done
```

Then perform

```
aws s3api --profile minio --endpoint-url http://localhost:9000 list-objects \
    --bucket testbucket --prefix dir/ --delimiter / --max-keys 1000
```

You shall see missing NextMarker, this would disallow listing beyond max-keys
requested and also disallow beyond 4500 (maxKeyObjectList) prefixes being listed
because client wouldn't know the NextMarker available.

This PR addresses this situation properly by making the implementation
more spec compatible. i.e NextMarker in-fact can be either an object, a prefix
with delimiter depending on the input operation.

This issue was introduced after the list caching changes and has been present
for a while.
 		2020-12-19 09:36:04 -08:00
..
build fix: delimiter based listing was broken without marker (#11136) 2020-12-19 09:36:04 -08:00
run/core fix: regression introduced in aws-sdk-go tests 2020-12-18 21:14:27 -08:00
.gitignore Add mint tests into MinIO repo (#7886) 2019-07-09 18:32:39 -07:00
README.md deprecate/remove global WORM mode (#9436) 2020-04-24 16:37:05 -07:00
create-data-files.sh Add mint tests into MinIO repo (#7886) 2019-07-09 18:32:39 -07:00
entrypoint.sh Add mint tests into MinIO repo (#7886) 2019-07-09 18:32:39 -07:00
install-packages.list Remove nuget from docker mint build (#9864) 2020-06-17 14:47:57 -07:00
mint.sh fix: delimiter based listing was broken without marker (#11136) 2020-12-19 09:36:04 -08:00
postinstall.sh Fix mint hub.docker.com builds (#7908) 2019-07-11 11:45:57 -07:00
preinstall.sh fix: move to go1.15 (#10660) 2020-10-13 18:33:18 -07:00
release.sh Fix mint hub.docker.com builds (#7908) 2019-07-11 11:45:57 -07:00
remove-packages.list Remove nuget from docker mint build (#9864) 2020-06-17 14:47:57 -07:00

README.md

Mint Slack Docker Pulls

Mint is a testing framework for Minio object server, available as a docker image. It runs correctness, benchmarking and stress tests. Following are the SDKs/tools used in correctness tests.

  • awscli
  • aws-sdk-go
  • aws-sdk-php
  • aws-sdk-ruby
  • aws-sdk-java
  • mc
  • minio-go
  • minio-java
  • minio-js
  • minio-py
  • minio-dotnet
  • s3cmd

Running Mint

Mint is run by docker run command which requires Docker to be installed. For Docker installation follow the steps here.

To run Mint with Minio Play server as test target,

$ docker run -e SERVER_ENDPOINT=play.minio.io:9000 -e ACCESS_KEY=Q3AM3UQ867SPQQA43P2F \
             -e SECRET_KEY=zuf+tfteSlswRu7BJ86wekitnifILbZam1KYY3TG -e ENABLE_HTTPS=1 minio/mint

After the tests are run, output is stored in /mint/log directory inside the container. To get these logs, use docker cp command. For example

docker cp <container-id>:/mint/log /tmp/logs

Mint environment variables

Below environment variables are required to be passed to the docker container. Supported environment variables:

Environment variable Description Example
SERVER_ENDPOINT Endpoint of Minio server in the format HOST:PORT; for virtual style IP:PORT play.minio.io:9000
ACCESS_KEY Access key of access SERVER_ENDPOINT Q3AM3UQ867SPQQA43P2F
SECRET_KEY Secret Key of access SERVER_ENDPOINT zuf+tfteSlswRu7BJ86wekitnifILbZam1KYY3TG
ENABLE_HTTPS (Optional) Set 1 to indicate to use HTTPS to access SERVER_ENDPOINT. Defaults to 0 (HTTP) 1
MINT_MODE (Optional) Set mode indicating what category of tests to be run by values core, full. Defaults to core full
DOMAIN (Optional) Value of MINIO_DOMAIN environment variable used in Minio server myminio.com
ENABLE_VIRTUAL_STYLE (Optional) Set 1 to indicate virtual style access . Defaults to 0 (Path style) 1

Test virtual style access against Minio server

To test Minio server virtual style access with Mint, follow these steps:

  • Set a domain in your Minio server using environment variable MINIO_DOMAIN. For example export MINIO_DOMAIN=myminio.com.
  • Start Minio server.
  • Execute Mint against Minio server (with MINIO_DOMAIN set to myminio.com) using this command
$ docker run -e "SERVER_ENDPOINT=192.168.86.133:9000" -e "DOMAIN=minio.com"  \
	     -e "ACCESS_KEY=minio" -e "SECRET_KEY=minio123" -e "ENABLE_HTTPS=0" \
	     -e "ENABLE_VIRTUAL_STYLE=1" minio/mint

Mint log format

All test logs are stored in /mint/log/log.json as multiple JSON document. Below is the JSON format for every entry in the log file.

JSON field Type Description Example
name string Testing tool/SDK name "aws-sdk-php"
function string Test function name "getBucketLocation ( array $params = [] )"
args object (Optional) Key/Value map of arguments passed to test function {"Bucket":"aws-sdk-php-bucket-20341"}
duration int Time taken in milliseconds to run the test 384
status string one of PASS, FAIL or NA "PASS"
alert string (Optional) Alert message indicating test failure "I/O error on create file"
message string (Optional) Any log message "validating checksum of downloaded object"
error string Detailed error message including stack trace on status FAIL "Error executing \"CompleteMultipartUpload\" on ...

For Developers

Running Mint development code

After making changes to Mint source code a local docker image can be built/run by

$ docker build -t minio/mint . -f Dockerfile.mint
$ docker run -e SERVER_ENDPOINT=play.minio.io:9000 -e ACCESS_KEY=Q3AM3UQ867SPQQA43P2F \
             -e SECRET_KEY=zuf+tfteSlswRu7BJ86wekitnifILbZam1KYY3TG \
             -e ENABLE_HTTPS=1 -e MINT_MODE=full minio/mint:latest

Adding tests with new tool/SDK

Below are the steps need to be followed

  • Create new app directory under build and run/core directories.
  • Create install.sh which does installation of required tool/SDK under app directory.
  • Any build and install time dependencies should be added to install-packages.list.
  • Build time dependencies should be added to remove-packages.list for removal to have clean Mint docker image.
  • Add run.sh in app directory under run/core which execute actual tests.

Test data

Tests may use pre-created data set to perform various object operations on Minio server. Below data files are available under /mint/data directory.

File name Size
datafile-0-b 0B
datafile-1-b 1B
datafile-1-kB 1KiB
datafile-10-kB 10KiB
datafile-33-kB 33KiB
datafile-100-kB 100KiB
datafile-1-MB 1MiB
datafile-1.03-MB 1.03MiB
datafile-5-MB 5MiB
datafile-6-MB 6MiB
datafile-10-MB 10MiB
datafile-11-MB 11MiB
datafile-65-MB 65MiB
datafile-129-MB 129MiB