MyFavMirrors/minio
1
0
Fork 0
mirror of https://github.com/minio/minio.git synced 2025-02-03 18:06:00 -05:00
Code Issues Packages Projects Releases Wiki Activity

S3 Select: Workaround java buffer size (#8312)

Browse Source 
Updates #7475

The Java implementation has a 128KB buffer and a message must be emitted before that is used. #7475 therefore limits the message size to 128KB. But up to 256 bytes are written to the buffer in each call. This means we must emit a message before shorter than 128KB.

Therefore we change the limit to 128KB minus 256 bytes.
This commit is contained in:
Klaus Post 2019-09-25 16:26:21 -07:00 committed by kannappanr
parent 704be85987
commit be313f1758
1 changed files with 7 additions and 2 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

9
pkg/s3select/message.go
View File

@ -62,7 +62,12 @@ var recordsHeader = []byte{
} }
const ( const (
maxRecordMessageLength = 128 * 1024 // Chosen for compatibility with AWS JAVA SDK // Chosen for compatibility with AWS JAVA SDK
// It has a a buffer size of 128K:
// https://github.com/aws/aws-sdk-java/blob/master/aws-java-sdk-s3/src/main/java/com/amazonaws/services/s3/internal/eventstreaming/MessageDecoder.java#L26
// but we must make sure there is always space to add 256 bytes:
// https://github.com/aws/aws-sdk-java/blob/master/aws-java-sdk-s3/src/main/java/com/amazonaws/services/s3/model/SelectObjectContentEventStream.java#L197
maxRecordMessageLength = (128 << 10) - 256
) )
var ( var (
@ -83,7 +88,7 @@ func newRecordsMessage(payload []byte) []byte {
} }
// payloadLenForMsgLen computes the length of the payload in a record // payloadLenForMsgLen computes the length of the payload in a record
// message given the length of the message. // message given the total length of the message.
func payloadLenForMsgLen(messageLength int) int { func payloadLenForMsgLen(messageLength int) int {
headerLength := len(recordsHeader) headerLength := len(recordsHeader)
payloadLength := messageLength - 4 - 4 - 4 - headerLength - 4 payloadLength := messageLength - 4 - 4 - 4 - headerLength - 4