MinIO

Commit Graph

Author	SHA1	Message	Date
Sveinn	91e1487de4	Add LDAP public key authentication to SFTP (#19833 )	1 year ago
Poorna	e7aa26dc29	fix: allow DeleteObject unversioned objects with insufficient read quorum (#19581 ) Since the object is being permanently deleted, the lack of read quorum should not matter as long as sufficient disks are online to complete the deletion with parity requirements. If several pools have the same object with insufficient read quorum, attempt to delete object from all the pools where it exists	1 year ago
Klaus Post	650efc2e96	Fix listing in objects split across pools (#19227 ) Merging same-object - multiple versions from different pools would not always result in correct ordering. When merging keep inputs separate. ``` λ mc ls --versions local/testbucket ------ before ------ [2024-03-05 20:17:19 CET] 228B STANDARD 1f163718-9bc5-4b01-bff7-5d8cf09caf10 v3 PUT hosts [2024-03-05 20:19:56 CET] 19KiB STANDARD null v2 PUT hosts [2024-03-05 20:17:15 CET] 228B STANDARD 73c9f651-f023-4566-b012-cc537fdb7ce2 v1 PUT hosts ------ after ------ λ mc ls --versions local/testbucket [2024-03-05 20:19:56 CET] 19KiB STANDARD null v3 PUT hosts [2024-03-05 20:17:19 CET] 228B STANDARD 1f163718-9bc5-4b01-bff7-5d8cf09caf10 v2 PUT hosts [2024-03-05 20:17:15 CET] 228B STANDARD 73c9f651-f023-4566-b012-cc537fdb7ce2 v1 PUT hosts ```	1 year ago
Anis Eleuch	2e23e61a45	Add support of conf file to pass arguments and options (#18592 )	2 years ago
Harshavardhana	380a59520b	add missing testdata for benchmarking	2 years ago
Anis Elleuch	77dc99e71d	Do not use inline data size in xl.meta quorum calculation (#14831 ) * Do not use inline data size in xl.meta quorum calculation Data shards of one object can different inline/not-inline decision in multiple disks. This happens with outdated disks when inline decision changes. For example, enabling bucket versioning configuration will change the small file threshold. When the parity of an object becomes low, GET object can return 503 because it is not unable to calculate the xl.meta quorum, just because some xl.meta has inline data and other are not. So this commit will be disable taking the size of the inline data into consideration when calculating the xl.meta quorum. * Add tests for simulatenous inline/notinline object Co-authored-by: Anis Elleuch <anis@min.io>	3 years ago
Harshavardhana	cf94d1f1f1	do not crash readXLMetaNoData - if the `xl.meta` has incorrect content (#14538 ) ``` tmp = buf[want:] ``` Would potentially crash when `buf` is truncated for some reason and does not have the expected bytes, this is of course considered not normal and is an odd situation. But we do not need to crash here instead allow for errors to be returned and let callers handle the errors.	3 years ago
Klaus Post	0012ca8ca5	Fix inconsistent metadata after healing (#14125 ) When calculating signatures empty part ETags were not discarded, leading to a different signature compared to freshly created ones. This would mean that after a heal signature of the healed metadata would be different. Fixing the calculation of signature will make these consistent. Furthermore when inconsistent entries, with zero version ID, with the same mod times but different signatures, the one with the lowest signature would be picked for quorum check. Since this is 50/50, we fall back to a simple quorum count on all signatures. Each of these fixes by themselves will lead to quorum. Tests were added for regressions and expected outcomes.	4 years ago
Klaus Post	faf013ec84	Improve performance on multiple versions (#13573 ) Existing: ```go type xlMetaV2 struct { Versions []xlMetaV2Version `json:"Versions" msg:"Versions"` } ``` Serialized as regular MessagePack. ```go //msgp:tuple xlMetaV2VersionHeader type xlMetaV2VersionHeader struct { VersionID [16]byte ModTime int64 Type VersionType Flags xlFlags } ``` Serialize as streaming MessagePack, format: ``` int(headerVersion) int(xlmetaVersion) int(nVersions) for each version { binary blob, xlMetaV2VersionHeader, serialized binary blob, xlMetaV2Version, serialized. } ``` xlMetaV2VersionHeader is <= 30 bytes serialized. Deserialized struct can easily be reused and does not contain pointers, so efficient as a slice (single allocation) This allows quickly parsing everything as slices of bytes (no copy). Versions are always saved sorted by modTime, newest first. No more need to sort on load. * Allows checking if a version exists. * Allows reading single version without unmarshal all. * Allows reading latest version of type without unmarshal all. * Allows reading latest version without unmarshal of all. * Allows checking if the latest is deleteMarker by reading first entry. * Allows adding/updating/deleting a version with only header deserialization. * Reduces allocations on conversion to FileInfo(s).	4 years ago
Harshavardhana	e4e117faab	fix: enable xl.json to xl.meta only if legacy drive is found (#11255 ) another optimization is renameLegacyMetadata() never needs to validate bucket with os.Stat() again, leading to reduction in one extra syscall.	5 years ago
Klaus Post	6b14c4ab1e	Optimize decryptObjectInfo (#10726 ) `decryptObjectInfo` is a significant bottleneck when listing objects. Reduce the allocations for a significant speedup. https://github.com/minio/sio/pull/40 ``` λ benchcmp before.txt after.txt benchmark old ns/op new ns/op delta Benchmark_decryptObjectInfo-32 24260928 808656 -96.67% benchmark old MB/s new MB/s speedup Benchmark_decryptObjectInfo-32 0.04 1.24 31.00x benchmark old allocs new allocs delta Benchmark_decryptObjectInfo-32 75112 48996 -34.77% benchmark old bytes new bytes delta Benchmark_decryptObjectInfo-32 287694772 4228076 -98.53% ```	5 years ago
Klaus Post	a982baff27	ListObjects Metadata Caching (#10648 ) Design: https://gist.github.com/klauspost/025c09b48ed4a1293c917cecfabdf21c Gist of improvements: * Cross-server caching and listing will use the same data across servers and requests. * Lists can be arbitrarily resumed at a constant speed. * Metadata for all files scanned is stored for streaming retrieval. * The existing bloom filters controlled by the crawler is used for validating caches. * Concurrent requests for the same data (or parts of it) will not spawn additional walkers. * Listing a subdirectory of an existing recursive cache will use the cache. * All listing operations are fully streamable so the number of objects in a bucket no longer dictates the amount of memory. * Listings can be handled by any server within the cluster. * Caches are cleaned up when out of date or superseded by a more recent one.	5 years ago

12 Commits (b9f0e8c7124ddd65f65f08d2c603f76f39a7ea09)