MinIO

Commit Graph

Author	SHA1	Message	Date
Harshavardhana	95c65f4e8f	do not panic on rebalance during server restarts (#19563 ) This PR makes a feasible approach to handle all the scenarios that we must face to avoid returning "panic." Instead, we must return "errServerNotInitialized" when a bucketMetadataSys.Get() is called, allowing the caller to retry their operation and wait. Bonus fix the way data-usage-cache stores the object. Instead of storing usage-cache.bin with the bucket as `.minio.sys/buckets`, the `buckets` must be relative to the bucket `.minio.sys` as part of the object name. Otherwise, there is no way to decommission entries at `.minio.sys/buckets` and their final erasure set positions. A bucket must never have a `/` in it. Adds code to read() from existing data-usage.bin upon upgrade.	1 year ago
Harshavardhana	8b8be2695f	optimize mkdir calls to avoid base-dir `Mkdir` attempts (#18021 ) Currently we have IOPs of these patterns ``` [OS] os.Mkdir play.min.io:9000 /disk1 2.718µs [OS] os.Mkdir play.min.io:9000 /disk1/data 2.406µs [OS] os.Mkdir play.min.io:9000 /disk1/data/.minio.sys 4.068µs [OS] os.Mkdir play.min.io:9000 /disk1/data/.minio.sys/tmp 2.843µs [OS] os.Mkdir play.min.io:9000 /disk1/data/.minio.sys/tmp/d89c8ceb-f8d1-4cc6-b483-280f87c4719f 20.152µs ``` It can be seen that we can save quite Nx levels such as if your drive is mounted at `/disk1/minio` you can simply skip sending an `Mkdir /disk1/` and `Mkdir /disk1/minio`. Since they are expected to exist already, this PR adds a way for us to ignore all paths upto the mount or a directory which ever has been provided to MinIO setup.	2 years ago
Harshavardhana	7725425e05	fix: fork os.MkdirAll to optimize cases where parent exists (#15379 ) a/b/c/d/ where `a/b/c/` exists results in additional syscalls such as an Lstat() call to verify if the `a/b/c/` exists and its a directory. We do not need to do this on MinIO since the parent prefixes if exist, we can simply return success without spending additional syscalls. Also this implementation attempts to simply use Access() calls to avoid os.Stat() calls since the latter does memory allocation for things we do not need to use. Access() is simpler since we have a predictable structure on the backend and we know exactly how our path structures are.	3 years ago
Harshavardhana	f527c708f2	run gofumpt cleanup across code-base (#14015 )	4 years ago
Harshavardhana	c791de0e1e	re-implement pickValidInfo dataDir, move to quorum calculation (#13681 ) dataDir loosely based on maxima is incorrect and does not work in all situations such as disks in the following order - xl.json migration to xl.meta there may be partial xl.json's leftover if some disks are not yet connected when the disk is yet to come up, since xl.json mtime and xl.meta is same the dataDir maxima doesn't work properly leading to quorum issues. - its also possible that XLV1 might be true among the disks available, make sure to keep FileInfo based on common quorum and skip unexpected disks with the older data format. Also, this PR tests upgrade from older to a newer release if the data is readable and matches the checksum. NOTE: this is just initial work we can build on top of this to do further tests.	4 years ago
Harshavardhana	4d876d03e8	fix: do not fail upon faulty/non-writable drives gracefully start the server, if there are other drives available - print enough information for administrator to notice the errors in console. Bonus: for really large streams use larger buffer for writes.	4 years ago
Harshavardhana	069432566f	update license change for MinIO Signed-off-by: Harshavardhana <harsha@minio.io>	4 years ago
Ritesh H Shukla	6a2ed44095	fix: optionally enable tracing posix calls	4 years ago
Harshavardhana	df93102235	fix: unwrapping issues with os.Is* functions (#10949 ) reduces 3 stat calls, reducing the overall startup time significantly.	5 years ago
Anis Elleuch	f44a960dcd	tests: Fix one multi-delete test failure in Windows CI (#9602 ) There is a disparency of behavior under Linux & Windows about the returned error when trying to rename a non existant path. err := os.Rename("/path/does/not/exist", "/tmp/copy") Linux: isSysErrNotDir(err) = false os.IsNotExist(err) = true Windows: isSysErrNotDir(err) = true os.IsNotExist(err) = true ENOTDIR in Linux is returned when the destination path of the rename call contains a file in one of the middle segments of the path (e.g. /tmp/file/dst, where /tmp/file is an actual file not a directory) However, as shown above, Windows has more scenarios when it returns ENOTDIR. For example, when the source path contains an inexistant directory in its path. In that case, we want errFileNotFound returned and not errFileAccessDenied, so this commit will add a further check to close the disparency between Windows & Linux.	5 years ago
Harshavardhana	5d3d57c12a	Start using error wrapping with fmt.Errorf (#8588 ) Use fatih/errwrap to fix all the code to use error wrapping with fmt.Errorf()	6 years ago
Daryl Finlay	9389a55e5d	Cancel PutObjectPart on upload abort (#7940 ) Calling ListMultipartUploads fails if an upload is aborted while a part is being uploaded because the directory for the upload exists (since fsRenameFile ends up calling os.MkdirAll) but the meta JSON file doesn't. To fix this we make sure an upload hasn't been aborted during PutObjectPart by checking the existence of the directory for the upload while moving the temporary part file into it.	6 years ago
kannappanr	5ecac91a55	Replace Minio refs in docs with MinIO and links (#7494 )	6 years ago
Harshavardhana	c184038b6a	Add proper custom errors object creations (#7387 ) In scenario 1 ``` - bucket/object-prefix - bucket/object-prefix/object ``` Server responds with `XMinioParentIsObject` In scenario 2 ``` - bucket/object-prefix/object - bucket/object-prefix ``` Server responds with `XMinioObjectExistsAsDirectory` Fixes #6566	6 years ago
Oleg Kovalov	37de2dbd3b	simplifying if-else chains to switches (#6208 )	7 years ago
Harshavardhana	2dede2fdc2	Add reliable RemoveAll to handle racy situations (#6227 )	7 years ago
Harshavardhana	12f67d47f1	Fix a possible race during PutObject() (#5376 ) Under any concurrent removeObjects in progress might have removed the parents of the same prefix for which there is an ongoing putObject request. An inconsistent situation may arise as explained below even under sufficient locking. PutObject is almost successful at the last stage when a temporary file is renamed to its actual namespace at `a/b/c/object1`. Concurrently a RemoveObject is also in progress at the same prefix for an `a/b/c/object2`. To create the object1 at location `a/b/c` PutObject has to create all the parents recursively. ``` a/b/c - os.MkdirAll loops through has now created 'a/' and 'b/' about to create 'c/' a/b/c/object2 - at this point 'c/' and 'object2' are deleted about to delete b/ ``` Now for os.MkdirAll loop the expected situation is that top level parent 'a/b/' exists which it created , such that it can create 'c/' - since removeObject and putObject do not compete for lock due to holding locks at different resources. removeObject proceeds to delete parent 'b/' since 'c/' is not yet present, once deleted 'os.MkdirAll' would receive an error as syscall.ENOENT which would fail the putObject request. This PR tries to address this issue by implementing a safer/guarded approach where we would retry an operation such as `os.MkdirAll` and `os.Rename` if both operations observe syscall.ENOENT. Fixes #5254	8 years ago

17 Commits (b9f0e8c7124ddd65f65f08d2c603f76f39a7ea09)