* In a `--resync --upload-only --single-directory 'dir'` scenario, and where the root 'dir' for --single-directory is a 'shared folder' we will not have the 'tie' DB entry created because of using --upload-only because we do not download the folder structure from OneDrive. As a result, query of the folder will fail and file uploads will fail.
Simulate the 'tie' DB record only when --resync --upload-only --single-directory 'dir' is being used, and if that folder is 'remote' and if we are using a 'personal' account.
The 'impact' of this however is, because of `--resync --upload-only` being used, local files are not in the local DB cache anymore, thus are treated as *new* files, thus, will be attempted to be re-uploaded.
* Fix getPathDetailsByDriveId query when using --dry-run and a nested path with --single-directory
* Fix fake response generation to use generated values for all account types to avoid DB lookup failures when using --dry-run for Personal account types
* When syncing OneDrive Business Shared Folders and using --single-directory, select correct driveId and itemId for the remote directory that needs to be synced
* Normally, the 'remoteItem' field will contain 'fileSystemInfo' however, if the user uses the 'Add Shortcut ..' option in OneDrive WebUI to create a 'link', this object, whilst remote, does not have 'fileSystemInfo' in the expected place, this leading to a application crash
* Add exception handling for when the API returns a 400 error when attempting to query a path on OneDrive. If the path generates a 'bad request' response, this needs to be correctly handled.
* Handle bad ShaprePoint data when the API does not return the expected data points when using those references to display what SharePoint sites are available when using --get-O365-drive-id
* Add a file check when using --upload-only --remove-source-files so that parental paths are added to the database, to allow child objects to be uploaded in this scenario
* Fix uploading documents to Shared Business Folders when shared folder exists on a SharePoint site due to Microsoft Sharepoint 'enrichment' of files
See: https://github.com/OneDrive/onedrive-api-docs/issues/935 for further details
* When in --monitor mode, and there are multiple driveId being used (shared folders), when a deletion event occurs, search the DB using all the known driveids rather than just the default one. When using just the default driveid, if the file to be deleted resides on another drive, the following error message is printed: 'Item cannot be deleted from OneDrive because it was not found in the local database' - which is not entirely accurate as the item is in the database, it is just not being searched for correctly.
* Use 'nextLink' value if present when searching for specific SharePoint site names
* Update error output to provide more details why an error occurred if a SharePoint site lacks the details we need to perform the match
* Revert the config & logpath change introduced via #1267 as the logging path should not be the value checked, the input path is what should be checked
* Fix logging output when processing downloaded files to remove potential ambiguity as to what events are actually occurring with individual file events post download
* After a file has been uploaded to SharePoint, use the response JSON and compare the quickXorHash to that of the local file before the upload. If there is a difference, Microsoft has modified the file post upload, thus we now need to download this 'updated' file. If there is no hash difference, we do not need to download the file.
* Add new DB query to return distinct driveId's from database
* Use all distinct drive ID's to query if path is in-sync with OneDrive before attempting to retreive the item file link from OneDrive
* Add debugging output when adding paths and items to an inotify watch
* Only if a path is a directory, perform a path walk
* Catch Exception as error when permissions cause access exception
* Allow config to specify a tenant id for non-multi-tenant applications
* Add instructions on how to find your O365 tenant identifier for National Cloud deployments.
Co-authored-by: Ryan Brown <ryan.brown@avarint.com>
Co-authored-by: abraunegg <alex.braunegg@gmail.com>
* Implement feature request to create a shareable file link. Default is to create a read-only anonymous link
Co-authored-by: Norbert Preining <norbert@preining.info>
* In certain scenarios quota details are restricted, thus, remaining free space tracking is not possible. Flag this is the case rather than report a zero value.
* In certain scenarios when using --single-directory (either very first run or with --resync) the parent path for the selected path will not be in the database, thus, all items that should be synced are not. Add another special case handler to handle this scenario.
* Fix 'Skipping uploading this new file as parent path is not in the database' when uploading to a Personal Shared Folder
* When syncing OneDrive Personal Shared Folders, add the driveId to the driveIdArray
* Rename function to be more representative of what function does
* Remove erroneous 'return' statement which could prematurely end processing all changes returned from OneDrive
* Improve --single-directory sync performance
This commit modifies code inside applyDifferences function,
the part responsible for deleting changes/moving them out
of the selected sync directories. This change makes an HTTP
request responsible for checking whether a changed item still
exists on OneDrive only be sent only if it may in fact influence
whether an item will be deleted from the synced folder or not
or just discarded.
This makes an enormous performance boost, because it limits
redundant HTTP requests that ask about changed items that will
be discarded or not.
Signed-off-by: Mateusz P-K <mateusz.kaplon@gmail.com>
* Update the 'Processing XXXX' output to display the full path, rather than just the item name from the database. In cases where you have the same file or directory name, having the full path where it is present is more informative
* In July 2020, Microsoft increased the maximum file size for uploads from 15GB / 20GB for Personal / Business Accounts to 100GB per file for all account types
* If a symbolic link is 'relative', it is relative to the path where the symbolic link exists, not the current working directory. Test if the failed symbolic link read is due to the link being relative, and advise accordingly.
* Strengthen bypass data preservation variable checks
* If a file is modified locally, and '--resync' is used, there is no way to tell if this file is a failed download or 'valid' thus potentially lead to a data loss scenario. In this case, and ONLY if --resync was issued, rename the local file for data preservation
* On some Linux distributions, the file system search tool locally modifies certain files after indexing. Even though the file contents has not changed, the file itself has, as the local modified timestamp has been updated. This then causes timestamp checks to be invalid. To ignore this in cases where this is occurring, configure 'bypass_data_preservation' to 'true' in the config file and local data protection rules will be ignored.
* Add warning to application startup if 'bypass_data_preservation' has been enabled
* Update database enforce check with conditional check on parent drive ID. If the parent is a shared folder, the parent ID will never be in the database as we are never provided that parent ID.
* Catch uphandled MonitorException when inotify throws an error
* Change monitor loop init full scan to false at start as fullScanOverride now correctly handled, negating need to true-up at application start
* Handle '100 Continue' response during upload
* Update handling of delta link being expired
* Update progress bar handling for uploads as #888 changed bar dynamics
* Change 'syncListConfigured' to 'syncListConfiguredFullScanOverride' as this is what this variable
* Update handling of fullScanRequired and syncListConfiguredFullScanOverride
* When a 429 or 504 is generated when querying for 'changes' inside a changeset bundle, dont restart scanning changes from the beginning, retry original request after a delay
* Change output for debugging
* Update setOneDriveFullScanTrigger to only be used when sync_list or skip_dir is used
* Unset oneDriveFullScanTrigger when it is currently set and no longer needed
* Before attempting to compute the complex path, check if the parent details are in the database
* Handle --dry-run and --no-remote-delete for shared folders
* Use the configured 'fullScanRequired' every 10th loop to ensure that the local repository actually is in sync with OneDrive when sync_list is not used
* Use 'oneDriveFullScanTrigger' if it is set to trigger a full scan earlier than wait for next full sync if in monitor mode
* OneDrive does not support the uploading of zero-byte files when uploading via the OneDrive Web UI, however it is possible to upload then via the API as a 'new file'. If a local zero-byte file is modified, upload this as a 'new file' rather than attempt to upload as a normal modified file.
* Handle .nosync directive when downloading new files to an existing directory that is (was) in sync
* Handle the situation where a .nosync is created after some sort of partial download, so that the file, whilst now unwanted, is not downloaded
* Update --single-directory handling when using --dry-run
* Update --single-directory handling when using --resync and items to sync are in a child folder rather than root, leading to parent database issues (foreign key constraint)
* Update the client identifier to 'd50ca740-c83f-4d1b-b616-12c519384f0c'
* Update User Agent identifier to comply with OneDrive traffic decoration requirements
* Provide 'config' file option to modify / update client identifier to override application default
* Overhaul OneDrive error response handling for 429 errors by utilising the HTTP response header Retry-After to configure the correct 'retry' window. If no retry window is set, defaults to 120 seconds.
* Update error response messaging when a 429 response is received
* Update how the original OneDrive query is retried when a 429 response is received
* Update the User Agent string to be more compliant with OneDrive decoration requirements to assist in avoiding 429 responses due to incorrect User Agent string being used. Updated to: ISV|abraunegg|OneDrive_Client_for_Linux/v%version_tag%
* If skip_dotfiles = true, moving files into a skipped .folder should not throw an error and should be removed from their previous location on OneDrive
* Add log message to indicate why delete operation is occurring on item which was moved
