* Expand on the edge case solution fix for #1586 to drop --single-directory qualifier in this very specific scenario, and add the DB tie record in all --resync --upload-only scenarios where the driveId is actually remote
* In fixing #1712 which was due to a OneDrive API change, the flagging that quota is being restricted was missed, thus, if this is truly being restricted may cause files to OneDrive Business Shared Folders be unable to be uploaded despite space being available.
* Resolve issue where --resync is not used and items.sqlite3 is not available at application startup, local files, whilst potentially newer, will not be preserved thus leading to a potential data loss scenario
* Add --disable-download-validation to allow downloading files from OneDrive where SharePoint and the OneDrive API mis-represents the values in the API as compared to the actual HTTP server response sent to the client. In the API JSON response we get a "size" value of X, but the HTTP Server Content Length reports a size of Y. When this occurs, the download will report as failed. This was seen as part understanding the cause of https://github.com/abraunegg/onedrive/discussions/1667
* In a `--resync --upload-only --single-directory 'dir'` scenario, and where the root 'dir' for --single-directory is a 'shared folder' we will not have the 'tie' DB entry created because of using --upload-only because we do not download the folder structure from OneDrive. As a result, query of the folder will fail and file uploads will fail.
Simulate the 'tie' DB record only when --resync --upload-only --single-directory 'dir' is being used, and if that folder is 'remote' and if we are using a 'personal' account.
The 'impact' of this however is, because of `--resync --upload-only` being used, local files are not in the local DB cache anymore, thus are treated as *new* files, thus, will be attempted to be re-uploaded.
* Fix getPathDetailsByDriveId query when using --dry-run and a nested path with --single-directory
* Fix fake response generation to use generated values for all account types to avoid DB lookup failures when using --dry-run for Personal account types
* When syncing OneDrive Business Shared Folders and using --single-directory, select correct driveId and itemId for the remote directory that needs to be synced
* Normally, the 'remoteItem' field will contain 'fileSystemInfo' however, if the user uses the 'Add Shortcut ..' option in OneDrive WebUI to create a 'link', this object, whilst remote, does not have 'fileSystemInfo' in the expected place, this leading to a application crash
* Add exception handling for when the API returns a 400 error when attempting to query a path on OneDrive. If the path generates a 'bad request' response, this needs to be correctly handled.
* Handle bad ShaprePoint data when the API does not return the expected data points when using those references to display what SharePoint sites are available when using --get-O365-drive-id
* Add a file check when using --upload-only --remove-source-files so that parental paths are added to the database, to allow child objects to be uploaded in this scenario
* Fix uploading documents to Shared Business Folders when shared folder exists on a SharePoint site due to Microsoft Sharepoint 'enrichment' of files
See: https://github.com/OneDrive/onedrive-api-docs/issues/935 for further details
* When in --monitor mode, and there are multiple driveId being used (shared folders), when a deletion event occurs, search the DB using all the known driveids rather than just the default one. When using just the default driveid, if the file to be deleted resides on another drive, the following error message is printed: 'Item cannot be deleted from OneDrive because it was not found in the local database' - which is not entirely accurate as the item is in the database, it is just not being searched for correctly.
* Use 'nextLink' value if present when searching for specific SharePoint site names
* Update error output to provide more details why an error occurred if a SharePoint site lacks the details we need to perform the match
* Revert the config & logpath change introduced via #1267 as the logging path should not be the value checked, the input path is what should be checked
* Fix logging output when processing downloaded files to remove potential ambiguity as to what events are actually occurring with individual file events post download
* After a file has been uploaded to SharePoint, use the response JSON and compare the quickXorHash to that of the local file before the upload. If there is a difference, Microsoft has modified the file post upload, thus we now need to download this 'updated' file. If there is no hash difference, we do not need to download the file.
* Add new DB query to return distinct driveId's from database
* Use all distinct drive ID's to query if path is in-sync with OneDrive before attempting to retreive the item file link from OneDrive
* Add debugging output when adding paths and items to an inotify watch
* Only if a path is a directory, perform a path walk
* Catch Exception as error when permissions cause access exception
* Allow config to specify a tenant id for non-multi-tenant applications
* Add instructions on how to find your O365 tenant identifier for National Cloud deployments.
Co-authored-by: Ryan Brown <ryan.brown@avarint.com>
Co-authored-by: abraunegg <alex.braunegg@gmail.com>
* Implement feature request to create a shareable file link. Default is to create a read-only anonymous link
Co-authored-by: Norbert Preining <norbert@preining.info>
* In certain scenarios quota details are restricted, thus, remaining free space tracking is not possible. Flag this is the case rather than report a zero value.
* In certain scenarios when using --single-directory (either very first run or with --resync) the parent path for the selected path will not be in the database, thus, all items that should be synced are not. Add another special case handler to handle this scenario.
* Fix 'Skipping uploading this new file as parent path is not in the database' when uploading to a Personal Shared Folder
* When syncing OneDrive Personal Shared Folders, add the driveId to the driveIdArray
* Rename function to be more representative of what function does
* Remove erroneous 'return' statement which could prematurely end processing all changes returned from OneDrive
* Improve --single-directory sync performance
This commit modifies code inside applyDifferences function,
the part responsible for deleting changes/moving them out
of the selected sync directories. This change makes an HTTP
request responsible for checking whether a changed item still
exists on OneDrive only be sent only if it may in fact influence
whether an item will be deleted from the synced folder or not
or just discarded.
This makes an enormous performance boost, because it limits
redundant HTTP requests that ask about changed items that will
be discarded or not.
Signed-off-by: Mateusz P-K <mateusz.kaplon@gmail.com>
* Update the 'Processing XXXX' output to display the full path, rather than just the item name from the database. In cases where you have the same file or directory name, having the full path where it is present is more informative
* In July 2020, Microsoft increased the maximum file size for uploads from 15GB / 20GB for Personal / Business Accounts to 100GB per file for all account types
* If a symbolic link is 'relative', it is relative to the path where the symbolic link exists, not the current working directory. Test if the failed symbolic link read is due to the link being relative, and advise accordingly.
* Strengthen bypass data preservation variable checks
* If a file is modified locally, and '--resync' is used, there is no way to tell if this file is a failed download or 'valid' thus potentially lead to a data loss scenario. In this case, and ONLY if --resync was issued, rename the local file for data preservation
* On some Linux distributions, the file system search tool locally modifies certain files after indexing. Even though the file contents has not changed, the file itself has, as the local modified timestamp has been updated. This then causes timestamp checks to be invalid. To ignore this in cases where this is occurring, configure 'bypass_data_preservation' to 'true' in the config file and local data protection rules will be ignored.
* Add warning to application startup if 'bypass_data_preservation' has been enabled
* Update database enforce check with conditional check on parent drive ID. If the parent is a shared folder, the parent ID will never be in the database as we are never provided that parent ID.
* Catch uphandled MonitorException when inotify throws an error
* Change monitor loop init full scan to false at start as fullScanOverride now correctly handled, negating need to true-up at application start
* Handle '100 Continue' response during upload
* Update handling of delta link being expired
* Update progress bar handling for uploads as #888 changed bar dynamics
* Change 'syncListConfigured' to 'syncListConfiguredFullScanOverride' as this is what this variable
* Update handling of fullScanRequired and syncListConfiguredFullScanOverride
* When a 429 or 504 is generated when querying for 'changes' inside a changeset bundle, dont restart scanning changes from the beginning, retry original request after a delay
* Change output for debugging
* Update setOneDriveFullScanTrigger to only be used when sync_list or skip_dir is used
* Unset oneDriveFullScanTrigger when it is currently set and no longer needed
* Before attempting to compute the complex path, check if the parent details are in the database
* Handle --dry-run and --no-remote-delete for shared folders
* Use the configured 'fullScanRequired' every 10th loop to ensure that the local repository actually is in sync with OneDrive when sync_list is not used
* Use 'oneDriveFullScanTrigger' if it is set to trigger a full scan earlier than wait for next full sync if in monitor mode