* Add try block for opening new file for writing when downloading a new file to catch any read only file systems
* Move set file attributes to master function, incase there is a download failure, exit scope cannot set attributes on a file that is non existent
* When ~/.config/onedrive/ gets created for the first time, directory permissions are not set, thus, effective permissions of 'd---------' get applied. This then causes issues attempting to update the 'refresh_token' as permission is denied.
* When permission is denied, the file exception error is not correctly handled
* Set refresh_token file permissions
* When using WAL and SHM files, certain thresholds are used to automatically determine when WAL and SHM data is committed to the main database file. If these thresholds are not met, the data is not written. Before terminating the program, commit the data, and at the end of each monitor loop, commit the data
* Only try and write data to DB file and destroy variable if it was set to begin with
* Only try and destroy 'onedrive' variable if set, rather than set to null
* Adding support for authentication with single tenant custom applications in OneDrive Business/Enterprise accounts.
When using a custom single tenant application the /common/ part on the authentication URLs needs to be replaced
by the tenant name. This can be configured on the azure_tenant_id setting, but this entry was being ignored unless
a custom azure_ad_endpoint was also set.
* Updated documentation to mention that, when configuring for Single Tenant Applications,
both the Tenant Name and Application ID must be included on the config file.
Co-authored-by: abraunegg <alex.braunegg@gmail.com>
* Allow config to specify a tenant id for non-multi-tenant applications
* Add instructions on how to find your O365 tenant identifier for National Cloud deployments.
Co-authored-by: Ryan Brown <ryan.brown@avarint.com>
Co-authored-by: abraunegg <alex.braunegg@gmail.com>
* Support negative patterns in sync_list where a new line is matched in a negative manner to be excluded is prepended with a '!' or '-' character to exclude an otherwise included path
* Implement feature request to create a shareable file link. Default is to create a read-only anonymous link
Co-authored-by: Norbert Preining <norbert@preining.info>
* In certain scenarios quota details are restricted, thus, remaining free space tracking is not possible. Flag this is the case rather than report a zero value.
* In certain scenarios when using --single-directory (either very first run or with --resync) the parent path for the selected path will not be in the database, thus, all items that should be synced are not. Add another special case handler to handle this scenario.
* Fix 'Skipping uploading this new file as parent path is not in the database' when uploading to a Personal Shared Folder
* When syncing OneDrive Personal Shared Folders, add the driveId to the driveIdArray
* Rename function to be more representative of what function does
* The original intention here was that 'Sync with OneDrive is ...' is outputted when the sync starts / ends in 'verbose' mode, and not added to when not using any 'verbose'. What is occurring is, this is being added when MONITOR_LOG_QUIET threshold is met, which was not the intention. This now fixes when this message is outputted so that it is more reflective of when a sync is being done, when in --monitor mode
* Remove erroneous 'return' statement which could prematurely end processing all changes returned from OneDrive
* Improve --single-directory sync performance
This commit modifies code inside applyDifferences function,
the part responsible for deleting changes/moving them out
of the selected sync directories. This change makes an HTTP
request responsible for checking whether a changed item still
exists on OneDrive only be sent only if it may in fact influence
whether an item will be deleted from the synced folder or not
or just discarded.
This makes an enormous performance boost, because it limits
redundant HTTP requests that ask about changed items that will
be discarded or not.
Signed-off-by: Mateusz P-K <mateusz.kaplon@gmail.com>
* Update the 'Processing XXXX' output to display the full path, rather than just the item name from the database. In cases where you have the same file or directory name, having the full path where it is present is more informative
* In July 2020, Microsoft increased the maximum file size for uploads from 15GB / 20GB for Personal / Business Accounts to 100GB per file for all account types
* Added support for loading system configs if there's no user config file supplied. Uses application defaults if corrupt.
* Update documentation to indicate where config files can be stored & read from
* If a symbolic link is 'relative', it is relative to the path where the symbolic link exists, not the current working directory. Test if the failed symbolic link read is due to the link being relative, and advise accordingly.
* Strengthen bypass data preservation variable checks
* If a file is modified locally, and '--resync' is used, there is no way to tell if this file is a failed download or 'valid' thus potentially lead to a data loss scenario. In this case, and ONLY if --resync was issued, rename the local file for data preservation
* It may be desirable to see what options are being passed in to performSync() without enabling the full verbose debug logging. This has been useful when tracking down 'sync_list' / sync issue & other performance related items without having to enable full verbose debugging to see what is going on
* Having a 45 second default between monitor sync loops is too short. If there are a large number of files on the system (30K+) local file integrity checks take longer than 1 minute to complete, thus, it will seem like the system is 'forever' processing. Changing this to 300 seconds between sync loops is a more sane default.
* On some Linux distributions, the file system search tool locally modifies certain files after indexing. Even though the file contents has not changed, the file itself has, as the local modified timestamp has been updated. This then causes timestamp checks to be invalid. To ignore this in cases where this is occurring, configure 'bypass_data_preservation' to 'true' in the config file and local data protection rules will be ignored.
* Add warning to application startup if 'bypass_data_preservation' has been enabled
* Update database enforce check with conditional check on parent drive ID. If the parent is a shared folder, the parent ID will never be in the database as we are never provided that parent ID.
* Catch uphandled MonitorException when inotify throws an error
* Change monitor loop init full scan to false at start as fullScanOverride now correctly handled, negating need to true-up at application start
* Handle '100 Continue' response during upload
* Update handling of delta link being expired
* Update progress bar handling for uploads as #888 changed bar dynamics
* Change 'syncListConfigured' to 'syncListConfiguredFullScanOverride' as this is what this variable
* Update handling of fullScanRequired and syncListConfiguredFullScanOverride
* When a 429 or 504 is generated when querying for 'changes' inside a changeset bundle, dont restart scanning changes from the beginning, retry original request after a delay
* Change output for debugging
* Update setOneDriveFullScanTrigger to only be used when sync_list or skip_dir is used
* Unset oneDriveFullScanTrigger when it is currently set and no longer needed
* Before attempting to compute the complex path, check if the parent details are in the database
* Handle --dry-run and --no-remote-delete for shared folders
* Change from round to floor, so % bar increases when downloaded data is at X% not potentially under, thus leading to under reporting
* Add debug output when each % increase when downloading a file to assist with validating progress
* Start displaying ETA starting at 5% rather than 10%
* Use the configured 'fullScanRequired' every 10th loop to ensure that the local repository actually is in sync with OneDrive when sync_list is not used
* Use 'oneDriveFullScanTrigger' if it is set to trigger a full scan earlier than wait for next full sync if in monitor mode
* OneDrive does not support the uploading of zero-byte files when uploading via the OneDrive Web UI, however it is possible to upload then via the API as a 'new file'. If a local zero-byte file is modified, upload this as a 'new file' rather than attempt to upload as a normal modified file.
* Handle .nosync directive when downloading new files to an existing directory that is (was) in sync
* Handle the situation where a .nosync is created after some sort of partial download, so that the file, whilst now unwanted, is not downloaded
* Update --single-directory handling when using --dry-run
* Update --single-directory handling when using --resync and items to sync are in a child folder rather than root, leading to parent database issues (foreign key constraint)
* Update application output when database is corrupt and how to resolve by using --resync
* Handle --dry-run and --resync correctly - do not copy the database, it may be corrupt
* Always check if items-dryrun.sqlite3 still exists, remove it as it maybe corrupt when --dry-run is used
* Update the client identifier to 'd50ca740-c83f-4d1b-b616-12c519384f0c'
* Update User Agent identifier to comply with OneDrive traffic decoration requirements
* Provide 'config' file option to modify / update client identifier to override application default
* Update regex that extracts the response code from the response URI to avoid potentially generating a bad request to OneDrive, leading to a 'AADSTS9002313: Invalid request. Request is malformed or invalid.' response. With thanks to @zfil for fix.
* Overhaul OneDrive error response handling for 429 errors by utilising the HTTP response header Retry-After to configure the correct 'retry' window. If no retry window is set, defaults to 120 seconds.
* Update error response messaging when a 429 response is received
* Update how the original OneDrive query is retried when a 429 response is received
* Update the User Agent string to be more compliant with OneDrive decoration requirements to assist in avoiding 429 responses due to incorrect User Agent string being used. Updated to: ISV|abraunegg|OneDrive_Client_for_Linux/v%version_tag%
* If skip_dotfiles = true, moving files into a skipped .folder should not throw an error and should be removed from their previous location on OneDrive
* Add log message to indicate why delete operation is occurring on item which was moved
* Update application output to be clearer when just authorising the application and --synchronize or --monitor not passed in
* Update usage.md with updated authorize details and example
