* Improve application startup when using --monitor when there is no network connection to the OneDrive API
* Retry network connection test when using --monitor
* Only initialise application if OneDrive API is reachable
* Add to debug log output the configured URL's which will be used throughout the application to remove any ambiguity as to using incorrect URL's when making API calls.
* Add debugging output when adding paths and items to an inotify watch
* Only if a path is a directory, perform a path walk
* Catch Exception as error when permissions cause access exception
* If using specific AD endpoints, but using the default application ID, the redirect URL does not match. Check if the application_id before setting the redirect URL.
* Remove items from direct config string update, as they are configurable via config file, thus set option via config file is not used
* Add debugging and application config output for used options
* Add try block for opening new file for writing when downloading a new file to catch any read only file systems
* Move set file attributes to master function, incase there is a download failure, exit scope cannot set attributes on a file that is non existent
* When ~/.config/onedrive/ gets created for the first time, directory permissions are not set, thus, effective permissions of 'd---------' get applied. This then causes issues attempting to update the 'refresh_token' as permission is denied.
* When permission is denied, the file exception error is not correctly handled
* Set refresh_token file permissions
* When using WAL and SHM files, certain thresholds are used to automatically determine when WAL and SHM data is committed to the main database file. If these thresholds are not met, the data is not written. Before terminating the program, commit the data, and at the end of each monitor loop, commit the data
* Only try and write data to DB file and destroy variable if it was set to begin with
* Only try and destroy 'onedrive' variable if set, rather than set to null
* Adding support for authentication with single tenant custom applications in OneDrive Business/Enterprise accounts.
When using a custom single tenant application the /common/ part on the authentication URLs needs to be replaced
by the tenant name. This can be configured on the azure_tenant_id setting, but this entry was being ignored unless
a custom azure_ad_endpoint was also set.
* Updated documentation to mention that, when configuring for Single Tenant Applications,
both the Tenant Name and Application ID must be included on the config file.
Co-authored-by: abraunegg <alex.braunegg@gmail.com>
* Allow config to specify a tenant id for non-multi-tenant applications
* Add instructions on how to find your O365 tenant identifier for National Cloud deployments.
Co-authored-by: Ryan Brown <ryan.brown@avarint.com>
Co-authored-by: abraunegg <alex.braunegg@gmail.com>
* Support negative patterns in sync_list where a new line is matched in a negative manner to be excluded is prepended with a '!' or '-' character to exclude an otherwise included path
* Implement feature request to create a shareable file link. Default is to create a read-only anonymous link
Co-authored-by: Norbert Preining <norbert@preining.info>
* In certain scenarios quota details are restricted, thus, remaining free space tracking is not possible. Flag this is the case rather than report a zero value.
* In certain scenarios when using --single-directory (either very first run or with --resync) the parent path for the selected path will not be in the database, thus, all items that should be synced are not. Add another special case handler to handle this scenario.
* Fix 'Skipping uploading this new file as parent path is not in the database' when uploading to a Personal Shared Folder
* When syncing OneDrive Personal Shared Folders, add the driveId to the driveIdArray
* Rename function to be more representative of what function does
* The original intention here was that 'Sync with OneDrive is ...' is outputted when the sync starts / ends in 'verbose' mode, and not added to when not using any 'verbose'. What is occurring is, this is being added when MONITOR_LOG_QUIET threshold is met, which was not the intention. This now fixes when this message is outputted so that it is more reflective of when a sync is being done, when in --monitor mode
* Remove erroneous 'return' statement which could prematurely end processing all changes returned from OneDrive
* Improve --single-directory sync performance
This commit modifies code inside applyDifferences function,
the part responsible for deleting changes/moving them out
of the selected sync directories. This change makes an HTTP
request responsible for checking whether a changed item still
exists on OneDrive only be sent only if it may in fact influence
whether an item will be deleted from the synced folder or not
or just discarded.
This makes an enormous performance boost, because it limits
redundant HTTP requests that ask about changed items that will
be discarded or not.
Signed-off-by: Mateusz P-K <mateusz.kaplon@gmail.com>
* Update the 'Processing XXXX' output to display the full path, rather than just the item name from the database. In cases where you have the same file or directory name, having the full path where it is present is more informative
* In July 2020, Microsoft increased the maximum file size for uploads from 15GB / 20GB for Personal / Business Accounts to 100GB per file for all account types
* Added support for loading system configs if there's no user config file supplied. Uses application defaults if corrupt.
* Update documentation to indicate where config files can be stored & read from
* If a symbolic link is 'relative', it is relative to the path where the symbolic link exists, not the current working directory. Test if the failed symbolic link read is due to the link being relative, and advise accordingly.
* Strengthen bypass data preservation variable checks
* If a file is modified locally, and '--resync' is used, there is no way to tell if this file is a failed download or 'valid' thus potentially lead to a data loss scenario. In this case, and ONLY if --resync was issued, rename the local file for data preservation
* It may be desirable to see what options are being passed in to performSync() without enabling the full verbose debug logging. This has been useful when tracking down 'sync_list' / sync issue & other performance related items without having to enable full verbose debugging to see what is going on
* Having a 45 second default between monitor sync loops is too short. If there are a large number of files on the system (30K+) local file integrity checks take longer than 1 minute to complete, thus, it will seem like the system is 'forever' processing. Changing this to 300 seconds between sync loops is a more sane default.
* On some Linux distributions, the file system search tool locally modifies certain files after indexing. Even though the file contents has not changed, the file itself has, as the local modified timestamp has been updated. This then causes timestamp checks to be invalid. To ignore this in cases where this is occurring, configure 'bypass_data_preservation' to 'true' in the config file and local data protection rules will be ignored.
* Add warning to application startup if 'bypass_data_preservation' has been enabled
* Update database enforce check with conditional check on parent drive ID. If the parent is a shared folder, the parent ID will never be in the database as we are never provided that parent ID.
* Catch uphandled MonitorException when inotify throws an error
* Change monitor loop init full scan to false at start as fullScanOverride now correctly handled, negating need to true-up at application start
* Handle '100 Continue' response during upload
* Update handling of delta link being expired
* Update progress bar handling for uploads as #888 changed bar dynamics
* Change 'syncListConfigured' to 'syncListConfiguredFullScanOverride' as this is what this variable
* Update handling of fullScanRequired and syncListConfiguredFullScanOverride
* When a 429 or 504 is generated when querying for 'changes' inside a changeset bundle, dont restart scanning changes from the beginning, retry original request after a delay
* Change output for debugging
* Update setOneDriveFullScanTrigger to only be used when sync_list or skip_dir is used
* Unset oneDriveFullScanTrigger when it is currently set and no longer needed
* Before attempting to compute the complex path, check if the parent details are in the database
* Handle --dry-run and --no-remote-delete for shared folders
* Change from round to floor, so % bar increases when downloaded data is at X% not potentially under, thus leading to under reporting
* Add debug output when each % increase when downloading a file to assist with validating progress
* Start displaying ETA starting at 5% rather than 10%
