Commit graph

594 commits

Author SHA1 Message Date
abraunegg 42b7945d9c
Support sync_list matching full path root wildcard with exclusions to simplify sync_list configuration (#1273)
* Support sync_list matching full path root wildcard with exclusions to simplify sync_list configuration
2021-02-14 08:55:49 +11:00
abraunegg cd7b7b0b68
Use correct path offset for sync_list exclusion matching (#1269)
* When processing sync_list entries, by default, if the path starts with '/', and offset of 1 is used to ensure correct path matching can be done. However, if the path starts with '!/' or '-/' to exclude a certain path, an offset of 1 was still being used, thus, exclusion matching was not occurring correctly. Use the correct offset, based on the 'sync_list' entry.
2021-02-14 06:43:55 +11:00
abraunegg 3ea16b619f
Revert the config & logpath change and check correct input entry (#1270)
* Revert the config & logpath change introduced via #1267 as the logging path should not be the value checked, the input path is what should be checked
2021-02-13 17:37:22 +11:00
abraunegg f37fdda836
Fix logging output when handing downloaded files (#1267)
* Fix logging output when processing downloaded files to remove potential ambiguity as to what events are actually occurring with individual file events post download
2021-02-11 04:07:45 +11:00
abraunegg 1bfd995153 Revert "Fix logging output when handing downloaded new files (#1265)"
This reverts commit 99a0f4bfed.
2021-02-10 13:33:32 +11:00
abraunegg 99a0f4bfed
Fix logging output when handing downloaded new files (#1265)
* Fix logging output when processing files to remove potential ambiguity as to what events are actually occuring with indivudual file events
2021-02-10 13:26:10 +11:00
abraunegg 66a172d13a
Correctly handle '~' when present in 'log_dir' configuration option (#1258)
* Handle '~' when present in 'log_dir' and expand based on user and shell environment  variables to ensure correct expansion.
2021-02-06 20:46:56 +11:00
abraunegg 2c253362bf
Improve upload handling of files for SharePoint sites (#1250)
* After a file has been uploaded to SharePoint, use the response JSON and compare the quickXorHash to that of the local file before the upload. If there is a difference, Microsoft has modified the file post upload, thus we now need to download this 'updated' file. If there is no hash difference, we do not need to download the file.
2021-02-01 16:46:12 +11:00
abraunegg 3b87602a81
Use correct driveId value to query for changes when using --single-directory (#1235)
* Use the driveId value rather than defaultDriveId when looking for changes when using --single-directory
2021-01-22 14:38:11 +11:00
abraunegg 6b0b6a66f7
Search all distinct drive id's rather than just default drive id for --get-file-link (#1230)
* Add new DB query to return distinct driveId's from database
* Use all distinct drive ID's to query if path is in-sync with OneDrive before attempting to retreive the item file link from OneDrive
2021-01-20 19:46:56 +11:00
abraunegg 15a84627e1
Support new file maximum upload size of 250GB (#1233)
* Support new file maximum upload size of 250GB
2021-01-20 09:25:59 +11:00
abraunegg cef00e74b2
Catch database assertion when item path cannot be calculated (#1217)
* Catch database assertion when item path cannot be calculated
2021-01-15 14:44:13 +11:00
abraunegg 96d3e68498
Add new config option to rate limit connection to OneDrive (#1210)
* Add new config option to rate limit connection to OneDrive
2021-01-04 08:28:34 +11:00
abraunegg be1757148f
Add upload speed metrics when files are uploaded (#1209)
* Add upload speed metrics when files are uploaded
* Clarify 'data to transfer' is what is needed to be downloaded from OneDrive
2021-01-03 12:20:43 +11:00
abraunegg 62a2b4f393
Fix application crash with --get-O365-drive-id when API response is restricted (#1198)
* Check OneDrive API response for required elements before attempting to use elements for next query
2020-12-23 15:25:59 +11:00
abraunegg 771a93242f
Fix application crash when calculating the path length due to invalid UTF characters (#1193)
* Fix application crash when calculating the path length due to invalid UTF characters
2020-12-21 07:27:27 +11:00
abraunegg ba94b92d8a
Improve application startup when using --monitor when there is no network connection to the OneDrive API (#1188)
* Improve application startup when using --monitor when there is no network connection to the OneDrive API
* Retry network connection test when using --monitor
* Only initialise application if OneDrive API is reachable
2020-12-18 06:17:52 +11:00
abraunegg 9b44bb1e96
Fix crash when unable to read a local file due to file permissions (Issue #1178) (#1179)
* Test if local file is readable based on user permissions before attempting to access file to determine the sync state or upload
2020-12-12 05:58:07 +11:00
abraunegg 5349340818
Add debug log output for configured API URL's (#1177)
* Add to debug log output the configured URL's which will be used throughout the application to remove any ambiguity as to using incorrect URL's when making API calls.
2020-12-10 11:18:13 +11:00
abraunegg 3b18638a63
Remove duplicate code for error output functions and enhance error logging output (#1170)
* Remove duplicate code for error output functions and enhance error logging output
2020-12-09 14:18:16 +11:00
abraunegg 335eab326d
Handle case where API provided deltaLink generates a further API error (#1175)
* Handle case where API provided deltaLink generates a further API error. Refer to: https://github.com/OneDrive/onedrive-api-docs/issues/1408
2020-12-09 14:03:49 +11:00
abraunegg 88f78c163f
Fix application crash in --monitor mode due to 'Failed to stat file' when setgid is used on a directory (#1157)
* Add debugging output when adding paths and items to an inotify watch
* Only if a path is a directory, perform a path walk
* Catch Exception as error when permissions cause access exception
2020-11-26 06:35:20 +11:00
abraunegg 48fc64a247
Check application_id before setting redirect URL when using specific Azure endpoints (#1165)
* If using specific AD endpoints, but using the default application ID, the redirect URL does not match. Check if the application_id before setting the redirect URL.
2020-11-26 05:51:29 +11:00
abraunegg a4a5a5f267
Fix logging output when authentication authorisation fails (#1156)
* If OneDrive generates an auth error, dont log that the auth was sucessful
2020-11-21 06:46:05 +11:00
abraunegg ac4da2e2ba
Update #1142 logging output handling (#1153)
* Update logging output to debug rather than standard log
2020-11-20 06:33:47 +11:00
abraunegg 8c23efc6c5
Fix changing permissions on pre-existing local directories (#1152)
* When attempting to create local directories, test to determine if they exist locally first before creating & setting file system permissions
2020-11-20 05:36:20 +11:00
abraunegg 17d5f97cb0
Merge branch 'master' into fix-issue-1137 2020-11-14 06:21:03 +11:00
abraunegg 550ab4b6dd
Use config set option for 'remove_source_files' and 'skip_dir_strict_match' rather than ignore (#1142)
* Remove items from direct config string update, as they are configurable via config file, thus set option via config file is not used
* Add debugging and application config output for used options
2020-11-14 06:17:53 +11:00
abraunegg aa533ef5a4 Update sync.d
* tweak skip message output
2020-11-13 09:52:48 +11:00
abraunegg b59596523e Update sync.d
* Full fix
2020-11-13 08:27:09 +11:00
abraunegg b57bb3fde5 Update sync.d
* Remove this
2020-11-13 08:02:28 +11:00
abraunegg dd989b73d1 Update sync.d
* Flag that this download failed
2020-11-13 07:48:14 +11:00
abraunegg ac02b768d3 catch folder creation errors due to file system permissions error
* catch folder creation errors due to file system permissions error
2020-11-13 07:34:52 +11:00
abraunegg cc2b8085f5 Update error catching and handling
* Update error catching and handling
2020-11-13 05:42:24 +11:00
abraunegg 68cc51ab4d Update onedrive.d
* Remove try block - incompatible with auto designation needed to open file ...
2020-11-12 21:51:07 +11:00
abraunegg aa47f1119f Fix download failure due to incorrect filesystem permissions
* Add try block for opening new file for writing when downloading a new file to catch any read only file systems
* Move set file attributes to master function, incase there is a download failure, exit scope cannot set attributes on a file that is non existent
2020-11-12 21:35:06 +11:00
abraunegg 2c80033c89
Correctly set config directory permissions at first start (#1132)
* When ~/.config/onedrive/ gets created for the first time, directory permissions are not set, thus, effective permissions of 'd---------' get applied. This then causes issues attempting to update the 'refresh_token' as permission is denied.
* When permission is denied, the file exception error is not correctly handled
* Set refresh_token file permissions
2020-11-09 08:06:48 +11:00
mp1994 7a4abfff40
Check if localModifiedTime >= item.mtime: avoid re-upload (#1131)
* Check if localModifiedTime is >= than item.mtime: avoid re-upload for equal modified time

Co-authored-by: abraunegg <alex.braunegg@gmail.com>
2020-11-08 08:44:57 +11:00
abraunegg 57d6753f80
Merge contents of SQLite WAL file into main database file on sync completion (#1128)
* When using WAL and SHM files, certain thresholds are used to automatically determine when WAL and SHM data is committed to the main database file. If these thresholds are not met, the data is not written. Before terminating the program, commit the data, and at the end of each monitor loop, commit the data
* Only try and write data to DB file and destroy variable if it was set to begin with
* Only try and destroy 'onedrive' variable if set, rather than set to null
2020-11-06 10:28:15 +11:00
abraunegg d2193c97f5
Fix uploading data to shared business folders when using --upload-only (#1127)
* Handle uploading new data to a OneDrive Shared Business Folder when using --upload-only and if the shared folder has never been synced before
2020-11-05 05:19:39 +11:00
abraunegg b4c3da6e10
Check the given local path in reverse order for a skip_dir match on --resync (#1125)
* When matching skip_dir entries, if there is no direct first match, perform a reverse directory walk to test each path block for a match
2020-11-05 04:51:35 +11:00
abraunegg 363f64df36
Check for the presence of .nosync when processing DB entries (#1124)
* Add check for .nosync if enabled for directory database objects as these might also be relevant mount points which have 'gone' away
2020-11-04 07:28:40 +11:00
abraunegg 106e114929
Update --list-shared-folders to only show folders (#1114)
* Update --list-shared-folders to only show folders
2020-11-01 10:06:08 +11:00
abraunegg b365ec6322
Cleanup dryrun shm and wal files if they exist (#1121)
* Cleanup dryrun shm and wal files if they exist
2020-11-01 06:55:17 +11:00
abraunegg cf4ee5017f
Configure specific File and Folder Permissions (#1108)
* Configure specific File and Folder Permissions, defaults for new items downloaded from OneDrive:
Directories: 700
Files: 600
2020-10-30 08:00:26 +11:00
abraunegg fe86dd1809
Fix Business Shared Folder handling to process only Business Shared Folders (#1104)
* Update Business Shared Folder handling to process only Shared Folders, not individually shared files
2020-10-26 17:23:24 +11:00
abraunegg 9f60b95eed
Fix logging output for modification comparison source (#1103)
* Fix logging output for modification comparison source
2020-10-23 06:36:14 +11:00
sebastiaopf a3c4ba1b3b
Adding support for authentication w/ single tenant custom applications (#1094)
* Adding support for authentication with single tenant custom applications in OneDrive Business/Enterprise accounts.

When using a custom single tenant application the /common/ part on the authentication URLs needs to be replaced
by the tenant name. This can be configured on the azure_tenant_id setting, but this entry was being ignored unless
a custom azure_ad_endpoint was also set.

* Updated documentation to mention that, when configuring for Single Tenant Applications,
both the Tenant Name and Application ID must be included on the config file.

Co-authored-by: abraunegg <alex.braunegg@gmail.com>
2020-10-10 15:39:04 +11:00
abraunegg b4328db2ef
Update debugging output for /delta changes available queries (#1089)
* Update debugging output for /delta changes available queries
2020-10-07 09:33:00 +11:00
themoox 2f35a33f4f
Allow config to specify a tenant id for non-multi-tenant applications (#1085)
* Allow config to specify a tenant id for non-multi-tenant applications
* Add instructions on how to find your O365 tenant identifier for National Cloud deployments.

Co-authored-by: Ryan Brown <ryan.brown@avarint.com>
Co-authored-by: abraunegg <alex.braunegg@gmail.com>
2020-10-06 08:06:04 +11:00