Commit graph

490 commits

Author SHA1 Message Date
abraunegg 585b35724f
Reduce change scan impact of fix for #658 (#691)
* If there are delta changes (meaning something changed on OneDrive) process the changes that have been presented
2019-10-23 05:29:11 +11:00
abraunegg 1ab36f4014
Update skip_dir pattern matching when using wildcards (Issue #689) (#692)
* Better handle skip_dir searching when wildcards are involved by passing in a better 'path' snippet from which to compare skip_dir entries against
* Enhance the directory to 'search' to include simple & complex matching in case long paths are used as part of skip_dir
2019-10-22 14:08:17 +11:00
abraunegg cfdb295823
Fix typo - retrySucess -> retrySuccess
* Fix typo - retrySucess -> retrySuccess
2019-10-21 18:53:00 +11:00
Rob A 44576ea4ca Handle curl exceptions and timeouts better with backoff/retry logic (Issue #693) (#695)
* Handle curl exceptions and timeouts better with backoff/retry logic
2019-10-21 18:04:08 +11:00
abraunegg cdad8631ec
Update JSON logging output (#696)
* Update invalid JSON object output handling as this really should be debug messaging not error messages
2019-10-21 13:25:52 +11:00
abraunegg 6a8d575941
Modify 'Processing X changes' output to be more reflective of actual activity (Issue #679) (#680)
* When using 'sync_list' do not output that we are processing 'changes' as this is misleading.
* When using sync_list perform a full scan only the first time when using --synchronize & --monitor to avoid double processing of all OneDrive items each and every time
* When using sync_list and --monitor only perform a full object scan when we are also configured to perform a full scan
2019-10-09 12:02:17 +11:00
abraunegg 808a2f331b
Fix log output to be consistent
Fix log output to be consistent with sync_list exclusion
2019-10-08 18:11:37 +11:00
abraunegg 5da97d0757
Fix typo - skip_list -> sync_list
Fix typo - skip_list -> sync_list
2019-10-08 17:48:51 +11:00
abraunegg 57cf29783a
Fix handling of skip_dir, skip_file & sync_list config options (#686)
* config option skip_dir should only be applied to directories
* config option skip_file should only be applied to files
* If 'directory' is in 'skip_file' then upload skip reason erroneously lists sync_list as reason for skipping
* update log output when something is excluded
* where a path is to be excluded via sync_list, only exclude via sync_list & not check skip_dir & skip_file as well
2019-10-08 17:34:35 +11:00
abraunegg ca9a6b6ac0
Make user-agent string a configuration option (#673)
* Make user-agent string a configuration option
* Set default User-Agent to OneDrive Client for Linux v{version}
* Add 'user_agent' to default config file
2019-09-24 16:42:40 +10:00
Jon Mills 3aa4d4965e Return remote name when 'case-insensitive match' (#670)
If there is a case-insensitive match error, also return the remote name from the response
2019-09-23 17:35:18 +10:00
abraunegg fd4547376c
Fix crash when resume_upload file is not a valid JSON (#664)
* Validate that there is JSON data in the string before the string is read via parseJSON()
* Add try block to catch file system exceptions if they are generated when attempting to update the file date & time
2019-09-22 06:40:39 +10:00
abraunegg 72467e32ca
Update OneDrive API status code 500 handling (#661)
* Update OneDrive API status code 500 handling when uploading files as error message is not correct
2019-09-17 11:15:03 +10:00
abraunegg 2abc05af1d
Query for all changes if using sync_list file (#659)
* If we are using a sync_list file, using deltaLink will actually 'miss' changes (moves & deletes) on OneDrive as using sync_list discards changes
2019-09-16 05:06:02 +10:00
abraunegg e32b8b3c7f
Move logging of 'initialisation' to actually when the attempt to initialise is performed (#656)
* Move logging of 'initialisation' to actually when the attempt to initialise is performed
2019-09-11 20:26:21 +10:00
abraunegg 42d8c454e2
Fix downloading of files from OneDrive Personal Shared Folders (Issue #635) (#637)
* Fix downloading of files from OneDrive Personal Shared Folders when the OneDrive API responds with unexpected additional path data
2019-09-09 12:30:59 +10:00
abraunegg 0424caaaf8
Fix abnormal termination when no internet connection (#643)
* Fix abnormal termination when no internet connection
2019-09-06 13:12:13 +10:00
abraunegg 90a28e0f6c
Fix 'key not found: name' when checking if item was synced (#645)
* Fix searching for 'name' when deleting a synced item, if the OneDrive API does not return the expected details in the API call
2019-09-06 12:30:33 +10:00
abraunegg 5cd860a398
Fix handling of 5xx responses from OneDrive when uploading via a session (Issue #632) (#631)
* Fix handling of 5xx responses from OneDrive when performing a session upload
* Switch to same checks when doing non session upload so that OneDrive exceptions are thrown correctly
* Remove a 'throw e' when curl times out
* Remove a needless 'throw e' when a session upload cannot be found
* Set the upload failed flag when OneDrive errors on session uploads
2019-08-31 05:27:49 +10:00
abraunegg eb04ad5ffa
Remove redundant logging output (#627)
* Remove redundant logging output
2019-08-24 17:28:16 +10:00
abraunegg ad0daf2df5
Detect the need for --resync when config changes (#617)
* Detect the need for --resync when config changes either via config file or cli override
2019-08-24 17:18:58 +10:00
abraunegg bc3853bc4f
Add additional JSON object validation (#623)
* Add additional JSON object validation for queries made against OneDrive where a JSON response is expected and where that response is to be used and expected to be valid
2019-08-24 16:26:08 +10:00
abraunegg 488f756ecd
Fix unhandled error exceptions that cause application to exit / crash when uploading files (Issue #624) (#625)
* Remove 'else throw e' which causes the application to exit / crash and replace with what the actual error message is from OneDrive
* Make logging of unhandled errors consistent
* Add extra error handling when uploading files for errors received from OneDrive or file system errors
* Flag that the upload failed if there is an exception error raised
* Update logging output for when an error occurs to indicate that the file was skipped
* Log the OneDrive response as log.vlog rather than log.error as it is now being handled correctly
* Throw an exception for 50x errors so that it gets caught correctly
* Clean up logging when exception is generated
2019-08-24 15:54:48 +10:00
abraunegg 437fe4d9e4
Remove double logging output for a OneDrive 412 error (#622)
* Remove this logging output as it creates a double entry
2019-08-22 20:07:33 +10:00
abraunegg 893e235151
Catch a 403 Forbidden exception when querying Sharepoint Library Names (#620)
* Catch a 403 Forbidden exception when attempting to query for an Office 365 Shared Library Name
* Validate that the returned response is a valid JSON object
* Update scopes as per https://github.com/OneDrive/onedrive-api-docs/issues/1119
* Update that if a 403 error is received, what is the right course of action to take
2019-08-09 18:14:10 +10:00
abraunegg 4e97f8810d
Implement --get-file-link (#612)
* Implement --get-file-link which will return the weburl of a file which has been synced to OneDrive
2019-08-02 18:43:31 +10:00
abraunegg 100e644ae0
Catch error when creating sync dir fails (#608)
* Catch error when creating the sync dir fails - we cant run if this is not created or present
2019-07-31 07:20:26 +10:00
abraunegg 8f71043f23
Add a check to handle moving items out of sync scope & delete locally if true (#604)
* Add a check to handle moving items out of sync scope & delete locally if true
2019-07-30 07:02:24 +10:00
abraunegg 3171318503
Resolve file / folder upload issues (Issue #598) (#602)
* Without this throw, uploadLastModifiedTime fails to catch that the 412 error was returned, thus, the retry with a null cTag / eTag is not performed which leads to 'OneDrive response missing required 'id' element' being generated.
* Fix 'Unexpected character '<'. (Line 1:1)' when OneDrive has an exception error
* Validate that the response from OneDrive is a valid JSON object
2019-07-27 05:24:59 +10:00
abraunegg b41c5eb05d
Update Sharepoint modified file handling for files > 4Mb in size (#594)
* Update Sharepoint modified file handling for files > 4Mb in size
2019-07-17 05:11:38 +10:00
abraunegg 56644c4cbf
Update Sharepoint Document Library modified file handling (#590)
* Update Sharepoint Document Library handling for .txt & .csv files
* Use replace function for modified Sharepoint Document Library files rather than delete and upload as new file, preserving file history
2019-07-14 07:12:27 +10:00
abraunegg 66764f279d
Add explicit check for hashes being returned from OneDrive (Issue #591) (#592)
* Add explicit check for hashes to be returned in cases where OneDrive API fails to provide them despite requested to do so
* Add comparison with sha1 if OneDrive provides that rather than quickXor
2019-07-13 15:42:35 +10:00
abraunegg 68c6c5d75d
Fix hidden directories in 'root' from having prefix removed (#586)
* Fix hidden directories in 'root' from having prefix removed
2019-07-12 13:12:26 +10:00
Norbert Preining ab7e321726 Remove -d shorthand for --download-only (#577)
* Remove -d shorthand for --download-only to avoid confusion with other GNU applications where -d stands for 'debug'
2019-07-10 13:23:44 +10:00
Matthew Fallon f39621c2da Change unauthorized-api exit code to 3 (Issue #567) (#572)
* Changes unauthorized-api exit code to 3
* Systemd will not restart on exit code 3
2019-07-06 08:01:04 +10:00
abraunegg 80675aa8b4
Fix unable to download all files (Issue #563) (#568)
* Update logging output for better clarity
* Update logging output from vdebug -> error for errors
* Remove restrictive checks on fileDetails - if missing, continue, not return
* Add debug lines for if / when this is missing
2019-07-05 16:17:06 +10:00
abraunegg 1d02b3cb7a
Fix not all files being downloaded due to query failure & false DB update (#564)
* If query for file details or file object generates an error, we return, but we also need to flag that downloadFailed = true so that item is not added to the database which may cause validation issues leading to thinking that the file was downloaded but then deleted
2019-07-03 18:33:51 +10:00
abraunegg 4a4611ccea
Update fix for Issue #555 (#559)
* Update fix for #555 as try | catch block for session creation appears to miss error response codes
* Update how JSONValue object is determined to be valid
* Add error logging when response is not a valid JSON object
2019-06-30 10:12:25 +10:00
abraunegg 2f3804a3c0
Add download size & hash check (Issue #558) (#560)
* Check if the file that was downloaded to local disk equals the reported size & reported hash of the file on OneDrive. If sizes do not match, report an error and remove the local file as it is most likely corrupt.
2019-06-28 13:13:32 +10:00
abraunegg 15a2749382
Handle OneDrive error response on creating upload session (#556)
* Add a try block when attempting to create the upload session and handle if there is an error response from OneDrive
* Should return a JSONValue regardless of error
2019-06-27 18:40:18 +10:00
abraunegg d939d38c4f
Implement HTTP/2 downgrade by default (#549)
* Implement HTTP/2 downgrade by default to increase application stability when using curl >= 7.62
2019-06-21 01:11:29 +10:00
abraunegg bb2986b222
Add object check for json value (#551)
* Add explicit check that variable is a valid object
* Add default handling if query throws an error
2019-06-21 00:45:04 +10:00
abraunegg 10dd0300e3
Update issue #540 fix (#543)
* Update how and when filesize is configured and used before file download
2019-06-18 10:32:27 +10:00
abraunegg 687080d435
Size check should only be used if configured (Issue #540) (#541)
* Size check should only be used if configured
* Rather than make 2 x calls to OneDrive service, driveItem already has the size value to check
* Add a try block when query for file details
2019-06-17 10:16:27 +10:00
abraunegg bbf35f12ea
Handle a directory in the sync_dir when no permission to access (Issue #532) (#536)
* Handle a directory in the sync_dir when no permission to access
* Log the error during sync rather that only in verbose mode
2019-06-15 09:23:32 +10:00
abraunegg 7cf2912d2d
Rework directory layout and refactor MD files for readability (#523)
* Move lots of stuff into contrib directory to clean up root folder
* Update autoconf files for new location of some files
* Refactor README split and move to docs directory
* Update INSTALL.md
* Update Makefile.in
* Install docs from new location
* Uninstall docs correctly
* Update Dockerfile
* Fix up Dockerfile spacing
* Move systemd stuff to contrib
* Move docker specific stuff to contrib/docker, update docs
* Fix Alpine and Stretch docker files
* Update configure after changed file locations
* Update INSTALL.md to include pkg-config as requirement
* Move also entrypoint.sh to contrib/docker
* Update 'min-notif-changes' to 'min-notify-changes' which was missed in PR #467
* Update to add --sync-root-files to usage.md and man page
* Update usage.md to include --auth-files ARG details
2019-06-11 18:40:05 +10:00
Norbert Preining aa2e2245f0 Support file based authorization (Issue: #524) (#525)
* support file based authorization
* remove generated files after authentication
2019-06-11 17:12:36 +10:00
luukvbaal ef6af8e5bc implement skip_size (Issue #213) (#517)
* Implement skip_size feature request (@luukvbaal)
2019-06-08 06:37:41 +10:00
abraunegg c617185c14
Update missing cTag patch (Issue #512) (#515)
* Update original patch to check for eTag in response to use, if not found set to empty value.
* Add cTag response validation for all cTag response use 'in-case' the cTag response is dropped from other responses
2019-06-03 10:43:16 +10:00
abraunegg 8d4a35e34d
Handle OneDrive API response when cTag key is missing (Issue #512) (#513)
* Handle response from OneDrive where cTag JSON elements are missing when uploading a locally modified file
* Update bug_report.md to fix up spelling errors
2019-05-30 12:41:26 +10:00
abraunegg bac9ab41ae
Enhance error message for curl CA SSL issue (Issue #507) (#508)
* Enhance error message when unable to connect to Microsoft OneDrive service when the CA SSL certificate has issues
2019-05-24 19:17:08 +10:00
abraunegg 239085f1fc
Check timestamp to ensure that latest timestamp is used when comparing OneDrive changes (Issue #505) (#506)
* If OneDrive presents a change to download and the file already exists, check the local modified timestamp to ensure that the local file is not technically newer than the file we are going to download
2019-05-23 05:38:32 +10:00
abraunegg 20eda60fdf
Catch a curl timeout exception (#499)
* Catch a curl timeout exception
* If we hit a curl exception, log that we hit this, as the handling of the exception in 'some' instances may actually not log an error as to 'why'
2019-05-13 09:10:19 -04:00
abraunegg f779881ff9
Check to ensure that file downloaded and is present locally before setting times (Issue #496)
* Check to see if the file was actually downloaded before attempting to set the times on the file.
* If file is not local, download failed
2019-05-13 08:22:15 -04:00
abraunegg 3e35e20cc5
Update sync.d (#493)
* Add an encoding validation check before any path length checks are performed as if the path contains any invalid UTF-8 sequences, the path walk to establish valid path length will fail
2019-05-13 07:52:49 -04:00
abraunegg caec1cb8cb
Implement --sync-root-files when using a sync_list file (Issue #491) (#492)
* Implement --sync-root-files to sync all files in the OneDrive root when using a sync_list file that would normally exclude these files from being synced
2019-05-09 21:18:49 +10:00
abraunegg bac5d6f1a9
Fix dry run reports "Deleting item from one drive" for remotely created file (Issue #489) (#490)
* Fix --dry-run logging output for faking downloading new files
2019-05-09 19:52:39 +10:00
abraunegg af43b77100
Better handling for initialisation errors (Issue #483) (#484)
* Better handle initialisation issues when OneDrive / MS Graph is experiencing problems that generate 401 & 5xx error codes
2019-05-05 21:06:55 +10:00
abraunegg fbad4b4835
Fix 'Local files not deleted' when using bad 'skip_file' entry (#480)
* Add a check for bad 'skip_file' entry that prevents incorrect searching of local changes (mainly deletes) from being synced
2019-04-25 11:00:23 +10:00
abraunegg ea26e4d830
Add logging as to why resume upload failed (Issue #468) (#470)
* Log 'why' the resume upload failed
* Remove the 'resume_upload' as it is invalid & contains bad data
2019-04-15 10:56:11 +10:00
abraunegg c6923cdf29
Update 'resume_upload' handling in the event of bad OneDrive response (Issue #468) (#469)
* Add JSON response checks in the event OneDrive sends malformed data
2019-04-15 08:10:48 +10:00
abraunegg 25d2f7284f
Update 'min_notif_changes' to 'min_notify_changes' (#467)
* Update 'min_notif_changes' to 'min_notify_changes'
2019-04-15 06:28:32 +10:00
abraunegg 521cf33001
Add 'quota' check for valid value (#466)
* If the 'quota' details are not provided when querying the OneDrive account details, we cannot set 'remainingFreeSpace' Add a flag that if this is 0 or negative, ignore validating if there is enough free space to upload a file and just upload blindly
2019-04-13 08:59:03 +10:00
abraunegg a2889098e9
Handle object error response from OneDrive (Issue #464) (#465)
* Add specific check to ensure that variable is of type object
* Add error logging to indicate error response from OneDrive was received
* Update error logging in saveItem to match
* Add a '401 Unauthorized' event handler when querying OneDrive if the file exists
2019-04-13 08:31:54 +10:00
Norbert Preining ea22d8fef5 config options for command line switches (#449)
* config options for command line switches to allow for better config handling in docker containers
2019-04-11 12:26:20 +10:00
abraunegg f12bcfb650
Add debugging for --get-O365-drive-id (#462)
* Add debugging for --get-O365-drive-id
2019-04-10 20:26:21 +10:00
abraunegg 92d2f14f15
Update 'root' item identification handling (#457)
* Add additional check to ensure item root we flag as 'root' is not just name checked as 'root'
2019-04-09 05:05:47 +10:00
abraunegg ea2e4a98bf
Add --upload-only check for sharepoint fix (Issue #452) (#453)
* Add a check for --upload-only use when trying to work around https://github.com/OneDrive/onedrive-api-docs/issues/935
* Update logging and print a warning message that the files are now technically different due to sharepoint bug and using --upload-only
2019-04-06 12:44:51 +11:00
abraunegg 7c2b3e853d
Implement more meaningful 5xx error responses (Issue #445) (#451)
* Issue #445 and https://github.com/OneDrive/onedrive-api-docs/issues/1035 highlight that the current 5xx error messages are too generic and need to be more specific. This patch makes those error messages more specific to the response code from OneDrive API
2019-04-04 17:13:15 +11:00
abraunegg 6a6b8f128f
Issue #432 and #441 combined (#447)
* Re-add original #432 and #441 fixes to master
2019-04-02 05:21:02 +11:00
abraunegg abb82868e4
Roll back to v2.3.1 (#446)
* Roll back to v2.3.1 after bad re-base merge with PR #389
2019-04-02 04:51:25 +11:00
abraunegg f38b13dd00
Resolve file creation loop when working directly in the synced folder with libreoffice (#442)
* Add debug line for flagging to disable upload validation checks
* Handle Microsoft 'enrichment' of certain documents when stored on Sharepoint
* Handle skip_dir checks when nothing to check against (false positive)
* Update default 'skip_file' to include tmp and lock files generated by LibreOffice
* Update code comments & logging output
* Update readme & default config file
* Update database version due to changing defaults of 'skip_file' which will force a rebuild and use of new skip_file default regex
2019-04-02 04:00:22 +11:00
abraunegg 4444b9cd8a
Reduce scanning of local filesystem needlessly every sync in monitor mode (#433)
* Reduce scanning the entire local system in monitor mode for local changes
* Add 'monitor_fullscan_frequency' to set the frequency of performing a full disk scan when in monitor mode
2019-04-02 03:58:56 +11:00
Norbert Preining b7cc540c2c fixes to the merge routine 2019-03-25 10:25:10 +09:00
Norbert Preining a433b44b56 Merge branch 'master' into norbert/config-options 2019-03-25 09:41:27 +09:00
abraunegg 79cc599057
Resolve high CPU usage when performing DB reads (#419)
* Disable automatic indexing as we specifically create the required indexes
* Tell SQLite to store temporary tables in memory. This will speed up many read operations that rely on temporary tables, indices, and views.
* Add links & reasoning behind other PRAGMA settings used
* Add new index specifically for driveId & parentId paring
* To force DB schema & index creation, bump DB schema version
* Update handling of skip_dir and skip_file parsing - should only check if the file is excluded if the parent directory is not
* Add another index for selectByPath database queries
* Add new build option to get more DEBUG symbolic information
* Use boolean values rather than on / off values
* Enable auto_vacuum for entry deletes / database cleanup
2019-03-24 11:12:40 +11:00
abraunegg 002f9b7aec
Prevent use of --synchronize & --monitor together (#431)
* Prevent use of --synchronize & --monitor together
2019-03-23 10:20:27 +11:00
abraunegg f5c3e7643d
Update 'skip_dir' handling to check against OneDrive new downloads (#427)
* Add check for directories to match skip_dir entries for OneDrive objects to download
* Update logging as to why a OneDrive object was skipped
2019-03-22 15:21:59 +11:00
Norbert Preining 8f28daec8a ignore a path containing multiple dirs in skip_dir 2019-03-22 12:05:30 +09:00
abraunegg de98fe8492
Fix JSONValue is not an object crash when a 5xx error is returned when uploading files (#426)
* Fix JSONValue is not an object crash when a 5xx error is returned when uploading files
2019-03-22 10:23:21 +11:00
abraunegg f2e007a1d7
Update skip_dir config handling (#418)
* Original skip_dir handling expected an explicit match to path to match. With this patch, wildcard matching for any directory matching the path entry will be skipped
2019-03-22 05:30:32 +11:00
abraunegg 6b8b51a7cc
Update sync.d (#422)
* Add a check to validate if 'size' is in the JSON response
2019-03-21 10:29:04 +11:00
Norbert Preining 3aff2e8251 move verbose check to first getopt to make it available early on 2019-03-17 13:10:41 +09:00
Norbert Preining ab918dd82d config directory management fixes 2019-03-17 12:41:26 +09:00
Norbert Preining 362bf35688 Merge branch 'master' into norbert/config-options 2019-03-17 12:25:27 +09:00
abraunegg 99c0267c7a
Implement 'skip_dir' config option (Issue #123) (#388)
* Implement config option 'skip_dir'
2019-03-15 06:55:05 +11:00
abraunegg ac7789df14
Update error logging around directory case-insensitive match (#412)
* Update error logging around directory case-insensitive match
2019-03-14 05:44:43 +11:00
abraunegg 4c3b959bf6
Resolve unhandled application crash when invalid auth response is used (Issue #399) (#410)
* Add a http 400 response error handler
* If the response uri generates a 400 error, the JSON response will not contain the access_token. Request to re-authenticate
2019-03-12 11:50:21 +11:00
abraunegg e849eb3de4
Resolve Key not found: nextExpectedRanges (Issue #408) (#409)
* Dont 'assume' that the key value pairs exist. Check if they do before attempting to use them
2019-03-12 11:49:15 +11:00
abraunegg 98624267c5
Implement --dry-run (#337)
* Implement new feature --dry-run
2019-03-11 17:57:47 +11:00
abraunegg 3f08e271af
Update util.d - reduce local FS read overheads (#407)
* Try and read 1 byte of the file rather than 10MB for each 'new' file to be uploaded to validate local file permissions
2019-03-11 17:56:26 +11:00
abraunegg efaaaadce2
Rework function uploadCreateDir (#397)
* Ensure that on query failure, default root and drive id is used
2019-03-08 08:40:29 +11:00
abraunegg ba834368ca
Implement --check-for-nosync to ignore folder when special file present (Issue #163) (#390)
* Implement a capability to ignore a folder when a special file (.nosync) is present
2019-03-03 05:58:36 +11:00
Norbert Preining e5b500a041 remove unnecessary code, cleanup 2019-02-27 08:57:04 +09:00
Norbert Preining 388a3bb102 Merge branch 'master' into norbert/config-options 2019-02-27 08:51:28 +09:00
Norbert Preining 09f328e63f configurable logging intervals in monitor mode (Issue: #378) (#391)
* Configurable interval of logging in monitor mode (Issue: #378)
2019-02-27 07:21:23 +11:00
Norbert Preining b668ecef71 Create new upload session on reinit (Issue: #379) (#392)
* By moving the UploadSession creation from the constructor to the init function we guarantee that a re-initialization also starts a new UploadSession.
2019-02-27 06:49:23 +11:00
Norbert Preining 8cabe38e49 update log.verbose from options 2019-02-25 11:55:32 +09:00
Norbert Preining 8b7d772680 remove work-around from previous versions 2019-02-24 16:58:02 +09:00
Norbert Preining d02be44b9a order getopt call options alphabetically 2019-02-24 16:56:50 +09:00