Commit graph

448 commits

Author SHA1 Message Date
abraunegg 10dd0300e3
Update issue #540 fix (#543)
* Update how and when filesize is configured and used before file download
2019-06-18 10:32:27 +10:00
abraunegg 687080d435
Size check should only be used if configured (Issue #540) (#541)
* Size check should only be used if configured
* Rather than make 2 x calls to OneDrive service, driveItem already has the size value to check
* Add a try block when query for file details
2019-06-17 10:16:27 +10:00
abraunegg bbf35f12ea
Handle a directory in the sync_dir when no permission to access (Issue #532) (#536)
* Handle a directory in the sync_dir when no permission to access
* Log the error during sync rather that only in verbose mode
2019-06-15 09:23:32 +10:00
abraunegg 7cf2912d2d
Rework directory layout and refactor MD files for readability (#523)
* Move lots of stuff into contrib directory to clean up root folder
* Update autoconf files for new location of some files
* Refactor README split and move to docs directory
* Update INSTALL.md
* Update Makefile.in
* Install docs from new location
* Uninstall docs correctly
* Update Dockerfile
* Fix up Dockerfile spacing
* Move systemd stuff to contrib
* Move docker specific stuff to contrib/docker, update docs
* Fix Alpine and Stretch docker files
* Update configure after changed file locations
* Update INSTALL.md to include pkg-config as requirement
* Move also entrypoint.sh to contrib/docker
* Update 'min-notif-changes' to 'min-notify-changes' which was missed in PR #467
* Update to add --sync-root-files to usage.md and man page
* Update usage.md to include --auth-files ARG details
2019-06-11 18:40:05 +10:00
Norbert Preining aa2e2245f0 Support file based authorization (Issue: #524) (#525)
* support file based authorization
* remove generated files after authentication
2019-06-11 17:12:36 +10:00
luukvbaal ef6af8e5bc implement skip_size (Issue #213) (#517)
* Implement skip_size feature request (@luukvbaal)
2019-06-08 06:37:41 +10:00
abraunegg c617185c14
Update missing cTag patch (Issue #512) (#515)
* Update original patch to check for eTag in response to use, if not found set to empty value.
* Add cTag response validation for all cTag response use 'in-case' the cTag response is dropped from other responses
2019-06-03 10:43:16 +10:00
abraunegg 8d4a35e34d
Handle OneDrive API response when cTag key is missing (Issue #512) (#513)
* Handle response from OneDrive where cTag JSON elements are missing when uploading a locally modified file
* Update bug_report.md to fix up spelling errors
2019-05-30 12:41:26 +10:00
abraunegg bac9ab41ae
Enhance error message for curl CA SSL issue (Issue #507) (#508)
* Enhance error message when unable to connect to Microsoft OneDrive service when the CA SSL certificate has issues
2019-05-24 19:17:08 +10:00
abraunegg 239085f1fc
Check timestamp to ensure that latest timestamp is used when comparing OneDrive changes (Issue #505) (#506)
* If OneDrive presents a change to download and the file already exists, check the local modified timestamp to ensure that the local file is not technically newer than the file we are going to download
2019-05-23 05:38:32 +10:00
abraunegg 20eda60fdf
Catch a curl timeout exception (#499)
* Catch a curl timeout exception
* If we hit a curl exception, log that we hit this, as the handling of the exception in 'some' instances may actually not log an error as to 'why'
2019-05-13 09:10:19 -04:00
abraunegg f779881ff9
Check to ensure that file downloaded and is present locally before setting times (Issue #496)
* Check to see if the file was actually downloaded before attempting to set the times on the file.
* If file is not local, download failed
2019-05-13 08:22:15 -04:00
abraunegg 3e35e20cc5
Update sync.d (#493)
* Add an encoding validation check before any path length checks are performed as if the path contains any invalid UTF-8 sequences, the path walk to establish valid path length will fail
2019-05-13 07:52:49 -04:00
abraunegg caec1cb8cb
Implement --sync-root-files when using a sync_list file (Issue #491) (#492)
* Implement --sync-root-files to sync all files in the OneDrive root when using a sync_list file that would normally exclude these files from being synced
2019-05-09 21:18:49 +10:00
abraunegg bac5d6f1a9
Fix dry run reports "Deleting item from one drive" for remotely created file (Issue #489) (#490)
* Fix --dry-run logging output for faking downloading new files
2019-05-09 19:52:39 +10:00
abraunegg af43b77100
Better handling for initialisation errors (Issue #483) (#484)
* Better handle initialisation issues when OneDrive / MS Graph is experiencing problems that generate 401 & 5xx error codes
2019-05-05 21:06:55 +10:00
abraunegg fbad4b4835
Fix 'Local files not deleted' when using bad 'skip_file' entry (#480)
* Add a check for bad 'skip_file' entry that prevents incorrect searching of local changes (mainly deletes) from being synced
2019-04-25 11:00:23 +10:00
abraunegg ea26e4d830
Add logging as to why resume upload failed (Issue #468) (#470)
* Log 'why' the resume upload failed
* Remove the 'resume_upload' as it is invalid & contains bad data
2019-04-15 10:56:11 +10:00
abraunegg c6923cdf29
Update 'resume_upload' handling in the event of bad OneDrive response (Issue #468) (#469)
* Add JSON response checks in the event OneDrive sends malformed data
2019-04-15 08:10:48 +10:00
abraunegg 25d2f7284f
Update 'min_notif_changes' to 'min_notify_changes' (#467)
* Update 'min_notif_changes' to 'min_notify_changes'
2019-04-15 06:28:32 +10:00
abraunegg 521cf33001
Add 'quota' check for valid value (#466)
* If the 'quota' details are not provided when querying the OneDrive account details, we cannot set 'remainingFreeSpace' Add a flag that if this is 0 or negative, ignore validating if there is enough free space to upload a file and just upload blindly
2019-04-13 08:59:03 +10:00
abraunegg a2889098e9
Handle object error response from OneDrive (Issue #464) (#465)
* Add specific check to ensure that variable is of type object
* Add error logging to indicate error response from OneDrive was received
* Update error logging in saveItem to match
* Add a '401 Unauthorized' event handler when querying OneDrive if the file exists
2019-04-13 08:31:54 +10:00
Norbert Preining ea22d8fef5 config options for command line switches (#449)
* config options for command line switches to allow for better config handling in docker containers
2019-04-11 12:26:20 +10:00
abraunegg f12bcfb650
Add debugging for --get-O365-drive-id (#462)
* Add debugging for --get-O365-drive-id
2019-04-10 20:26:21 +10:00
abraunegg 92d2f14f15
Update 'root' item identification handling (#457)
* Add additional check to ensure item root we flag as 'root' is not just name checked as 'root'
2019-04-09 05:05:47 +10:00
abraunegg ea2e4a98bf
Add --upload-only check for sharepoint fix (Issue #452) (#453)
* Add a check for --upload-only use when trying to work around https://github.com/OneDrive/onedrive-api-docs/issues/935
* Update logging and print a warning message that the files are now technically different due to sharepoint bug and using --upload-only
2019-04-06 12:44:51 +11:00
abraunegg 7c2b3e853d
Implement more meaningful 5xx error responses (Issue #445) (#451)
* Issue #445 and https://github.com/OneDrive/onedrive-api-docs/issues/1035 highlight that the current 5xx error messages are too generic and need to be more specific. This patch makes those error messages more specific to the response code from OneDrive API
2019-04-04 17:13:15 +11:00
abraunegg 6a6b8f128f
Issue #432 and #441 combined (#447)
* Re-add original #432 and #441 fixes to master
2019-04-02 05:21:02 +11:00
abraunegg abb82868e4
Roll back to v2.3.1 (#446)
* Roll back to v2.3.1 after bad re-base merge with PR #389
2019-04-02 04:51:25 +11:00
abraunegg f38b13dd00
Resolve file creation loop when working directly in the synced folder with libreoffice (#442)
* Add debug line for flagging to disable upload validation checks
* Handle Microsoft 'enrichment' of certain documents when stored on Sharepoint
* Handle skip_dir checks when nothing to check against (false positive)
* Update default 'skip_file' to include tmp and lock files generated by LibreOffice
* Update code comments & logging output
* Update readme & default config file
* Update database version due to changing defaults of 'skip_file' which will force a rebuild and use of new skip_file default regex
2019-04-02 04:00:22 +11:00
abraunegg 4444b9cd8a
Reduce scanning of local filesystem needlessly every sync in monitor mode (#433)
* Reduce scanning the entire local system in monitor mode for local changes
* Add 'monitor_fullscan_frequency' to set the frequency of performing a full disk scan when in monitor mode
2019-04-02 03:58:56 +11:00
Norbert Preining b7cc540c2c fixes to the merge routine 2019-03-25 10:25:10 +09:00
Norbert Preining a433b44b56 Merge branch 'master' into norbert/config-options 2019-03-25 09:41:27 +09:00
abraunegg 79cc599057
Resolve high CPU usage when performing DB reads (#419)
* Disable automatic indexing as we specifically create the required indexes
* Tell SQLite to store temporary tables in memory. This will speed up many read operations that rely on temporary tables, indices, and views.
* Add links & reasoning behind other PRAGMA settings used
* Add new index specifically for driveId & parentId paring
* To force DB schema & index creation, bump DB schema version
* Update handling of skip_dir and skip_file parsing - should only check if the file is excluded if the parent directory is not
* Add another index for selectByPath database queries
* Add new build option to get more DEBUG symbolic information
* Use boolean values rather than on / off values
* Enable auto_vacuum for entry deletes / database cleanup
2019-03-24 11:12:40 +11:00
abraunegg 002f9b7aec
Prevent use of --synchronize & --monitor together (#431)
* Prevent use of --synchronize & --monitor together
2019-03-23 10:20:27 +11:00
abraunegg f5c3e7643d
Update 'skip_dir' handling to check against OneDrive new downloads (#427)
* Add check for directories to match skip_dir entries for OneDrive objects to download
* Update logging as to why a OneDrive object was skipped
2019-03-22 15:21:59 +11:00
Norbert Preining 8f28daec8a ignore a path containing multiple dirs in skip_dir 2019-03-22 12:05:30 +09:00
abraunegg de98fe8492
Fix JSONValue is not an object crash when a 5xx error is returned when uploading files (#426)
* Fix JSONValue is not an object crash when a 5xx error is returned when uploading files
2019-03-22 10:23:21 +11:00
abraunegg f2e007a1d7
Update skip_dir config handling (#418)
* Original skip_dir handling expected an explicit match to path to match. With this patch, wildcard matching for any directory matching the path entry will be skipped
2019-03-22 05:30:32 +11:00
abraunegg 6b8b51a7cc
Update sync.d (#422)
* Add a check to validate if 'size' is in the JSON response
2019-03-21 10:29:04 +11:00
Norbert Preining 3aff2e8251 move verbose check to first getopt to make it available early on 2019-03-17 13:10:41 +09:00
Norbert Preining ab918dd82d config directory management fixes 2019-03-17 12:41:26 +09:00
Norbert Preining 362bf35688 Merge branch 'master' into norbert/config-options 2019-03-17 12:25:27 +09:00
abraunegg 99c0267c7a
Implement 'skip_dir' config option (Issue #123) (#388)
* Implement config option 'skip_dir'
2019-03-15 06:55:05 +11:00
abraunegg ac7789df14
Update error logging around directory case-insensitive match (#412)
* Update error logging around directory case-insensitive match
2019-03-14 05:44:43 +11:00
abraunegg 4c3b959bf6
Resolve unhandled application crash when invalid auth response is used (Issue #399) (#410)
* Add a http 400 response error handler
* If the response uri generates a 400 error, the JSON response will not contain the access_token. Request to re-authenticate
2019-03-12 11:50:21 +11:00
abraunegg e849eb3de4
Resolve Key not found: nextExpectedRanges (Issue #408) (#409)
* Dont 'assume' that the key value pairs exist. Check if they do before attempting to use them
2019-03-12 11:49:15 +11:00
abraunegg 98624267c5
Implement --dry-run (#337)
* Implement new feature --dry-run
2019-03-11 17:57:47 +11:00
abraunegg 3f08e271af
Update util.d - reduce local FS read overheads (#407)
* Try and read 1 byte of the file rather than 10MB for each 'new' file to be uploaded to validate local file permissions
2019-03-11 17:56:26 +11:00
abraunegg efaaaadce2
Rework function uploadCreateDir (#397)
* Ensure that on query failure, default root and drive id is used
2019-03-08 08:40:29 +11:00
abraunegg ba834368ca
Implement --check-for-nosync to ignore folder when special file present (Issue #163) (#390)
* Implement a capability to ignore a folder when a special file (.nosync) is present
2019-03-03 05:58:36 +11:00
Norbert Preining e5b500a041 remove unnecessary code, cleanup 2019-02-27 08:57:04 +09:00
Norbert Preining 388a3bb102 Merge branch 'master' into norbert/config-options 2019-02-27 08:51:28 +09:00
Norbert Preining 09f328e63f configurable logging intervals in monitor mode (Issue: #378) (#391)
* Configurable interval of logging in monitor mode (Issue: #378)
2019-02-27 07:21:23 +11:00
Norbert Preining b668ecef71 Create new upload session on reinit (Issue: #379) (#392)
* By moving the UploadSession creation from the constructor to the init function we guarantee that a re-initialization also starts a new UploadSession.
2019-02-27 06:49:23 +11:00
Norbert Preining 8cabe38e49 update log.verbose from options 2019-02-25 11:55:32 +09:00
Norbert Preining 8b7d772680 remove work-around from previous versions 2019-02-24 16:58:02 +09:00
Norbert Preining d02be44b9a order getopt call options alphabetically 2019-02-24 16:56:50 +09:00
Norbert Preining d49a598f4b clean up, move help function 2019-02-24 16:33:10 +09:00
Norbert Preining a71b9bf789 Merge branch 'master' into norbert/config-options 2019-02-24 16:27:53 +09:00
abraunegg 7a47df7a38
Implement --skip-dot-files (#386)
* Implement new CLI option --skip-dot-files to skip .files and .folders if option is used
2019-02-24 17:19:45 +11:00
Norbert Preining f2f272a8f8 move more parts of the getopt calls into config module 2019-02-22 22:00:57 +09:00
Norbert Preining a635cab0ad config options: working version 2019-02-22 16:15:10 +09:00
Norbert Preining 25b09775ab more playing around 2019-02-22 10:41:07 +09:00
Norbert Preining 7c9ea56293 some playground 2019-02-22 10:39:16 +09:00
Norbert Preining 18611c833f initial work on various config options for cmd line options 2019-02-22 10:39:16 +09:00
Norbert Preining 0d3b891177 don't annoy with notifs when offline and cannot act on changes (#385)
* Don't annoy with notifs when offline and cannot act on changes
2019-02-21 19:51:15 +11:00
Jack Thomasson 460596dd36 Add smaller footprint Docker container builds using Alpine and Debian Stretch (#369)
* Add additional Docker container builds utilising a smaller OS footprint
* Resolve compiling with LDC on Alpine as musl lacks some standard interfaces
2019-02-17 06:13:19 +11:00
abraunegg 51615bff0b
Rework PR #373 to reduce needless logging in monitor mode (#376)
* Rework PR #373 to reduce needless logging in monitor mode
2019-02-15 07:03:15 +11:00
abraunegg ff822a9ab9
Update handling of incomplete OneDrive JSON responses (#375)
* Re-order handling of response JSON to ensure that 'id' key element is always checked for.
* Print errors when there is an issue with saveItem & articulate why
2019-02-13 05:54:06 +11:00
Norbert Preining 1f955c82be
deal with unavailable dbus server (Issue: #365)
If the dbus server cannot be started or is not available, calling a notification
would result in an exception. During initialization check that we can get the
server information, and disable notifications if this is not possible.

Note: we change dnotify.d from the originally distributed version to include
a check for availability function.
2019-01-30 17:34:07 +09:00
Norbert Preining 1c6fd5df39 notification on incoming changes (#355)
* When notifications are enabled, display the number of OneDrive changes to process if any are found
* Add 'config' option 'min_notif_changes' for minimum number of changes to notify on, default = 5
* Update readme, manual page for new option
2019-01-29 04:54:03 +11:00
Norbert Preining 16da1b4994
treat mv of new file as upload of mv target (Issue: #357) 2019-01-22 15:14:53 +09:00
abraunegg 7532c9d8cc
Update handling of --get-O365-drive-id for multi matching (#354)
* Update handling of --get-O365-drive-id to print out all 'site names' that match the explicit search entry rather than just the last match
* Add webUrl for site to output so that the 'right' site drive_id can be identified
* Update O365 readme with updated output
2019-01-19 13:01:41 +11:00
abraunegg c27ff936fb
Resolve build warning & support CentOS 6.x for installation (Issue #350) (#353)
* Support install on CentOS / RHEL 6.x
* Support uninstall on CentOS / RHEL 6.x
* Add /usr/local/bin/ to search path when starting init.d service
* Fix dmd-2.084.0 deprecation warning: loop index implicitly converted from size_t to int
* Update readme based on CentOS 6 / RHEL 6 additional dependencies
* Resolve warnings about "bashisms" in init.d script (Issue #349) - fixed here as we are updating the init file already, rather than have multiple PR's
2019-01-19 13:01:01 +11:00
abraunegg 83cc2e419b
Add business retention policy handler (Issue #338)
* When OneDrive Business is configured with Retention Policies, attempts to delete a 'non empty' folder results in an an error response. Correctly handle the error response to print a warning message that OneDrive content must be manually deleted. 

**Note:** A future version of this client will handle the removal of 'non empty' folders when Retention Policies are enabled.
2019-01-19 12:59:39 +11:00
abraunegg c91dcc3e49
Add handling for JSON response error when OneDrive API returns a 404 due to API regression (#340)
* Update sync.d to handle OneDrive API Regression & provide details on workaround
2019-01-16 20:14:55 +11:00
abraunegg 1d902a5beb
Handle Item not found exception when unable to query OneDrive 'root' for changes (#336)
* Add a 404 error handler when OneDrive cannot query what changes are available for a particular 'root' folder id
* Add additional verbose debug logging to assist in the future
* Change to not throw an exception when OneDrive cannot query what changes are available for a particular 'root' folder - rather display some meaningful details instead of crashing the application
* As we know the actual 'items.sqlite3' location, display that as the file to remove rather than a 'generic' message as this is more helpful
2019-01-11 05:31:10 +11:00
abraunegg bf43ecda9e
Ensure that remote deletes are handled correctly (#333)
* Fix remote deletes the fix for #323, #324 & #331 introduced a bug where an item when remote deleted, would actually be re-inserted to the database & eventually cause a database assertion
2019-01-07 04:22:09 +11:00
abraunegg b75163245f
Update handling of 'remote' folder designation as 'root' items (Issue #323, #324, #331) (#328)
* Add extra verbose debugging
* Update handling of 'remote' folder designation as 'root' items
2019-01-07 03:02:08 +11:00
abraunegg 1fa7e5f20c
Add extra debugging & http protocol downgrade (Issue #314) (#327)
* Add --force-http-1.1 flag to downgrade any HTTP/2 curl operations to HTTP 1.1 protocol
* Explicitly set all bool items to false at initialisation
* Update --display-config to display sync_list if configured
* Add debug handling to display when sync_list is loaded
* Add debug handling to output the handling of OneDrive changes
2019-01-06 05:43:44 +11:00
abraunegg ddc5d602da
Update sync.d (#329)
* Update HTTP 412 error notifications to better identify where 412 error is occuring
* Add a check for 'id' key on metadata update to prevent 'std.json.JSONException@std/json.d(494): Key not found: id' (Issue #325)
2019-01-06 05:35:55 +11:00
abraunegg 46ef8ed376 Revert "Update handling of HTTP 412 - Precondition Failed errors (#issue 325) (#326)"
This reverts commit 388cc1ded5.
2019-01-05 17:18:28 +11:00
abraunegg 388cc1ded5
Update handling of HTTP 412 - Precondition Failed errors (#issue 325) (#326)
* Update HTTP 412 - Precondition Failed messaging
* Add debug messaging when 5xx errors occur
* Update 5xx message to include
2019-01-05 07:57:10 +11:00
Norbert Preining b9890ae2e5
better help output formatting (Issue #298) 2018-12-28 18:01:50 +09:00
Norbert Preining fcf1ef2e1f
signal handling and database shutdown (#315)
Add a signal handler and shutdown database connection on SIGINT and SIGTERM
to make sure that .waf files are properly incorportated into the sqlite database.
2018-12-28 11:19:20 +09:00
abraunegg 2553366a89
Implement Feature Request: Add status command or switch (Issue #112) (#307)
* Implement Feature Request: Add status command or switch
2018-12-28 12:26:03 +11:00
abraunegg 7a6b507367
Update sync.d (#316)
* Update logic handling of --single-directory to prevent inadvertent local data loss
2018-12-28 11:57:32 +11:00
Norbert Preining 2ec6aa3df6
make sure sqlite checkpointing works by properly finalizing statements (#310)
If an sqlite statement is prepared and reused by using reset again and again, the connection
keeps a lock on the database, and checkpointing cannot ensure that transaction from the
wal file are carried over into the main database.

By preparing the statement every time it is used, the destructor calls finalize and thus
the lock is released.

Possible impacts during high frequency changes via monitor mode etc might arise.
2018-12-26 16:11:06 +09:00
abraunegg 5d388581b1
Resolve 'sync_dir' not read from 'config' file when run in Docker container #306 (#308)
* Fix logic for sync_dir handling on headless systems
* Fix logic where potentially a 'default' ~/OneDrive sync_dir could be set despite 'config' file configured for an alternate
* Add debug handling for sync_dir operations
* Add debug handling for homePath calculation
* Add debug handling for configDirBase calculation
* Reorder main() so that log.vdebug works where required
* Rework configDirName as this had the same issue as syncDir
* Update configDirName & syncDir '~' handling & replacement to be more robust and only replace if '~' is first char in string
* Add additional debug logging if syncDir is created
2018-12-23 11:15:10 +11:00
abraunegg 1060d85118
Resolve JSONException when supplying --get-O365-drive-id option with a string containing spaces (#305)
* JSONException when supplying --get-O365-drive-id option with a string containing spaces
2018-12-21 11:28:16 +11:00
abraunegg 594bfcec54
Fix syncdir option is ignored (#302)
* Explicitly set value for syncdir if the option is passed in via CLI rather than 'config' file
2018-12-20 17:27:34 +11:00
abraunegg dfd3045ce4
Fix exception when missing parameter from usage option (#299)
* Handle the exception generated when an option is used which is expecting an input parameter but none is given
2018-12-20 11:05:32 +11:00
Norbert Preining 025a3b2878
check config file keys for validity (#296)
* check config file keys for validity, use setValue instead of direct access
* Update config.d
  Add 'drive_id' to be initialised, set to an empty string
* exit application if there is a configuration file error
* Issue #293 was caused by a spelling error in the configuration file. 
  If the configuration file has errors, we should not load it or run
  using the application defaults as this may have undesirable consequences
  for users data
* missed returning false on key issue
* Missed this edit of the file
2018-12-20 08:51:21 +09:00
abraunegg 7d522583ef
Implement option --display-config (Issue #100) (#292)
* Implement --display-config to show the application configuration without actually performing a sync or application init
2018-12-20 05:42:28 +11:00
abraunegg 53df30810e check displayName rather than description (#290)
* In testing, 'description' and 'displayName' were the same, but in reality we are checking against the 'displayName' value as this is what we are asking the user to input, thus we need to check against displayName
2018-12-18 13:20:17 +09:00
Norbert Preining 91aaf91aa6 Reinitialize sync engine after three failed trials (#283)
* After extended suspend, or bad connection, the connection to OD might
go stale - thus no network connection to Microsoft OneDrive Service, skipping sync
and this continues ad infinitum. Change the last action to re-initialize the sync engine.
2018-12-13 07:08:18 +11:00
Norbert Preining 6a62406db7 Deal with deleted move targets (Issue: 273) (#278)
if the move target has been deleted before the monitor action completes,
the monitor action throws an exception when stat-ing the target for
mtime. Do not do anything in case the move target has disappeared
2018-12-08 05:02:47 +11:00
Norbert Preining 3608dcc4c2 Add very verbose (debug) mode by double -v -v (#279)
* Add very verbose (debug) mode by double -v -v - currently only used for debugging the inotify events
2018-12-08 05:01:22 +11:00
Norbert Preining 87cd18379d Deal with items that disappear during upload (Issue: #258) (#276)
* Deal with disappearing items during upload (Issue: #258)
2018-12-06 20:28:03 +11:00