Commit graph

194 commits

Author SHA1 Message Date
abraunegg 7a18393ed0
Update skip_dotfiles handling of .folders when in monitor mode (#848)
* Prevent monitoring files inside .folders when using skip_dotfiles in monitor mode
* Handle '.folder' -> 'folder' move when skip_dotfiles is enabled
2020-04-06 20:05:06 +10:00
abraunegg 50fb2d8bfa
Update application output when using --dry-run and --resync and database is corrupt (Issue#844) (#845)
* Update application output when database is corrupt and how to resolve by using --resync
* Handle --dry-run and --resync correctly - do not copy the database, it may be corrupt
* Always check if items-dryrun.sqlite3 still exists, remove it as it maybe corrupt when --dry-run is used
2020-03-30 06:55:27 +11:00
abraunegg 1e8395fb73
Handle moving files into a skipped .folder when skip_dotfiles = true (#828)
* If skip_dotfiles = true, moving files into a skipped .folder should not throw an error and should be removed from their previous location on OneDrive
* Add log message to indicate why delete operation is occurring on item which was moved
2020-03-20 05:35:21 +11:00
abraunegg 4cb0ebbfde
Update logging output (Issue #818) (#819)
* Tighten up the logging output to better clarify what is occurring when sync_list is being used
2020-03-15 06:58:57 +11:00
abraunegg b1fcb814c3
Update application output when just authorising application (Issue #820) (#821)
* Update application output to be clearer when just authorising the application and --synchronize or --monitor not passed in
* Update usage.md with updated authorize details and example
2020-03-15 06:29:44 +11:00
abraunegg 14cd47b56f
Implement #699 - Perform skip_dir explicit match only (#768)
* Implement #699 - Perform skip_dir explicit match only
2020-01-29 16:37:50 +11:00
abraunegg f2fe718389
Implement #763 - Delete local files after sync (#767)
* Implement #763 - Delete local files after sync
2020-01-27 08:42:00 +11:00
abraunegg 5c2ad041bd
Implement warning on big deletes to safeguard data on OneDrive (#621)
* When testing changes to onedrive client configuration, the new configuration might be invalid (see #458 for example) and you remove all your data on OneDrive accidentally. This new feature attempts to protect your data on OneDrive when performing large deletes, so that a large delete is detected and asks for confirmation before actually processing this request. This feature does not impact `--monitor` mode of operation, only standalone mode of operation.
2020-01-03 07:46:58 +11:00
abraunegg 217d72486d
Change synclist override flag to false as default when not using sync_list (#745)
* Set 'syncListConfiguredOverride' to false by default, only set to true at start if sync_list is true
* Remove logAndNotify as it is excessive for each change bundle to inform the desktop
* Add a log entry when a monitor sync loop with OneDrive starts & completes
2019-12-17 05:32:38 +11:00
abraunegg db5ceccffb
Check database for excluded sync_list items previously in scope (Issue #723) (#724)
* When using sync_list and files are moved out of sync scope online via OneDrive website, check if the item was previously in-sync in database. if true, flag for local delete.
* Rework fix for #568 so that if a directory is moved / created in sync_list scope, on the 'true-up' sync, any files that were moved with the folder are now downloaded
* Update when sync_list override is done
* Add debugging for each loop for sync_list scan handling
2019-11-21 19:58:34 +11:00
Tudor Brindus 7ba59e0b1d Fix --monitor when used with --download-only (#708)
* Fix --monitor when used with --download-only. This fixes the regression introduced in 12947d160f.
2019-10-31 14:59:44 +11:00
abraunegg 12947d160f
Fix unhandled monitor initialisation exception (Issue #704) (#705)
* Catch MonitorException when initialisation failure occurs, print error and exit ... cant enter monitor loop if we cant initialise correctly.
* Cleanup and add documentation update
2019-10-30 17:46:02 +11:00
abraunegg 9b3179540f Revert "Fix unhandled monitor initialisation exception"
This reverts commit e1be2b1e55.
2019-10-30 06:33:11 +11:00
abraunegg e1be2b1e55 Fix unhandled monitor initialisation exception
* Catch MonitorException when initialisation failure occurs, print error and exit ... cant enter monitor loop if we cant initialise correctly.
2019-10-30 06:32:17 +11:00
abraunegg 585b35724f
Reduce change scan impact of fix for #658 (#691)
* If there are delta changes (meaning something changed on OneDrive) process the changes that have been presented
2019-10-23 05:29:11 +11:00
abraunegg 6a8d575941
Modify 'Processing X changes' output to be more reflective of actual activity (Issue #679) (#680)
* When using 'sync_list' do not output that we are processing 'changes' as this is misleading.
* When using sync_list perform a full scan only the first time when using --synchronize & --monitor to avoid double processing of all OneDrive items each and every time
* When using sync_list and --monitor only perform a full object scan when we are also configured to perform a full scan
2019-10-09 12:02:17 +11:00
abraunegg e32b8b3c7f
Move logging of 'initialisation' to actually when the attempt to initialise is performed (#656)
* Move logging of 'initialisation' to actually when the attempt to initialise is performed
2019-09-11 20:26:21 +10:00
abraunegg 42d8c454e2
Fix downloading of files from OneDrive Personal Shared Folders (Issue #635) (#637)
* Fix downloading of files from OneDrive Personal Shared Folders when the OneDrive API responds with unexpected additional path data
2019-09-09 12:30:59 +10:00
abraunegg ad0daf2df5
Detect the need for --resync when config changes (#617)
* Detect the need for --resync when config changes either via config file or cli override
2019-08-24 17:18:58 +10:00
abraunegg 4e97f8810d
Implement --get-file-link (#612)
* Implement --get-file-link which will return the weburl of a file which has been synced to OneDrive
2019-08-02 18:43:31 +10:00
abraunegg 100e644ae0
Catch error when creating sync dir fails (#608)
* Catch error when creating the sync dir fails - we cant run if this is not created or present
2019-07-31 07:20:26 +10:00
Matthew Fallon f39621c2da Change unauthorized-api exit code to 3 (Issue #567) (#572)
* Changes unauthorized-api exit code to 3
* Systemd will not restart on exit code 3
2019-07-06 08:01:04 +10:00
abraunegg d939d38c4f
Implement HTTP/2 downgrade by default (#549)
* Implement HTTP/2 downgrade by default to increase application stability when using curl >= 7.62
2019-06-21 01:11:29 +10:00
abraunegg bac9ab41ae
Enhance error message for curl CA SSL issue (Issue #507) (#508)
* Enhance error message when unable to connect to Microsoft OneDrive service when the CA SSL certificate has issues
2019-05-24 19:17:08 +10:00
abraunegg caec1cb8cb
Implement --sync-root-files when using a sync_list file (Issue #491) (#492)
* Implement --sync-root-files to sync all files in the OneDrive root when using a sync_list file that would normally exclude these files from being synced
2019-05-09 21:18:49 +10:00
abraunegg fbad4b4835
Fix 'Local files not deleted' when using bad 'skip_file' entry (#480)
* Add a check for bad 'skip_file' entry that prevents incorrect searching of local changes (mainly deletes) from being synced
2019-04-25 11:00:23 +10:00
abraunegg 25d2f7284f
Update 'min_notif_changes' to 'min_notify_changes' (#467)
* Update 'min_notif_changes' to 'min_notify_changes'
2019-04-15 06:28:32 +10:00
Norbert Preining ea22d8fef5 config options for command line switches (#449)
* config options for command line switches to allow for better config handling in docker containers
2019-04-11 12:26:20 +10:00
abraunegg ea2e4a98bf
Add --upload-only check for sharepoint fix (Issue #452) (#453)
* Add a check for --upload-only use when trying to work around https://github.com/OneDrive/onedrive-api-docs/issues/935
* Update logging and print a warning message that the files are now technically different due to sharepoint bug and using --upload-only
2019-04-06 12:44:51 +11:00
abraunegg 6a6b8f128f
Issue #432 and #441 combined (#447)
* Re-add original #432 and #441 fixes to master
2019-04-02 05:21:02 +11:00
abraunegg abb82868e4
Roll back to v2.3.1 (#446)
* Roll back to v2.3.1 after bad re-base merge with PR #389
2019-04-02 04:51:25 +11:00
abraunegg 4444b9cd8a
Reduce scanning of local filesystem needlessly every sync in monitor mode (#433)
* Reduce scanning the entire local system in monitor mode for local changes
* Add 'monitor_fullscan_frequency' to set the frequency of performing a full disk scan when in monitor mode
2019-04-02 03:58:56 +11:00
Norbert Preining b7cc540c2c fixes to the merge routine 2019-03-25 10:25:10 +09:00
Norbert Preining a433b44b56 Merge branch 'master' into norbert/config-options 2019-03-25 09:41:27 +09:00
abraunegg 002f9b7aec
Prevent use of --synchronize & --monitor together (#431)
* Prevent use of --synchronize & --monitor together
2019-03-23 10:20:27 +11:00
Norbert Preining 3aff2e8251 move verbose check to first getopt to make it available early on 2019-03-17 13:10:41 +09:00
Norbert Preining ab918dd82d config directory management fixes 2019-03-17 12:41:26 +09:00
Norbert Preining 362bf35688 Merge branch 'master' into norbert/config-options 2019-03-17 12:25:27 +09:00
abraunegg 99c0267c7a
Implement 'skip_dir' config option (Issue #123) (#388)
* Implement config option 'skip_dir'
2019-03-15 06:55:05 +11:00
abraunegg 98624267c5
Implement --dry-run (#337)
* Implement new feature --dry-run
2019-03-11 17:57:47 +11:00
abraunegg ba834368ca
Implement --check-for-nosync to ignore folder when special file present (Issue #163) (#390)
* Implement a capability to ignore a folder when a special file (.nosync) is present
2019-03-03 05:58:36 +11:00
Norbert Preining e5b500a041 remove unnecessary code, cleanup 2019-02-27 08:57:04 +09:00
Norbert Preining 388a3bb102 Merge branch 'master' into norbert/config-options 2019-02-27 08:51:28 +09:00
Norbert Preining 09f328e63f configurable logging intervals in monitor mode (Issue: #378) (#391)
* Configurable interval of logging in monitor mode (Issue: #378)
2019-02-27 07:21:23 +11:00
Norbert Preining b668ecef71 Create new upload session on reinit (Issue: #379) (#392)
* By moving the UploadSession creation from the constructor to the init function we guarantee that a re-initialization also starts a new UploadSession.
2019-02-27 06:49:23 +11:00
Norbert Preining d49a598f4b clean up, move help function 2019-02-24 16:33:10 +09:00
Norbert Preining a71b9bf789 Merge branch 'master' into norbert/config-options 2019-02-24 16:27:53 +09:00
abraunegg 7a47df7a38
Implement --skip-dot-files (#386)
* Implement new CLI option --skip-dot-files to skip .files and .folders if option is used
2019-02-24 17:19:45 +11:00
Norbert Preining f2f272a8f8 move more parts of the getopt calls into config module 2019-02-22 22:00:57 +09:00
Norbert Preining a635cab0ad config options: working version 2019-02-22 16:15:10 +09:00
Norbert Preining 25b09775ab more playing around 2019-02-22 10:41:07 +09:00
Norbert Preining 7c9ea56293 some playground 2019-02-22 10:39:16 +09:00
Norbert Preining 18611c833f initial work on various config options for cmd line options 2019-02-22 10:39:16 +09:00
Norbert Preining 0d3b891177 don't annoy with notifs when offline and cannot act on changes (#385)
* Don't annoy with notifs when offline and cannot act on changes
2019-02-21 19:51:15 +11:00
abraunegg 51615bff0b
Rework PR #373 to reduce needless logging in monitor mode (#376)
* Rework PR #373 to reduce needless logging in monitor mode
2019-02-15 07:03:15 +11:00
Norbert Preining 1c6fd5df39 notification on incoming changes (#355)
* When notifications are enabled, display the number of OneDrive changes to process if any are found
* Add 'config' option 'min_notif_changes' for minimum number of changes to notify on, default = 5
* Update readme, manual page for new option
2019-01-29 04:54:03 +11:00
abraunegg 1fa7e5f20c
Add extra debugging & http protocol downgrade (Issue #314) (#327)
* Add --force-http-1.1 flag to downgrade any HTTP/2 curl operations to HTTP 1.1 protocol
* Explicitly set all bool items to false at initialisation
* Update --display-config to display sync_list if configured
* Add debug handling to display when sync_list is loaded
* Add debug handling to output the handling of OneDrive changes
2019-01-06 05:43:44 +11:00
Norbert Preining b9890ae2e5
better help output formatting (Issue #298) 2018-12-28 18:01:50 +09:00
Norbert Preining fcf1ef2e1f
signal handling and database shutdown (#315)
Add a signal handler and shutdown database connection on SIGINT and SIGTERM
to make sure that .waf files are properly incorportated into the sqlite database.
2018-12-28 11:19:20 +09:00
abraunegg 2553366a89
Implement Feature Request: Add status command or switch (Issue #112) (#307)
* Implement Feature Request: Add status command or switch
2018-12-28 12:26:03 +11:00
Norbert Preining 2ec6aa3df6
make sure sqlite checkpointing works by properly finalizing statements (#310)
If an sqlite statement is prepared and reused by using reset again and again, the connection
keeps a lock on the database, and checkpointing cannot ensure that transaction from the
wal file are carried over into the main database.

By preparing the statement every time it is used, the destructor calls finalize and thus
the lock is released.

Possible impacts during high frequency changes via monitor mode etc might arise.
2018-12-26 16:11:06 +09:00
abraunegg 5d388581b1
Resolve 'sync_dir' not read from 'config' file when run in Docker container #306 (#308)
* Fix logic for sync_dir handling on headless systems
* Fix logic where potentially a 'default' ~/OneDrive sync_dir could be set despite 'config' file configured for an alternate
* Add debug handling for sync_dir operations
* Add debug handling for homePath calculation
* Add debug handling for configDirBase calculation
* Reorder main() so that log.vdebug works where required
* Rework configDirName as this had the same issue as syncDir
* Update configDirName & syncDir '~' handling & replacement to be more robust and only replace if '~' is first char in string
* Add additional debug logging if syncDir is created
2018-12-23 11:15:10 +11:00
abraunegg 594bfcec54
Fix syncdir option is ignored (#302)
* Explicitly set value for syncdir if the option is passed in via CLI rather than 'config' file
2018-12-20 17:27:34 +11:00
abraunegg dfd3045ce4
Fix exception when missing parameter from usage option (#299)
* Handle the exception generated when an option is used which is expecting an input parameter but none is given
2018-12-20 11:05:32 +11:00
Norbert Preining 025a3b2878
check config file keys for validity (#296)
* check config file keys for validity, use setValue instead of direct access
* Update config.d
  Add 'drive_id' to be initialised, set to an empty string
* exit application if there is a configuration file error
* Issue #293 was caused by a spelling error in the configuration file. 
  If the configuration file has errors, we should not load it or run
  using the application defaults as this may have undesirable consequences
  for users data
* missed returning false on key issue
* Missed this edit of the file
2018-12-20 08:51:21 +09:00
abraunegg 7d522583ef
Implement option --display-config (Issue #100) (#292)
* Implement --display-config to show the application configuration without actually performing a sync or application init
2018-12-20 05:42:28 +11:00
Norbert Preining 91aaf91aa6 Reinitialize sync engine after three failed trials (#283)
* After extended suspend, or bad connection, the connection to OD might
go stale - thus no network connection to Microsoft OneDrive Service, skipping sync
and this continues ad infinitum. Change the last action to re-initialize the sync engine.
2018-12-13 07:08:18 +11:00
Norbert Preining 3608dcc4c2 Add very verbose (debug) mode by double -v -v (#279)
* Add very verbose (debug) mode by double -v -v - currently only used for debugging the inotify events
2018-12-08 05:01:22 +11:00
Norbert Preining 06aa42f6d1 better log messages, less noise on temporary timeouts (#275)
* better log messages, less noise on temporary timeouts
2018-12-06 18:05:52 +11:00
abraunegg 0562c52492 Handle short lived files (Issue #273) (PR #274)
* Handle short lived files (Issue #273)
* Update README.md
* Add --disable-notifications to readme
2018-12-06 08:50:46 +09:00
Norbert Preining cc6cbf5ac7 Notification support via libnotify (#270)
* add external sources for (d)notify with README and license statements
* add new logAndNotify function, make building a compile time option
* use logAndNotify and main.d, make timeout message silent
* update documentation for notification support
* add command line switch --disable-notifications
* add build-deps for libnotify in README.md
2018-12-06 06:19:00 +11:00
Norbert Preining b10e641fe9 allow starting offline in monitor mode (#266)
* SyncEngine: allow for multiple calls to init by saving the state
* Factor out syncengine initialization code so that it can be reused
* Don't exit in monitor when started offline
* Always call initSyncEngine during regular syncs, this will call the SyncEngine init routine which caches the state, thus a quasi no-op.
* Add forgotten http.shutdown
2018-12-04 11:15:44 +11:00
abraunegg 29808c905c
Implement --get-O365-drive-id to get correct SharePoint Shared Library (Issue #248) (#260)
* Implement --get-O365-drive-id to return the correct drive_id of a SharePoint Shared Library to use for syncing that repository
2018-12-04 10:59:23 +11:00
Norbert Preining 9817104ca1 continue in monitor mode when sync timed out (#265)
* update handling of connection timeouts in monitor mode & allow application continue rather than exit
2018-12-03 13:41:31 +11:00
Norbert Preining 11138602ed gracefully handle connection errors in monitor mode (#253)
* gracefully handle connection errors, resolve issue #234
2018-11-29 20:48:24 +11:00
abraunegg a26a69ce9f
Change logging to a separate logfile is no longer the default (#239)
* Change '--download' to '--download-only' to align with '--upload-only'
* Enable logging to a separate file only if the '--enable-logging' flag is passed through at client run time
* Implement configuration option for logfile location, if logging is enabled
2018-11-24 07:13:16 +11:00
abraunegg c5ee62efd8
Sync with shared folders/drives for onedrive business (#206)
* Refactor PR104 for latest code base to enable client use with SharePoint libraries
* Implement '--disable-upload-validation' as SharePoint modifies files when uploaded
* Enable '--disable-upload-validation' by default if account type is documentLibrary - as most likely this is a SharePoint repository
2018-11-24 06:26:30 +11:00
abraunegg a6336e70b3
Resolve Key not found: driveType (Issue #152) (#166)
* Add try & catch in the case of a 504 error being generated to upload the file. Currently, if a 504 is generated, it is handled gracefully, but further files to upload are stopped
* Update where ever there is a 404 try block, add a if statement to capture if a 5xx error is returned as well
2018-09-25 05:25:40 +10:00
Marcus Ball 3a4c71d7ee Fix incorrectly nested configDir in X11 systems (Issue #181) (#182)
* Fix incorrectly nested configDir in X11 systems
2018-09-21 05:45:06 +10:00
abraunegg be1cb5e5ab
Update how default path is set when running under systemd (#170)
* Update how default path is set when running under systemd
2018-09-13 08:46:27 +10:00
abraunegg 564a77fb4e
Initial fix to high CPU usage when running in monitor mode (#165)
* Add skilion commit c2cdb24131
* Add skilion commit b702afda50
2018-09-13 08:43:29 +10:00
abraunegg a3f69d4a2e
Resolve systemd/user is not supported on CentOS / RHEL (Issue #131) (#132)
* Resolve systemd/user is not supported on CentOS / RHEL (Issue #131)
2018-08-27 10:35:58 +10:00
abraunegg b98c9386f3
Resolve skipping of symlinks in monitor mode (#125)
* Resolve skipping of symlinks in monitor mode
* Fix up some spelling errors
2018-08-17 08:03:21 +10:00
abraunegg 6a5ab5607a
test for curl exception when checking online status (#117)
* Check for a curl exception when checking the online status of the client at startup
2018-08-14 07:21:11 +10:00
abraunegg 2bb5dce752
Fix --no-remote-delete (#110)
* Original implementation of --no-remote-delete used a cfg value, however this introduced 2 issues - value could be set via a config parameter which was not the intention, but also could be set to a non true value causing application issues. This patch resolves how  --no-remote-delete should have originally been implemented
2018-08-10 07:40:17 +10:00
abraunegg e922a7d85e
Implement Feature Request: Make checkinterval for monitor configurable (Issue #31) (#97)
* Implement Feature: Make checkinterval for monitor configurable
* Update monitor mode processing logic
2018-08-08 05:35:18 +10:00
abraunegg 2e04bf534a
Implement Feature Request: Upload Only Option that does not perform remote delete (Issue #49) (#96)
* Implement Feature Request: Upload Only Option that does not perform remote deletes
2018-08-05 10:43:31 +10:00
Leif Denby c0074638da Add ability to skip symlinks (Issue #87) (#92)
* Add ability to skip symlinks
2018-08-03 06:02:10 +10:00
abraunegg d0209d8dbc
Gracefully handle OneDrive account password change (#73)
* Gracefully handle OneDrive account password change rather than crashing if authorization token is now invalid
2018-07-16 09:58:36 +10:00
abraunegg 54ae6eacad
Update logic --upload-only and --local-first (#71)
* Update the logic handling of --upload-only and --local-first to be more consistent regardless of using --single-directory or not.
2018-07-15 15:22:08 +10:00
abraunegg fed53e8612
Update sync_dir handling (#55)
* Update sync_dir handling to use the absolute path for setting parameter to something other than ~/OneDrive via config file or command line
2018-07-10 05:34:27 +10:00
abraunegg 77ac29acdf Update feature flag
* Update feature flag from 'check-mount' to 'check-for-nomount' as this is what this option does
* Add information into readme file
2018-06-27 06:23:17 +10:00
abraunegg 1cfe137a92 Implement feature request: Detect when sync-folder is missing
* Provide a new switch 'check-mount' which will check for the presence of a hidden file '.nosync' file. If this file is found in the sync dir, then the sync process will shutdown. Refer to #8 for further details.
2018-06-17 08:27:43 +10:00
abraunegg 2827467296 Resolve std.file.FileException when checking logfile path
* Resolve std.file.FileException@std/file.d(2954): /var/log/onedrive/: Permission denied when there is no permission to create the directory on application startup
2018-05-16 19:19:43 +10:00
Matthias C. M. Troffaes c863d2c3c9 Fix config folder. 2018-05-16 09:11:50 +01:00
abraunegg bc15d7f77f Display informative message when not using --synchronize
* Prior to all the changes in #314, the client would sync regardless of
options. #314 introduced a flag that must be included if you want to
sync, however there was no feedback if '--synchronize' was not included
and the sync was not occuring. This patch provides that notification &
feedback.
2018-04-24 12:14:36 +10:00
abraunegg 5cb92c01b9 Expand HTTPS debug to include GET JSON responses
* Change debug flag from --debug-http to --debug-https as OneDrive only
uses HTTPS, not HTTP
* Expand --debug-https to print the JSON responses to GET operations.
This is to assist with debugging #334 where a change in the MS API
(https://github.com/OneDrive/onedrive-api-docs/issues/834) no longer
sends 'lastModifiedDateTime' with responses to some GET's. This issue
seems to manefest itself with OneDrive Business accounts currently.
2018-04-15 20:02:39 +10:00
abraunegg 21f4d8e858 Add init.d service file & modify how configDirName is set
* Add init.d service file & helper script to start service
* Change how configDirName is set as XDG_CONFIG_HOME does not exist on
systems where X11 is not present. When using init scripts and using
XDG_CONFIG_HOME, ~ is not expanded thus existing config cannot be found.
Using ~ in --confdir also does not work when running under init.d
2018-04-13 09:33:16 +10:00
abraunegg 091e675d85 Update README.md based on current changes made
* Update readme
* Tweak logging output based on sync options selected
2018-04-12 10:18:18 +10:00
abraunegg f1aa190a91 Add uploadOnly flag and add username to logfile name
* Add specific uploadOnly flag so that the onedrive client only uploads
data from the local directory and does not download changes from
OneDrive
* Change the logfile name to include the username running the onedrive
client
2018-04-07 17:06:57 +10:00