Commit graph

490 commits

Author SHA1 Message Date
Norbert Preining 0d05274f32 Fix renaming of files sync issues (#252)
* issue #249 do not safeRename the target if it is in sync
* issue #249 only remove actual path if the associated ids agree
2018-11-29 21:31:44 +11:00
Norbert Preining 11138602ed gracefully handle connection errors in monitor mode (#253)
* gracefully handle connection errors, resolve issue #234
2018-11-29 20:48:24 +11:00
abraunegg a26a69ce9f
Change logging to a separate logfile is no longer the default (#239)
* Change '--download' to '--download-only' to align with '--upload-only'
* Enable logging to a separate file only if the '--enable-logging' flag is passed through at client run time
* Implement configuration option for logfile location, if logging is enabled
2018-11-24 07:13:16 +11:00
abraunegg c5ee62efd8
Sync with shared folders/drives for onedrive business (#206)
* Refactor PR104 for latest code base to enable client use with SharePoint libraries
* Implement '--disable-upload-validation' as SharePoint modifies files when uploaded
* Enable '--disable-upload-validation' by default if account type is documentLibrary - as most likely this is a SharePoint repository
2018-11-24 06:26:30 +11:00
abraunegg 23e79d39e1 Update sync.d
* Update exit codes on sync.init() to -1
2018-11-22 07:11:14 +11:00
abraunegg cbd3ecbf71
Resolve unable to create folders on shared OneDrive accounts (Issue #217) (#244)
* Update logic to use remote item details for folder creation
* Backport sync.d changes from PR #246 to this PR before merge
2018-11-22 07:03:04 +11:00
abraunegg fc162b94e2
Resolve database assertion failure due to authentication HTTP/1.1 400 Bad Request (#246)
* Enhance initialisation error handling to catch 4xx errors
2018-11-21 18:54:08 +11:00
abraunegg eaedfac34a
Update onedrive.d for additional debug logging (#238)
* In case the user does not use the --verbose tag, if we are doing --https-debug we want this printed to the console
2018-11-18 07:05:01 +11:00
abraunegg e70722862c
update files for 2.1.6 release (#235)
* update files for 2.1.6 release
2018-11-15 06:43:58 +11:00
abraunegg f785396643
Update HTTP/2 handling for session uploads (#233)
* Remove HTTP 1.1 downgrade for session uploads
* Handle HTTP/2 0 (zero) response code for successful session data upload
* Add debugging for actual server response, not curl interpreted response when using --debug-https
2018-11-15 06:08:55 +11:00
abraunegg 8a98bf3e30
Add handling for HTTP Code 302 generated by HTTP/2 calls (#229)
* Update initial Curl 7.62.0 fix with a better solution which handles the redirect's given by HTTP/2 connections (credit @Popa21)
* Use HTTP/1.1 for session uploads otherwise the response when using HTTP/2 generates a 'JSONValue is not an object' error
2018-11-12 06:01:57 +11:00
abraunegg 86ea576144
Handle HTTP request returned status code 412 (Precondition Failed) for session uploads (#227)
* Handle HTTP request returned status code 412 (Precondition Failed) for session uploads to OneDrive Personal Accounts
* Fix Failed to remove file /root/.config/onedrive/resume_upload: No such file or directory if there is a session upload error and the resume file does not get created
* Handle response codes when using 2 different systems using --upload-only but the same OneDrive account and uploading the same filename to the same location
2018-11-11 11:15:48 +11:00
abraunegg 11e477f045
Resolve files will not download when using curl 7.62.0 (http2 issue) (#225)
* Force connections to use HTTP 1.1 as Curl 7.62.0 defaults to using http2 if h2 support is built in which causes issues
2018-11-11 06:38:41 +11:00
abraunegg eb2cbfd0a8
Resolve 'Key not found: path' when syncing from some shared folders (#211)
* Add a check to validate if the parentReference object has a path object
* Fix spelling error - trough -> through
* Only upload changes on remote folder if the item is in the database - dont assert if false
2018-10-30 06:20:52 +11:00
abraunegg 53b72e0d9d
Resolve syncing of Shared Folders due to OneDrive API change - Issue #144 (#191)
* Update handling of shared folders and items which was previously broken to a OneDrive API change
* Extract syncFolderName from either local or remote folder - not just local folder.
2018-10-10 19:31:21 +11:00
abraunegg 74b9163b06
Resolve Key not found: expirationDateTime on session resume (Issue #174) (#176)
* Handle an invalid response on session resume when a 4xx / 5xx response is generated from the OneDrive service
2018-10-04 09:38:23 +10:00
abraunegg 14b2de8f4c
Resolve crash with strange filename (Issue #151) (#178)
* Update file handling to look for HTML ASCII codes which will cause uploads to fail
2018-10-04 09:33:39 +10:00
abraunegg 31fc6a4d4f
Validate filename length before download to conform with Linux FS limits #142 (#180)
* Validate filename length before download to conform with Linux FS limits
2018-10-04 09:31:28 +10:00
abraunegg a6336e70b3
Resolve Key not found: driveType (Issue #152) (#166)
* Add try & catch in the case of a 504 error being generated to upload the file. Currently, if a 504 is generated, it is handled gracefully, but further files to upload are stopped
* Update where ever there is a 404 try block, add a if statement to capture if a 5xx error is returned as well
2018-09-25 05:25:40 +10:00
Marcus Ball 3a4c71d7ee Fix incorrectly nested configDir in X11 systems (Issue #181) (#182)
* Fix incorrectly nested configDir in X11 systems
2018-09-21 05:45:06 +10:00
abraunegg be1cb5e5ab
Update how default path is set when running under systemd (#170)
* Update how default path is set when running under systemd
2018-09-13 08:46:27 +10:00
abraunegg 564a77fb4e
Initial fix to high CPU usage when running in monitor mode (#165)
* Add skilion commit c2cdb24131
* Add skilion commit b702afda50
2018-09-13 08:43:29 +10:00
abraunegg 4919a58246
Resolve File download fails if the file is marked as malware in OneDrive (Issue #153) (#155)
* Gracefully handle the situation where OneDrive has marked a file as malware and will not provide the file for download
2018-09-13 08:41:46 +10:00
abraunegg ce311db18d
Resolve Correctly handle file case sensitivity issues in same folder (Issue #146) (#147)
* Resolve to correctly handle file case sensitivity issues in same folder by using the same mechanism as Issue #139
2018-08-27 13:11:56 +10:00
abraunegg b2c9e041be
Resolve 'The parent item is not in the local database' (Issue #139) (#143)
* Resolve 'The parent item is not in the local database' where data cannot be uploaded / saved to the local database because there are folders in the same path with the same name due to case sensitivity issues
2018-08-27 10:47:01 +10:00
abraunegg b9faa5cd1f
Resolve HTTP request returned status code 429 (Too Many Requests) (Issue #133) (#138)
* Adding a retry method for downloading files when a HTTP request returned status code 429 (Too Many Requests) response is returned from OneDrive
2018-08-27 10:45:26 +10:00
borouhin a08df3d7fd Resolve handling of maximum path length calculation (Issue #134) (#135)
* Update the handling of maximum path length calculation
2018-08-27 10:43:49 +10:00
abraunegg a3f69d4a2e
Resolve systemd/user is not supported on CentOS / RHEL (Issue #131) (#132)
* Resolve systemd/user is not supported on CentOS / RHEL (Issue #131)
2018-08-27 10:35:58 +10:00
abraunegg 8031f9491a
Resolve Gateway Timeout - JSONValue is not an object (Issue #127) (#129)
* Resolve Gateway Timeout - JSONValue is not an object (Issue #127)
2018-08-27 10:35:15 +10:00
abraunegg b98c9386f3
Resolve skipping of symlinks in monitor mode (#125)
* Resolve skipping of symlinks in monitor mode
* Fix up some spelling errors
2018-08-17 08:03:21 +10:00
abraunegg e0b1b595e4
Add file access check for file upload (#116)
* Add a check to validate that we can read the file by reading the first 10MB when uploading a new file or resuming an upload
2018-08-14 18:30:13 +10:00
abraunegg 6a5ab5607a
test for curl exception when checking online status (#117)
* Check for a curl exception when checking the online status of the client at startup
2018-08-14 07:21:11 +10:00
abraunegg 6a6e0079af
Update --no-remote-delete to handle directories (#119)
* Update --no-remote-delete to handle not to delete remote directories if deleted locally
2018-08-14 07:16:20 +10:00
abraunegg bed2b6c75f
Implement Feature Request: Add a progress bar for large uploads & downloads (Issue #12) (#99)
* Implement feature request to add a progress bar for large file uploads & downloads
2018-08-10 07:46:27 +10:00
abraunegg 2bb5dce752
Fix --no-remote-delete (#110)
* Original implementation of --no-remote-delete used a cfg value, however this introduced 2 issues - value could be set via a config parameter which was not the intention, but also could be set to a non true value causing application issues. This patch resolves how  --no-remote-delete should have originally been implemented
2018-08-10 07:40:17 +10:00
abraunegg e922a7d85e
Implement Feature Request: Make checkinterval for monitor configurable (Issue #31) (#97)
* Implement Feature: Make checkinterval for monitor configurable
* Update monitor mode processing logic
2018-08-08 05:35:18 +10:00
abraunegg 2a113de66c
Resolve missing config value: no-remote-delete (#103)
* Resolve missing config value: no-remote-delete
2018-08-07 19:27:21 +10:00
abraunegg 2e04bf534a
Implement Feature Request: Upload Only Option that does not perform remote delete (Issue #49) (#96)
* Implement Feature Request: Upload Only Option that does not perform remote deletes
2018-08-05 10:43:31 +10:00
Leif Denby c0074638da Add ability to skip symlinks (Issue #87) (#92)
* Add ability to skip symlinks
2018-08-03 06:02:10 +10:00
abraunegg aa995eb3d3
Implement better handling of database errors & database disk space issues (Issue #77) (#80)
* Implement better handling of database exit scenarios when there is zero disk space left on drive where the items database resides (Issue #77)
* Implement better handling of incorrect database permissions
* Implement better handling of different database versions to automatically re-create tables if version mis-match (RHBZ #1598934)
2018-08-02 08:28:52 +10:00
abraunegg 91d8b7ab93
Gracefully handle a timeout when accessing the Microsoft OneDrive Service (Issue #82) (#86)
* Gracefully handle a timeout when accessing the Microsoft OneDrive Service
* Update HTTP connection handling for long running operations based on https://forum.dlang.org/post/k2vbk5$re7$1@digitalmars.com & https://github.com/dlang/phobos/pull/797
2018-08-02 08:25:37 +10:00
abraunegg 6b670df686
Fix localFileModifiedTime to not use fraction seconds (#91)
* Fix localFileModifiedTime when uploading a new file to compare using no fraction seconds
2018-08-02 08:22:26 +10:00
abraunegg a491620c1a Update logging message for single directory removal
* Update logging message for single directory removal - we are trying to delete not create
2018-07-19 06:43:30 +10:00
abraunegg d0209d8dbc
Gracefully handle OneDrive account password change (#73)
* Gracefully handle OneDrive account password change rather than crashing if authorization token is now invalid
2018-07-16 09:58:36 +10:00
abraunegg a4e055fdc3
Update 'HTTP 412 - Precondition Failed' error handling (#67)
* Update 'HTTP 412 - Precondition Failed' error handling based on local testing to ensure a http 412 response is gracefully handled & the modified file is subsequently uploaded to OneDrive successfully.
2018-07-16 06:36:31 +10:00
abraunegg 54ae6eacad
Update logic --upload-only and --local-first (#71)
* Update the logic handling of --upload-only and --local-first to be more consistent regardless of using --single-directory or not.
2018-07-15 15:22:08 +10:00
Robert Foster f3f216c1a8 Fix for Issue #23 (#60)
* Resove computeQuickXorHash generates a different hash when files are > 64Kb
2018-07-11 06:43:45 +10:00
abraunegg c0d2874ace
Fix: skip_file configuration doesn't handle spaces or specified directory paths (Issue #52) (#57)
* Update skip_file handling by using updated wild2regex to check files & paths of items to skip ro resolve issue #52
2018-07-10 13:08:17 +10:00
abraunegg 4667ecad12
Fix: Crash if file is locked by online editing (status code 423) (Issue #36) (#50)
* Resolve application crash if file is locked by online editing (status code 423)
2018-07-10 10:30:22 +10:00
abraunegg 0215774726 Import only atoi
* Import only atoi
2018-07-10 09:21:47 +10:00
abraunegg 9cb8c297fe comment out meaningless output for the moment
* item.name is not available, so we get a bunch of meaningless log output when deleting items. This will get fixed when wider logging changes are synced in
2018-07-10 09:14:35 +10:00
abraunegg fed53e8612
Update sync_dir handling (#55)
* Update sync_dir handling to use the absolute path for setting parameter to something other than ~/OneDrive via config file or command line
2018-07-10 05:34:27 +10:00
abraunegg 0109ef6a0c
Resolve compilation issue with dmd-2.081.0 (#48)
* Resolve: Error: module `stdlib` is in file 'std/c/stdlib.d' which cannot be read when using dmd-2.081.0 due to deprication of function - https://dlang.org/changelog/2.081.0.html#remove_std_c
2018-07-04 11:37:00 +10:00
abraunegg 83c28ba3b6
Fix: Multiple versions of file shown on website after single upload (Issue #2) (#40)
* Resolve multiple versions of file shown on website after single upload - however Issue #23 & OneDrive API bug (https://github.com/OneDrive/onedrive-api-docs/issues/877) will still create 2 file versions on OneDrive Business. The work around is to disable file versions until #877 is resolved.
2018-07-03 07:24:57 +10:00
abraunegg 5e48ba96ac
Fix: Error when trying to upload a file with weird non printable characters present (Issue #35) (#41)
* Implement an invalid whitespace filename check which currently checks for a 'new line' whitespace entry in a filename. Regex can be expanded if other files with similar characteristics are found.
2018-07-02 19:40:54 +10:00
abraunegg 871b3d72c4
Fix: Name checking rule not exactly right (Issue #34) (#38)
* Fix regex for name checks to be more explicit
* Update name validation checks to be more compliant with Microsoft requirements
2018-07-02 14:05:21 +10:00
abraunegg 6036283d20 Add '+' as a character to escape
* Add '+' as a character to escape when using the wild2regex function
2018-06-28 13:46:20 +10:00
abraunegg 63c5dd86b7
Merge pull request #28 from abraunegg/Issue-#8
Implement feature request: Detect when sync-folder is missing
2018-06-27 16:27:41 +10:00
abraunegg 77ac29acdf Update feature flag
* Update feature flag from 'check-mount' to 'check-for-nomount' as this is what this option does
* Add information into readme file
2018-06-27 06:23:17 +10:00
abraunegg 17e5138b5a
Resolve sync issues with Shared Folders & Personal OneDrive Accounts (#26)
* Resolve Shared Folders not syncing (OneDrive Personal)
* Resolve 'Key not found: fileSystemInfo' when then item is a remote item (OneDrive Personal)
* Resolve where file names are being truncated by 5 characters with each sync (OneDrive Business)
* remove extra // not required
* Remove commented out code to avoid confusion
2018-06-17 12:02:58 +10:00
abraunegg 1cfe137a92 Implement feature request: Detect when sync-folder is missing
* Provide a new switch 'check-mount' which will check for the presence of a hidden file '.nosync' file. If this file is found in the sync dir, then the sync process will shutdown. Refer to #8 for further details.
2018-06-17 08:27:43 +10:00
abraunegg d6b4930d0a Revert "Resolve: Key not found: fileSystemInfo when syncing shared folders"
This reverts commit e7267e597b.
2018-06-13 16:35:52 +10:00
abraunegg 2fe63f581d Revert "Encode syncFolderName for easier 'canFind' processing"
This reverts commit 943fcc032b.
2018-06-13 06:46:56 +10:00
abraunegg 943fcc032b Encode syncFolderName for easier 'canFind' processing
* Resolve "Remote change discarded - not in --single-directory scope" when syncing Shared Folder sub directories
2018-06-10 11:10:49 +10:00
abraunegg e7267e597b Resolve: Key not found: fileSystemInfo when syncing shared folders
* Resolve #11 where shared folders were unable to be sync'd due to fileSystemInfo data being within the remoteItem object
* Initial work on resolving #2, but fix not validated or complete
2018-06-09 21:43:39 +10:00
abraunegg 2576b69a88 Resolve 'database is locked' message and crash
* Resolve 'database is locked' when another onedrive process is running and has exclusive access to the database
2018-06-02 06:19:29 +10:00
abraunegg dc4bfb62ff
Update sync.d
Update comment - 256 -> 400
2018-05-25 14:17:28 +10:00
abraunegg 3381511e19 Update path and naming limitations
* Resolve https://github.com/abraunegg/onedrive/issues/6
* Update OneDrive Business path length to 400 characters
* Update allowed characters for path and file names
2018-05-25 14:12:39 +10:00
abraunegg 18ee4649e0 Resolve #356 by adding additional check for 409 response
* Add an additional check for 409 (Conflict) being returned when creating directories even when the directory is only attempted to be created when it cannot be found.
2018-05-23 03:46:13 +10:00
abraunegg 2827467296 Resolve std.file.FileException when checking logfile path
* Resolve std.file.FileException@std/file.d(2954): /var/log/onedrive/: Permission denied when there is no permission to create the directory on application startup
2018-05-16 19:19:43 +10:00
Matthias C. M. Troffaes c863d2c3c9 Fix config folder. 2018-05-16 09:11:50 +01:00
abraunegg 0b1ca6dbfd Fix incorrect id tag
* Fix incorrect id tage when checking database if item exists
2018-05-15 07:15:54 +10:00
abraunegg 8b0e9513e3 Resolve another itemdb.d(307): Assertion
* Clean up --single-directory logging as we should not be spilling out the actual item details
* Fix some spelling errors
* Fix another core.exception.AssertError@src/itemdb.d(307): Assertion failure when attempting to compute the path
* Fix logging when items are being filtered out as non relevant
* Add upload validation that file uploaded equals the size of the file locally as this is a contributing factor to 412 errors
2018-05-15 06:59:17 +10:00
abraunegg 9520fe7716 Resolve itemdb.d(295): Assertion failure
* Resolve core.exception.AssertError@src/itemdb.d(295): Assertion failure when performing a --single-directory sync
2018-05-09 16:39:23 +10:00
abraunegg 41976ed216 Update long path handling
* Update long path handling to account for differences between OneDrive Personal and OneDrive Business Accounts
* Update logging to be cleaner on fragment uploads
2018-05-09 06:47:03 +10:00
abraunegg accb62bdeb Update logging of 500 & 504 errors
* Update logging of 500 & 504 errors. For 504 errors this was 'breaking the formatting of the log line. For 500 errors - this needs to be logged regardless of verbosity.
2018-05-07 05:37:47 +10:00
abraunegg 0c4def83f8 Resolve 500 & 504 HTTP Status Codes
* When completing some operations on OneDrive, a 504 error can be returned when Microsoft Graph times out when communicating with internal resources. Rather than failing, handle the 504 and retry the request.
* When performing numerous changes via OneDrive online, Microsoft Graph has an issue in interpreting the the response, thus generates a 500 internal server error. As this is something we cannot fix, handle the 500 error by gracefully exiting.
2018-05-05 17:00:50 +10:00
abraunegg 4e0c3a4cf0 Integrate #346 for sqlite performance enhancement
* Update database journal mode to WAL as per
https://www.sqlite.org/wal.html
2018-05-03 16:38:40 +10:00
abraunegg 0b5dc66507 Rework OneDrive generating a 412 'Precondition Failed' Error
* Rework the 412 fix so that the OneDrive client gracefully handles the
error & retries the metadata update without the eTag/cTag causing the
issue.
* Fix the double logging to console of 'upload' and 'download' items
2018-05-03 16:21:53 +10:00
abraunegg 90dec8d6f1 Resolve build deprecation warning
* Resolve build warning: Deprecation: `std.net.curl.dur` is not visible
from module by adding std.datetime & reordering dependancy import order
2018-04-30 19:41:54 +10:00
abraunegg ccd4dfbaf7 Resolve #341 '--single-directory' edge case
* Resolve where '--single-directory' is used to sync a single directory,
but a file / folder is 'moved' on OneDrive to outside the scope of the
focus of the '--single-directory' path. Before this change, the file
would remain in the local path (original location) whilst it would
reside in the new OneDrive path. This patch looks for this mis match and
deletes the local file / folder to reflect it is no longer in the same
OneDrive path.
2018-04-29 12:39:27 +10:00
abraunegg 6fe06d03ac Resolve 'Directory not empty' issues on bulk deletes
* In some cases OneDrive does not send all delete changes or changes are
out of order. When deleting local files, this can lead to a 'Directory
not empty' error. This change adds to check for any remaining children
of a directory that is to be deleted and delete them before
2018-04-26 09:45:18 +10:00
abraunegg 65a1181fcf Resolve std.exception.ErrnoException when user not part of users group
* Resolve: std.exception.ErrnoException@std/stdio.d(423): Cannot open
file `/var/log/onedrive/xxxxx.onedrive.log' in mode `a' (Permission
denied). This issue occurs if the user running 'onedrive' is not part of
the standard 'users' group (GID 100)
2018-04-25 12:11:34 +10:00
abraunegg bc15d7f77f Display informative message when not using --synchronize
* Prior to all the changes in #314, the client would sync regardless of
options. #314 introduced a flag that must be included if you want to
sync, however there was no feedback if '--synchronize' was not included
and the sync was not occuring. This patch provides that notification &
feedback.
2018-04-24 12:14:36 +10:00
abraunegg 034eccfaec Resolve #132 - Couldn't resolve host name on handle
* By default the DNS timeout on HTTP.method is too short. If DNS
resolution takes too long, a resolution error is thrown. Increase DNS
timeout to 5 seconds.
* Cleanup unittest - double declaration
2018-04-24 12:10:27 +10:00
abraunegg dff245d29b Resolve 'view.delta can only be called on the root'
* Resolve the error 'view.delta can only be called on the root' when
using OneDrive Business Account & --single-directory switch
* Change checkDatabaseForOneDriveRoot so that it is performed at init()
and not needed to be called a number of other times throughout the code
- leads to less HTTPS calls being made
* Cleanup comments that are no longer relevant due to
checkDatabaseForOneDriveRoot being run at init()
2018-04-23 10:58:47 +10:00
abraunegg 63f991a64e Reduce HTTPS calls for rootId & driveId by using init() set values
* Rather than make repeated HTTPS calls for the OneDrive rootId &
driveId, reuse the rootId & driveId as set via init() where possible
2018-04-21 16:30:35 +10:00
abraunegg 7a394a065f Resolve issue #73 - OneDrive Error 507 Insufficient Storage Exception
* Resolve https://github.com/skilion/onedrive/issues/73 by by tracking
the change in available free disk space when uploading a new file to
OneDrive
2018-04-21 14:54:50 +10:00
abraunegg 82ef982240 Configure at sync.init() global variables for driveId, rootId, accountType & space available
* Configure at sync.init() global variables for driveId, rootId,
accountType & space available
2018-04-21 14:32:39 +10:00
abraunegg 134c14e459 Resolve 'Key not found: lastModifiedDateTime' (OneDrive API change)
* Original patch (77c0cdbe24) whilst it
resolved the issue of a OneDrive API change, when a file was deleted on
OneDrive, it was not actually deleted on the local file system. This
patch re-implements the fix for 'Key not found: lastModifiedDateTime' by
taking into account the deleted state of the item and correctly removes
the local file that is now in the deleted state.
2018-04-21 14:03:02 +10:00
abraunegg 751d790631 Revert "Resolve 'Key not found: lastModifiedDateTime'"
This reverts commit 77c0cdbe24.
2018-04-21 07:32:31 +10:00
abraunegg 076b180e63 Revert "Manually revert lastModifiedDateTime"
This reverts commit 4a8456fa96.
2018-04-21 07:32:18 +10:00
abraunegg 4a8456fa96 Manually revert lastModifiedDateTime
* Manually revert lastModifiedDateTime
2018-04-21 07:24:19 +10:00
abraunegg 8a059dc0e1 Revert "Switch to using list_children for an item id's children"
This reverts commit fedaedacd7.
2018-04-21 07:19:04 +10:00
abraunegg 3d224d0595 Revert "Resolve issue #73 - 507 Insufficiente Storage Exception"
This reverts commit 648c1ef094.
2018-04-21 07:19:00 +10:00
abraunegg 894bdcde08 Revert "Fix itemdb.d assertion"
This reverts commit 66d00c5ec9.
2018-04-21 07:18:53 +10:00
abraunegg 095aa64cd6 Revert "Revert 77c0cdbe24 temporarily"
This reverts commit be03fa48df.
2018-04-21 07:18:46 +10:00
abraunegg be03fa48df Revert 77c0cdbe24 temporarily
* The fix in 77c0cdbe24 whilst resolves
'Key not found: lastModifiedDateTime' introduces a bug where files
deleted by onedrive, are not deleted locally. Reverting the change to
implement a fuller fix for 'Key not found: lastModifiedDateTime'.
2018-04-21 07:09:18 +10:00
abraunegg 66d00c5ec9 Fix itemdb.d assertion
* Fix core.exception.AssertError@src/itemdb.d(293): Assertion failure
introduced after onedrive.viewChildrenById(driveId, id) change
(fedaedacd7)
2018-04-20 18:41:49 +10:00
abraunegg 648c1ef094 Resolve issue #73 - 507 Insufficiente Storage Exception
* Resove https://github.com/skilion/onedrive/issues/73 by implementing a
check, initialised on init() to track the space used by uploads vs space
available
2018-04-19 16:14:42 +10:00
abraunegg fedaedacd7 Switch to using list_children for an item id's children
* Change how onedrive get's the children from a particular item id.
Previously view.delta was used to return the children, however
view.delta is not implemented on children when using OneDrive Business
accounts. By using list_children, we can get the children from any id
correctly when using either a 'personal' or 'business' account
2018-04-19 14:31:35 +10:00
abraunegg 77c0cdbe24 Resolve 'Key not found: lastModifiedDateTime'
* Fix 'Key not found: lastModifiedDateTime'
* Handling of Tombstoned items in the database
* Add for debugging purposes the rootID, driveID and Account Type
* Code cleanup - remove old commented out debugging items that are not
needed
2018-04-18 06:28:08 +10:00
abraunegg 835806f0eb Update "hidden" directory & skip_file handling
* Partial rollback of 570d42269e to
original logic behind isNameExcluded()
* Removed .* from default skip_file configuration
2018-04-16 21:39:45 +10:00
abraunegg 5cb92c01b9 Expand HTTPS debug to include GET JSON responses
* Change debug flag from --debug-http to --debug-https as OneDrive only
uses HTTPS, not HTTP
* Expand --debug-https to print the JSON responses to GET operations.
This is to assist with debugging #334 where a change in the MS API
(https://github.com/OneDrive/onedrive-api-docs/issues/834) no longer
sends 'lastModifiedDateTime' with responses to some GET's. This issue
seems to manefest itself with OneDrive Business accounts currently.
2018-04-15 20:02:39 +10:00
abraunegg 0febd67bd3 Add Microsoft naming conventions and namespace validation for items that will be uploaded
* Add Pull #104 (@JunpeiAnzai) to ensure that uploaded files conform to
Windows naming conventions and namespaces. Refer to
https://msdn.microsoft.com/en-us/library/aa365247 for further details
regarding Windows naming conventions and namespaces.
2018-04-13 11:20:38 +10:00
abraunegg 21f4d8e858 Add init.d service file & modify how configDirName is set
* Add init.d service file & helper script to start service
* Change how configDirName is set as XDG_CONFIG_HOME does not exist on
systems where X11 is not present. When using init scripts and using
XDG_CONFIG_HOME, ~ is not expanded thus existing config cannot be found.
Using ~ in --confdir also does not work when running under init.d
2018-04-13 09:33:16 +10:00
abraunegg 091e675d85 Update README.md based on current changes made
* Update readme
* Tweak logging output based on sync options selected
2018-04-12 10:18:18 +10:00
abraunegg 7eb11f0b44 Log why we are skipping a large file for upload
Log the reason why a file is not being uploaded (missed this in fec7654)
2018-04-12 08:44:52 +10:00
abraunegg fec7654118 Resolve Issues #121, #294, #329
Resolve onedrive crash when uploading files that exceed the allowed
OneDrive individual file limit
2018-04-12 07:31:42 +10:00
abraunegg 570d42269e Fix onedrive not syncing "hidden" directories
* Fix the handling of hidden filenames & directories (.hello, .git etc)
so that these are uploaded / downloaded without issue.
* Add verbose logging for uploaded files as to why they were skipped
2018-04-11 13:02:06 +10:00
abraunegg 5e0e763317 Change logging of upload, download and deletion activities
Prior to this change logging of upload, download and deletion activities
were only written to the log file when using the --verbose flag, however
they would be written to console / syslog. This corrects that behaviour.
2018-04-10 20:03:28 +10:00
abraunegg f1aa190a91 Add uploadOnly flag and add username to logfile name
* Add specific uploadOnly flag so that the onedrive client only uploads
data from the local directory and does not download changes from
OneDrive
* Change the logfile name to include the username running the onedrive
client
2018-04-07 17:06:57 +10:00
abraunegg 8463e60acc Update onedrive.d to remove '&'
Change '&error' to 'error' on line 37 of onedrive.d

Issue uncovered where prior code with '&' would compile without issue on
DMD reference compiler v2.078.3 and below, but using '2.079.0' this
would throw an error if '&' was present.

The original patch added the '&' to get the code to compile, this simple
reverts this change.

Note: For future reference compile this code with DMD v2.079.0 or later.
2018-03-30 08:14:00 +11:00
abraunegg dd73ae3c4b OneDrive Client Changes
* Fix 4xx errors including (412 pre-condition)
* Add Logging - log to a file (/var/log/onedrive/onedrive.log)
* Add http(s) debugging as a flag
* Add dont sync when just blindly running the application
* Add individual folder sync - ie ~/OneDrive/blah/ vs. syncing
everything in ~/OneDrive/
* Add sync from local directory first rather than download first then
upload
* Add upload long path check (430 character limitation)
2018-03-14 15:43:40 +11:00
skilion c231b13ad4 Disabled buffering on stdout 2018-02-18 18:24:46 +01:00
skilion 008a5756b3 Fix crash when the delta link is expired 2018-02-18 18:02:38 +01:00
skilion d9a5d1b3b9 fix regex for parsing authentication uri 2018-01-20 21:13:01 +01:00
Mikael Göransson a1e326f9b4 invalid_grant, error 70000
when trying to redeem an code, this error always occured, no matter
which parts of the response uri/url that was used.

the old regexp for parsing the code contained 3 groups, where as the
actual code always was the last group, and the second group was either ?
or &, and the first group would've been everything up until "code=".

changed from a matching group to character class so there would only be
two matching groups, so calling popFront() would actually leave the
authorization code in the front.
2018-01-20 20:51:33 +01:00
skilion 867c8ed26c improved regex 2018-01-19 18:56:14 +01:00
skilion 03f746a650 Grammar fix 2018-01-19 17:56:36 +01:00
skilion c7e09304a6 fix for https://github.com/skilion/onedrive/issues/269 2018-01-06 21:59:05 +01:00
skilion abad626b3d ensures that the parentId of the root is null 2018-01-06 19:27:27 +01:00
skilion bb3121e10d fix for https://github.com/skilion/onedrive/issues/269 2018-01-06 17:50:36 +01:00
skilion 9f7d9249bc restore prev redirect url 2018-01-02 16:32:16 +01:00
skilion 96c364c5fc removed tmp instruction 2018-01-02 15:06:20 +01:00
skilion d7f6674f60 removed redundant key parentDriveId 2018-01-02 15:05:32 +01:00
skilion 4ebc4a8544 handle move and delete of remote folders in monitor mode 2018-01-02 13:41:56 +01:00
skilion 9bedec0d7c handle moving items between drives 2018-01-01 18:38:08 +01:00
skilion c54d1927e9 do not throw in makeItem 2018-01-01 16:20:28 +01:00
skilion 764975e224 download only option 2017-12-31 17:11:17 +01:00
skilion 80cfdf62cf handle the case of deleting a remote folder locally 2017-12-31 16:56:56 +01:00
skilion fba3ed999e handle large uploads with parent id 2017-12-31 16:11:02 +01:00
skilion c8d5e03be8 upload files new with parent id 2017-12-31 13:47:18 +01:00
skilion 6907daa5e8 create dir with parent id 2017-12-31 13:18:11 +01:00
skilion 95c952fe62 handle deleting remote folders 2017-12-31 12:44:41 +01:00
skilion b7adc4d0cc WIP on remote folders 2017-12-31 02:30:31 +01:00
skilion b250214577 fix path handling of direct children of the root 2017-12-29 16:51:08 +01:00
skilion dd3e1b2861 fix wrong parameters in call 2017-12-29 12:24:26 +01:00
skilion 48559c592d fix handling root of remote item 2017-12-28 19:58:31 +01:00
skilion 90ab54cfc6 restore foreign keys and indices 2017-12-28 19:57:51 +01:00
skilion 8e91f57cb3 fix select children 2017-12-28 15:42:33 +01:00
skilion e3f011b724 write logs on stdout instead of stderr 2017-12-28 15:21:41 +01:00
skilion 35ce743b39 removed exception chaining 2017-12-28 15:03:15 +01:00
skilion 789ec85e0c partial remote items support 2017-12-27 15:13:28 +01:00
skilion d9c9915bc3 update API to use full item addressing 2017-12-27 15:12:38 +01:00
skilion 22db459ac4 fixed bug where remotely deleted and recreated directories would make the application crash 2017-08-01 19:20:02 +02:00
skilion 9765b27902 new parameter syncdir 2017-08-01 19:11:50 +02:00
skilion 7fae9c1bef versioning 2017-07-14 11:40:57 +02:00
skilion 849dd36276 check if the item is in the db before executing operations on it 2017-07-02 15:38:21 +02:00
lanhin 8f5f54f6a1 Merge branch 'master' of github.com:skilion/onedrive 2017-06-22 22:25:04 +08:00
skilion a61abc4e15 fix issue #202 2017-06-20 22:01:57 +02:00
skilion 1581c6989a fix error in QuickXorHash implementation 2017-06-16 11:40:07 +02:00
skilion 974be0884b fixed bug where an item is incorrectly assumed to be a file 2017-06-15 15:17:47 +02:00
skilion 126f2719b8 hack for onedrive biz 2017-06-15 14:19:23 +02:00
skilion 3529a4f78b make makeItem() more resiliant 2017-06-15 13:45:14 +02:00
skilion 9e2123249a do not check token status during initialization 2017-06-15 13:02:04 +02:00
skilion b703a824c7 compute path before performing selective sync 2017-06-15 12:59:33 +02:00
skilion 65bb14ad37 WIP for remote items support 2017-06-14 22:49:00 +02:00
skilion 7de0d97e6e fix error message 2017-06-14 22:30:29 +02:00
skilion 35f81b2115 itemdb support for remote items 2017-06-14 15:50:02 +02:00
skilion 74c931cdba added test for NULL values 2017-06-14 15:48:15 +02:00
skilion c5a9b8b480 fix bug where remote items are wrongly identified as deleted 2017-06-12 17:54:33 +02:00
skilion 4f542800a7 fix bug when using personalized config dir 2017-06-12 17:54:33 +02:00
skilion c2815d6498 fix indentation 2017-05-29 01:14:22 +02:00
skilion ad5441ccdc fix indentation 2017-05-29 00:23:55 +02:00
skilion b5cbaa13cc cleaned comments 2017-05-28 23:15:03 +02:00
skilion 8b6c99b0d9 notify deletion in verbose mode 2017-05-28 23:14:37 +02:00
skilion 305242d8a1 fix typo 2017-05-28 23:01:58 +02:00
skilion 809ccde562 workaround for old OneDrive files 2017-05-28 22:45:09 +02:00
skilion 8360d6508d align message 2017-05-28 22:17:21 +02:00
skilion 44fc36fd8d support for SHA1 and QuickXor hash algorithms 2017-05-28 22:13:19 +02:00
skilion 691862b18f implement the recommended way to enumerate changes 2017-05-28 20:54:57 +02:00
skilion 3d8daa086d added --print-token 2017-05-28 20:14:50 +02:00
skilion 80c60beef9 added newline at the end 2017-05-28 19:55:00 +02:00
skilion 7fec22bef1 fix variable type in for loop 2017-05-28 19:51:02 +02:00
skilion 9030a107b7 fix third person in comments 2017-05-28 19:49:55 +02:00
skilion f19b86e4c7 QuickXor utility function 2017-05-28 19:37:51 +02:00
skilion a2298be257 QuickXor implementation 2017-05-28 19:28:10 +02:00
lanhin 79139600ac Merge branch 'master' of github.com:skilion/onedrive
This may fix some up load problems.
2017-04-24 13:45:52 +08:00
skilion 97a9d53914 refactored selective sync code
fixed bug in selective sync
2017-03-24 22:31:56 +01:00
skilion 0d69ed805d wip for OneDrive Biz 2017-03-21 18:11:32 +01:00
skilion 1c7b726994 fix different time resolution when comparing 2017-03-21 18:11:32 +01:00
skilion 88d2a94078 added selective sync to monitor.d 2017-03-14 17:20:18 +01:00
skilion 79ed0a7883 fix different time resolution when comparing 2017-03-12 17:17:38 +01:00
skilion 2c1c168fab fix root dir skipped when searching for new files 2017-03-12 16:35:47 +01:00
skilion 9dd944286f new feature selective sync
removed system wide config file
hardcoded sane default config
updated README
2017-03-12 16:11:51 +01:00
skilion 8b204d18b4 hardcode the client id 2017-03-12 10:30:31 +01:00
skilion ac555fd574 added missing scope 2017-03-11 14:52:33 +01:00
skilion 957b034805 exit early if the item is to be deleted 2017-03-11 14:21:00 +01:00
skilion 1260fcfcc1 fix addressing root by path 2017-03-11 13:34:07 +01:00
skilion a3b25c45f5 catch CurlTimeoutException 2017-03-11 12:07:21 +01:00
skilion 1beadf2577 switch to Microsoft Graph API 2017-03-11 11:44:53 +01:00
skilion 203062fc6b added missing item fields in saveItem() 2016-12-29 00:36:42 +01:00
skilion 0afb6fa392 workaround for error 412 (Precondition Failed) on item delete 2016-12-28 23:47:29 +01:00
skilion 6a7c30e34f added missing checkAccessTokenExpired() in createByPath 2016-12-28 16:29:55 +01:00
skilion 3a10f0f4b5 fixed bug that allowed files marked for deletion to be keeped 2016-12-25 23:34:35 +01:00
skilion ff58a86b7f use libcurl to check connection 2016-12-25 23:34:35 +01:00
skilion bbbfeb68f3 fix condition in applyDifferences loop 2016-12-25 20:19:54 +01:00
skilion 6913c87d5c schema upgrade 2016-12-25 19:23:33 +01:00
skilion c9016845c3 removed time comparison hack 2016-12-25 17:40:43 +01:00
skilion ad4e910e55 check if the token is valid on init 2016-12-25 17:40:43 +01:00
skilion 82b2b63c75 added functions to get and set the db version 2016-12-25 17:40:43 +01:00
skilion 9ad9394b98 better handle the case when cTag is null 2016-12-25 17:40:43 +01:00
skilion e33f566ecf bug fix 2016-12-14 19:53:25 +01:00
skilion 7257c4c9bf do not throw on expired status token 2016-12-14 15:17:20 +01:00
skilion 947136cf62 show onedrive error response 2016-12-13 18:04:16 +01:00
lanhin 7c112304b5 Bug fixed: 1. Replace mkdirRecurse with mkdir to deal nested directory;2. Before download an item, check if its dirName exists, if not, build it. 2016-11-23 23:10:37 +08:00
skilion 022ba09e41 added multiGlobMatch() 2016-09-18 11:50:10 +02:00
skilion f5fc13cbad getValue() with default return value 2016-09-18 11:37:41 +02:00
Jumpei Anzai 9209a649d7 avoid syncing broken symlink 2016-08-22 17:29:43 +09:00
skilion c91decaa16 added logout cmd line option 2016-08-05 00:12:58 +02:00
skilion 738536736a Refactoring
- unified configs
- logging module
- new cmd line option to change the config dir
2016-08-04 23:43:10 +02:00
skilion 7d700e1a4c ask for the response uri one time only 2016-06-30 12:51:44 +02:00
skilion 18b279a339 restored AVOverride 2016-06-29 21:41:44 +02:00
skilion 09af80d843 create sync_dir if it doesn't exists 2016-06-28 14:21:48 +02:00
skilion 0efc2fe382 use view.delta instead of view.changes
removed dependency on client_secret
2016-06-27 19:23:14 +02:00
Jonathon M. Abbott 8b68adf8dd Ignore antivirus errors when downloading files
- closes #31
2016-04-01 22:45:42 +10:00
skilion 880af3ec03 changed redirect_url to redirect_uri 2016-03-07 14:34:35 +01:00
skilion 89452985ec workaround for segfault in std.net.curl.Curl.shutdown() on exit 2016-03-07 14:04:08 +01:00
skilion 5ed7dfcf3c Merge pull request #27 from astanin/fix-missing-ctag
fix #24: crash on sync if cTag is missing
2016-02-11 22:41:29 +01:00
Sergey Astanin ab2923df4d fix #29: add a user-friendly message how to change fs.inotify.max_user_watches 2016-02-04 15:36:25 +01:00
Sergey Astanin b824f87e7e fix #24: crash on sync if cTag is missing
According to https://dev.onedrive.com/resources/item.htm
cTag property is not returned if the Item is a folder.
2016-02-03 15:29:30 +01:00
skilion c2b0b7f733 intercept curl exceptions 2015-12-29 19:38:15 +01:00
skilion ea970890a2 capability to run as service 2015-11-29 21:12:44 +01:00
skilion 3f40728779 redirect stdout to stderr 2015-10-18 10:12:10 +02:00
skilion 288d1feac1 avoid full inotify restart in the monitor loop 2015-10-10 22:18:33 +02:00
skilion 32009e2747 responses are encoded in UTF-8 2015-10-04 17:33:48 +02:00
skilion 789bdf8390 ignore 404 errors on delete 2015-10-04 16:27:02 +02:00
skilion e7d493807d reimplemented std.net.curl basic methods in order to intercept http status codes 2015-10-04 16:24:45 +02:00
skilion 9da9b7240d better way to check if a JSONValue has a member 2015-10-03 10:45:43 +02:00
skilion 2acd00209f rename the local item if it is unsynced and there is a new version of it 2015-09-30 15:53:49 +02:00
skilion 9a50174c08 missing parameters were making uploads fail 2015-09-30 15:14:39 +02:00
skilion 10cfa83d39 fixed bug which caused dirs to not get deleted 2015-09-28 13:42:58 +02:00
skilion 9b80d99ad4 support for files bigger than 100 MB and resumable uploads 2015-09-28 13:25:25 +02:00
skilion 88b11433a7 removed space from auth url 2015-09-27 18:56:50 +02:00
skilion b4aeda818a regex simplified 2015-09-25 21:39:18 +02:00
skilion 7afae8fbb8 correct way to test if a value is in the AA 2015-09-24 19:12:19 +02:00
skilion 3489827431 alert when no config file is found 2015-09-24 18:59:17 +02:00
skilion f7e4b2f1e3 fixed acces to json nested objects 2015-09-22 16:14:16 +02:00
skilion 07f5f1ce78 create config dir if it doesn't exist 2015-09-22 15:26:45 +02:00
skilion 7cfa936db8 changed how config files works 2015-09-22 14:48:18 +02:00
skilion a032ef9671 changed help message 2015-09-22 11:52:57 +02:00
skilion 1c21474301 more error handling 2015-09-22 11:52:28 +02:00
skilion 85cd44cf73 better help message 2015-09-22 11:20:54 +02:00
skilion da2cfbde9c additional fix for wildcards 2015-09-22 11:16:06 +02:00
skilion 21993b36ee removed useless destructor 2015-09-22 10:56:14 +02:00
skilion c74282879b fix Precondition Failed after upload 2015-09-21 17:23:12 +02:00
skilion 7ea7c0f9cd case insensitive patterns 2015-09-21 13:04:05 +02:00
skilion d2ec6c688f better exception management 2015-09-20 21:21:51 +02:00
skilion 4498c03dec skip slash from regex match 2015-09-20 20:23:09 +02:00