* Fix handling of 5xx responses from OneDrive when performing a session upload
* Switch to same checks when doing non session upload so that OneDrive exceptions are thrown correctly
* Remove a 'throw e' when curl times out
* Remove a needless 'throw e' when a session upload cannot be found
* Set the upload failed flag when OneDrive errors on session uploads
* Add additional JSON object validation for queries made against OneDrive where a JSON response is expected and where that response is to be used and expected to be valid
* Remove 'else throw e' which causes the application to exit / crash and replace with what the actual error message is from OneDrive
* Make logging of unhandled errors consistent
* Add extra error handling when uploading files for errors received from OneDrive or file system errors
* Flag that the upload failed if there is an exception error raised
* Update logging output for when an error occurs to indicate that the file was skipped
* Log the OneDrive response as log.vlog rather than log.error as it is now being handled correctly
* Throw an exception for 50x errors so that it gets caught correctly
* Clean up logging when exception is generated
* Catch a 403 Forbidden exception when attempting to query for an Office 365 Shared Library Name
* Validate that the returned response is a valid JSON object
* Update scopes as per https://github.com/OneDrive/onedrive-api-docs/issues/1119
* Update that if a 403 error is received, what is the right course of action to take
* Without this throw, uploadLastModifiedTime fails to catch that the 412 error was returned, thus, the retry with a null cTag / eTag is not performed which leads to 'OneDrive response missing required 'id' element' being generated.
* Fix 'Unexpected character '<'. (Line 1:1)' when OneDrive has an exception error
* Validate that the response from OneDrive is a valid JSON object
* Update Sharepoint Document Library handling for .txt & .csv files
* Use replace function for modified Sharepoint Document Library files rather than delete and upload as new file, preserving file history
* Add explicit check for hashes to be returned in cases where OneDrive API fails to provide them despite requested to do so
* Add comparison with sha1 if OneDrive provides that rather than quickXor
* Update logging output for better clarity
* Update logging output from vdebug -> error for errors
* Remove restrictive checks on fileDetails - if missing, continue, not return
* Add debug lines for if / when this is missing
* If query for file details or file object generates an error, we return, but we also need to flag that downloadFailed = true so that item is not added to the database which may cause validation issues leading to thinking that the file was downloaded but then deleted
* Update fix for #555 as try | catch block for session creation appears to miss error response codes
* Update how JSONValue object is determined to be valid
* Add error logging when response is not a valid JSON object
* Check if the file that was downloaded to local disk equals the reported size & reported hash of the file on OneDrive. If sizes do not match, report an error and remove the local file as it is most likely corrupt.
* Add a try block when attempting to create the upload session and handle if there is an error response from OneDrive
* Should return a JSONValue regardless of error
* Size check should only be used if configured
* Rather than make 2 x calls to OneDrive service, driveItem already has the size value to check
* Add a try block when query for file details
* Move lots of stuff into contrib directory to clean up root folder
* Update autoconf files for new location of some files
* Refactor README split and move to docs directory
* Update INSTALL.md
* Update Makefile.in
* Install docs from new location
* Uninstall docs correctly
* Update Dockerfile
* Fix up Dockerfile spacing
* Move systemd stuff to contrib
* Move docker specific stuff to contrib/docker, update docs
* Fix Alpine and Stretch docker files
* Update configure after changed file locations
* Update INSTALL.md to include pkg-config as requirement
* Move also entrypoint.sh to contrib/docker
* Update 'min-notif-changes' to 'min-notify-changes' which was missed in PR #467
* Update to add --sync-root-files to usage.md and man page
* Update usage.md to include --auth-files ARG details
* Update original patch to check for eTag in response to use, if not found set to empty value.
* Add cTag response validation for all cTag response use 'in-case' the cTag response is dropped from other responses
* Handle response from OneDrive where cTag JSON elements are missing when uploading a locally modified file
* Update bug_report.md to fix up spelling errors
* If OneDrive presents a change to download and the file already exists, check the local modified timestamp to ensure that the local file is not technically newer than the file we are going to download
* Catch a curl timeout exception
* If we hit a curl exception, log that we hit this, as the handling of the exception in 'some' instances may actually not log an error as to 'why'
* Add an encoding validation check before any path length checks are performed as if the path contains any invalid UTF-8 sequences, the path walk to establish valid path length will fail
* Implement --sync-root-files to sync all files in the OneDrive root when using a sync_list file that would normally exclude these files from being synced
* If the 'quota' details are not provided when querying the OneDrive account details, we cannot set 'remainingFreeSpace' Add a flag that if this is 0 or negative, ignore validating if there is enough free space to upload a file and just upload blindly
* Add specific check to ensure that variable is of type object
* Add error logging to indicate error response from OneDrive was received
* Update error logging in saveItem to match
* Add a '401 Unauthorized' event handler when querying OneDrive if the file exists
* Add a check for --upload-only use when trying to work around https://github.com/OneDrive/onedrive-api-docs/issues/935
* Update logging and print a warning message that the files are now technically different due to sharepoint bug and using --upload-only
* Issue #445 and https://github.com/OneDrive/onedrive-api-docs/issues/1035 highlight that the current 5xx error messages are too generic and need to be more specific. This patch makes those error messages more specific to the response code from OneDrive API
* Add debug line for flagging to disable upload validation checks
* Handle Microsoft 'enrichment' of certain documents when stored on Sharepoint
* Handle skip_dir checks when nothing to check against (false positive)
* Update default 'skip_file' to include tmp and lock files generated by LibreOffice
* Update code comments & logging output
* Update readme & default config file
* Update database version due to changing defaults of 'skip_file' which will force a rebuild and use of new skip_file default regex
* Reduce scanning the entire local system in monitor mode for local changes
* Add 'monitor_fullscan_frequency' to set the frequency of performing a full disk scan when in monitor mode
* Disable automatic indexing as we specifically create the required indexes
* Tell SQLite to store temporary tables in memory. This will speed up many read operations that rely on temporary tables, indices, and views.
* Add links & reasoning behind other PRAGMA settings used
* Add new index specifically for driveId & parentId paring
* To force DB schema & index creation, bump DB schema version
* Update handling of skip_dir and skip_file parsing - should only check if the file is excluded if the parent directory is not
* Add another index for selectByPath database queries
* Add new build option to get more DEBUG symbolic information
* Use boolean values rather than on / off values
* Enable auto_vacuum for entry deletes / database cleanup
* Original skip_dir handling expected an explicit match to path to match. With this patch, wildcard matching for any directory matching the path entry will be skipped
* Add a http 400 response error handler
* If the response uri generates a 400 error, the JSON response will not contain the access_token. Request to re-authenticate
* By moving the UploadSession creation from the constructor to the init function we guarantee that a re-initialization also starts a new UploadSession.
* Add additional Docker container builds utilising a smaller OS footprint
* Resolve compiling with LDC on Alpine as musl lacks some standard interfaces
* Re-order handling of response JSON to ensure that 'id' key element is always checked for.
* Print errors when there is an issue with saveItem & articulate why
If the dbus server cannot be started or is not available, calling a notification
would result in an exception. During initialization check that we can get the
server information, and disable notifications if this is not possible.
Note: we change dnotify.d from the originally distributed version to include
a check for availability function.
* When notifications are enabled, display the number of OneDrive changes to process if any are found
* Add 'config' option 'min_notif_changes' for minimum number of changes to notify on, default = 5
* Update readme, manual page for new option
* Update handling of --get-O365-drive-id to print out all 'site names' that match the explicit search entry rather than just the last match
* Add webUrl for site to output so that the 'right' site drive_id can be identified
* Update O365 readme with updated output
* Support install on CentOS / RHEL 6.x
* Support uninstall on CentOS / RHEL 6.x
* Add /usr/local/bin/ to search path when starting init.d service
* Fix dmd-2.084.0 deprecation warning: loop index implicitly converted from size_t to int
* Update readme based on CentOS 6 / RHEL 6 additional dependencies
* Resolve warnings about "bashisms" in init.d script (Issue #349) - fixed here as we are updating the init file already, rather than have multiple PR's
* When OneDrive Business is configured with Retention Policies, attempts to delete a 'non empty' folder results in an an error response. Correctly handle the error response to print a warning message that OneDrive content must be manually deleted.
**Note:** A future version of this client will handle the removal of 'non empty' folders when Retention Policies are enabled.
* Add a 404 error handler when OneDrive cannot query what changes are available for a particular 'root' folder id
* Add additional verbose debug logging to assist in the future
* Change to not throw an exception when OneDrive cannot query what changes are available for a particular 'root' folder - rather display some meaningful details instead of crashing the application
* As we know the actual 'items.sqlite3' location, display that as the file to remove rather than a 'generic' message as this is more helpful
* Fix remote deletes the fix for #323, #324 & #331 introduced a bug where an item when remote deleted, would actually be re-inserted to the database & eventually cause a database assertion
* Add --force-http-1.1 flag to downgrade any HTTP/2 curl operations to HTTP 1.1 protocol
* Explicitly set all bool items to false at initialisation
* Update --display-config to display sync_list if configured
* Add debug handling to display when sync_list is loaded
* Add debug handling to output the handling of OneDrive changes
* Update HTTP 412 error notifications to better identify where 412 error is occuring
* Add a check for 'id' key on metadata update to prevent 'std.json.JSONException@std/json.d(494): Key not found: id' (Issue #325)
Add a signal handler and shutdown database connection on SIGINT and SIGTERM
to make sure that .waf files are properly incorportated into the sqlite database.
If an sqlite statement is prepared and reused by using reset again and again, the connection
keeps a lock on the database, and checkpointing cannot ensure that transaction from the
wal file are carried over into the main database.
By preparing the statement every time it is used, the destructor calls finalize and thus
the lock is released.
Possible impacts during high frequency changes via monitor mode etc might arise.
* Fix logic for sync_dir handling on headless systems
* Fix logic where potentially a 'default' ~/OneDrive sync_dir could be set despite 'config' file configured for an alternate
* Add debug handling for sync_dir operations
* Add debug handling for homePath calculation
* Add debug handling for configDirBase calculation
* Reorder main() so that log.vdebug works where required
* Rework configDirName as this had the same issue as syncDir
* Update configDirName & syncDir '~' handling & replacement to be more robust and only replace if '~' is first char in string
* Add additional debug logging if syncDir is created
* check config file keys for validity, use setValue instead of direct access
* Update config.d
Add 'drive_id' to be initialised, set to an empty string
* exit application if there is a configuration file error
* Issue #293 was caused by a spelling error in the configuration file.
If the configuration file has errors, we should not load it or run
using the application defaults as this may have undesirable consequences
for users data
* missed returning false on key issue
* Missed this edit of the file
* In testing, 'description' and 'displayName' were the same, but in reality we are checking against the 'displayName' value as this is what we are asking the user to input, thus we need to check against displayName
* After extended suspend, or bad connection, the connection to OD might
go stale - thus no network connection to Microsoft OneDrive Service, skipping sync
and this continues ad infinitum. Change the last action to re-initialize the sync engine.
if the move target has been deleted before the monitor action completes,
the monitor action throws an exception when stat-ing the target for
mtime. Do not do anything in case the move target has disappeared
* add external sources for (d)notify with README and license statements
* add new logAndNotify function, make building a compile time option
* use logAndNotify and main.d, make timeout message silent
* update documentation for notification support
* add command line switch --disable-notifications
* add build-deps for libnotify in README.md
* SyncEngine: allow for multiple calls to init by saving the state
* Factor out syncengine initialization code so that it can be reused
* Don't exit in monitor when started offline
* Always call initSyncEngine during regular syncs, this will call the SyncEngine init routine which caches the state, thus a quasi no-op.
* Add forgotten http.shutdown
This change adjusts the two core timeout parameters of libcurl operation
- dataTimeout: changed from 3600sec to 300sec
this timeout controls the max time when there is no data incoming
(actually below CURLOPT_LOW_SPEED_LIMIT)
- operationTimeout: added 3600sec
this timeout controls the maximally allowed connection timeout
* Change '--download' to '--download-only' to align with '--upload-only'
* Enable logging to a separate file only if the '--enable-logging' flag is passed through at client run time
* Implement configuration option for logfile location, if logging is enabled
* Refactor PR104 for latest code base to enable client use with SharePoint libraries
* Implement '--disable-upload-validation' as SharePoint modifies files when uploaded
* Enable '--disable-upload-validation' by default if account type is documentLibrary - as most likely this is a SharePoint repository
* Remove HTTP 1.1 downgrade for session uploads
* Handle HTTP/2 0 (zero) response code for successful session data upload
* Add debugging for actual server response, not curl interpreted response when using --debug-https
* Update initial Curl 7.62.0 fix with a better solution which handles the redirect's given by HTTP/2 connections (credit @Popa21)
* Use HTTP/1.1 for session uploads otherwise the response when using HTTP/2 generates a 'JSONValue is not an object' error
* Handle HTTP request returned status code 412 (Precondition Failed) for session uploads to OneDrive Personal Accounts
* Fix Failed to remove file /root/.config/onedrive/resume_upload: No such file or directory if there is a session upload error and the resume file does not get created
* Handle response codes when using 2 different systems using --upload-only but the same OneDrive account and uploading the same filename to the same location
* Add a check to validate if the parentReference object has a path object
* Fix spelling error - trough -> through
* Only upload changes on remote folder if the item is in the database - dont assert if false
* Update handling of shared folders and items which was previously broken to a OneDrive API change
* Extract syncFolderName from either local or remote folder - not just local folder.
* Add try & catch in the case of a 504 error being generated to upload the file. Currently, if a 504 is generated, it is handled gracefully, but further files to upload are stopped
* Update where ever there is a 404 try block, add a if statement to capture if a 5xx error is returned as well
* Resolve 'The parent item is not in the local database' where data cannot be uploaded / saved to the local database because there are folders in the same path with the same name due to case sensitivity issues
* Original implementation of --no-remote-delete used a cfg value, however this introduced 2 issues - value could be set via a config parameter which was not the intention, but also could be set to a non true value causing application issues. This patch resolves how --no-remote-delete should have originally been implemented
* Implement better handling of database exit scenarios when there is zero disk space left on drive where the items database resides (Issue #77)
* Implement better handling of incorrect database permissions
* Implement better handling of different database versions to automatically re-create tables if version mis-match (RHBZ #1598934)
* Update 'HTTP 412 - Precondition Failed' error handling based on local testing to ensure a http 412 response is gracefully handled & the modified file is subsequently uploaded to OneDrive successfully.
* item.name is not available, so we get a bunch of meaningless log output when deleting items. This will get fixed when wider logging changes are synced in
* Resolve multiple versions of file shown on website after single upload - however Issue #23 & OneDrive API bug (https://github.com/OneDrive/onedrive-api-docs/issues/877) will still create 2 file versions on OneDrive Business. The work around is to disable file versions until #877 is resolved.
* Implement an invalid whitespace filename check which currently checks for a 'new line' whitespace entry in a filename. Regex can be expanded if other files with similar characteristics are found.
* Resolve Shared Folders not syncing (OneDrive Personal)
* Resolve 'Key not found: fileSystemInfo' when then item is a remote item (OneDrive Personal)
* Resolve where file names are being truncated by 5 characters with each sync (OneDrive Business)
* remove extra // not required
* Remove commented out code to avoid confusion
* Provide a new switch 'check-mount' which will check for the presence of a hidden file '.nosync' file. If this file is found in the sync dir, then the sync process will shutdown. Refer to #8 for further details.
* Resolve#11 where shared folders were unable to be sync'd due to fileSystemInfo data being within the remoteItem object
* Initial work on resolving #2, but fix not validated or complete
* Add an additional check for 409 (Conflict) being returned when creating directories even when the directory is only attempted to be created when it cannot be found.
* Resolve std.file.FileException@std/file.d(2954): /var/log/onedrive/: Permission denied when there is no permission to create the directory on application startup
* Clean up --single-directory logging as we should not be spilling out the actual item details
* Fix some spelling errors
* Fix another core.exception.AssertError@src/itemdb.d(307): Assertion failure when attempting to compute the path
* Fix logging when items are being filtered out as non relevant
* Add upload validation that file uploaded equals the size of the file locally as this is a contributing factor to 412 errors
* Update long path handling to account for differences between OneDrive Personal and OneDrive Business Accounts
* Update logging to be cleaner on fragment uploads
* Update logging of 500 & 504 errors. For 504 errors this was 'breaking the formatting of the log line. For 500 errors - this needs to be logged regardless of verbosity.
* When completing some operations on OneDrive, a 504 error can be returned when Microsoft Graph times out when communicating with internal resources. Rather than failing, handle the 504 and retry the request.
* When performing numerous changes via OneDrive online, Microsoft Graph has an issue in interpreting the the response, thus generates a 500 internal server error. As this is something we cannot fix, handle the 500 error by gracefully exiting.
* Rework the 412 fix so that the OneDrive client gracefully handles the
error & retries the metadata update without the eTag/cTag causing the
issue.
* Fix the double logging to console of 'upload' and 'download' items
* Resolve where '--single-directory' is used to sync a single directory,
but a file / folder is 'moved' on OneDrive to outside the scope of the
focus of the '--single-directory' path. Before this change, the file
would remain in the local path (original location) whilst it would
reside in the new OneDrive path. This patch looks for this mis match and
deletes the local file / folder to reflect it is no longer in the same
OneDrive path.
* In some cases OneDrive does not send all delete changes or changes are
out of order. When deleting local files, this can lead to a 'Directory
not empty' error. This change adds to check for any remaining children
of a directory that is to be deleted and delete them before
* Resolve: std.exception.ErrnoException@std/stdio.d(423): Cannot open
file `/var/log/onedrive/xxxxx.onedrive.log' in mode `a' (Permission
denied). This issue occurs if the user running 'onedrive' is not part of
the standard 'users' group (GID 100)
* Prior to all the changes in #314, the client would sync regardless of
options. #314 introduced a flag that must be included if you want to
sync, however there was no feedback if '--synchronize' was not included
and the sync was not occuring. This patch provides that notification &
feedback.
* By default the DNS timeout on HTTP.method is too short. If DNS
resolution takes too long, a resolution error is thrown. Increase DNS
timeout to 5 seconds.
* Cleanup unittest - double declaration
* Resolve the error 'view.delta can only be called on the root' when
using OneDrive Business Account & --single-directory switch
* Change checkDatabaseForOneDriveRoot so that it is performed at init()
and not needed to be called a number of other times throughout the code
- leads to less HTTPS calls being made
* Cleanup comments that are no longer relevant due to
checkDatabaseForOneDriveRoot being run at init()
* Original patch (77c0cdbe24) whilst it
resolved the issue of a OneDrive API change, when a file was deleted on
OneDrive, it was not actually deleted on the local file system. This
patch re-implements the fix for 'Key not found: lastModifiedDateTime' by
taking into account the deleted state of the item and correctly removes
the local file that is now in the deleted state.
* The fix in 77c0cdbe24 whilst resolves
'Key not found: lastModifiedDateTime' introduces a bug where files
deleted by onedrive, are not deleted locally. Reverting the change to
implement a fuller fix for 'Key not found: lastModifiedDateTime'.
* Change how onedrive get's the children from a particular item id.
Previously view.delta was used to return the children, however
view.delta is not implemented on children when using OneDrive Business
accounts. By using list_children, we can get the children from any id
correctly when using either a 'personal' or 'business' account
* Fix 'Key not found: lastModifiedDateTime'
* Handling of Tombstoned items in the database
* Add for debugging purposes the rootID, driveID and Account Type
* Code cleanup - remove old commented out debugging items that are not
needed
* Change debug flag from --debug-http to --debug-https as OneDrive only
uses HTTPS, not HTTP
* Expand --debug-https to print the JSON responses to GET operations.
This is to assist with debugging #334 where a change in the MS API
(https://github.com/OneDrive/onedrive-api-docs/issues/834) no longer
sends 'lastModifiedDateTime' with responses to some GET's. This issue
seems to manefest itself with OneDrive Business accounts currently.
* Add Pull #104 (@JunpeiAnzai) to ensure that uploaded files conform to
Windows naming conventions and namespaces. Refer to
https://msdn.microsoft.com/en-us/library/aa365247 for further details
regarding Windows naming conventions and namespaces.
* Add init.d service file & helper script to start service
* Change how configDirName is set as XDG_CONFIG_HOME does not exist on
systems where X11 is not present. When using init scripts and using
XDG_CONFIG_HOME, ~ is not expanded thus existing config cannot be found.
Using ~ in --confdir also does not work when running under init.d
* Fix the handling of hidden filenames & directories (.hello, .git etc)
so that these are uploaded / downloaded without issue.
* Add verbose logging for uploaded files as to why they were skipped
Prior to this change logging of upload, download and deletion activities
were only written to the log file when using the --verbose flag, however
they would be written to console / syslog. This corrects that behaviour.
* Add specific uploadOnly flag so that the onedrive client only uploads
data from the local directory and does not download changes from
OneDrive
* Change the logfile name to include the username running the onedrive
client
Change '&error' to 'error' on line 37 of onedrive.d
Issue uncovered where prior code with '&' would compile without issue on
DMD reference compiler v2.078.3 and below, but using '2.079.0' this
would throw an error if '&' was present.
The original patch added the '&' to get the code to compile, this simple
reverts this change.
Note: For future reference compile this code with DMD v2.079.0 or later.
* Fix 4xx errors including (412 pre-condition)
* Add Logging - log to a file (/var/log/onedrive/onedrive.log)
* Add http(s) debugging as a flag
* Add dont sync when just blindly running the application
* Add individual folder sync - ie ~/OneDrive/blah/ vs. syncing
everything in ~/OneDrive/
* Add sync from local directory first rather than download first then
upload
* Add upload long path check (430 character limitation)
when trying to redeem an code, this error always occured, no matter
which parts of the response uri/url that was used.
the old regexp for parsing the code contained 3 groups, where as the
actual code always was the last group, and the second group was either ?
or &, and the first group would've been everything up until "code=".
changed from a matching group to character class so there would only be
two matching groups, so calling popFront() would actually leave the
authorization code in the front.
