* Implement --display-running-config to display the running configuration as used at application startup
* Add ONEDRIVE_DISPLAY_CONFIG variable to control --display-running-config in Docker environments
* Only safe remove the files from the local file system if these are exclusively available for this running instance to perform a delete - otherwise this allows multiple clients to perform a --resync on the same data, which, could lead to a data loss scenario
* When the client needs to exit due to an issue, ensure that the curl http instance is shutdown before the exit is performed. This also potentially solves some segmentation faults seen on Ubuntu|Debian platforms due to issues in the shared library libphobos2-ldc-shared.so.X
* contrib: remove bash hashbang from completion
This file starts with the #! sequence that marks interpreted scripts, but
it is a bash completion script that is merely intended to be sourced.
* src: spelling error (Attemtping => Attempting)
* src: spelling error (reponse => response)
* src: spelling error (sucessfully => successfully)
* Update OneDrive API response handling for National Cloud Deployments
* Add developer option to allow easy switch between /children and /delta to query OneDrive for changes
In some OneDrive Business scenarios, the shared folder /delta response lacks the 'root' drive details. When this occurs, this creates the following error: A database statement execution error occurred: foreign key constraint failed. Ensure we query independently the root details for this shared folder and ensure that it is added before we process the /delta response
* Only update the database if the item modified time is different, otherwise we are needlessly updating the database with data that is the same
* Generating a file hash, during the file integrity check is computationally expensive. Only generate a file hash if the modified time of the file is different, otherwise it is pointless to generate the file hash during each integrity check
* Only flush SHM and WAL post integrity check
* Implement a check to validate local filesystem available space before attempting file download
* Implement 'space_reservation' config option with a default value of 50 MB
* Prevent the original run-away logging error message 'Failed initialization on handle XXXX' from occurring if the system is out of space
* Update documentation and man page
* Fix where if the --confdir as specified has incorrect parent permissions, the client is unable to read the required hash files, thus cause an application crash
* When opening files, we should only be opening them as read-only
* When we write out any hash files. they should only be readable|writeable by the userid that is running the application
* Force a synchronization of a specific folder, only when using --synchronize --single-directory and ignoring all non-default skip_dir and skip_file rules
* When using the application with 'defaults' with logging enabled and no verbose mode, the application output that a sync process has started & finished is only written to the logfile every 5 sync times (25 mins). This change updates this so that, if defaults are being used with logging enabled and no verbose mode, the application will always, at a minimum, write out to the application log file when a sync was started and when it was completed. This provides important reference in the log as to when an activity began and was completed.
* In some scenarios users may have a running background service syncing data (systemd or otherwise). In these events, running the application in standalone mode can create a conflict when attempting to perform database queries when there are significant changes to data structures occurring. This PR specifically checks if the database operation files are present, and, if these are present (meaning the application is most likely currently operational) fail fast and not execute the application as a second instance against the same active configuration.
Potentially also resolves RBZ2061430 and RBZ2075468
* When a new folder is created locally, use the lastModifiedDateTime as provided by OneDrive for the folders timestamp instead of the time when the client created the directory
Note: This only will impact new folder that are created, will not touch or modify existing folders
* Update logging output when running in --monitor --verbose mode in regards to the inotify events
* Add additional logging specifically for delete event to denote in log output the source of a deletion event when running in --monitor mode
* Extend GitHub version check to be done once per day when running in --monitor
* Change notification to also notify GUI if notifications are being utilized
* Add application version check, to check GitHub latest release tag and compare to the application version being run. If the application version is older, display a warning message that the client is obsolete and unsupported and that the user needs to upgrade their client version.
* Explicitly set libcurl options to ignore SIGPIPE, use TCP NAGLE and ensure we are re-using connections. These should be 'enabled by default' by libcurl, but we need to ensure we are setting them to their supposed default
Potentially resolves the following issues: #494, #753, #792, #884, #1162, #1408, #1520, #1526 + others.
* Update application logging configuration instructions
* Update error notification handling when unable to write to required logfile due to access permission issues
* Expand on the edge case solution fix for #1586 to drop --single-directory qualifier in this very specific scenario, and add the DB tie record in all --resync --upload-only scenarios where the driveId is actually remote
* This change is to create an argument version of the config operation_timeout.
There are many times where one may need to temporarily increase the timeout for an operation of a single large file that takes longer than the configured timeout. The command line argument is an excellent way to override the default or 'config' set option if required.
* In fixing #1712 which was due to a OneDrive API change, the flagging that quota is being restricted was missed, thus, if this is truly being restricted may cause files to OneDrive Business Shared Folders be unable to be uploaded despite space being available.
* Resolve issue where --resync is not used and items.sqlite3 is not available at application startup, local files, whilst potentially newer, will not be preserved thus leading to a potential data loss scenario
* Add --disable-download-validation to allow downloading files from OneDrive where SharePoint and the OneDrive API mis-represents the values in the API as compared to the actual HTTP server response sent to the client. In the API JSON response we get a "size" value of X, but the HTTP Server Content Length reports a size of Y. When this occurs, the download will report as failed. This was seen as part understanding the cause of https://github.com/abraunegg/onedrive/discussions/1667
* In a `--resync --upload-only --single-directory 'dir'` scenario, and where the root 'dir' for --single-directory is a 'shared folder' we will not have the 'tie' DB entry created because of using --upload-only because we do not download the folder structure from OneDrive. As a result, query of the folder will fail and file uploads will fail.
Simulate the 'tie' DB record only when --resync --upload-only --single-directory 'dir' is being used, and if that folder is 'remote' and if we are using a 'personal' account.
The 'impact' of this however is, because of `--resync --upload-only` being used, local files are not in the local DB cache anymore, thus are treated as *new* files, thus, will be attempted to be re-uploaded.
* Fix getPathDetailsByDriveId query when using --dry-run and a nested path with --single-directory
* Fix fake response generation to use generated values for all account types to avoid DB lookup failures when using --dry-run for Personal account types
* Catch a Invalid UTF-8 handling error when attempting to initialise the application. This is caused by the 'path' to the application configuration (typically ~/.config/onedrive) contains a bad UTF-8 character thus cannot be read / initialised
* When syncing OneDrive Business Shared Folders and using --single-directory, select correct driveId and itemId for the remote directory that needs to be synced
* Normally, the 'remoteItem' field will contain 'fileSystemInfo' however, if the user uses the 'Add Shortcut ..' option in OneDrive WebUI to create a 'link', this object, whilst remote, does not have 'fileSystemInfo' in the expected place, this leading to a application crash
* Add exception handling for when the API returns a 400 error when attempting to query a path on OneDrive. If the path generates a 'bad request' response, this needs to be correctly handled.
* Handle bad ShaprePoint data when the API does not return the expected data points when using those references to display what SharePoint sites are available when using --get-O365-drive-id
* Add a file check when using --upload-only --remove-source-files so that parental paths are added to the database, to allow child objects to be uploaded in this scenario
* Fix 'sync_list' handling of inclusions when name is included in another folders name to avoid creating empty parental root folders. When a 'sync_list' entry ends with '/*' the common path matching causes a sub path match to occur. Before directly hitting as an exact match, perform an additional check of the path to sync against the sync_list entry without the '/*' being present.
* Fix uploading documents to Shared Business Folders when shared folder exists on a SharePoint site due to Microsoft Sharepoint 'enrichment' of files
See: https://github.com/OneDrive/onedrive-api-docs/issues/935 for further details
* When in --monitor mode, and there are multiple driveId being used (shared folders), when a deletion event occurs, search the DB using all the known driveids rather than just the default one. When using just the default driveid, if the file to be deleted resides on another drive, the following error message is printed: 'Item cannot be deleted from OneDrive because it was not found in the local database' - which is not entirely accurate as the item is in the database, it is just not being searched for correctly.
* Use 'nextLink' value if present when searching for specific SharePoint site names
* Update error output to provide more details why an error occurred if a SharePoint site lacks the details we need to perform the match
* When processing sync_list entries, by default, if the path starts with '/', and offset of 1 is used to ensure correct path matching can be done. However, if the path starts with '!/' or '-/' to exclude a certain path, an offset of 1 was still being used, thus, exclusion matching was not occurring correctly. Use the correct offset, based on the 'sync_list' entry.
* Revert the config & logpath change introduced via #1267 as the logging path should not be the value checked, the input path is what should be checked
* Fix logging output when processing downloaded files to remove potential ambiguity as to what events are actually occurring with individual file events post download
* After a file has been uploaded to SharePoint, use the response JSON and compare the quickXorHash to that of the local file before the upload. If there is a difference, Microsoft has modified the file post upload, thus we now need to download this 'updated' file. If there is no hash difference, we do not need to download the file.
* Add new DB query to return distinct driveId's from database
* Use all distinct drive ID's to query if path is in-sync with OneDrive before attempting to retreive the item file link from OneDrive
* Improve application startup when using --monitor when there is no network connection to the OneDrive API
* Retry network connection test when using --monitor
* Only initialise application if OneDrive API is reachable
* Add to debug log output the configured URL's which will be used throughout the application to remove any ambiguity as to using incorrect URL's when making API calls.
* Add debugging output when adding paths and items to an inotify watch
* Only if a path is a directory, perform a path walk
* Catch Exception as error when permissions cause access exception
* If using specific AD endpoints, but using the default application ID, the redirect URL does not match. Check if the application_id before setting the redirect URL.
* Remove items from direct config string update, as they are configurable via config file, thus set option via config file is not used
* Add debugging and application config output for used options
* Add try block for opening new file for writing when downloading a new file to catch any read only file systems
* Move set file attributes to master function, incase there is a download failure, exit scope cannot set attributes on a file that is non existent
* When ~/.config/onedrive/ gets created for the first time, directory permissions are not set, thus, effective permissions of 'd---------' get applied. This then causes issues attempting to update the 'refresh_token' as permission is denied.
* When permission is denied, the file exception error is not correctly handled
* Set refresh_token file permissions
* When using WAL and SHM files, certain thresholds are used to automatically determine when WAL and SHM data is committed to the main database file. If these thresholds are not met, the data is not written. Before terminating the program, commit the data, and at the end of each monitor loop, commit the data
* Only try and write data to DB file and destroy variable if it was set to begin with
* Only try and destroy 'onedrive' variable if set, rather than set to null
* Adding support for authentication with single tenant custom applications in OneDrive Business/Enterprise accounts.
When using a custom single tenant application the /common/ part on the authentication URLs needs to be replaced
by the tenant name. This can be configured on the azure_tenant_id setting, but this entry was being ignored unless
a custom azure_ad_endpoint was also set.
* Updated documentation to mention that, when configuring for Single Tenant Applications,
both the Tenant Name and Application ID must be included on the config file.
Co-authored-by: abraunegg <alex.braunegg@gmail.com>
* Allow config to specify a tenant id for non-multi-tenant applications
* Add instructions on how to find your O365 tenant identifier for National Cloud deployments.
Co-authored-by: Ryan Brown <ryan.brown@avarint.com>
Co-authored-by: abraunegg <alex.braunegg@gmail.com>
* Support negative patterns in sync_list where a new line is matched in a negative manner to be excluded is prepended with a '!' or '-' character to exclude an otherwise included path
* Implement feature request to create a shareable file link. Default is to create a read-only anonymous link
Co-authored-by: Norbert Preining <norbert@preining.info>