From f7eb9f90019684c7b46d6705c93f2f1fe6d0b42b Mon Sep 17 00:00:00 2001 From: abraunegg Date: Sat, 6 Mar 2021 07:07:54 +1100 Subject: [PATCH 01/19] Update Docker.md * Update document instructions based on validation on multiple platforms --- docs/Docker.md | 112 ++++++++++++++++++++++++++++++++----------------- 1 file changed, 73 insertions(+), 39 deletions(-) diff --git a/docs/Docker.md b/docs/Docker.md index 4b05d9ba..b0b02cd7 100644 --- a/docs/Docker.md +++ b/docs/Docker.md @@ -1,56 +1,89 @@ -# onedrive docker image +# Run the OneDrive Client for Linux under Docker +This client can be run as a Docker container, with 3 available options for you to choose from: +1. Container based on CentOS 7 - Docker Tag: latest +2. Container based on Debian Stretch - Docker Tag: stretch +3. Container based on Alpine Linux - Docker Tag: alpine -Thats right folks onedrive is now dockerized ;) +These containers offer a simple monitoring-mode service for the OneDrive Client for Linux. + +The instructions below have been validated on: +* Red Hat Enterprise Linux 8.x +* Ubuntu Server 20.04 + +The instructions below will utilise the 'latest' tag, however this can be substituted for 'stretch' or 'alpine' if desired. -This container offers simple monitoring-mode service for 'Free Client for OneDrive on Linux'. ## Basic Setup +### 0. Install docker using your distribution platform's instructions +1. Ensure that SELinux has been disabled on your system. A reboot may be required to ensure that this is correctly disabled. +2. Install Docker as per requried for your platform +3. Obtain your normal, non-root user UID and GID by using the `id` command +4. As your normal, non-root user, ensure that you can run `docker run hello-world` *without* using `sudo` -### 0. Install docker under your own platform's instructions +Once the above 4 steps are complete and you can successfully run `docker run hello-world` without sudo, only then proceed to 'Pulling and Running the Docker Image' +## Pulling and Running the Docker Image ### 1. Pull the image - ```bash docker pull driveone/onedrive:latest ``` -**NOTE:** SELinux context needs to be configured or disabled for Docker, to be able to write to OneDrive host directory. +**NOTE:** SELinux context needs to be configured or disabled for Docker to be able to write to OneDrive host directory. ### 2. Prepare config volume +The Docker container requries 2 Docker volumes: +* Config Volume +* Data Volume -Onedrive needs two volumes. One of them is the config volume. Create it with: - +Create the config volume with the following command: ```bash docker volume create onedrive_conf ``` This will create a docker volume labeled `onedrive_conf`, where all configuration of your onedrive account will be stored. You can add a custom config file and other things later. -The second docker volume is for your data folder and is created in the next step. It needs the path to a folder on your filesystem that you want to keep in sync with OneDrive. Keep in mind that: +The second docker volume is for your data folder and is created in the next step. This volume needs to be a path to a directory on your local filesystem, and this is where your data will be stored from OneDrive. Keep in mind that: -- The owner of your specified folder must not be root - -- The owner of your specified folder must have permissions for its parent directory - -### 3. First run - -Onedrive needs to be authorized with your Microsoft account. This is achieved by running docker in interactive mode. Run the docker image with the two commands below and **make sure to change `onedriveDir` to the onedrive data directory on your filesystem (e.g. `"/home/abraunegg/OneDrive"`)**. -Additionally, the user id and group id should be added to remove any potential user conflicts, denoted by the environment variables `${ONEDRIVE_UID}` and `${ONEDRIVE_GID}`. +* The owner of this specified folder must not be root +* The owner of this specified folder must have permissions for its parent directory +**NOTE:** Issues occur when this target folder is a mounted folder of an external system (NAS, SMB mount, USB Drive etc) as the 'mount' itself is owed by 'root'. If this is your use case, you *must* ensure your normal user can mount your desired target without having the target mounted by 'root'. If you do not fix this, your Docker container will fail to start with the following error message: ```bash -onedriveDir="${HOME}/OneDrive" -docker run -it --name onedrive -v onedrive_conf:/onedrive/conf -v "${onedriveDir}:/onedrive/data" -e "ONEDRIVE_UID:${ONEDRIVE_UID}" -e "ONEDRIVE_GID:${ONEDRIVE_GID}" driveone/onedrive:latest +ROOT level privileges prohibited! ``` -- You will be asked to open a specific link using your web browser -- Login to your Microsoft Account and give the application the permission -- After giving the permission, you will be redirected to a blank page. -- Copy the URI of the blank page into the application. +### 3. First run +The 'onedrive' client within the Docker container needs to be authorized with your Microsoft account. This is achieved by initially running docker in interactive mode. -The onedrive monitor is configured to start with your host system. If your onedrive is working as expected, you can detach from the container with Ctrl+p, Ctrl+q. +Run the docker image with the commands below and make sure to change `ONEDRIVE_DATA_DIR` to the actual onedrive data directory on your filesystem that you wish to use (e.g. `"/home/abraunegg/OneDrive"`)**. +```bash +export ONEDRIVE_DATA_DIR="${HOME}/OneDrive" +mkdir -p ${ONEDRIVE_DATA_DIR} +docker run -it --name onedrive -v onedrive_conf:/onedrive/conf -v "${ONEDRIVE_DATA_DIR}:/onedrive/data" -e "ONEDRIVE_UID:${ONEDRIVE_UID}" -e "ONEDRIVE_GID:${ONEDRIVE_GID}" driveone/onedrive:latest +``` +**NOTE:** It is also highly advisable for you to replace `${ONEDRIVE_UID}` and `${ONEDRIVE_GID}` with your actual UID and GID as specified by your `id` command output to avoid any any potential user or group conflicts. -### 4. Status, stop, and restart +**Important:** The 'target' folder of `ONEDRIVE_DATA_DIR` must exist before running the Docker container, otherwise, Docker will create the target folder, and the folder will be given 'root' permissions, which then causes the Docker container to fail upon startup with the following error message: +```bash +ROOT level privileges prohibited! +``` +**Example:** +``` +docker run -it --name onedrive -v onedrive_conf:/onedrive/conf -v "${ONEDRIVE_DATA_DIR}:/onedrive/data" -e "ONEDRIVE_UID:1000" -e "ONEDRIVE_GID:1000" driveone/onedrive:latest +``` + +When the Docker container successfully starts: +* You will be asked to open a specific link using your web browser +* Login to your Microsoft Account and give the application the permission +* After giving the permission, you will be redirected to a blank page +* Copy the URI of the blank page into the application prompt to authorise the application + +Once the 'onedrive' application is authorised, the client will automatically start monitoring your `ONEDRIVE_DATA_DIR` for data changes to be uploaded to OneDrive. Files stored on OneDrive will be downloaded to this location. + +If the client is working as expected, you can detach from the container with Ctrl+p, Ctrl+q. + +### 4. Docker Container Status, stop, and restart Check if the monitor service is running ```bash @@ -75,7 +108,7 @@ Resume monitor docker start onedrive ``` -Remove onedrive monitor +Remove onedrive Docker container ```bash docker rm -f onedrive @@ -85,7 +118,7 @@ docker rm -f onedrive ### 5. Docker-compose Also supports docker-compose schemas > 3. -In the following example it is assumed you have a `onedriveDir` environment variable and a `onedrive_conf` volume. +In the following example it is assumed you have a `ONEDRIVE_DATA_DIR` environment variable and a `onedrive_conf` volume. However, you can also use bind mounts for the configuration folder, e.g. `export ONEDRIVE_CONF="${HOME}/OneDriveConfig"`. ``` @@ -99,14 +132,14 @@ services: - ONEDRIVE_GID=${PGID} volumes: - onedrive_conf:/onedrive/conf - - ${onedriveDir}:/onedrive/data + - ${ONEDRIVE_DATA_DIR}:/onedrive/data ``` Note that you still have to perform step 3: First Run. ### 6. Edit the config -Onedrive should run in default configuration, however you can change your configuration by placing a custom config file in the `onedrive_conf` docker volume. First download the default config from [here](https://raw.githubusercontent.com/abraunegg/onedrive/master/config) +The 'onedrive' client should run in default configuration, however you can change this default configuration by placing a custom config file in the `onedrive_conf` docker volume. First download the default config from [here](https://raw.githubusercontent.com/abraunegg/onedrive/master/config) Then put it into your onedrive_conf volume path, which can be found with: ```bash @@ -123,8 +156,9 @@ There are many ways to do this, the easiest is probably to 1. Create a second docker config volume (replace `Work` with your desired name): `docker volume create onedrive_conf_Work` 2. And start a second docker monitor container (again replace `Work` with your desired name): ``` -onedriveDirWork="/home/abraunegg/OneDriveWork" -docker run -it --restart unless-stopped --name onedrive_Work -v onedrive_conf_Work:/onedrive/conf -v "${onedriveDirWork}:/onedrive/data" driveone/onedrive:latest +export ONEDRIVE_DATA_DIR_WORK="/home/abraunegg/OneDriveWork" +mkdir -p ${ONEDRIVE_DATA_DIR_WORK} +docker run -it --restart unless-stopped --name onedrive_Work -v onedrive_conf_Work:/onedrive/conf -v "${ONEDRIVE_DATA_DIR_WORK}:/onedrive/data" driveone/onedrive:latest ``` ## Run or update with one script @@ -132,14 +166,14 @@ docker run -it --restart unless-stopped --name onedrive_Work -v onedrive_conf_Wo If you are experienced with docker and onedrive, you can use the following script: ```bash -# Update onedriveDir with correct existing OneDrive directory path -onedriveDir="${HOME}/OneDrive" +# Update ONEDRIVE_DATA_DIR with correct existing OneDrive directory path +ONEDRIVE_DATA_DIR="${HOME}/OneDrive" firstRun='-d' docker pull driveone/onedrive:latest docker inspect onedrive_conf > /dev/null || { docker volume create onedrive_conf; firstRun='-it'; } docker inspect onedrive > /dev/null && docker rm -f onedrive -docker run $firstRun --restart unless-stopped --name onedrive -v onedrive_conf:/onedrive/conf -v "${onedriveDir}:/onedrive/data" driveone/onedrive:latest +docker run $firstRun --restart unless-stopped --name onedrive -v onedrive_conf:/onedrive/conf -v "${ONEDRIVE_DATA_DIR}:/onedrive/data" driveone/onedrive:latest ``` @@ -160,24 +194,24 @@ docker run $firstRun --restart unless-stopped --name onedrive -v onedrive_conf:/ ### Usage Examples **Verbose Output:** ```bash -docker container run -e ONEDRIVE_VERBOSE=1 -v onedrive_conf:/onedrive/conf -v "${onedriveDir}:/onedrive/data" driveone/onedrive:latest +docker container run -e ONEDRIVE_VERBOSE=1 -v onedrive_conf:/onedrive/conf -v "${ONEDRIVE_DATA_DIR}:/onedrive/data" driveone/onedrive:latest ``` **Debug Output:** ```bash -docker container run -e ONEDRIVE_DEBUG=1 -v onedrive_conf:/onedrive/conf -v "${onedriveDir}:/onedrive/data" driveone/onedrive:latest +docker container run -e ONEDRIVE_DEBUG=1 -v onedrive_conf:/onedrive/conf -v "${ONEDRIVE_DATA_DIR}:/onedrive/data" driveone/onedrive:latest ``` **Perform a --resync:** ```bash -docker container run -e ONEDRIVE_RESYNC=1 -v onedrive_conf:/onedrive/conf -v "${onedriveDir}:/onedrive/data" driveone/onedrive:latest +docker container run -e ONEDRIVE_RESYNC=1 -v onedrive_conf:/onedrive/conf -v "${ONEDRIVE_DATA_DIR}:/onedrive/data" driveone/onedrive:latest ``` **Perform a --resync and --verbose:** ```bash -docker container run -e ONEDRIVE_RESYNC=1 -e ONEDRIVE_VERBOSE=1 -v onedrive_conf:/onedrive/conf -v "${onedriveDir}:/onedrive/data" driveone/onedrive:latest +docker container run -e ONEDRIVE_RESYNC=1 -e ONEDRIVE_VERBOSE=1 -v onedrive_conf:/onedrive/conf -v "${ONEDRIVE_DATA_DIR}:/onedrive/data" driveone/onedrive:latest ``` **Perform a --logout and re-authenticate:** ```bash -docker container run -e ONEDRIVE_LOGOUT=1 -v onedrive_conf:/onedrive/conf -v "${onedriveDir}:/onedrive/data" driveone/onedrive:latest +docker container run -e ONEDRIVE_LOGOUT=1 -v onedrive_conf:/onedrive/conf -v "${ONEDRIVE_DATA_DIR}:/onedrive/data" driveone/onedrive:latest ``` ## Build instructions From 42ef8d72b56608ef41147ea2f7884a882a7c20a7 Mon Sep 17 00:00:00 2001 From: Yonn Trimoreau Date: Sat, 6 Mar 2021 20:30:24 +0100 Subject: [PATCH 02/19] Fix typo in advanced-usage.md (#1322) * Fix typo in advanced-usage.md --- docs/advanced-usage.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/advanced-usage.md b/docs/advanced-usage.md index 1f7f8070..fa58d690 100644 --- a/docs/advanced-usage.md +++ b/docs/advanced-usage.md @@ -61,7 +61,7 @@ Test the configuration using '--display-config' and '--dry-run'. By doing so, th #### Display the configuration ```text -onedrive --confdir="~/.config/my-new-config --display-config" +onedrive --confdir="~/.config/my-new-config" --display-config ``` #### Test the configuration by performing a dry-run From 4ef6c7e331485640702fed0650960a77ea233cb3 Mon Sep 17 00:00:00 2001 From: abraunegg Date: Mon, 8 Mar 2021 05:44:22 +1100 Subject: [PATCH 03/19] Update Docker.md * Remove '**' to clean up document as it was left in by mistake after other edits --- docs/Docker.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/Docker.md b/docs/Docker.md index b0b02cd7..f64c3cf0 100644 --- a/docs/Docker.md +++ b/docs/Docker.md @@ -55,7 +55,7 @@ ROOT level privileges prohibited! ### 3. First run The 'onedrive' client within the Docker container needs to be authorized with your Microsoft account. This is achieved by initially running docker in interactive mode. -Run the docker image with the commands below and make sure to change `ONEDRIVE_DATA_DIR` to the actual onedrive data directory on your filesystem that you wish to use (e.g. `"/home/abraunegg/OneDrive"`)**. +Run the docker image with the commands below and make sure to change `ONEDRIVE_DATA_DIR` to the actual onedrive data directory on your filesystem that you wish to use (e.g. `"/home/abraunegg/OneDrive"`). ```bash export ONEDRIVE_DATA_DIR="${HOME}/OneDrive" mkdir -p ${ONEDRIVE_DATA_DIR} From 7786a17915d51323195c160e52a5030095310dea Mon Sep 17 00:00:00 2001 From: abraunegg Date: Mon, 8 Mar 2021 07:17:48 +1100 Subject: [PATCH 04/19] Remove depreciated config option 'force_http_11' (#1323) * Remove depreciated config option 'force_http_11' which was flagged as depreciated by PR #549 in v2.3.6 (June 2019). --- config | 1 - docs/USAGE.md | 1 - onedrive.1.in | 5 ----- src/config.d | 4 ---- src/main.d | 5 ----- 5 files changed, 16 deletions(-) diff --git a/config b/config index 7ba34365..1747024e 100644 --- a/config +++ b/config @@ -18,7 +18,6 @@ # disable_notifications = "false" # disable_upload_validation = "false" # enable_logging = "false" -# force_http_11 = "false" # force_http_2 = "false" # local_first = "false" # no_remote_delete = "false" diff --git a/docs/USAGE.md b/docs/USAGE.md index 8770cedb..b27d7b64 100644 --- a/docs/USAGE.md +++ b/docs/USAGE.md @@ -326,7 +326,6 @@ See the [config](https://raw.githubusercontent.com/abraunegg/onedrive/master/con # disable_notifications = "false" # disable_upload_validation = "false" # enable_logging = "false" -# force_http_11 = "false" # force_http_2 = "false" # local_first = "false" # no_remote_delete = "false" diff --git a/onedrive.1.in b/onedrive.1.in index a97a3847..0e24c519 100644 --- a/onedrive.1.in +++ b/onedrive.1.in @@ -89,11 +89,6 @@ Configuration file key: \fBenable_logging\fP (default: \fBfalse\fP) \fB\-\-force\fP Force the deletion of data when a 'big delete' is detected .TP -\fB\-\-force\-http\-1.1\fP -Force the use of HTTP 1.1 for all operations (DEPRECIATED) -.br -Configuration file key: \fBforce_http_11\fP (default: \fBfalse\fP) -.TP \fB\-\-force\-http\-2\fP Force the use of HTTP/2 for all operations where applicable .br diff --git a/src/config.d b/src/config.d index 33e7ba81..4b13c4e6 100644 --- a/src/config.d +++ b/src/config.d @@ -61,7 +61,6 @@ final class Config boolValues["disable_notifications"] = false; boolValues["disable_upload_validation"] = false; boolValues["enable_logging"] = false; - boolValues["force_http_11"] = false; boolValues["force_http_2"] = false; boolValues["local_first"] = false; boolValues["no_remote_delete"] = false; @@ -343,9 +342,6 @@ final class Config "enable-logging", "Enable client activity to a separate log file", &boolValues["enable_logging"], - "force-http-1.1", - "Force the use of HTTP/1.1 for all operations (DEPRECIATED)", - &boolValues["force_http_11"], "force-http-2", "Force the use of HTTP/2 for all operations where applicable", &boolValues["force_http_2"], diff --git a/src/main.d b/src/main.d index 18199c20..02f72313 100644 --- a/src/main.d +++ b/src/main.d @@ -580,11 +580,6 @@ int main(string[] args) return EXIT_SUCCESS; } - // If the user is still using --force-http-1.1 advise its no longer required - if (cfg.getValueBool("force_http_11")) { - log.log("NOTE: The use of --force-http-1.1 is depreciated"); - } - // Test if OneDrive service can be reached, exit if it cant be reached log.vdebug("Testing network to ensure network connectivity to Microsoft OneDrive Service"); online = testNetwork(); From 13d6522be884d9ea128172276420e6a448009a83 Mon Sep 17 00:00:00 2001 From: abraunegg Date: Wed, 10 Mar 2021 07:18:18 +1100 Subject: [PATCH 05/19] Update INSTALL.md * Add Alpine Linux package details --- docs/INSTALL.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/INSTALL.md b/docs/INSTALL.md index c9a2bcf4..f254bbac 100644 --- a/docs/INSTALL.md +++ b/docs/INSTALL.md @@ -5,6 +5,7 @@ This project has been packaged for the following Linux distributions: | Distribution | Package Name & Package Link |  i686  | x86_64 | ARMHF | AARCH64 | Extra Details | |---------------------------------|------------------------------------------------------------------------------|:----:|:------:|:-----:|:-------:|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| Alpine Linux | [onedrive](https://pkgs.alpinelinux.org/packages?name=onedrive&branch=edge) |not_supported|supported|not_supported|supported| | Arch Linux

Manjaro Linux | [onedrive-abraunegg](https://aur.archlinux.org/packages/onedrive-abraunegg/) |supported|supported|supported|supported| Install via: `pamac build onedrive-abraunegg` from the Arch Linux User Repository (AUR)

**Note:** If asked regarding a provider for 'd-runtime' and 'd-compiler', select 'liblphobos' and 'ldc'

**Note:** System must have at least 1GB of memory & 1GB swap space | Debian | [onedrive](https://packages.debian.org/search?keywords=onedrive) |supported|supported|supported|supported| | | Fedora | [onedrive](https://koji.fedoraproject.org/koji/packageinfo?packageID=26044) |supported|supported|supported|supported| | From cefb3169fafd289907fe82cf412076b264faf419 Mon Sep 17 00:00:00 2001 From: abraunegg Date: Thu, 11 Mar 2021 09:57:36 +1100 Subject: [PATCH 06/19] Update README.md * Add 'Have a question?' --- README.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 495662b2..e4a0145a 100644 --- a/README.md +++ b/README.md @@ -33,7 +33,10 @@ This client is a 'fork' of the [skilion](https://github.com/skilion/onedrive) cl ## Frequently Asked Questions Refer to [Frequently Asked Questions](https://github.com/abraunegg/onedrive/wiki/Frequently-Asked-Questions) -## Reporting issues +## Have a question? +If you have a question or need something clarified, please raise a new disscussion post [here](https://github.com/abraunegg/onedrive/discussions) + +## Reporting an Issue or Bug If you encounter any bugs you can report them here on Github. Before filing an issue be sure to: 1. Check the version of the application you are using `onedrive --version` and ensure that you are running either the latest [release](https://github.com/abraunegg/onedrive/releases) or built from master. From b8717fbc536fed5786fb5700b14443ade0249cf9 Mon Sep 17 00:00:00 2001 From: abraunegg Date: Sat, 13 Mar 2021 05:11:44 +1100 Subject: [PATCH 07/19] Use 'nextLink' value if present when searching for specific SharePoint site names (#1329) * Use 'nextLink' value if present when searching for specific SharePoint site names * Update error output to provide more details why an error occurred if a SharePoint site lacks the details we need to perform the match --- README.md | 2 +- src/onedrive.d | 9 ++- src/sync.d | 213 ++++++++++++++++++++++++++++++++----------------- 3 files changed, 147 insertions(+), 77 deletions(-) diff --git a/README.md b/README.md index e4a0145a..f9b4b1c8 100644 --- a/README.md +++ b/README.md @@ -33,7 +33,7 @@ This client is a 'fork' of the [skilion](https://github.com/skilion/onedrive) cl ## Frequently Asked Questions Refer to [Frequently Asked Questions](https://github.com/abraunegg/onedrive/wiki/Frequently-Asked-Questions) -## Have a question? +## Have a question If you have a question or need something clarified, please raise a new disscussion post [here](https://github.com/abraunegg/onedrive/discussions) ## Reporting an Issue or Bug diff --git a/src/onedrive.d b/src/onedrive.d index 40996f68..1714ddaf 100644 --- a/src/onedrive.d +++ b/src/onedrive.d @@ -780,10 +780,15 @@ final class OneDriveApi } // https://docs.microsoft.com/en-us/onedrive/developer/rest-api/api/site_search?view=odsp-graph-online - JSONValue o365SiteSearch(){ + JSONValue o365SiteSearch(const(char)[] nextLink){ checkAccessTokenExpired(); const(char)[] url; - url = siteSearchUrl ~ "=*"; + // configure URL to query + if (nextLink.empty) { + url = siteSearchUrl ~ "=*"; + } else { + url = nextLink; + } return get(url); } diff --git a/src/sync.d b/src/sync.d index f1b687a5..66773594 100644 --- a/src/sync.d +++ b/src/sync.d @@ -5392,91 +5392,156 @@ final class SyncEngine string drive_id; string webUrl; bool found = false; - JSONValue siteQuery; + JSONValue siteQuery; + string nextLink; + string[] siteSearchResults; log.log("Office 365 Library Name Query: ", o365SharedLibraryName); - try { - siteQuery = onedrive.o365SiteSearch(); - } catch (OneDriveException e) { - log.error("ERROR: Query of OneDrive for Office 365 Library Name failed"); - if (e.httpStatusCode == 403) { - // Forbidden - most likely authentication scope needs to be updated - log.error("ERROR: Authentication scope needs to be updated. Use --logout and re-authenticate client."); - return; - } else { - // display what the error is - displayOneDriveErrorMessage(e.msg, getFunctionName!({})); - return; - } - } - - // is siteQuery a valid JSON object & contain data we can use? - if ((siteQuery.type() == JSONType.object) && ("value" in siteQuery)) { - // valid JSON object - log.vdebug("O365 Query Response: ", siteQuery); - - foreach (searchResult; siteQuery["value"].array) { - // Need an 'exclusive' match here with o365SharedLibraryName as entered - log.vdebug("Found O365 Site: ", searchResult); - - // 'displayName', 'id' and 'webUrl' have to be present in the search result record - if (("displayName" in searchResult) && ("id" in searchResult) && ("webUrl" in searchResult)) { - if (o365SharedLibraryName == searchResult["displayName"].str){ - // 'displayName' matches search request - site_id = searchResult["id"].str; - webUrl = searchResult["webUrl"].str; - JSONValue siteDriveQuery; - - try { - siteDriveQuery = onedrive.o365SiteDrives(site_id); - } catch (OneDriveException e) { - log.error("ERROR: Query of OneDrive for Office Site ID failed"); - // display what the error is - displayOneDriveErrorMessage(e.msg, getFunctionName!({})); - return; - } - - // is siteDriveQuery a valid JSON object & contain data we can use? - if ((siteDriveQuery.type() == JSONType.object) && ("value" in siteDriveQuery)) { - // valid JSON object - foreach (driveResult; siteDriveQuery["value"].array) { - // Display results - found = true; - writeln("SiteName: ", searchResult["displayName"].str); - writeln("drive_id: ", driveResult["id"].str); - writeln("URL: ", webUrl); - } - } else { - // not a valid JSON object - log.error("ERROR: There was an error performing this operation on OneDrive"); - log.error("ERROR: Increase logging verbosity to assist determining why."); - return; - } + for (;;) { + try { + siteQuery = onedrive.o365SiteSearch(nextLink); + } catch (OneDriveException e) { + log.error("ERROR: Query of OneDrive for Office 365 Library Name failed"); + if (e.httpStatusCode == 403) { + // Forbidden - most likely authentication scope needs to be updated + log.error("ERROR: Authentication scope needs to be updated. Use --logout and re-authenticate client."); + return; + } + // HTTP request returned status code 429 (Too Many Requests) + if (e.httpStatusCode == 429) { + // HTTP request returned status code 429 (Too Many Requests). We need to leverage the response Retry-After HTTP header to ensure minimum delay until the throttle is removed. + handleOneDriveThrottleRequest(); + log.vdebug("Retrying original request that generated the OneDrive HTTP 429 Response Code (Too Many Requests) - attempting to query OneDrive drive children"); + } + // HTTP request returned status code 504 (Gateway Timeout) or 429 retry + if ((e.httpStatusCode == 429) || (e.httpStatusCode == 504)) { + // re-try the specific changes queries + if (e.httpStatusCode == 504) { + log.log("OneDrive returned a 'HTTP 504 - Gateway Timeout' when attempting to query Sharepoint Sites - retrying applicable request"); + log.vdebug("siteQuery = onedrive.o365SiteSearch(nextLink) previously threw an error - retrying"); + // The server, while acting as a proxy, did not receive a timely response from the upstream server it needed to access in attempting to complete the request. + log.vdebug("Thread sleeping for 30 seconds as the server did not receive a timely response from the upstream server it needed to access in attempting to complete the request"); + Thread.sleep(dur!"seconds"(30)); + } + // re-try original request - retried for 429 and 504 + try { + log.vdebug("Retrying Query: siteQuery = onedrive.o365SiteSearch(nextLink)"); + siteQuery = onedrive.o365SiteSearch(nextLink); + log.vdebug("Query 'siteQuery = onedrive.o365SiteSearch(nextLink)' performed successfully on re-try"); + } catch (OneDriveException e) { + // display what the error is + log.vdebug("Query Error: siteQuery = onedrive.o365SiteSearch(nextLink) on re-try after delay"); + // error was not a 504 this time + displayOneDriveErrorMessage(e.msg, getFunctionName!({})); + return; } } else { - // 'displayName' not present in JSON results - log.error("ERROR: The results returned from OneDrive API do not contain the required items to match. Please check your permissions with your site administrator."); - log.error("ERROR: Your site security settings is preventing the following details from being accessed: 'displayName', 'id' and 'webUrl'"); - log.error("ERROR: To debug this further, please use --verbose --verbose to provide insight as to what details are actually returned."); + // display what the error is + displayOneDriveErrorMessage(e.msg, getFunctionName!({})); return; } } - if(!found) { - log.error("ERROR: The requested SharePoint site could not be found. Please check it's name and your permissions to access the site."); - // List all sites returned to assist user - log.log("\nThe following SharePoint site names were returned:"); + // is siteQuery a valid JSON object & contain data we can use? + if ((siteQuery.type() == JSONType.object) && ("value" in siteQuery)) { + // valid JSON object + log.vdebug("O365 Query Response: ", siteQuery); + foreach (searchResult; siteQuery["value"].array) { - // list the display name that we use to match against the user query - log.log(" * ", searchResult["displayName"].str); + // Need an 'exclusive' match here with o365SharedLibraryName as entered + log.vdebug("Found O365 Site: ", searchResult); + + // 'displayName', 'id' and 'webUrl' have to be present in the search result record + if (("displayName" in searchResult) && ("id" in searchResult) && ("webUrl" in searchResult)) { + if (o365SharedLibraryName == searchResult["displayName"].str){ + // 'displayName' matches search request + site_id = searchResult["id"].str; + webUrl = searchResult["webUrl"].str; + JSONValue siteDriveQuery; + + try { + siteDriveQuery = onedrive.o365SiteDrives(site_id); + } catch (OneDriveException e) { + log.error("ERROR: Query of OneDrive for Office Site ID failed"); + // display what the error is + displayOneDriveErrorMessage(e.msg, getFunctionName!({})); + return; + } + + // is siteDriveQuery a valid JSON object & contain data we can use? + if ((siteDriveQuery.type() == JSONType.object) && ("value" in siteDriveQuery)) { + // valid JSON object + foreach (driveResult; siteDriveQuery["value"].array) { + // Display results + found = true; + writeln("SiteName: ", searchResult["displayName"].str); + writeln("drive_id: ", driveResult["id"].str); + writeln("URL: ", webUrl); + } + } else { + // not a valid JSON object + log.error("ERROR: There was an error performing this operation on OneDrive"); + log.error("ERROR: Increase logging verbosity to assist determining why."); + return; + } + } + } else { + // 'displayName', 'id' or ''webUrl' not present in JSON results for a specific site + string siteNameAvailable = "Site 'name' was restricted by OneDrive API permissions"; + bool displayNameAvailable = false; + bool idAvailable = false; + bool webUrlAvailable = false; + if ("name" in searchResult) siteNameAvailable = searchResult["name"].str; + if ("displayName" in searchResult) displayNameAvailable = true; + if ("id" in searchResult) idAvailable = true; + if ("webUrl" in searchResult) webUrlAvailable = true; + + // Display error details for this site data + log.error("\nERROR: SharePoint Site details not provided for: ", siteNameAvailable); + log.error("ERROR: The SharePoint Site results returned from OneDrive API do not contain the required items to match. Please check your permissions with your site administrator."); + log.error("ERROR: Your site security settings is preventing the following details from being accessed: 'displayName', 'id' and 'webUrl'"); + log.vlog(" - Is 'displayName' available = ", displayNameAvailable); + log.vlog(" - Is 'id' available = ", idAvailable); + log.vlog(" - Is 'webUrl' available = ", webUrlAvailable); + log.error("ERROR: To debug this further, please increase verbosity (--verbose or --verbose --verbose) to provide further insight as to what details are actually being returned."); + } } + + if(!found) { + // The SharePoint site we are searching for was not found in this bundle set + // Add to siteSearchResults so we can display what we did find + string siteSearchResultsEntry; + foreach (searchResult; siteQuery["value"].array) { + siteSearchResultsEntry = " * " ~ searchResult["displayName"].str; + siteSearchResults ~= siteSearchResultsEntry; + } + } + } else { + // not a valid JSON object + log.error("ERROR: There was an error performing this operation on OneDrive"); + log.error("ERROR: Increase logging verbosity to assist determining why."); + return; + } + + // If a collection exceeds the default page size (200 items), the @odata.nextLink property is returned in the response + // to indicate more items are available and provide the request URL for the next page of items. + if ("@odata.nextLink" in siteQuery) { + // Update nextLink to next set of SharePoint library names + nextLink = siteQuery["@odata.nextLink"].str; + log.vdebug("Setting nextLink to (@odata.nextLink): ", nextLink); + } else break; + } + + // Was the intended target found? + if(!found) { + log.error("\nERROR: The requested SharePoint site could not be found. Please check it's name and your permissions to access the site."); + // List all sites returned to assist user + log.log("\nThe following SharePoint site names were returned:"); + foreach (searchResultEntry; siteSearchResults) { + // list the display name that we use to match against the user query + log.log(searchResultEntry); } - } else { - // not a valid JSON object - log.error("ERROR: There was an error performing this operation on OneDrive"); - log.error("ERROR: Increase logging verbosity to assist determining why."); - return; } } @@ -6181,8 +6246,8 @@ final class SyncEngine // to indicate more items are available and provide the request URL for the next page of items. if ("@odata.nextLink" in thisLevelChildren) { // Update nextLink to next changeSet bundle - log.vdebug("Setting nextLink to (@odata.nextLink): ", nextLink); nextLink = thisLevelChildren["@odata.nextLink"].str; + log.vdebug("Setting nextLink to (@odata.nextLink): ", nextLink); } else break; } From 10606293f74885a7838a4c8736139ef0f1954277 Mon Sep 17 00:00:00 2001 From: abraunegg Date: Sun, 14 Mar 2021 12:59:17 +1100 Subject: [PATCH 08/19] Update Docker files for Raspberry Pi (#1335) * Revert PR #1259 * Provide dedicated armhf & aarch64 Dockerfiles * Document updates --- contrib/docker/Dockerfile-aarch64 | 21 ++++++++++++++++++++ contrib/docker/Dockerfile-rpi | 7 +++---- docs/Docker.md | 32 +++++++++++++++++++------------ 3 files changed, 44 insertions(+), 16 deletions(-) create mode 100644 contrib/docker/Dockerfile-aarch64 diff --git a/contrib/docker/Dockerfile-aarch64 b/contrib/docker/Dockerfile-aarch64 new file mode 100644 index 00000000..84ed407a --- /dev/null +++ b/contrib/docker/Dockerfile-aarch64 @@ -0,0 +1,21 @@ +# -*-Dockerfile-*- +FROM debian:stretch +RUN apt update && \ + apt install -y build-essential curl libcurl4-openssl-dev libsqlite3-dev pkg-config wget git +RUN wget https://github.com/ldc-developers/ldc/releases/download/v1.16.0/ldc2-1.16.0-linux-aarch64.tar.xz && \ + tar -xvf ldc2-1.16.0-linux-aarch64.tar.xz +COPY . /usr/src/onedrive +RUN cd /usr/src/onedrive/ && \ + ./configure DC=/ldc2-1.16.0-linux-aarch64/bin/ldmd2 && \ + make clean && \ + make && \ + make install + +FROM debian:stretch-slim +ENTRYPOINT ["/entrypoint.sh"] +RUN apt update && \ + apt install -y gosu libcurl3 libsqlite3-0 && \ + rm -rf /var/*/apt && \ + mkdir -p /onedrive/conf /onedrive/data +COPY contrib/docker/entrypoint.sh / +COPY --from=0 /usr/local/bin/onedrive /usr/local/bin/ diff --git a/contrib/docker/Dockerfile-rpi b/contrib/docker/Dockerfile-rpi index 4c88c9bc..3e37ac77 100644 --- a/contrib/docker/Dockerfile-rpi +++ b/contrib/docker/Dockerfile-rpi @@ -1,13 +1,12 @@ # -*-Dockerfile-*- FROM debian:stretch -ARG ARCH armhf # or aarch64 RUN apt update && \ apt install -y build-essential curl libcurl4-openssl-dev libsqlite3-dev pkg-config wget git -RUN wget https://github.com/ldc-developers/ldc/releases/download/v1.16.0/ldc2-1.16.0-linux-${ARCH}.tar.xz && \ - tar -xvf ldc2-1.16.0-linux-${ARCH}.tar.xz +RUN wget https://github.com/ldc-developers/ldc/releases/download/v1.16.0/ldc2-1.16.0-linux-armhf.tar.xz && \ + tar -xvf ldc2-1.16.0-linux-armhf.tar.xz COPY . /usr/src/onedrive RUN cd /usr/src/onedrive/ && \ - ./configure DC=/ldc2-1.16.0-linux-${ARCH}/bin/ldmd2 && \ + ./configure DC=/ldc2-1.16.0-linux-armhf/bin/ldmd2 && \ make clean && \ make && \ make install diff --git a/docs/Docker.md b/docs/Docker.md index f64c3cf0..3c686a9d 100644 --- a/docs/Docker.md +++ b/docs/Docker.md @@ -116,7 +116,6 @@ docker rm -f onedrive ## Advanced Setup ### 5. Docker-compose - Also supports docker-compose schemas > 3. In the following example it is assumed you have a `ONEDRIVE_DATA_DIR` environment variable and a `onedrive_conf` volume. However, you can also use bind mounts for the configuration folder, e.g. `export ONEDRIVE_CONF="${HOME}/OneDriveConfig"`. @@ -138,7 +137,6 @@ services: Note that you still have to perform step 3: First Run. ### 6. Edit the config - The 'onedrive' client should run in default configuration, however you can change this default configuration by placing a custom config file in the `onedrive_conf` docker volume. First download the default config from [here](https://raw.githubusercontent.com/abraunegg/onedrive/master/config) Then put it into your onedrive_conf volume path, which can be found with: @@ -151,7 +149,6 @@ Or you can map your own config folder to the config volume. Make sure to copy al The detailed document for the config can be found here: [Configuration](https://github.com/abraunegg/onedrive/blob/master/docs/USAGE.md#configuration) ### 7. Sync multiple accounts - There are many ways to do this, the easiest is probably to 1. Create a second docker config volume (replace `Work` with your desired name): `docker volume create onedrive_conf_Work` 2. And start a second docker monitor container (again replace `Work` with your desired name): @@ -162,7 +159,6 @@ docker run -it --restart unless-stopped --name onedrive_Work -v onedrive_conf_Wo ``` ## Run or update with one script - If you are experienced with docker and onedrive, you can use the following script: ```bash @@ -178,8 +174,6 @@ docker run $firstRun --restart unless-stopped --name onedrive -v onedrive_conf:/ ## Environment Variables - - | Variable | Purpose | Sample Value | | ---------------- | --------------------------------------------------- |:-------------:| | ONEDRIVE_UID | UserID (UID) to run as | 1000 | @@ -219,7 +213,7 @@ docker container run -e ONEDRIVE_LOGOUT=1 -v onedrive_conf:/onedrive/conf -v "${ * Build environment must have at least 1GB of memory & 2GB swap space There are 2 ways to validate this requirement: -* Modify the file `/etc/dphys-swapfile` and edit the `CONF_SWAPSIZE`, for example: `CONF_SWAPSIZE=2024`. A reboot is required to make this change effective. +* Modify the file `/etc/dphys-swapfile` and edit the `CONF_SWAPSIZE`, for example: `CONF_SWAPSIZE=2048`. A reboot is required to make this change effective. * Dynamically allocate a swapfile for building: ```bash cd /var @@ -235,7 +229,7 @@ swapon -s free -h ``` -### Building the Docker image +### Building a custom Docker image You can also build your own image instead of pulling the one from [hub.docker.com](https://hub.docker.com/r/driveone/onedrive): ```bash git clone https://github.com/abraunegg/onedrive @@ -248,18 +242,32 @@ Dockerfile-stretch or Dockerfile-alpine. These [multi-stage builder pattern](https://docs.docker.com/develop/develop-images/multistage-build/) Dockerfiles require Docker version at least 17.05. -#### How to build a Docker image based on Debian Stretch - +#### How to build and run a custom Docker image based on Debian Stretch ``` bash docker build . -t local-ondrive-stretch -f contrib/docker/Dockerfile-stretch +docker container run -v onedrive_conf:/onedrive/conf -v "${ONEDRIVE_DATA_DIR}:/onedrive/data" local-ondrive-stretch:latest ``` -#### How to build a Docker image based on Alpine Linux +#### How to build and run a custom Docker image based on Alpine Linux ``` bash docker build . -t local-ondrive-alpine -f contrib/docker/Dockerfile-alpine +docker container run -v onedrive_conf:/onedrive/conf -v "${ONEDRIVE_DATA_DIR}:/onedrive/data" local-ondrive-alpine:latest ``` -#### How to build a Docker image for ARMHF (Raspberry Pi) +#### How to build and run a custom Docker image for ARMHF (Raspberry Pi) +Compatible with: +* Raspberry Pi +* Raspberry Pi 2 +* Raspberry Pi Zero +* Raspberry Pi 3 +* Raspberry Pi 4 ``` bash docker build . -t local-onedrive-rpi -f contrib/docker/Dockerfile-rpi +docker container run -v onedrive_conf:/onedrive/conf -v "${ONEDRIVE_DATA_DIR}:/onedrive/data" local-ondrive-rpi:latest +``` + +#### How to build and run a custom Docker image for AARCH64 Platforms +``` bash +docker build . -t local-onedrive-aarch64 -f contrib/docker/Dockerfile-aarch64 +docker container run -v onedrive_conf:/onedrive/conf -v "${ONEDRIVE_DATA_DIR}:/onedrive/data" local-onedrive-aarch64:latest ``` From 6bec0ddc642617cc1a436833213a1f3ffb1cac31 Mon Sep 17 00:00:00 2001 From: abraunegg Date: Mon, 15 Mar 2021 16:24:55 +1100 Subject: [PATCH 09/19] Correctly handle files which contain '%' as invalid files (#1343) * Fix checking filenames which contain '%' which should be classified as invalid based on the MS OneDrive API Invalid characters designation --- src/util.d | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/util.d b/src/util.d index cd20af0e..db5f237f 100644 --- a/src/util.d +++ b/src/util.d @@ -205,7 +205,7 @@ bool isValidName(string path) // Leading whitespace and trailing whitespace/dot `^\s.*|^.*[\s\.]$|` ~ // Invalid characters - `.*[<>:"\|\?*/\\].*|` ~ + `.*[<>:"\|\?*/\\%].*|` ~ // Reserved device name and trailing .~ `(?:^CON|^PRN|^AUX|^NUL|^COM[0-9]|^LPT[0-9])(?:[.].+)?$` ); From 037a6b43e1631552ae44f2aea042649773b58bd0 Mon Sep 17 00:00:00 2001 From: abraunegg Date: Wed, 17 Mar 2021 18:23:01 +1100 Subject: [PATCH 10/19] Support multiple 'document libraries' within a single Shared Library Site (#1350) * Support providing the 'drive_id' for multiple 'document libraries' within a single Shared Library Site --- docs/SharePoint-Shared-Libraries.md | 31 +++++++++++++++++++++++------ src/sync.d | 22 ++++++++++---------- 2 files changed, 36 insertions(+), 17 deletions(-) diff --git a/docs/SharePoint-Shared-Libraries.md b/docs/SharePoint-Shared-Libraries.md index 811e662b..8a141931 100644 --- a/docs/SharePoint-Shared-Libraries.md +++ b/docs/SharePoint-Shared-Libraries.md @@ -14,18 +14,38 @@ Syncing a OneDrive SharePoint library requires additional configuration for your ## Query that shared library name using the client to obtain the required configuration details 2. Run the following command using the 'onedrive' client ```text -onedrive --get-O365-drive-id '' +onedrive --get-O365-drive-id '' ``` This will return something similar to the following: ```text Configuration file successfully loaded Configuring Global Azure AD Endpoints Initializing the Synchronization Engine ... -Office 365 Library Name Query: -SiteName: -drive_id: b!6H_y8B...xU5 -URL: +Office 365 Library Name Query: +----------------------------------------------- +Site Name: +Library Name: +drive_id: b!6H_y8B...xU5 +Library URL: +----------------------------------------------- ``` +If there are no matches to the site you are attempting to search, the following will be displayed: +```text +Configuration file successfully loaded +Configuring Global Azure AD Endpoints +Initializing the Synchronization Engine ... +Office 365 Library Name Query: blah + +ERROR: The requested SharePoint site could not be found. Please check it's name and your permissions to access the site. + +The following SharePoint site names were returned: + * + * + ... + * +``` +This list of site names can be used as a basis to search for the correct site for which you are searching + ## Configure the client's config file with the required 'drive_id' & 'sync_dir' options 3. Create a new local folder to store the SharePoint Library data in @@ -54,6 +74,5 @@ The OneDrive client will now be configured to sync this SharePoint shared librar ## Sync the SharePoint Library as required 6. Sync the SharePoint Library to your system with either `--synchronize` or `--monitor` operations - # How to configure multiple OneDrive SharePoint Shared Library sync Refer to [./advanced-usage.md](advanced-usage.md) for configuration assistance. diff --git a/src/sync.d b/src/sync.d index 66773594..7345e136 100644 --- a/src/sync.d +++ b/src/sync.d @@ -5390,7 +5390,6 @@ final class SyncEngine string site_id; string drive_id; - string webUrl; bool found = false; JSONValue siteQuery; string nextLink; @@ -5452,12 +5451,11 @@ final class SyncEngine // Need an 'exclusive' match here with o365SharedLibraryName as entered log.vdebug("Found O365 Site: ", searchResult); - // 'displayName', 'id' and 'webUrl' have to be present in the search result record - if (("displayName" in searchResult) && ("id" in searchResult) && ("webUrl" in searchResult)) { + // 'displayName' and 'id' have to be present in the search result record in order to query the site + if (("displayName" in searchResult) && ("id" in searchResult)) { if (o365SharedLibraryName == searchResult["displayName"].str){ // 'displayName' matches search request site_id = searchResult["id"].str; - webUrl = searchResult["webUrl"].str; JSONValue siteDriveQuery; try { @@ -5474,11 +5472,16 @@ final class SyncEngine // valid JSON object foreach (driveResult; siteDriveQuery["value"].array) { // Display results + writeln("-----------------------------------------------"); + log.vdebug("Site Details: ", driveResult); found = true; - writeln("SiteName: ", searchResult["displayName"].str); - writeln("drive_id: ", driveResult["id"].str); - writeln("URL: ", webUrl); + writeln("Site Name: ", searchResult["displayName"].str); + writeln("Library Name: ", driveResult["name"].str); + writeln("drive_id: ", driveResult["id"].str); + writeln("Library URL: ", driveResult["webUrl"].str); } + // closeout + writeln("-----------------------------------------------"); } else { // not a valid JSON object log.error("ERROR: There was an error performing this operation on OneDrive"); @@ -5491,19 +5494,16 @@ final class SyncEngine string siteNameAvailable = "Site 'name' was restricted by OneDrive API permissions"; bool displayNameAvailable = false; bool idAvailable = false; - bool webUrlAvailable = false; if ("name" in searchResult) siteNameAvailable = searchResult["name"].str; if ("displayName" in searchResult) displayNameAvailable = true; if ("id" in searchResult) idAvailable = true; - if ("webUrl" in searchResult) webUrlAvailable = true; // Display error details for this site data log.error("\nERROR: SharePoint Site details not provided for: ", siteNameAvailable); log.error("ERROR: The SharePoint Site results returned from OneDrive API do not contain the required items to match. Please check your permissions with your site administrator."); - log.error("ERROR: Your site security settings is preventing the following details from being accessed: 'displayName', 'id' and 'webUrl'"); + log.error("ERROR: Your site security settings is preventing the following details from being accessed: 'displayName' or 'id'"); log.vlog(" - Is 'displayName' available = ", displayNameAvailable); log.vlog(" - Is 'id' available = ", idAvailable); - log.vlog(" - Is 'webUrl' available = ", webUrlAvailable); log.error("ERROR: To debug this further, please increase verbosity (--verbose or --verbose --verbose) to provide further insight as to what details are actually being returned."); } } From 0dcc76f9f1a0a7a5574c2c12a7722b8f41b5dc11 Mon Sep 17 00:00:00 2001 From: abraunegg Date: Thu, 18 Mar 2021 13:24:26 +1100 Subject: [PATCH 11/19] Update 'Skipping shared folder due to existing name conflict' check for Business Shared Folders (#1351) * Update 'Skipping shared folder due to existing name conflict' check for Business Shared Folders --- src/sync.d | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/src/sync.d b/src/sync.d index 7345e136..c8d781cb 100644 --- a/src/sync.d +++ b/src/sync.d @@ -639,9 +639,12 @@ final class SyncEngine // for each driveid in the existing driveIDsArray foreach (searchDriveId; driveIDsArray) { log.vdebug("searching database for: ", searchDriveId, " ", sharedFolderName); - if (itemdb.selectByPath(sharedFolderName, searchDriveId, databaseItem)) { + if (itemdb.idInLocalDatabase(searchDriveId, searchResult["remoteItem"]["id"].str)){ + // Shared folder is present log.vdebug("Found shared folder name in database"); itemInDatabase = true; + // Query the DB for the details of this item + itemdb.selectByPath(sharedFolderName, searchDriveId, databaseItem); log.vdebug("databaseItem: ", databaseItem); // Does the databaseItem.driveId == defaultDriveId? if (databaseItem.driveId == defaultDriveId) { From 6b20478635cdbfe3bb24b79c35ce52325c77e837 Mon Sep 17 00:00:00 2001 From: abraunegg Date: Fri, 19 Mar 2021 17:29:55 +1100 Subject: [PATCH 12/19] Fix 'Item cannot be deleted from OneDrive because it was not found in the local database' (#1354) * When in --monitor mode, and there are multiple driveId being used (shared folders), when a deletion event occurs, search the DB using all the known driveids rather than just the default one. When using just the default driveid, if the file to be deleted resides on another drive, the following error message is printed: 'Item cannot be deleted from OneDrive because it was not found in the local database' - which is not entirely accurate as the item is in the database, it is just not being searched for correctly. --- src/main.d | 2 +- src/sync.d | 24 ++++++++++++++++++++++-- 2 files changed, 23 insertions(+), 3 deletions(-) diff --git a/src/main.d b/src/main.d index 02f72313..3584a022 100644 --- a/src/main.d +++ b/src/main.d @@ -1043,7 +1043,7 @@ int main(string[] args) log.vlog("Offline, cannot delete item!"); } catch(SyncException e) { if (e.msg == "The item to delete is not in the local database") { - log.vlog("Item cannot be deleted from OneDrive because not found in the local database"); + log.vlog("Item cannot be deleted from OneDrive because it was not found in the local database"); } else { log.logAndNotify("Cannot delete remote item: ", e.msg); } diff --git a/src/sync.d b/src/sync.d index c8d781cb..0569edf7 100644 --- a/src/sync.d +++ b/src/sync.d @@ -933,7 +933,17 @@ final class SyncEngine } Item item; - if (!itemdb.selectByPath(path, defaultDriveId, item)) { + // Need to check all driveid's we know about, not just the defaultDriveId + bool itemInDB = false; + foreach (searchDriveId; driveIDsArray) { + if (itemdb.selectByPath(path, searchDriveId, item)) { + // item was found in the DB + itemInDB = true; + break; + } + } + // Was the item found in the DB + if (!itemInDB) { // this is odd .. this directory is not in the local database - just go delete it log.vlog("The requested directory to delete was not found in the local database - pushing delete request direct to OneDrive"); uploadDeleteItem(item, path); @@ -5331,9 +5341,19 @@ final class SyncEngine void deleteByPath(const(string) path) { Item item; - if (!itemdb.selectByPath(path, defaultDriveId, item)) { + // Need to check all driveid's we know about, not just the defaultDriveId + bool itemInDB = false; + foreach (searchDriveId; driveIDsArray) { + if (itemdb.selectByPath(path, searchDriveId, item)) { + // item was found in the DB + itemInDB = true; + break; + } + } + if (!itemInDB) { throw new SyncException("The item to delete is not in the local database"); } + if (item.parentId == null) { // the item is a remote folder, need to do the operation on the parent enforce(itemdb.selectByPathWithoutRemote(path, defaultDriveId, item)); From 71601d324061111c3316d36e341e4824c3274601 Mon Sep 17 00:00:00 2001 From: abraunegg Date: Fri, 26 Mar 2021 07:19:43 +1100 Subject: [PATCH 13/19] Update INSTALL.md and USAGE.md (#1363) * Update ARMHF and ARM64 build instructions after validating on various platforms and confirming application operation / results --- docs/INSTALL.md | 44 +++++++++++++++++++++++++++++--------------- docs/USAGE.md | 32 ++++++++++++++++++++++++++++---- 2 files changed, 57 insertions(+), 19 deletions(-) diff --git a/docs/INSTALL.md b/docs/INSTALL.md index f254bbac..82177097 100644 --- a/docs/INSTALL.md +++ b/docs/INSTALL.md @@ -237,28 +237,42 @@ sudo pacman -S libnotify ``` ### Dependencies: Raspbian (ARMHF) +Validated using: +* `Linux raspberrypi 5.4.79-v7+ #1373 SMP Mon Nov 23 13:22:33 GMT 2020 armv7l GNU/Linux` (2020-12-02-raspios-buster-armhf) using Raspberry Pi 2 Model B +* `Linux raspberrypi 5.4.83-v8+ #1379 SMP PREEMPT Mon Dec 14 13:15:14 GMT 2020 aarch64` (2021-01-11-raspios-buster-armhf) using Raspberry Pi 3 Model B+ + **Note:** Build environment must have at least 1GB of memory & 1GB swap space. Check with `swapon`. + ```text -sudo apt-get install libcurl4-openssl-dev -sudo apt-get install libsqlite3-dev -sudo apt-get install libxml2 -sudo apt-get install pkg-config -wget https://github.com/ldc-developers/ldc/releases/download/v1.16.0/ldc2-1.16.0-linux-armhf.tar.xz -tar -xvf ldc2-1.16.0-linux-armhf.tar.xz +sudo apt install build-essential +sudo apt install libcurl4-openssl-dev +sudo apt install libsqlite3-dev +sudo apt install pkg-config +sudo apt install git +sudo apt install curl +wget https://github.com/ldc-developers/ldc/releases/download/v1.17.0/ldc2-1.17.0-linux-armhf.tar.xz +tar -xvf ldc2-1.17.0-linux-armhf.tar.xz ``` For notifications the following is also necessary: ```text sudo apt install libnotify-dev ``` -### Dependencies: Debian (ARM64) +### Dependencies: Ubuntu 20.x / Debian 10 (ARM64) +Validated using: +* `Ubuntu 20.04.2 LTS (GNU/Linux 5.4.0-1028-raspi aarch64)` (ubuntu-20.04.2-preinstalled-server-arm64+raspi) using Raspberry Pi 3 Model B+ + +**Note:** Build environment must have at least 1GB of memory & 1GB swap space. Check with `swapon`. + ```text -sudo apt-get install libcurl4-openssl-dev -sudo apt-get install libsqlite3-dev -sudo apt-get install libxml2 -sudo apt-get install pkg-config -wget https://github.com/ldc-developers/ldc/releases/download/v1.16.0/ldc2-1.16.0-linux-aarch64.tar.xz -tar -xvf ldc2-1.16.0-linux-aarch64.tar.xz +sudo apt install build-essential +sudo apt install libcurl4-openssl-dev +sudo apt install libsqlite3-dev +sudo apt install pkg-config +sudo apt install git +sudo apt install curl +wget https://github.com/ldc-developers/ldc/releases/download/v1.25.1/ldc2-1.25.1-linux-aarch64.tar.xz +tar -xvf ldc2-1.25.1-linux-aarch64.tar.xz ``` For notifications the following is also necessary: ```text @@ -338,7 +352,7 @@ as far as possible automatically, but can be overridden by passing ```text git clone https://github.com/abraunegg/onedrive.git cd onedrive -./configure DC=~/ldc2-1.16.0-linux-armhf/bin/ldmd2 +./configure DC=~/ldc2-1.17.0-linux-armhf/bin/ldmd2 make clean; make sudo make install ``` @@ -348,7 +362,7 @@ sudo make install ```text git clone https://github.com/abraunegg/onedrive.git cd onedrive -./configure DC=~/ldc2-1.16.0-linux-aarch64/bin/ldmd2 +./configure DC=~/ldc2-1.25.1-linux-aarch64/bin/ldmd2 make clean; make sudo make install ``` diff --git a/docs/USAGE.md b/docs/USAGE.md index b27d7b64..2a06d52d 100644 --- a/docs/USAGE.md +++ b/docs/USAGE.md @@ -697,11 +697,35 @@ systemctl --user start onedrive **Note:** This will run the 'onedrive' process with a UID/GID of '0', thus, any files or folders that are created will be owned by 'root' -To see the logs run: +To see the systemd application logs run: ```text journalctl --user-unit=onedrive -f ``` +**Note:** It is a 'systemd' requirement that the XDG environment variables exist for correct enablement and operation of systemd services. If you receive this error when enabling the systemd service: +``` +Failed to connect to bus: No such file or directory +``` +The most likely cause is that the XDG environment variables are missing. To fix this, you must add the following to `.bashrc` or any other file which is run on user login: +``` +export XDG_RUNTIME_DIR="/run/user/$UID" +export DBUS_SESSION_BUS_ADDRESS="unix:path=${XDG_RUNTIME_DIR}/bus" +``` + +To make this change effective, you must logout of all user accounts where this change has been made. + +**Note:** On some systems (for example - Raspbian / Ubuntu / Debian on Raspberry Pi) the above XDG fix may not be reliable after system reboots. The potential alternative to start the client via systemd as root, is to perform the following: +1. Create a symbolic link from `/home/root/.config/onedrive` pointing to `/root/.config/onedrive/` +2. Create a systemd service using the '@' service file: `systemctl enable onedrive@root.service` +3. Start the root@service: `systemctl start onedrive@root.service` + +This will ensure that the service will correctly restart on system reboot. + +To see the systemd application logs run: +```text +journalctl --unit=onedrive@ -f +``` + ### OneDrive service running as root user via systemd (Red Hat Enterprise Linux, CentOS Linux) ```text systemctl enable onedrive @@ -709,7 +733,7 @@ systemctl start onedrive ``` **Note:** This will run the 'onedrive' process with a UID/GID of '0', thus, any files or folders that are created will be owned by 'root' -To see the logs run: +To see the systemd application logs run: ```text journalctl --unit=onedrive -f ``` @@ -731,7 +755,7 @@ systemctl start onedrive@.service systemctl status onedrive@.service ``` -To see the logs run: +To see the systemd application logs run: ```text journalctl --unit=onedrive@ -f ``` @@ -751,7 +775,7 @@ systemctl --user enable onedrive systemctl --user start onedrive ``` -To see the logs run: +To see the systemd application logs run: ```text journalctl --user-unit=onedrive -f ``` From fd04d632d6f6ee8a909c02b8c322b39320ce2d50 Mon Sep 17 00:00:00 2001 From: abraunegg Date: Sat, 27 Mar 2021 07:28:40 +1100 Subject: [PATCH 14/19] Fix application crash when unable to rename folder structure due to unhandled file-system issue (#1366) * Fix application crash when unable to rename folder structure due to unhandled file-system issue --- src/sync.d | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/src/sync.d b/src/sync.d index 0569edf7..dd8b9673 100644 --- a/src/sync.d +++ b/src/sync.d @@ -2523,7 +2523,14 @@ final class SyncEngine safeRename(newPath); } } - rename(oldPath, newPath); + // try and rename path, catch exception + try { + log.vdebug("Calling rename(oldPath, newPath)"); + rename(oldPath, newPath); + } catch (FileException e) { + // display the error message + displayFileSystemErrorMessage(e.msg, getFunctionName!({})); + } } // handle changed content and mtime // HACK: use mtime+hash instead of cTag because of https://github.com/OneDrive/onedrive-api-docs/issues/765 From 5c8083001a38a45137ac19794e0b9e60d79a9cb9 Mon Sep 17 00:00:00 2001 From: abraunegg Date: Sat, 27 Mar 2021 07:29:08 +1100 Subject: [PATCH 15/19] Update logging output to handle directory entries only (#1364) * When in --monitor mode, avoid outputting misleading logging when the new or modified item is a file, not a directory --- src/sync.d | 15 +++++++++++++-- 1 file changed, 13 insertions(+), 2 deletions(-) diff --git a/src/sync.d b/src/sync.d index dd8b9673..30fd3e02 100644 --- a/src/sync.d +++ b/src/sync.d @@ -2915,7 +2915,12 @@ final class SyncEngine } // scan for changes in the path provided - log.log("Uploading differences of ", logPath); + if (isDir(path)) { + // if this path is a directory, output this message. + // if a file, potentially leads to confusion as to what the client is actually doing + log.log("Uploading differences of ", logPath); + } + Item item; // For each unique OneDrive driveID we know about foreach (driveId; driveIDsArray) { @@ -2935,7 +2940,13 @@ final class SyncEngine } } - log.log("Uploading new items of ", logPath); + // scan for changes in the path provided + if (isDir(path)) { + // if this path is a directory, output this message. + // if a file, potentially leads to confusion as to what the client is actually doing + log.log("Uploading new items of ", logPath); + } + // Filesystem walk to find new files not uploaded uploadNewItems(path); // clean up idsToDelete only if --dry-run is set From b1b814f10a8d0f3ed1d62683019031ee29e8831c Mon Sep 17 00:00:00 2001 From: abraunegg Date: Sat, 27 Mar 2021 07:31:03 +1100 Subject: [PATCH 16/19] Fix uploading documents to Shared Business Folders when shared folder exists on a SharePoint site (#1352) * Fix uploading documents to Shared Business Folders when shared folder exists on a SharePoint site due to Microsoft Sharepoint 'enrichment' of files See: https://github.com/OneDrive/onedrive-api-docs/issues/935 for further details --- src/sync.d | 458 +++++++++++++++++++++++++++++++---------------------- 1 file changed, 272 insertions(+), 186 deletions(-) diff --git a/src/sync.d b/src/sync.d index 30fd3e02..e2150f7b 100644 --- a/src/sync.d +++ b/src/sync.d @@ -3508,10 +3508,18 @@ final class SyncEngine itemdb.deleteById(item.driveId, item.id); return; } else { - // For logging consistency - writeln(""); - // normal session upload - response = session.upload(path, item.driveId, item.parentId, baseName(path), item.eTag); + if ((!syncBusinessFolders) || (item.driveId == defaultDriveId)) { + // For logging consistency + writeln(""); + // If we are not syncing Shared Business Folders, or this change is going to the 'users' default drive, handle normally + // Perform a normal session upload + response = session.upload(path, item.driveId, item.parentId, baseName(path), item.eTag); + } else { + // If we are uploading to a shared business folder, there are a couple of corner cases here: + // 1. Shared Folder is a 'users' folder + // 2. Shared Folder is a 'SharePoint Library' folder, meaning we get hit by this stupidity: https://github.com/OneDrive/onedrive-api-docs/issues/935 + response = handleSharePointMetadataAdditionBug(item, path); + } } } catch (OneDriveException e) { if (e.httpStatusCode == 401) { @@ -3545,12 +3553,17 @@ final class SyncEngine uploadFailed = true; return; } - // upload done without error - writeln("done."); - - // As the session.upload includes the last modified time, save the response - // Is the response a valid JSON object - validation checking done in saveItem - saveItem(response); + // Did the upload fail? + if (!uploadFailed){ + // upload done without error or failure + writeln("done."); + // As the session.upload includes the last modified time, save the response + // Is the response a valid JSON object - validation checking done in saveItem + saveItem(response); + } else { + // uploadFailed, return + return; + } } // OneDrive documentLibrary @@ -3568,59 +3581,19 @@ final class SyncEngine itemdb.deleteById(item.driveId, item.id); return; } else { - // Handle certain file types differently - if ((extension(path) == ".txt") || (extension(path) == ".csv")) { - // .txt and .csv are unaffected by https://github.com/OneDrive/onedrive-api-docs/issues/935 - // For logging consistency - writeln(""); - try { - response = session.upload(path, item.driveId, item.parentId, baseName(path), item.eTag); - } catch (OneDriveException e) { - if (e.httpStatusCode == 401) { - // OneDrive returned a 'HTTP/1.1 401 Unauthorized Error' - file failed to be uploaded - writeln("skipped."); - log.vlog("OneDrive returned a 'HTTP 401 - Unauthorized' - gracefully handling error"); - uploadFailed = true; - return; - } - // Resolve https://github.com/abraunegg/onedrive/issues/36 - if ((e.httpStatusCode == 409) || (e.httpStatusCode == 423)) { - // The file is currently checked out or locked for editing by another user - // We cant upload this file at this time - writeln("skipped."); - log.fileOnly("Uploading modified file ", path, " ... skipped."); - writeln("", path, " is currently checked out or locked for editing by another user."); - log.fileOnly(path, " is currently checked out or locked for editing by another user."); - uploadFailed = true; - return; - } else { - // display what the error is - writeln("skipped."); - displayOneDriveErrorMessage(e.msg, getFunctionName!({})); - uploadFailed = true; - return; - } - } catch (FileException e) { - // display the error message - writeln("skipped."); - displayFileSystemErrorMessage(e.msg, getFunctionName!({})); - uploadFailed = true; - return; - } - // upload done without error + // Due to https://github.com/OneDrive/onedrive-api-docs/issues/935 Microsoft modifies all PDF, MS Office & HTML files with added XML content. It is a 'feature' of SharePoint. + // This means, as a session upload, on 'completion' the file is 'moved' and generates a 404 ...... + response = handleSharePointMetadataAdditionBug(item, path); + + // Did the upload fail? + if (!uploadFailed){ + // upload done without error or failure writeln("done."); // As the session.upload includes the last modified time, save the response // Is the response a valid JSON object - validation checking done in saveItem saveItem(response); - } else { - // Due to https://github.com/OneDrive/onedrive-api-docs/issues/935 Microsoft modifies all PDF, MS Office & HTML files with added XML content. It is a 'feature' of SharePoint. - // This means, as a session upload, on 'completion' the file is 'moved' and generates a 404 ...... - writeln("skipped."); - log.fileOnly("Uploading modified file ", path, " ... skipped."); - log.vlog("Skip Reason: Microsoft Sharepoint 'enrichment' after upload issue"); - log.vlog("See: https://github.com/OneDrive/onedrive-api-docs/issues/935 for further details"); - // Delete record from the local database - file will be uploaded as a new file - itemdb.deleteById(item.driveId, item.id); + } else { + // uploadFailed, return return; } } @@ -3735,6 +3708,68 @@ final class SyncEngine } } } + + private JSONValue handleSharePointMetadataAdditionBug(const ref Item item, const(string) path) + { + // Explicit function for handling https://github.com/OneDrive/onedrive-api-docs/issues/935 + JSONValue response; + // Handle certain file types differently + if ((extension(path) == ".txt") || (extension(path) == ".csv")) { + // .txt and .csv are unaffected by https://github.com/OneDrive/onedrive-api-docs/issues/935 + // For logging consistency + writeln(""); + try { + response = session.upload(path, item.driveId, item.parentId, baseName(path), item.eTag); + } catch (OneDriveException e) { + if (e.httpStatusCode == 401) { + // OneDrive returned a 'HTTP/1.1 401 Unauthorized Error' - file failed to be uploaded + writeln("skipped."); + log.vlog("OneDrive returned a 'HTTP 401 - Unauthorized' - gracefully handling error"); + uploadFailed = true; + return response; + } + // Resolve https://github.com/abraunegg/onedrive/issues/36 + if ((e.httpStatusCode == 409) || (e.httpStatusCode == 423)) { + // The file is currently checked out or locked for editing by another user + // We cant upload this file at this time + writeln("skipped."); + log.fileOnly("Uploading modified file ", path, " ... skipped."); + writeln("", path, " is currently checked out or locked for editing by another user."); + log.fileOnly(path, " is currently checked out or locked for editing by another user."); + uploadFailed = true; + return response; + } else { + // display what the error is + writeln("skipped."); + displayOneDriveErrorMessage(e.msg, getFunctionName!({})); + uploadFailed = true; + return response; + } + } catch (FileException e) { + // display the error message + writeln("skipped."); + displayFileSystemErrorMessage(e.msg, getFunctionName!({})); + uploadFailed = true; + return response; + } + // upload done without error + writeln("done."); + } else { + // Due to https://github.com/OneDrive/onedrive-api-docs/issues/935 Microsoft modifies all PDF, MS Office & HTML files with added XML content. It is a 'feature' of SharePoint. + // This means, as a session upload, on 'completion' the file is 'moved' and generates a 404 ...... + writeln("skipped."); + log.fileOnly("Uploading modified file ", path, " ... skipped."); + log.vlog("Skip Reason: Microsoft Sharepoint 'enrichment' after upload issue"); + log.vlog("See: https://github.com/OneDrive/onedrive-api-docs/issues/935 for further details"); + // Delete record from the local database - file will be uploaded as a new file + itemdb.deleteById(item.driveId, item.id); + uploadFailed = true; + return response; + } + + // return a JSON response so that it can be used and saved + return response; + } // upload new items to OneDrive private void uploadNewItems(const(string) path) @@ -4554,9 +4589,20 @@ final class SyncEngine // This has been seen with PNG / JPG files mainly, which then contributes to generating a 412 error when we attempt to update the metadata // Validate here that the file uploaded, at least in size, matches in the response to what the size is on disk if (thisFileSize != uploadFileSize){ - if(disableUploadValidation){ - // Print a warning message + // Upload size did not match local size + // There are 2 scenarios where this happens: + // 1. Failed Transfer + // 2. Upload file is going to a SharePoint Site, where Microsoft enriches the file with additional metadata with no way to disable + // For this client: + // - If a SharePoint Library, disableUploadValidation gets flagged as True + // - If we are syncing a business shared folder, this folder could reside on a Users Path (there should be no upload issue) or SharePoint (upload issue) + if ((disableUploadValidation)|| (syncBusinessFolders && (parent.driveId != defaultDriveId))){ + // Print a warning message - should only be triggered if: + // - disableUploadValidation gets flagged (documentLibrary account type) + // - syncBusinessFolders is being used & parent.driveId != defaultDriveId log.log("WARNING: Uploaded file size does not match local file - skipping upload validation"); + log.vlog("WARNING: Due to Microsoft Sharepoint 'enrichment' of files, this file is now technically different to your local copy"); + log.vlog("See: https://github.com/OneDrive/onedrive-api-docs/issues/935 for further details"); } else { // OK .. the uploaded file does not match and we did not disable this validation log.log("Uploaded file size does not match local file - upload failure - retrying"); @@ -4823,53 +4869,71 @@ final class SyncEngine } else { // OneDrive Business account modified file upload handling if (accountType == "business"){ - // OneDrive Business Account - always use a session to upload - writeln(""); - try { - response = session.upload(path, parent.driveId, parent.id, baseName(path), fileDetailsFromOneDrive["eTag"].str); - } catch (OneDriveException e) { - log.vdebug("response = session.upload(path, parent.driveId, parent.id, baseName(path), fileDetailsFromOneDrive['eTag'].str); generated a OneDriveException"); - if (e.httpStatusCode == 401) { - // OneDrive returned a 'HTTP/1.1 401 Unauthorized Error' - file failed to be uploaded + // OneDrive Business Account + if ((!syncBusinessFolders) || (parent.driveId == defaultDriveId)) { + // If we are not syncing Shared Business Folders, or this change is going to the 'users' default drive, handle normally + // For logging consistency + writeln(""); + try { + response = session.upload(path, parent.driveId, parent.id, baseName(path), fileDetailsFromOneDrive["eTag"].str); + } catch (OneDriveException e) { + log.vdebug("response = session.upload(path, parent.driveId, parent.id, baseName(path), fileDetailsFromOneDrive['eTag'].str); generated a OneDriveException"); + if (e.httpStatusCode == 401) { + // OneDrive returned a 'HTTP/1.1 401 Unauthorized Error' - file failed to be uploaded + writeln("skipped."); + log.vlog("OneDrive returned a 'HTTP 401 - Unauthorized' - gracefully handling error"); + uploadFailed = true; + return; + } + if (e.httpStatusCode == 429) { + // HTTP request returned status code 429 (Too Many Requests). We need to leverage the response Retry-After HTTP header to ensure minimum delay until the throttle is removed. + handleOneDriveThrottleRequest(); + // Retry original request by calling function again to avoid replicating any further error handling + log.vdebug("Retrying original request that generated the OneDrive HTTP 429 Response Code (Too Many Requests) - calling uploadNewFile(path);"); + uploadNewFile(path); + // return back to original call + return; + } + if (e.httpStatusCode == 504) { + // HTTP request returned status code 504 (Gateway Timeout) + log.log("OneDrive returned a 'HTTP 504 - Gateway Timeout' - retrying upload request"); + // Retry original request by calling function again to avoid replicating any further error handling + uploadNewFile(path); + // return back to original call + return; + } else { + // error uploading file + // display what the error is + writeln("skipped."); + displayOneDriveErrorMessage(e.msg, getFunctionName!({})); + uploadFailed = true; + return; + } + } catch (FileException e) { + // display the error message writeln("skipped."); - log.vlog("OneDrive returned a 'HTTP 401 - Unauthorized' - gracefully handling error"); + displayFileSystemErrorMessage(e.msg, getFunctionName!({})); uploadFailed = true; return; } - if (e.httpStatusCode == 429) { - // HTTP request returned status code 429 (Too Many Requests). We need to leverage the response Retry-After HTTP header to ensure minimum delay until the throttle is removed. - handleOneDriveThrottleRequest(); - // Retry original request by calling function again to avoid replicating any further error handling - log.vdebug("Retrying original request that generated the OneDrive HTTP 429 Response Code (Too Many Requests) - calling uploadNewFile(path);"); - uploadNewFile(path); - // return back to original call - return; - } - if (e.httpStatusCode == 504) { - // HTTP request returned status code 504 (Gateway Timeout) - log.log("OneDrive returned a 'HTTP 504 - Gateway Timeout' - retrying upload request"); - // Retry original request by calling function again to avoid replicating any further error handling - uploadNewFile(path); - // return back to original call - return; + // upload complete + writeln("done."); + saveItem(response); + } else { + // If we are uploading to a shared business folder, there are a couple of corner cases here: + // 1. Shared Folder is a 'users' folder + // 2. Shared Folder is a 'SharePoint Library' folder, meaning we get hit by this stupidity: https://github.com/OneDrive/onedrive-api-docs/issues/935 + + // Need try{} & catch (OneDriveException e) { & catch (FileException e) { handler for this query + response = handleSharePointMetadataAdditionBugReplaceFile(fileDetailsFromOneDrive, parent, path); + if (!uploadFailed){ + // Is the response a valid JSON object - validation checking done in saveItem + saveItem(response); } else { - // error uploading file - // display what the error is - writeln("skipped."); - displayOneDriveErrorMessage(e.msg, getFunctionName!({})); - uploadFailed = true; + // uploadFailed, return return; } - } catch (FileException e) { - // display the error message - writeln("skipped."); - displayFileSystemErrorMessage(e.msg, getFunctionName!({})); - uploadFailed = true; - return; } - // upload complete - writeln("done."); - saveItem(response); } // OneDrive SharePoint account modified file upload handling @@ -4878,93 +4942,15 @@ final class SyncEngine // as if too large, the following error will be generated by OneDrive: // HTTP request returned status code 413 (Request Entity Too Large) // We also cant use a session to upload the file, we have to use simpleUploadReplace - - // Calculate existing hash for this file - string existingFileHash = computeQuickXorHash(path); - - if (getSize(path) <= thresholdFileSize) { - // Upload file via simpleUploadReplace as below threshold size - try { - response = onedrive.simpleUploadReplace(path, fileDetailsFromOneDrive["parentReference"]["driveId"].str, fileDetailsFromOneDrive["id"].str, fileDetailsFromOneDrive["eTag"].str); - } catch (OneDriveException e) { - if (e.httpStatusCode == 401) { - // OneDrive returned a 'HTTP/1.1 401 Unauthorized Error' - file failed to be uploaded - writeln("skipped."); - log.vlog("OneDrive returned a 'HTTP 401 - Unauthorized' - gracefully handling error"); - uploadFailed = true; - return; - } else { - // display what the error is - writeln("skipped."); - displayOneDriveErrorMessage(e.msg, getFunctionName!({})); - uploadFailed = true; - return; - } - } catch (FileException e) { - // display the error message - writeln("skipped."); - displayFileSystemErrorMessage(e.msg, getFunctionName!({})); - uploadFailed = true; - return; - } + + // Need try{} & catch (OneDriveException e) { & catch (FileException e) { handler for this query + response = handleSharePointMetadataAdditionBugReplaceFile(fileDetailsFromOneDrive, parent, path); + if (!uploadFailed){ + // Is the response a valid JSON object - validation checking done in saveItem + saveItem(response); } else { - // Have to upload via a session, however we have to delete the file first otherwise this will generate a 404 error post session upload - // Remove the existing file - onedrive.deleteById(fileDetailsFromOneDrive["parentReference"]["driveId"].str, fileDetailsFromOneDrive["id"].str, fileDetailsFromOneDrive["eTag"].str); - // Upload as a session, as a new file - writeln(""); - try { - response = session.upload(path, parent.driveId, parent.id, baseName(path)); - } catch (OneDriveException e) { - if (e.httpStatusCode == 401) { - // OneDrive returned a 'HTTP/1.1 401 Unauthorized Error' - file failed to be uploaded - writeln("skipped."); - log.vlog("OneDrive returned a 'HTTP 401 - Unauthorized' - gracefully handling error"); - uploadFailed = true; - return; - } else { - // display what the error is - writeln("skipped."); - displayOneDriveErrorMessage(e.msg, getFunctionName!({})); - uploadFailed = true; - return; - } - } catch (FileException e) { - // display the error message - writeln("skipped."); - displayFileSystemErrorMessage(e.msg, getFunctionName!({})); - uploadFailed = true; - return; - } - } - writeln(" done."); - // Is the response a valid JSON object - validation checking done in saveItem - saveItem(response); - - // Due to https://github.com/OneDrive/onedrive-api-docs/issues/935 Microsoft modifies all PDF, MS Office & HTML files with added XML content. It is a 'feature' of SharePoint. - // So - now the 'local' and 'remote' file is technically DIFFERENT ... thanks Microsoft .. NO way to disable this stupidity - string uploadNewFileHash; - if (hasQuickXorHash(response)) { - // use the response json hash detail to compare - uploadNewFileHash = response["file"]["hashes"]["quickXorHash"].str; - } - - if (existingFileHash != uploadNewFileHash) { - // file was modified by Microsoft post upload to SharePoint site - log.vdebug("Existing Local File Hash: ", existingFileHash); - log.vdebug("New Remote File Hash: ", uploadNewFileHash); - - if(!uploadOnly){ - // Download the Microsoft 'modified' file so 'local' is now in sync - log.vlog("Due to Microsoft Sharepoint 'enrichment' of files, downloading 'enriched' file to ensure local file is in-sync"); - log.vlog("See: https://github.com/OneDrive/onedrive-api-docs/issues/935 for further details"); - auto fileSize = response["size"].integer; - onedrive.downloadById(response["parentReference"]["driveId"].str, response["id"].str, path, fileSize); - } else { - // we are not downloading a file, warn that file differences will exist - log.vlog("WARNING: Due to Microsoft Sharepoint 'enrichment' of files, this file is now technically different to your local copy"); - log.vlog("See: https://github.com/OneDrive/onedrive-api-docs/issues/935 for further details"); - } + // uploadFailed, return + return; } } } @@ -5041,6 +5027,106 @@ final class SyncEngine } } + private JSONValue handleSharePointMetadataAdditionBugReplaceFile(JSONValue fileDetailsFromOneDrive, const ref Item parent, const(string) path) + { + // Explicit function for handling https://github.com/OneDrive/onedrive-api-docs/issues/935 + // Replace existing file + JSONValue response; + + // Depending on the file size, this will depend on how best to handle the modified local file + // as if too large, the following error will be generated by OneDrive: + // HTTP request returned status code 413 (Request Entity Too Large) + // We also cant use a session to upload the file, we have to use simpleUploadReplace + + // Calculate existing hash for this file + string existingFileHash = computeQuickXorHash(path); + + if (getSize(path) <= thresholdFileSize) { + // Upload file via simpleUploadReplace as below threshold size + try { + response = onedrive.simpleUploadReplace(path, fileDetailsFromOneDrive["parentReference"]["driveId"].str, fileDetailsFromOneDrive["id"].str, fileDetailsFromOneDrive["eTag"].str); + } catch (OneDriveException e) { + if (e.httpStatusCode == 401) { + // OneDrive returned a 'HTTP/1.1 401 Unauthorized Error' - file failed to be uploaded + writeln("skipped."); + log.vlog("OneDrive returned a 'HTTP 401 - Unauthorized' - gracefully handling error"); + uploadFailed = true; + return response; + } else { + // display what the error is + writeln("skipped."); + displayOneDriveErrorMessage(e.msg, getFunctionName!({})); + uploadFailed = true; + return response; + } + } catch (FileException e) { + // display the error message + writeln("skipped."); + displayFileSystemErrorMessage(e.msg, getFunctionName!({})); + uploadFailed = true; + return response; + } + } else { + // Have to upload via a session, however we have to delete the file first otherwise this will generate a 404 error post session upload + // Remove the existing file + onedrive.deleteById(fileDetailsFromOneDrive["parentReference"]["driveId"].str, fileDetailsFromOneDrive["id"].str, fileDetailsFromOneDrive["eTag"].str); + // Upload as a session, as a new file + writeln(""); + try { + response = session.upload(path, parent.driveId, parent.id, baseName(path)); + } catch (OneDriveException e) { + if (e.httpStatusCode == 401) { + // OneDrive returned a 'HTTP/1.1 401 Unauthorized Error' - file failed to be uploaded + writeln("skipped."); + log.vlog("OneDrive returned a 'HTTP 401 - Unauthorized' - gracefully handling error"); + uploadFailed = true; + return response; + } else { + // display what the error is + writeln("skipped."); + displayOneDriveErrorMessage(e.msg, getFunctionName!({})); + uploadFailed = true; + return response; + } + } catch (FileException e) { + // display the error message + writeln("skipped."); + displayFileSystemErrorMessage(e.msg, getFunctionName!({})); + uploadFailed = true; + return response; + } + } + writeln("done."); + // Due to https://github.com/OneDrive/onedrive-api-docs/issues/935 Microsoft modifies all PDF, MS Office & HTML files with added XML content. It is a 'feature' of SharePoint. + // So - now the 'local' and 'remote' file is technically DIFFERENT ... thanks Microsoft .. NO way to disable this stupidity + string uploadNewFileHash; + if (hasQuickXorHash(response)) { + // use the response json hash detail to compare + uploadNewFileHash = response["file"]["hashes"]["quickXorHash"].str; + } + + if (existingFileHash != uploadNewFileHash) { + // file was modified by Microsoft post upload to SharePoint site + log.vdebug("Existing Local File Hash: ", existingFileHash); + log.vdebug("New Remote File Hash: ", uploadNewFileHash); + + if(!uploadOnly){ + // Download the Microsoft 'modified' file so 'local' is now in sync + log.vlog("Due to Microsoft Sharepoint 'enrichment' of files, downloading 'enriched' file to ensure local file is in-sync"); + log.vlog("See: https://github.com/OneDrive/onedrive-api-docs/issues/935 for further details"); + auto fileSize = response["size"].integer; + onedrive.downloadById(response["parentReference"]["driveId"].str, response["id"].str, path, fileSize); + } else { + // we are not downloading a file, warn that file differences will exist + log.vlog("WARNING: Due to Microsoft Sharepoint 'enrichment' of files, this file is now technically different to your local copy"); + log.vlog("See: https://github.com/OneDrive/onedrive-api-docs/issues/935 for further details"); + } + } + + // return a JSON response so that it can be used and saved + return response; + } + // delete an item on OneDrive private void uploadDeleteItem(Item item, const(string) path) { From 1203aebeb7fccea895450fa3a5e58e49a3f4081d Mon Sep 17 00:00:00 2001 From: abraunegg Date: Tue, 6 Apr 2021 06:43:19 +1000 Subject: [PATCH 17/19] Fix file kept in database when using --no-remote-delete & --remove-source-files (#1383) * Fix file kept in database when using --no-remote-delete & --remove-source-files --- src/sync.d | 38 +++++++++++++++++++++----------------- 1 file changed, 21 insertions(+), 17 deletions(-) diff --git a/src/sync.d b/src/sync.d index e2150f7b..0879f0db 100644 --- a/src/sync.d +++ b/src/sync.d @@ -4011,18 +4011,15 @@ final class SyncEngine if (!uploadFailed) { // Upload did not fail // Issue #763 - Delete local files after sync handling - // are we in an --upload-only scenario? - if (uploadOnly) { - // are we in a delete local file after upload? - if (localDeleteAfterUpload) { - // Log that we are deleting a local item - log.log("Removing local file as --upload-only & --remove-source-files configured"); - // are we in a --dry-run scenario? - if (!dryRun) { - // No --dry-run ... process local file delete - log.vdebug("Removing local file: ", path); - safeRemove(path); - } + // are we in an --upload-only & --remove-source-files scenario? + if ((uploadOnly) && (localDeleteAfterUpload)) { + // Log that we are deleting a local item + log.log("Removing local file as --upload-only & --remove-source-files configured"); + // are we in a --dry-run scenario? + if (!dryRun) { + // No --dry-run ... process local file delete + log.vdebug("Removing local file: ", path); + safeRemove(path); } } } @@ -5301,11 +5298,18 @@ final class SyncEngine if (jsonItem.type() == JSONType.object){ // Check if the response JSON has an 'id', otherwise makeItem() fails with 'Key not found: id' if (hasId(jsonItem)) { - // Takes a JSON input and formats to an item which can be used by the database - Item item = makeItem(jsonItem); - // Add to the local database - log.vdebug("Adding to database: ", item); - itemdb.upsert(item); + // Are we in a --upload-only & --remove-source-files scenario? + // We do not want to add the item to the database in this situation as there is no local reference to the file post file deletion + if ((uploadOnly) && (localDeleteAfterUpload)) { + // Log that we are deleting a local item + log.vdebug("Skipping adding to database as --upload-only & --remove-source-files configured"); + } else { + // Takes a JSON input and formats to an item which can be used by the database + Item item = makeItem(jsonItem); + // Add to the local database + log.vdebug("Adding to database: ", item); + itemdb.upsert(item); + } } else { // log error log.error("ERROR: OneDrive response missing required 'id' element"); From 94cc6ee3d63d6071171fb430faa5c37adebe6202 Mon Sep 17 00:00:00 2001 From: abraunegg Date: Tue, 6 Apr 2021 20:34:03 +1000 Subject: [PATCH 18/19] Revert #1343 as OneDrive API is now fixed in relation to handling '%' within filenames (#1386) * Revert #1343 as OneDrive API is now fixed in relation to handling '%' within filenames --- src/util.d | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/util.d b/src/util.d index db5f237f..cd20af0e 100644 --- a/src/util.d +++ b/src/util.d @@ -205,7 +205,7 @@ bool isValidName(string path) // Leading whitespace and trailing whitespace/dot `^\s.*|^.*[\s\.]$|` ~ // Invalid characters - `.*[<>:"\|\?*/\\%].*|` ~ + `.*[<>:"\|\?*/\\].*|` ~ // Reserved device name and trailing .~ `(?:^CON|^PRN|^AUX|^NUL|^COM[0-9]|^LPT[0-9])(?:[.].+)?$` ); From 6c64bec287deb01bd19331ff117f34e76f06e9d4 Mon Sep 17 00:00:00 2001 From: abraunegg Date: Wed, 7 Apr 2021 07:07:06 +1000 Subject: [PATCH 19/19] Release files for 2.4.11 (#1387) * Release files for 2.4.11 --- CHANGELOG.md | 24 +++++++++++++++++++ configure | 20 ++++++++-------- configure.ac | 2 +- ...e-2.4.10.ebuild => onedrive-2.4.11.ebuild} | 0 contrib/spec/onedrive.spec.in | 2 +- 5 files changed, 36 insertions(+), 12 deletions(-) rename contrib/gentoo/{onedrive-2.4.10.ebuild => onedrive-2.4.11.ebuild} (100%) diff --git a/CHANGELOG.md b/CHANGELOG.md index b32fb98e..5b4ffed1 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,30 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/) and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.html). +## 2.4.11 - 2021-4-07 +### Fixed +* Fix support for '/*' regardless of location within sync_list file +* Fix 429 response handling correctly check for 'retry-after' response header and use set value +* Fix 'sync_list' path handling for sub item matching, so that items in parent are not implicitly matched when there is no wildcard present +* Fix --get-O365-drive-id to use 'nextLink' value if present when searching for specific SharePoint site names +* Fix OneDrive Business Shared Folder existing name conflict check +* Fix incorrect error message 'Item cannot be deleted from OneDrive because it was not found in the local database' when item is actually present +* Fix application crash when unable to rename folder structure due to unhandled file-system issue +* Fix uploading documents to Shared Business Folders when the shared folder exists on a SharePoint site due to Microsoft Sharepoint 'enrichment' of files +* Fix that a file record is kept in database when using --no-remote-delete & --remove-source-files + +### Added +* Added support in --get-O365-drive-id to provide the 'drive_id' for multiple 'document libraries' within a single Shared Library Site + +### Removed +* Removed the depreciated config option 'force_http_11' which was flagged as depreciated by PR #549 in v2.3.6 (June 2019) + +### Updated +* Updated error output of --get-O365-drive-id to provide more details why an error occurred if a SharePoint site lacks the details we need to perform the match +* Updated Docker build files for Raspberry Pi to dedicated armhf & aarch64 Dockerfiles +* Updated logging output when in --monitor mode, avoid outputting misleading logging when the new or modified item is a file, not a directory +* Updated documentation (various) + ## 2.4.10 - 2021-2-19 ### Fixed * Catch database assertion when item path cannot be calculated diff --git a/configure b/configure index ee8c26e4..1737b191 100755 --- a/configure +++ b/configure @@ -1,6 +1,6 @@ #! /bin/sh # Guess values for system-dependent variables and create Makefiles. -# Generated by GNU Autoconf 2.69 for onedrive v2.4.11-dev. +# Generated by GNU Autoconf 2.69 for onedrive v2.4.11. # # Report bugs to . # @@ -579,8 +579,8 @@ MAKEFLAGS= # Identity of this package. PACKAGE_NAME='onedrive' PACKAGE_TARNAME='onedrive' -PACKAGE_VERSION='v2.4.11-dev' -PACKAGE_STRING='onedrive v2.4.11-dev' +PACKAGE_VERSION='v2.4.11' +PACKAGE_STRING='onedrive v2.4.11' PACKAGE_BUGREPORT='https://github.com/abraunegg/onedrive' PACKAGE_URL='' @@ -1219,7 +1219,7 @@ if test "$ac_init_help" = "long"; then # Omit some internal or obsolete options to make the list less imposing. # This message is too long to be a string in the A/UX 3.1 sh. cat <<_ACEOF -\`configure' configures onedrive v2.4.11-dev to adapt to many kinds of systems. +\`configure' configures onedrive v2.4.11 to adapt to many kinds of systems. Usage: $0 [OPTION]... [VAR=VALUE]... @@ -1280,7 +1280,7 @@ fi if test -n "$ac_init_help"; then case $ac_init_help in - short | recursive ) echo "Configuration of onedrive v2.4.11-dev:";; + short | recursive ) echo "Configuration of onedrive v2.4.11:";; esac cat <<\_ACEOF @@ -1393,7 +1393,7 @@ fi test -n "$ac_init_help" && exit $ac_status if $ac_init_version; then cat <<\_ACEOF -onedrive configure v2.4.11-dev +onedrive configure v2.4.11 generated by GNU Autoconf 2.69 Copyright (C) 2012 Free Software Foundation, Inc. @@ -1410,7 +1410,7 @@ cat >config.log <<_ACEOF This file contains any messages produced by compilers while running configure, to aid debugging if configure makes a mistake. -It was created by onedrive $as_me v2.4.11-dev, which was +It was created by onedrive $as_me v2.4.11, which was generated by GNU Autoconf 2.69. Invocation command line was $ $0 $@ @@ -2162,7 +2162,7 @@ fi -PACKAGE_DATE="February 2021" +PACKAGE_DATE="April 2021" @@ -3159,7 +3159,7 @@ cat >>$CONFIG_STATUS <<\_ACEOF || ac_write_fail=1 # report actual input values of CONFIG_FILES etc. instead of their # values after options handling. ac_log=" -This file was extended by onedrive $as_me v2.4.11-dev, which was +This file was extended by onedrive $as_me v2.4.11, which was generated by GNU Autoconf 2.69. Invocation command line was CONFIG_FILES = $CONFIG_FILES @@ -3212,7 +3212,7 @@ _ACEOF cat >>$CONFIG_STATUS <<_ACEOF || ac_write_fail=1 ac_cs_config="`$as_echo "$ac_configure_args" | sed 's/^ //; s/[\\""\`\$]/\\\\&/g'`" ac_cs_version="\\ -onedrive config.status v2.4.11-dev +onedrive config.status v2.4.11 configured by $0, generated by GNU Autoconf 2.69, with options \\"\$ac_cs_config\\" diff --git a/configure.ac b/configure.ac index eda79e9f..dfb0ba04 100644 --- a/configure.ac +++ b/configure.ac @@ -9,7 +9,7 @@ dnl - commit the changed files (configure.ac, configure) dnl - tag the release AC_PREREQ([2.69]) -AC_INIT([onedrive],[v2.4.11-dev], [https://github.com/abraunegg/onedrive], [onedrive]) +AC_INIT([onedrive],[v2.4.11], [https://github.com/abraunegg/onedrive], [onedrive]) AC_CONFIG_SRCDIR([src/main.d]) diff --git a/contrib/gentoo/onedrive-2.4.10.ebuild b/contrib/gentoo/onedrive-2.4.11.ebuild similarity index 100% rename from contrib/gentoo/onedrive-2.4.10.ebuild rename to contrib/gentoo/onedrive-2.4.11.ebuild diff --git a/contrib/spec/onedrive.spec.in b/contrib/spec/onedrive.spec.in index ca7861ad..439430fa 100644 --- a/contrib/spec/onedrive.spec.in +++ b/contrib/spec/onedrive.spec.in @@ -6,7 +6,7 @@ %endif Name: onedrive -Version: 2.4.10 +Version: 2.4.11 Release: 1%{?dist} Summary: Microsoft OneDrive Client Group: System Environment/Network