deblan-mirror/abraunegg-onedrive
deblan-mirror/abraunegg-onedrive
1
0
Fork 0
mirror of https://github.com/abraunegg/onedrive synced 2024-06-11 02:12:17 +02:00
Code Issues Projects Releases Wiki Activity

Merge branch 'master' into fix-issue-966

Browse source
This commit is contained in:
abraunegg 2021-04-07 07:39:09 +10:00 committed by GitHub
parent a8bec4bb1c 6c64bec287
commit c4807fd25e
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
19 changed files with 729 additions and 397 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

24
CHANGELOG.md
View file

@ -2,6 +2,30 @@
The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/) The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/)
and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.html). and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.html).
## 2.4.11 - 2021-4-07
### Fixed
* Fix support for '/*' regardless of location within sync_list file
* Fix 429 response handling correctly check for 'retry-after' response header and use set value
* Fix 'sync_list' path handling for sub item matching, so that items in parent are not implicitly matched when there is no wildcard present
* Fix --get-O365-drive-id to use 'nextLink' value if present when searching for specific SharePoint site names
* Fix OneDrive Business Shared Folder existing name conflict check
* Fix incorrect error message 'Item cannot be deleted from OneDrive because it was not found in the local database' when item is actually present
* Fix application crash when unable to rename folder structure due to unhandled file-system issue
* Fix uploading documents to Shared Business Folders when the shared folder exists on a SharePoint site due to Microsoft Sharepoint 'enrichment' of files
* Fix that a file record is kept in database when using --no-remote-delete & --remove-source-files
### Added
* Added support in --get-O365-drive-id to provide the 'drive_id' for multiple 'document libraries' within a single Shared Library Site
### Removed
* Removed the depreciated config option 'force_http_11' which was flagged as depreciated by PR #549 in v2.3.6 (June 2019)
### Updated
* Updated error output of --get-O365-drive-id to provide more details why an error occurred if a SharePoint site lacks the details we need to perform the match
* Updated Docker build files for Raspberry Pi to dedicated armhf & aarch64 Dockerfiles
* Updated logging output when in --monitor mode, avoid outputting misleading logging when the new or modified item is a file, not a directory
* Updated documentation (various)
## 2.4.10 - 2021-2-19 ## 2.4.10 - 2021-2-19
### Fixed ### Fixed
* Catch database assertion when item path cannot be calculated * Catch database assertion when item path cannot be calculated

5
README.md
View file

@ -33,7 +33,10 @@ This client is a 'fork' of the [skilion](https://github.com/skilion/onedrive) cl
## Frequently Asked Questions ## Frequently Asked Questions
Refer to [Frequently Asked Questions](https://github.com/abraunegg/onedrive/wiki/Frequently-Asked-Questions) Refer to [Frequently Asked Questions](https://github.com/abraunegg/onedrive/wiki/Frequently-Asked-Questions)
## Reporting issues ## Have a question
If you have a question or need something clarified, please raise a new disscussion post [here](https://github.com/abraunegg/onedrive/discussions)
## Reporting an Issue or Bug
If you encounter any bugs you can report them here on Github. Before filing an issue be sure to: If you encounter any bugs you can report them here on Github. Before filing an issue be sure to:
1. Check the version of the application you are using `onedrive --version` and ensure that you are running either the latest [release](https://github.com/abraunegg/onedrive/releases) or built from master. 1. Check the version of the application you are using `onedrive --version` and ensure that you are running either the latest [release](https://github.com/abraunegg/onedrive/releases) or built from master.

1
config
View file

@ -18,7 +18,6 @@
# disable_notifications = "false" # disable_notifications = "false"
# disable_upload_validation = "false" # disable_upload_validation = "false"
# enable_logging = "false" # enable_logging = "false"
# force_http_11 = "false"
# force_http_2 = "false" # force_http_2 = "false"
# local_first = "false" # local_first = "false"
# no_remote_delete = "false" # no_remote_delete = "false"

20
configure vendored
View file

@ -1,6 +1,6 @@
#! /bin/sh #! /bin/sh
# Guess values for system-dependent variables and create Makefiles. # Guess values for system-dependent variables and create Makefiles.
# Generated by GNU Autoconf 2.69 for onedrive v2.4.11-dev. # Generated by GNU Autoconf 2.69 for onedrive v2.4.11.
# #
# Report bugs to <https://github.com/abraunegg/onedrive>. # Report bugs to <https://github.com/abraunegg/onedrive>.
# #
@ -579,8 +579,8 @@ MAKEFLAGS=
# Identity of this package. # Identity of this package.
PACKAGE_NAME='onedrive' PACKAGE_NAME='onedrive'
PACKAGE_TARNAME='onedrive' PACKAGE_TARNAME='onedrive'
PACKAGE_VERSION='v2.4.11-dev' PACKAGE_VERSION='v2.4.11'
PACKAGE_STRING='onedrive v2.4.11-dev' PACKAGE_STRING='onedrive v2.4.11'
PACKAGE_BUGREPORT='https://github.com/abraunegg/onedrive' PACKAGE_BUGREPORT='https://github.com/abraunegg/onedrive'
PACKAGE_URL='' PACKAGE_URL=''
@ -1219,7 +1219,7 @@ if test "$ac_init_help" = "long"; then
# Omit some internal or obsolete options to make the list less imposing. # Omit some internal or obsolete options to make the list less imposing.
# This message is too long to be a string in the A/UX 3.1 sh. # This message is too long to be a string in the A/UX 3.1 sh.
cat <<_ACEOF cat <<_ACEOF
\`configure' configures onedrive v2.4.11-dev to adapt to many kinds of systems. \`configure' configures onedrive v2.4.11 to adapt to many kinds of systems.
Usage: $0 [OPTION]... [VAR=VALUE]... Usage: $0 [OPTION]... [VAR=VALUE]...
@ -1280,7 +1280,7 @@ fi
if test -n "$ac_init_help"; then if test -n "$ac_init_help"; then
case $ac_init_help in case $ac_init_help in
short | recursive ) echo "Configuration of onedrive v2.4.11-dev:";; short | recursive ) echo "Configuration of onedrive v2.4.11:";;
esac esac
cat <<\_ACEOF cat <<\_ACEOF
@ -1393,7 +1393,7 @@ fi
test -n "$ac_init_help" && exit $ac_status test -n "$ac_init_help" && exit $ac_status
if $ac_init_version; then if $ac_init_version; then
cat <<\_ACEOF cat <<\_ACEOF
onedrive configure v2.4.11-dev onedrive configure v2.4.11
generated by GNU Autoconf 2.69 generated by GNU Autoconf 2.69
Copyright (C) 2012 Free Software Foundation, Inc. Copyright (C) 2012 Free Software Foundation, Inc.
@ -1410,7 +1410,7 @@ cat >config.log <<_ACEOF
This file contains any messages produced by compilers while This file contains any messages produced by compilers while
running configure, to aid debugging if configure makes a mistake. running configure, to aid debugging if configure makes a mistake.
It was created by onedrive $as_me v2.4.11-dev, which was It was created by onedrive $as_me v2.4.11, which was
generated by GNU Autoconf 2.69. Invocation command line was generated by GNU Autoconf 2.69. Invocation command line was
$ $0 $@ $ $0 $@
@ -2162,7 +2162,7 @@ fi
PACKAGE_DATE="February 2021" PACKAGE_DATE="April 2021"
@ -3159,7 +3159,7 @@ cat >>$CONFIG_STATUS <<\_ACEOF || ac_write_fail=1
# report actual input values of CONFIG_FILES etc. instead of their # report actual input values of CONFIG_FILES etc. instead of their
# values after options handling. # values after options handling.
ac_log=" ac_log="
This file was extended by onedrive $as_me v2.4.11-dev, which was This file was extended by onedrive $as_me v2.4.11, which was
generated by GNU Autoconf 2.69. Invocation command line was generated by GNU Autoconf 2.69. Invocation command line was
CONFIG_FILES = $CONFIG_FILES CONFIG_FILES = $CONFIG_FILES
@ -3212,7 +3212,7 @@ _ACEOF
cat >>$CONFIG_STATUS <<_ACEOF || ac_write_fail=1 cat >>$CONFIG_STATUS <<_ACEOF || ac_write_fail=1
ac_cs_config="`$as_echo "$ac_configure_args" | sed 's/^ //; s/[\\""\`\$]/\\\\&/g'`" ac_cs_config="`$as_echo "$ac_configure_args" | sed 's/^ //; s/[\\""\`\$]/\\\\&/g'`"
ac_cs_version="\\ ac_cs_version="\\
onedrive config.status v2.4.11-dev onedrive config.status v2.4.11
configured by $0, generated by GNU Autoconf 2.69, configured by $0, generated by GNU Autoconf 2.69,
with options \\"\$ac_cs_config\\" with options \\"\$ac_cs_config\\"

2
configure.ac
View file

@ -9,7 +9,7 @@ dnl - commit the changed files (configure.ac, configure)
dnl - tag the release dnl - tag the release
AC_PREREQ([2.69]) AC_PREREQ([2.69])
AC_INIT([onedrive],[v2.4.11-dev], [https://github.com/abraunegg/onedrive], [onedrive]) AC_INIT([onedrive],[v2.4.11], [https://github.com/abraunegg/onedrive], [onedrive])
AC_CONFIG_SRCDIR([src/main.d]) AC_CONFIG_SRCDIR([src/main.d])

21
contrib/docker/Dockerfile-aarch64 Normal file
View file

@ -0,0 +1,21 @@
# -*-Dockerfile-*-
FROM debian:stretch
RUN apt update && \
apt install -y build-essential curl libcurl4-openssl-dev libsqlite3-dev pkg-config wget git
RUN wget https://github.com/ldc-developers/ldc/releases/download/v1.16.0/ldc2-1.16.0-linux-aarch64.tar.xz && \
tar -xvf ldc2-1.16.0-linux-aarch64.tar.xz
COPY . /usr/src/onedrive
RUN cd /usr/src/onedrive/ && \
./configure DC=/ldc2-1.16.0-linux-aarch64/bin/ldmd2 && \
make clean && \
make && \
make install
FROM debian:stretch-slim
ENTRYPOINT ["/entrypoint.sh"]
RUN apt update && \
apt install -y gosu libcurl3 libsqlite3-0 && \
rm -rf /var/*/apt && \
mkdir -p /onedrive/conf /onedrive/data
COPY contrib/docker/entrypoint.sh /
COPY --from=0 /usr/local/bin/onedrive /usr/local/bin/

7
contrib/docker/Dockerfile-rpi
View file

@ -1,13 +1,12 @@
# -*-Dockerfile-*- # -*-Dockerfile-*-
FROM debian:stretch FROM debian:stretch
ARG ARCH armhf # or aarch64
RUN apt update && \ RUN apt update && \
apt install -y build-essential curl libcurl4-openssl-dev libsqlite3-dev pkg-config wget git apt install -y build-essential curl libcurl4-openssl-dev libsqlite3-dev pkg-config wget git
RUN wget https://github.com/ldc-developers/ldc/releases/download/v1.16.0/ldc2-1.16.0-linux-${ARCH}.tar.xz && \ RUN wget https://github.com/ldc-developers/ldc/releases/download/v1.16.0/ldc2-1.16.0-linux-armhf.tar.xz && \
tar -xvf ldc2-1.16.0-linux-${ARCH}.tar.xz tar -xvf ldc2-1.16.0-linux-armhf.tar.xz
COPY . /usr/src/onedrive COPY . /usr/src/onedrive
RUN cd /usr/src/onedrive/ && \ RUN cd /usr/src/onedrive/ && \
./configure DC=/ldc2-1.16.0-linux-${ARCH}/bin/ldmd2 && \ ./configure DC=/ldc2-1.16.0-linux-armhf/bin/ldmd2 && \
make clean && \ make clean && \
make && \ make && \
make install make install

0
contrib/gentoo/onedrive-2.4.10.ebuild → contrib/gentoo/onedrive-2.4.11.ebuild
View file

2
contrib/spec/onedrive.spec.in
View file

@ -6,7 +6,7 @@
%endif %endif
Name: onedrive Name: onedrive
Version: 2.4.10 Version: 2.4.11
Release: 1%{?dist} Release: 1%{?dist}
Summary: Microsoft OneDrive Client Summary: Microsoft OneDrive Client
Group: System Environment/Network Group: System Environment/Network

144
docs/Docker.md
View file

@ -1,56 +1,89 @@
# onedrive docker image # Run the OneDrive Client for Linux under Docker
This client can be run as a Docker container, with 3 available options for you to choose from:
1. Container based on CentOS 7 - Docker Tag: latest
2. Container based on Debian Stretch - Docker Tag: stretch
3. Container based on Alpine Linux - Docker Tag: alpine
Thats right folks onedrive is now dockerized ;) These containers offer a simple monitoring-mode service for the OneDrive Client for Linux.
The instructions below have been validated on:
* Red Hat Enterprise Linux 8.x
* Ubuntu Server 20.04
The instructions below will utilise the 'latest' tag, however this can be substituted for 'stretch' or 'alpine' if desired.
This container offers simple monitoring-mode service for 'Free Client for OneDrive on Linux'.
## Basic Setup ## Basic Setup
### 0. Install docker using your distribution platform's instructions
1. Ensure that SELinux has been disabled on your system. A reboot may be required to ensure that this is correctly disabled.
2. Install Docker as per requried for your platform
3. Obtain your normal, non-root user UID and GID by using the `id` command
4. As your normal, non-root user, ensure that you can run `docker run hello-world` *without* using `sudo`
### 0. Install docker under your own platform's instructions Once the above 4 steps are complete and you can successfully run `docker run hello-world` without sudo, only then proceed to 'Pulling and Running the Docker Image'
## Pulling and Running the Docker Image
### 1. Pull the image ### 1. Pull the image
```bash ```bash
docker pull driveone/onedrive:latest docker pull driveone/onedrive:latest
``` ```
**NOTE:** SELinux context needs to be configured or disabled for Docker, to be able to write to OneDrive host directory. **NOTE:** SELinux context needs to be configured or disabled for Docker to be able to write to OneDrive host directory.
### 2. Prepare config volume ### 2. Prepare config volume
The Docker container requries 2 Docker volumes:
* Config Volume
* Data Volume
Onedrive needs two volumes. One of them is the config volume. Create it with: Create the config volume with the following command:
```bash ```bash
docker volume create onedrive_conf docker volume create onedrive_conf
``` ```
This will create a docker volume labeled `onedrive_conf`, where all configuration of your onedrive account will be stored. You can add a custom config file and other things later. This will create a docker volume labeled `onedrive_conf`, where all configuration of your onedrive account will be stored. You can add a custom config file and other things later.
The second docker volume is for your data folder and is created in the next step. It needs the path to a folder on your filesystem that you want to keep in sync with OneDrive. Keep in mind that: The second docker volume is for your data folder and is created in the next step. This volume needs to be a path to a directory on your local filesystem, and this is where your data will be stored from OneDrive. Keep in mind that:
- The owner of your specified folder must not be root * The owner of this specified folder must not be root
* The owner of this specified folder must have permissions for its parent directory
- The owner of your specified folder must have permissions for its parent directory
### 3. First run
Onedrive needs to be authorized with your Microsoft account. This is achieved by running docker in interactive mode. Run the docker image with the two commands below and **make sure to change `onedriveDir` to the onedrive data directory on your filesystem (e.g. `"/home/abraunegg/OneDrive"`)**.
Additionally, the user id and group id should be added to remove any potential user conflicts, denoted by the environment variables `${ONEDRIVE_UID}` and `${ONEDRIVE_GID}`.
**NOTE:** Issues occur when this target folder is a mounted folder of an external system (NAS, SMB mount, USB Drive etc) as the 'mount' itself is owed by 'root'. If this is your use case, you *must* ensure your normal user can mount your desired target without having the target mounted by 'root'. If you do not fix this, your Docker container will fail to start with the following error message:
```bash ```bash
onedriveDir="${HOME}/OneDrive" ROOT level privileges prohibited!
docker run -it --name onedrive -v onedrive_conf:/onedrive/conf -v "${onedriveDir}:/onedrive/data" -e "ONEDRIVE_UID:${ONEDRIVE_UID}" -e "ONEDRIVE_GID:${ONEDRIVE_GID}" driveone/onedrive:latest
``` ```
- You will be asked to open a specific link using your web browser ### 3. First run
- Login to your Microsoft Account and give the application the permission The 'onedrive' client within the Docker container needs to be authorized with your Microsoft account. This is achieved by initially running docker in interactive mode.
- After giving the permission, you will be redirected to a blank page.
- Copy the URI of the blank page into the application.
The onedrive monitor is configured to start with your host system. If your onedrive is working as expected, you can detach from the container with Ctrl+p, Ctrl+q. Run the docker image with the commands below and make sure to change `ONEDRIVE_DATA_DIR` to the actual onedrive data directory on your filesystem that you wish to use (e.g. `"/home/abraunegg/OneDrive"`).
```bash
export ONEDRIVE_DATA_DIR="${HOME}/OneDrive"
mkdir -p ${ONEDRIVE_DATA_DIR}
docker run -it --name onedrive -v onedrive_conf:/onedrive/conf -v "${ONEDRIVE_DATA_DIR}:/onedrive/data" -e "ONEDRIVE_UID:${ONEDRIVE_UID}" -e "ONEDRIVE_GID:${ONEDRIVE_GID}" driveone/onedrive:latest
```
**NOTE:** It is also highly advisable for you to replace `${ONEDRIVE_UID}` and `${ONEDRIVE_GID}` with your actual UID and GID as specified by your `id` command output to avoid any any potential user or group conflicts.
### 4. Status, stop, and restart **Important:** The 'target' folder of `ONEDRIVE_DATA_DIR` must exist before running the Docker container, otherwise, Docker will create the target folder, and the folder will be given 'root' permissions, which then causes the Docker container to fail upon startup with the following error message:
```bash
ROOT level privileges prohibited!
```
**Example:**
```
docker run -it --name onedrive -v onedrive_conf:/onedrive/conf -v "${ONEDRIVE_DATA_DIR}:/onedrive/data" -e "ONEDRIVE_UID:1000" -e "ONEDRIVE_GID:1000" driveone/onedrive:latest
```
When the Docker container successfully starts:
* You will be asked to open a specific link using your web browser
* Login to your Microsoft Account and give the application the permission
* After giving the permission, you will be redirected to a blank page
* Copy the URI of the blank page into the application prompt to authorise the application
Once the 'onedrive' application is authorised, the client will automatically start monitoring your `ONEDRIVE_DATA_DIR` for data changes to be uploaded to OneDrive. Files stored on OneDrive will be downloaded to this location.
If the client is working as expected, you can detach from the container with Ctrl+p, Ctrl+q.
### 4. Docker Container Status, stop, and restart
Check if the monitor service is running Check if the monitor service is running
```bash ```bash
@ -75,7 +108,7 @@ Resume monitor
docker start onedrive docker start onedrive
``` ```
Remove onedrive monitor Remove onedrive Docker container
```bash ```bash
docker rm -f onedrive docker rm -f onedrive
@ -83,9 +116,8 @@ docker rm -f onedrive
## Advanced Setup ## Advanced Setup
### 5. Docker-compose ### 5. Docker-compose
Also supports docker-compose schemas > 3. Also supports docker-compose schemas > 3.
In the following example it is assumed you have a `onedriveDir` environment variable and a `onedrive_conf` volume. In the following example it is assumed you have a `ONEDRIVE_DATA_DIR` environment variable and a `onedrive_conf` volume.
However, you can also use bind mounts for the configuration folder, e.g. `export ONEDRIVE_CONF="${HOME}/OneDriveConfig"`. However, you can also use bind mounts for the configuration folder, e.g. `export ONEDRIVE_CONF="${HOME}/OneDriveConfig"`.
``` ```
@ -99,14 +131,13 @@ services:
- ONEDRIVE_GID=${PGID} - ONEDRIVE_GID=${PGID}
volumes: volumes:
- onedrive_conf:/onedrive/conf - onedrive_conf:/onedrive/conf
- ${onedriveDir}:/onedrive/data - ${ONEDRIVE_DATA_DIR}:/onedrive/data
``` ```
Note that you still have to perform step 3: First Run. Note that you still have to perform step 3: First Run.
### 6. Edit the config ### 6. Edit the config
The 'onedrive' client should run in default configuration, however you can change this default configuration by placing a custom config file in the `onedrive_conf` docker volume. First download the default config from [here](https://raw.githubusercontent.com/abraunegg/onedrive/master/config)
Onedrive should run in default configuration, however you can change your configuration by placing a custom config file in the `onedrive_conf` docker volume. First download the default config from [here](https://raw.githubusercontent.com/abraunegg/onedrive/master/config)
Then put it into your onedrive_conf volume path, which can be found with: Then put it into your onedrive_conf volume path, which can be found with:
```bash ```bash
@ -118,34 +149,31 @@ Or you can map your own config folder to the config volume. Make sure to copy al
The detailed document for the config can be found here: [Configuration](https://github.com/abraunegg/onedrive/blob/master/docs/USAGE.md#configuration) The detailed document for the config can be found here: [Configuration](https://github.com/abraunegg/onedrive/blob/master/docs/USAGE.md#configuration)
### 7. Sync multiple accounts ### 7. Sync multiple accounts
There are many ways to do this, the easiest is probably to There are many ways to do this, the easiest is probably to
1. Create a second docker config volume (replace `Work` with your desired name): `docker volume create onedrive_conf_Work` 1. Create a second docker config volume (replace `Work` with your desired name): `docker volume create onedrive_conf_Work`
2. And start a second docker monitor container (again replace `Work` with your desired name): 2. And start a second docker monitor container (again replace `Work` with your desired name):
``` ```
onedriveDirWork="/home/abraunegg/OneDriveWork" export ONEDRIVE_DATA_DIR_WORK="/home/abraunegg/OneDriveWork"
docker run -it --restart unless-stopped --name onedrive_Work -v onedrive_conf_Work:/onedrive/conf -v "${onedriveDirWork}:/onedrive/data" driveone/onedrive:latest mkdir -p ${ONEDRIVE_DATA_DIR_WORK}
docker run -it --restart unless-stopped --name onedrive_Work -v onedrive_conf_Work:/onedrive/conf -v "${ONEDRIVE_DATA_DIR_WORK}:/onedrive/data" driveone/onedrive:latest
``` ```
## Run or update with one script ## Run or update with one script
If you are experienced with docker and onedrive, you can use the following script: If you are experienced with docker and onedrive, you can use the following script:
```bash ```bash
# Update onedriveDir with correct existing OneDrive directory path # Update ONEDRIVE_DATA_DIR with correct existing OneDrive directory path
onedriveDir="${HOME}/OneDrive" ONEDRIVE_DATA_DIR="${HOME}/OneDrive"
firstRun='-d' firstRun='-d'
docker pull driveone/onedrive:latest docker pull driveone/onedrive:latest
docker inspect onedrive_conf > /dev/null || { docker volume create onedrive_conf; firstRun='-it'; } docker inspect onedrive_conf > /dev/null || { docker volume create onedrive_conf; firstRun='-it'; }
docker inspect onedrive > /dev/null && docker rm -f onedrive docker inspect onedrive > /dev/null && docker rm -f onedrive
docker run $firstRun --restart unless-stopped --name onedrive -v onedrive_conf:/onedrive/conf -v "${onedriveDir}:/onedrive/data" driveone/onedrive:latest docker run $firstRun --restart unless-stopped --name onedrive -v onedrive_conf:/onedrive/conf -v "${ONEDRIVE_DATA_DIR}:/onedrive/data" driveone/onedrive:latest
``` ```
## Environment Variables ## Environment Variables
| Variable | Purpose | Sample Value | | Variable | Purpose | Sample Value |
| ---------------- | --------------------------------------------------- |:-------------:| | ---------------- | --------------------------------------------------- |:-------------:|
| <B>ONEDRIVE_UID</B> | UserID (UID) to run as | 1000 | | <B>ONEDRIVE_UID</B> | UserID (UID) to run as | 1000 |
@ -160,24 +188,24 @@ docker run $firstRun --restart unless-stopped --name onedrive -v onedrive_conf:/
### Usage Examples ### Usage Examples
**Verbose Output:** **Verbose Output:**
```bash ```bash
docker container run -e ONEDRIVE_VERBOSE=1 -v onedrive_conf:/onedrive/conf -v "${onedriveDir}:/onedrive/data" driveone/onedrive:latest docker container run -e ONEDRIVE_VERBOSE=1 -v onedrive_conf:/onedrive/conf -v "${ONEDRIVE_DATA_DIR}:/onedrive/data" driveone/onedrive:latest
``` ```
**Debug Output:** **Debug Output:**
```bash ```bash
docker container run -e ONEDRIVE_DEBUG=1 -v onedrive_conf:/onedrive/conf -v "${onedriveDir}:/onedrive/data" driveone/onedrive:latest docker container run -e ONEDRIVE_DEBUG=1 -v onedrive_conf:/onedrive/conf -v "${ONEDRIVE_DATA_DIR}:/onedrive/data" driveone/onedrive:latest
``` ```
**Perform a --resync:** **Perform a --resync:**
```bash ```bash
docker container run -e ONEDRIVE_RESYNC=1 -v onedrive_conf:/onedrive/conf -v "${onedriveDir}:/onedrive/data" driveone/onedrive:latest docker container run -e ONEDRIVE_RESYNC=1 -v onedrive_conf:/onedrive/conf -v "${ONEDRIVE_DATA_DIR}:/onedrive/data" driveone/onedrive:latest
``` ```
**Perform a --resync and --verbose:** **Perform a --resync and --verbose:**
```bash ```bash
docker container run -e ONEDRIVE_RESYNC=1 -e ONEDRIVE_VERBOSE=1 -v onedrive_conf:/onedrive/conf -v "${onedriveDir}:/onedrive/data" driveone/onedrive:latest docker container run -e ONEDRIVE_RESYNC=1 -e ONEDRIVE_VERBOSE=1 -v onedrive_conf:/onedrive/conf -v "${ONEDRIVE_DATA_DIR}:/onedrive/data" driveone/onedrive:latest
``` ```
**Perform a --logout and re-authenticate:** **Perform a --logout and re-authenticate:**
```bash ```bash
docker container run -e ONEDRIVE_LOGOUT=1 -v onedrive_conf:/onedrive/conf -v "${onedriveDir}:/onedrive/data" driveone/onedrive:latest docker container run -e ONEDRIVE_LOGOUT=1 -v onedrive_conf:/onedrive/conf -v "${ONEDRIVE_DATA_DIR}:/onedrive/data" driveone/onedrive:latest
``` ```
## Build instructions ## Build instructions
@ -185,7 +213,7 @@ docker container run -e ONEDRIVE_LOGOUT=1 -v onedrive_conf:/onedrive/conf -v "${
* Build environment must have at least 1GB of memory & 2GB swap space * Build environment must have at least 1GB of memory & 2GB swap space
There are 2 ways to validate this requirement: There are 2 ways to validate this requirement:
* Modify the file `/etc/dphys-swapfile` and edit the `CONF_SWAPSIZE`, for example: `CONF_SWAPSIZE=2024`. A reboot is required to make this change effective. * Modify the file `/etc/dphys-swapfile` and edit the `CONF_SWAPSIZE`, for example: `CONF_SWAPSIZE=2048`. A reboot is required to make this change effective.
* Dynamically allocate a swapfile for building: * Dynamically allocate a swapfile for building:
```bash ```bash
cd /var cd /var
@ -201,7 +229,7 @@ swapon -s
free -h free -h
``` ```
### Building the Docker image ### Building a custom Docker image
You can also build your own image instead of pulling the one from [hub.docker.com](https://hub.docker.com/r/driveone/onedrive): You can also build your own image instead of pulling the one from [hub.docker.com](https://hub.docker.com/r/driveone/onedrive):
```bash ```bash
git clone https://github.com/abraunegg/onedrive git clone https://github.com/abraunegg/onedrive
@ -214,18 +242,32 @@ Dockerfile-stretch or Dockerfile-alpine. These [multi-stage builder
pattern](https://docs.docker.com/develop/develop-images/multistage-build/) pattern](https://docs.docker.com/develop/develop-images/multistage-build/)
Dockerfiles require Docker version at least 17.05. Dockerfiles require Docker version at least 17.05.
#### How to build a Docker image based on Debian Stretch #### How to build and run a custom Docker image based on Debian Stretch
``` bash ``` bash
docker build . -t local-ondrive-stretch -f contrib/docker/Dockerfile-stretch docker build . -t local-ondrive-stretch -f contrib/docker/Dockerfile-stretch
docker container run -v onedrive_conf:/onedrive/conf -v "${ONEDRIVE_DATA_DIR}:/onedrive/data" local-ondrive-stretch:latest
``` ```
#### How to build a Docker image based on Alpine Linux
#### How to build and run a custom Docker image based on Alpine Linux
``` bash ``` bash
docker build . -t local-ondrive-alpine -f contrib/docker/Dockerfile-alpine docker build . -t local-ondrive-alpine -f contrib/docker/Dockerfile-alpine
docker container run -v onedrive_conf:/onedrive/conf -v "${ONEDRIVE_DATA_DIR}:/onedrive/data" local-ondrive-alpine:latest
``` ```
#### How to build a Docker image for ARMHF (Raspberry Pi)
#### How to build and run a custom Docker image for ARMHF (Raspberry Pi)
Compatible with:
* Raspberry Pi
* Raspberry Pi 2
* Raspberry Pi Zero
* Raspberry Pi 3
* Raspberry Pi 4
``` bash ``` bash
docker build . -t local-onedrive-rpi -f contrib/docker/Dockerfile-rpi docker build . -t local-onedrive-rpi -f contrib/docker/Dockerfile-rpi
docker container run -v onedrive_conf:/onedrive/conf -v "${ONEDRIVE_DATA_DIR}:/onedrive/data" local-ondrive-rpi:latest
```
#### How to build and run a custom Docker image for AARCH64 Platforms
``` bash
docker build . -t local-onedrive-aarch64 -f contrib/docker/Dockerfile-aarch64
docker container run -v onedrive_conf:/onedrive/conf -v "${ONEDRIVE_DATA_DIR}:/onedrive/data" local-onedrive-aarch64:latest
``` ```

45
docs/INSTALL.md
View file

@ -5,6 +5,7 @@ This project has been packaged for the following Linux distributions:
| Distribution | Package Name & Package Link | &nbsp;i686&nbsp; | x86_64 | ARMHF | AARCH64 | Extra Details | | Distribution | Package Name & Package Link | &nbsp;i686&nbsp; | x86_64 | ARMHF | AARCH64 | Extra Details |
|---------------------------------|------------------------------------------------------------------------------|:----:|:------:|:-----:|:-------:|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| |---------------------------------|------------------------------------------------------------------------------|:----:|:------:|:-----:|:-------:|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Alpine Linux | [onedrive](https://pkgs.alpinelinux.org/packages?name=onedrive&branch=edge) |<img src="./images/cross.gif" alt="not_supported" width="39" height="39"/>|<img src="./images/tick.gif" alt="supported" width="39" height="39"/>|<img src="./images/cross.gif" alt="not_supported" width="39" height="39"/>|<img src="./images/tick.gif" alt="supported" width="39" height="39"/>|
| Arch Linux<br><br>Manjaro Linux | [onedrive-abraunegg](https://aur.archlinux.org/packages/onedrive-abraunegg/) |<img src="./images/tick.gif" alt="supported" width="39" height="39"/>|<img src="./images/tick.gif" alt="supported" width="39" height="39"/>|<img src="./images/tick.gif" alt="supported" width="39" height="39"/>|<img src="./images/tick.gif" alt="supported" width="39" height="39"/>| Install via: `pamac build onedrive-abraunegg` from the Arch Linux User Repository (AUR)<br><br>**Note:** If asked regarding a provider for 'd-runtime' and 'd-compiler', select 'liblphobos' and 'ldc'<br><br>**Note:** System must have at least 1GB of memory & 1GB swap space | Arch Linux<br><br>Manjaro Linux | [onedrive-abraunegg](https://aur.archlinux.org/packages/onedrive-abraunegg/) |<img src="./images/tick.gif" alt="supported" width="39" height="39"/>|<img src="./images/tick.gif" alt="supported" width="39" height="39"/>|<img src="./images/tick.gif" alt="supported" width="39" height="39"/>|<img src="./images/tick.gif" alt="supported" width="39" height="39"/>| Install via: `pamac build onedrive-abraunegg` from the Arch Linux User Repository (AUR)<br><br>**Note:** If asked regarding a provider for 'd-runtime' and 'd-compiler', select 'liblphobos' and 'ldc'<br><br>**Note:** System must have at least 1GB of memory & 1GB swap space
| Debian | [onedrive](https://packages.debian.org/search?keywords=onedrive) |<img src="./images/tick.gif" alt="supported" width="39" height="39"/>|<img src="./images/tick.gif" alt="supported" width="39" height="39"/>|<img src="./images/tick.gif" alt="supported" width="39" height="39"/>|<img src="./images/tick.gif" alt="supported" width="39" height="39"/>| | | Debian | [onedrive](https://packages.debian.org/search?keywords=onedrive) |<img src="./images/tick.gif" alt="supported" width="39" height="39"/>|<img src="./images/tick.gif" alt="supported" width="39" height="39"/>|<img src="./images/tick.gif" alt="supported" width="39" height="39"/>|<img src="./images/tick.gif" alt="supported" width="39" height="39"/>| |
| Fedora | [onedrive](https://koji.fedoraproject.org/koji/packageinfo?packageID=26044) |<img src="./images/tick.gif" alt="supported" width="39" height="39"/>|<img src="./images/tick.gif" alt="supported" width="39" height="39"/>|<img src="./images/tick.gif" alt="supported" width="39" height="39"/>|<img src="./images/tick.gif" alt="supported" width="39" height="39"/>| | | Fedora | [onedrive](https://koji.fedoraproject.org/koji/packageinfo?packageID=26044) |<img src="./images/tick.gif" alt="supported" width="39" height="39"/>|<img src="./images/tick.gif" alt="supported" width="39" height="39"/>|<img src="./images/tick.gif" alt="supported" width="39" height="39"/>|<img src="./images/tick.gif" alt="supported" width="39" height="39"/>| |
@ -236,28 +237,42 @@ sudo pacman -S libnotify
``` ```
### Dependencies: Raspbian (ARMHF) ### Dependencies: Raspbian (ARMHF)
Validated using:
* `Linux raspberrypi 5.4.79-v7+ #1373 SMP Mon Nov 23 13:22:33 GMT 2020 armv7l GNU/Linux` (2020-12-02-raspios-buster-armhf) using Raspberry Pi 2 Model B
* `Linux raspberrypi 5.4.83-v8+ #1379 SMP PREEMPT Mon Dec 14 13:15:14 GMT 2020 aarch64` (2021-01-11-raspios-buster-armhf) using Raspberry Pi 3 Model B+
**Note:** Build environment must have at least 1GB of memory & 1GB swap space. Check with `swapon`. **Note:** Build environment must have at least 1GB of memory & 1GB swap space. Check with `swapon`.
```text ```text
sudo apt-get install libcurl4-openssl-dev sudo apt install build-essential
sudo apt-get install libsqlite3-dev sudo apt install libcurl4-openssl-dev
sudo apt-get install libxml2 sudo apt install libsqlite3-dev
sudo apt-get install pkg-config sudo apt install pkg-config
wget https://github.com/ldc-developers/ldc/releases/download/v1.16.0/ldc2-1.16.0-linux-armhf.tar.xz sudo apt install git
tar -xvf ldc2-1.16.0-linux-armhf.tar.xz sudo apt install curl
wget https://github.com/ldc-developers/ldc/releases/download/v1.17.0/ldc2-1.17.0-linux-armhf.tar.xz
tar -xvf ldc2-1.17.0-linux-armhf.tar.xz
``` ```
For notifications the following is also necessary: For notifications the following is also necessary:
```text ```text
sudo apt install libnotify-dev sudo apt install libnotify-dev
``` ```
### Dependencies: Debian (ARM64) ### Dependencies: Ubuntu 20.x / Debian 10 (ARM64)
Validated using:
* `Ubuntu 20.04.2 LTS (GNU/Linux 5.4.0-1028-raspi aarch64)` (ubuntu-20.04.2-preinstalled-server-arm64+raspi) using Raspberry Pi 3 Model B+
**Note:** Build environment must have at least 1GB of memory & 1GB swap space. Check with `swapon`.
```text ```text
sudo apt-get install libcurl4-openssl-dev sudo apt install build-essential
sudo apt-get install libsqlite3-dev sudo apt install libcurl4-openssl-dev
sudo apt-get install libxml2 sudo apt install libsqlite3-dev
sudo apt-get install pkg-config sudo apt install pkg-config
wget https://github.com/ldc-developers/ldc/releases/download/v1.16.0/ldc2-1.16.0-linux-aarch64.tar.xz sudo apt install git
tar -xvf ldc2-1.16.0-linux-aarch64.tar.xz sudo apt install curl
wget https://github.com/ldc-developers/ldc/releases/download/v1.25.1/ldc2-1.25.1-linux-aarch64.tar.xz
tar -xvf ldc2-1.25.1-linux-aarch64.tar.xz
``` ```
For notifications the following is also necessary: For notifications the following is also necessary:
```text ```text
@ -337,7 +352,7 @@ as far as possible automatically, but can be overridden by passing
```text ```text
git clone https://github.com/abraunegg/onedrive.git git clone https://github.com/abraunegg/onedrive.git
cd onedrive cd onedrive
./configure DC=~/ldc2-1.16.0-linux-armhf/bin/ldmd2 ./configure DC=~/ldc2-1.17.0-linux-armhf/bin/ldmd2
make clean; make make clean; make
sudo make install sudo make install
``` ```
@ -347,7 +362,7 @@ sudo make install
```text ```text
git clone https://github.com/abraunegg/onedrive.git git clone https://github.com/abraunegg/onedrive.git
cd onedrive cd onedrive
./configure DC=~/ldc2-1.16.0-linux-aarch64/bin/ldmd2 ./configure DC=~/ldc2-1.25.1-linux-aarch64/bin/ldmd2
make clean; make make clean; make
sudo make install sudo make install
``` ```

31
docs/SharePoint-Shared-Libraries.md
View file

@ -14,18 +14,38 @@ Syncing a OneDrive SharePoint library requires additional configuration for your
## Query that shared library name using the client to obtain the required configuration details ## Query that shared library name using the client to obtain the required configuration details
2. Run the following command using the 'onedrive' client 2. Run the following command using the 'onedrive' client
```text ```text
onedrive --get-O365-drive-id '<your library name>' onedrive --get-O365-drive-id '<your site name to search>'
``` ```
This will return something similar to the following: This will return something similar to the following:
```text ```text
Configuration file successfully loaded Configuration file successfully loaded
Configuring Global Azure AD Endpoints Configuring Global Azure AD Endpoints
Initializing the Synchronization Engine ... Initializing the Synchronization Engine ...
Office 365 Library Name Query: <your library name> Office 365 Library Name Query: <your site name to search>
SiteName: <your library name> -----------------------------------------------
drive_id: b!6H_y8B...xU5 Site Name: <your site name>
URL: <your site URL> Library Name: <your library name>
drive_id: b!6H_y8B...xU5
Library URL: <your library URL>
-----------------------------------------------
``` ```
If there are no matches to the site you are attempting to search, the following will be displayed:
```text
Configuration file successfully loaded
Configuring Global Azure AD Endpoints
Initializing the Synchronization Engine ...
Office 365 Library Name Query: blah
ERROR: The requested SharePoint site could not be found. Please check it's name and your permissions to access the site.
The following SharePoint site names were returned:
* <site name 1>
* <site name 2>
...
* <site name X>
```
This list of site names can be used as a basis to search for the correct site for which you are searching
## Configure the client's config file with the required 'drive_id' & 'sync_dir' options ## Configure the client's config file with the required 'drive_id' & 'sync_dir' options
3. Create a new local folder to store the SharePoint Library data in 3. Create a new local folder to store the SharePoint Library data in
@ -54,6 +74,5 @@ The OneDrive client will now be configured to sync this SharePoint shared librar
## Sync the SharePoint Library as required ## Sync the SharePoint Library as required
6. Sync the SharePoint Library to your system with either `--synchronize` or `--monitor` operations 6. Sync the SharePoint Library to your system with either `--synchronize` or `--monitor` operations
# How to configure multiple OneDrive SharePoint Shared Library sync # How to configure multiple OneDrive SharePoint Shared Library sync
Refer to [./advanced-usage.md](advanced-usage.md) for configuration assistance. Refer to [./advanced-usage.md](advanced-usage.md) for configuration assistance.

33
docs/USAGE.md
View file

@ -326,7 +326,6 @@ See the [config](https://raw.githubusercontent.com/abraunegg/onedrive/master/con
# disable_notifications = "false" # disable_notifications = "false"
# disable_upload_validation = "false" # disable_upload_validation = "false"
# enable_logging = "false" # enable_logging = "false"
# force_http_11 = "false"
# force_http_2 = "false" # force_http_2 = "false"
# local_first = "false" # local_first = "false"
# no_remote_delete = "false" # no_remote_delete = "false"
@ -698,11 +697,35 @@ systemctl --user start onedrive
**Note:** This will run the 'onedrive' process with a UID/GID of '0', thus, any files or folders that are created will be owned by 'root' **Note:** This will run the 'onedrive' process with a UID/GID of '0', thus, any files or folders that are created will be owned by 'root'
To see the logs run: To see the systemd application logs run:
```text ```text
journalctl --user-unit=onedrive -f journalctl --user-unit=onedrive -f
``` ```
**Note:** It is a 'systemd' requirement that the XDG environment variables exist for correct enablement and operation of systemd services. If you receive this error when enabling the systemd service:
```
Failed to connect to bus: No such file or directory
```
The most likely cause is that the XDG environment variables are missing. To fix this, you must add the following to `.bashrc` or any other file which is run on user login:
```
export XDG_RUNTIME_DIR="/run/user/$UID"
export DBUS_SESSION_BUS_ADDRESS="unix:path=${XDG_RUNTIME_DIR}/bus"
```
To make this change effective, you must logout of all user accounts where this change has been made.
**Note:** On some systems (for example - Raspbian / Ubuntu / Debian on Raspberry Pi) the above XDG fix may not be reliable after system reboots. The potential alternative to start the client via systemd as root, is to perform the following:
1. Create a symbolic link from `/home/root/.config/onedrive` pointing to `/root/.config/onedrive/`
2. Create a systemd service using the '@' service file: `systemctl enable onedrive@root.service`
3. Start the root@service: `systemctl start onedrive@root.service`
This will ensure that the service will correctly restart on system reboot.
To see the systemd application logs run:
```text
journalctl --unit=onedrive@<username> -f
```
### OneDrive service running as root user via systemd (Red Hat Enterprise Linux, CentOS Linux) ### OneDrive service running as root user via systemd (Red Hat Enterprise Linux, CentOS Linux)
```text ```text
systemctl enable onedrive systemctl enable onedrive
@ -710,7 +733,7 @@ systemctl start onedrive
``` ```
**Note:** This will run the 'onedrive' process with a UID/GID of '0', thus, any files or folders that are created will be owned by 'root' **Note:** This will run the 'onedrive' process with a UID/GID of '0', thus, any files or folders that are created will be owned by 'root'
To see the logs run: To see the systemd application logs run:
```text ```text
journalctl --unit=onedrive -f journalctl --unit=onedrive -f
``` ```
@ -732,7 +755,7 @@ systemctl start onedrive@<username>.service
systemctl status onedrive@<username>.service systemctl status onedrive@<username>.service
``` ```
To see the logs run: To see the systemd application logs run:
```text ```text
journalctl --unit=onedrive@<username> -f journalctl --unit=onedrive@<username> -f
``` ```
@ -752,7 +775,7 @@ systemctl --user enable onedrive
systemctl --user start onedrive systemctl --user start onedrive
``` ```
To see the logs run: To see the systemd application logs run:
```text ```text
journalctl --user-unit=onedrive -f journalctl --user-unit=onedrive -f
``` ```

2
docs/advanced-usage.md
View file

@ -61,7 +61,7 @@ Test the configuration using '--display-config' and '--dry-run'. By doing so, th
#### Display the configuration #### Display the configuration
```text ```text
onedrive --confdir="~/.config/my-new-config --display-config" onedrive --confdir="~/.config/my-new-config" --display-config
``` ```
#### Test the configuration by performing a dry-run #### Test the configuration by performing a dry-run

5
onedrive.1.in
View file

@ -89,11 +89,6 @@ Configuration file key: \fBenable_logging\fP (default: \fBfalse\fP)
\fB\-\-force\fP \fB\-\-force\fP
Force the deletion of data when a 'big delete' is detected Force the deletion of data when a 'big delete' is detected
.TP .TP
\fB\-\-force\-http\-1.1\fP
Force the use of HTTP 1.1 for all operations (DEPRECIATED)
.br
Configuration file key: \fBforce_http_11\fP (default: \fBfalse\fP)
.TP
\fB\-\-force\-http\-2\fP \fB\-\-force\-http\-2\fP
Force the use of HTTP/2 for all operations where applicable Force the use of HTTP/2 for all operations where applicable
.br .br

4
src/config.d
View file

@ -61,7 +61,6 @@ final class Config
boolValues["disable_notifications"] = false; boolValues["disable_notifications"] = false;
boolValues["disable_upload_validation"] = false; boolValues["disable_upload_validation"] = false;
boolValues["enable_logging"] = false; boolValues["enable_logging"] = false;
boolValues["force_http_11"] = false;
boolValues["force_http_2"] = false; boolValues["force_http_2"] = false;
boolValues["local_first"] = false; boolValues["local_first"] = false;
boolValues["no_remote_delete"] = false; boolValues["no_remote_delete"] = false;
@ -343,9 +342,6 @@ final class Config
"enable-logging", "enable-logging",
"Enable client activity to a separate log file", "Enable client activity to a separate log file",
&boolValues["enable_logging"], &boolValues["enable_logging"],
"force-http-1.1",
"Force the use of HTTP/1.1 for all operations (DEPRECIATED)",
&boolValues["force_http_11"],
"force-http-2", "force-http-2",
"Force the use of HTTP/2 for all operations where applicable", "Force the use of HTTP/2 for all operations where applicable",
&boolValues["force_http_2"], &boolValues["force_http_2"],

7
src/main.d
View file

@ -580,11 +580,6 @@ int main(string[] args)
return EXIT_SUCCESS; return EXIT_SUCCESS;
} }
// If the user is still using --force-http-1.1 advise its no longer required
if (cfg.getValueBool("force_http_11")) {
log.log("NOTE: The use of --force-http-1.1 is depreciated");
}
// Test if OneDrive service can be reached, exit if it cant be reached // Test if OneDrive service can be reached, exit if it cant be reached
log.vdebug("Testing network to ensure network connectivity to Microsoft OneDrive Service"); log.vdebug("Testing network to ensure network connectivity to Microsoft OneDrive Service");
online = testNetwork(); online = testNetwork();
@ -1048,7 +1043,7 @@ int main(string[] args)
log.vlog("Offline, cannot delete item!"); log.vlog("Offline, cannot delete item!");
} catch(SyncException e) { } catch(SyncException e) {
if (e.msg == "The item to delete is not in the local database") { if (e.msg == "The item to delete is not in the local database") {
log.vlog("Item cannot be deleted from OneDrive because not found in the local database"); log.vlog("Item cannot be deleted from OneDrive because it was not found in the local database");
} else { } else {
log.logAndNotify("Cannot delete remote item: ", e.msg); log.logAndNotify("Cannot delete remote item: ", e.msg);
} }

9
src/onedrive.d
View file

@ -852,10 +852,15 @@ final class OneDriveApi
} }
// https://docs.microsoft.com/en-us/onedrive/developer/rest-api/api/site_search?view=odsp-graph-online // https://docs.microsoft.com/en-us/onedrive/developer/rest-api/api/site_search?view=odsp-graph-online
JSONValue o365SiteSearch(){ JSONValue o365SiteSearch(const(char)[] nextLink){
checkAccessTokenExpired(); checkAccessTokenExpired();
const(char)[] url; const(char)[] url;
url = siteSearchUrl ~ "=*"; // configure URL to query
if (nextLink.empty) {
url = siteSearchUrl ~ "=*";
} else {
url = nextLink;
}
return get(url); return get(url);
} }

764
src/sync.d
View file

@ -661,9 +661,12 @@ final class SyncEngine
Item databaseItem; Item databaseItem;
foreach (searchDriveId; driveIDsArray) { foreach (searchDriveId; driveIDsArray) {
log.vdebug("searching database for: ", searchDriveId, " ", sharedFolderName); log.vdebug("searching database for: ", searchDriveId, " ", sharedFolderName);
if (itemdb.selectByPath(sharedFolderName, searchDriveId, databaseItem)) { if (itemdb.idInLocalDatabase(searchDriveId, searchResult["remoteItem"]["id"].str)){
// Shared folder is present
log.vdebug("Found shared folder name in database"); log.vdebug("Found shared folder name in database");
itemInDatabase = true; itemInDatabase = true;
// Query the DB for the details of this item
itemdb.selectByPath(sharedFolderName, searchDriveId, databaseItem);
log.vdebug("databaseItem: ", databaseItem); log.vdebug("databaseItem: ", databaseItem);
// Does the databaseItem.driveId == defaultDriveId? // Does the databaseItem.driveId == defaultDriveId?
if (databaseItem.driveId == defaultDriveId) { if (databaseItem.driveId == defaultDriveId) {
@ -958,7 +961,17 @@ final class SyncEngine
} }
Item item; Item item;
if (!itemdb.selectByPath(path, defaultDriveId, item)) { // Need to check all driveid's we know about, not just the defaultDriveId
bool itemInDB = false;
foreach (searchDriveId; driveIDsArray) {
if (itemdb.selectByPath(path, searchDriveId, item)) {
// item was found in the DB
itemInDB = true;
break;
}
}
// Was the item found in the DB
if (!itemInDB) {
// this is odd .. this directory is not in the local database - just go delete it // this is odd .. this directory is not in the local database - just go delete it
log.vlog("The requested directory to delete was not found in the local database - pushing delete request direct to OneDrive"); log.vlog("The requested directory to delete was not found in the local database - pushing delete request direct to OneDrive");
uploadDeleteItem(item, path); uploadDeleteItem(item, path);
@ -2538,7 +2551,14 @@ final class SyncEngine
safeRename(newPath); safeRename(newPath);
} }
} }
rename(oldPath, newPath); // try and rename path, catch exception
try {
log.vdebug("Calling rename(oldPath, newPath)");
rename(oldPath, newPath);
} catch (FileException e) {
// display the error message
displayFileSystemErrorMessage(e.msg, getFunctionName!({}));
}
} }
// handle changed content and mtime // handle changed content and mtime
// HACK: use mtime+hash instead of cTag because of https://github.com/OneDrive/onedrive-api-docs/issues/765 // HACK: use mtime+hash instead of cTag because of https://github.com/OneDrive/onedrive-api-docs/issues/765
@ -2923,7 +2943,12 @@ final class SyncEngine
} }
// scan for changes in the path provided // scan for changes in the path provided
log.log("Uploading differences of ", logPath); if (isDir(path)) {
// if this path is a directory, output this message.
// if a file, potentially leads to confusion as to what the client is actually doing
log.log("Uploading differences of ", logPath);
}
Item item; Item item;
// For each unique OneDrive driveID we know about // For each unique OneDrive driveID we know about
foreach (driveId; driveIDsArray) { foreach (driveId; driveIDsArray) {
@ -2943,7 +2968,13 @@ final class SyncEngine
} }
} }
log.log("Uploading new items of ", logPath); // scan for changes in the path provided
if (isDir(path)) {
// if this path is a directory, output this message.
// if a file, potentially leads to confusion as to what the client is actually doing
log.log("Uploading new items of ", logPath);
}
// Filesystem walk to find new files not uploaded // Filesystem walk to find new files not uploaded
uploadNewItems(path); uploadNewItems(path);
// clean up idsToDelete only if --dry-run is set // clean up idsToDelete only if --dry-run is set
@ -3505,10 +3536,18 @@ final class SyncEngine
itemdb.deleteById(item.driveId, item.id); itemdb.deleteById(item.driveId, item.id);
return; return;
} else { } else {
// For logging consistency if ((!syncBusinessFolders) || (item.driveId == defaultDriveId)) {
writeln(""); // For logging consistency
// normal session upload writeln("");
response = session.upload(path, item.driveId, item.parentId, baseName(path), item.eTag); // If we are not syncing Shared Business Folders, or this change is going to the 'users' default drive, handle normally
// Perform a normal session upload
response = session.upload(path, item.driveId, item.parentId, baseName(path), item.eTag);
} else {
// If we are uploading to a shared business folder, there are a couple of corner cases here:
// 1. Shared Folder is a 'users' folder
// 2. Shared Folder is a 'SharePoint Library' folder, meaning we get hit by this stupidity: https://github.com/OneDrive/onedrive-api-docs/issues/935
response = handleSharePointMetadataAdditionBug(item, path);
}
} }
} catch (OneDriveException e) { } catch (OneDriveException e) {
if (e.httpStatusCode == 401) { if (e.httpStatusCode == 401) {
@ -3542,12 +3581,17 @@ final class SyncEngine
uploadFailed = true; uploadFailed = true;
return; return;
} }
// upload done without error // Did the upload fail?
writeln("done."); if (!uploadFailed){
// upload done without error or failure
// As the session.upload includes the last modified time, save the response writeln("done.");
// Is the response a valid JSON object - validation checking done in saveItem // As the session.upload includes the last modified time, save the response
saveItem(response); // Is the response a valid JSON object - validation checking done in saveItem
saveItem(response);
} else {
// uploadFailed, return
return;
}
} }
// OneDrive documentLibrary // OneDrive documentLibrary
@ -3565,59 +3609,19 @@ final class SyncEngine
itemdb.deleteById(item.driveId, item.id); itemdb.deleteById(item.driveId, item.id);
return; return;
} else { } else {
// Handle certain file types differently // Due to https://github.com/OneDrive/onedrive-api-docs/issues/935 Microsoft modifies all PDF, MS Office & HTML files with added XML content. It is a 'feature' of SharePoint.
if ((extension(path) == ".txt") || (extension(path) == ".csv")) { // This means, as a session upload, on 'completion' the file is 'moved' and generates a 404 ......
// .txt and .csv are unaffected by https://github.com/OneDrive/onedrive-api-docs/issues/935 response = handleSharePointMetadataAdditionBug(item, path);
// For logging consistency
writeln(""); // Did the upload fail?
try { if (!uploadFailed){
response = session.upload(path, item.driveId, item.parentId, baseName(path), item.eTag); // upload done without error or failure
} catch (OneDriveException e) {
if (e.httpStatusCode == 401) {
// OneDrive returned a 'HTTP/1.1 401 Unauthorized Error' - file failed to be uploaded
writeln("skipped.");
log.vlog("OneDrive returned a 'HTTP 401 - Unauthorized' - gracefully handling error");
uploadFailed = true;
return;
}
// Resolve https://github.com/abraunegg/onedrive/issues/36
if ((e.httpStatusCode == 409) || (e.httpStatusCode == 423)) {
// The file is currently checked out or locked for editing by another user
// We cant upload this file at this time
writeln("skipped.");
log.fileOnly("Uploading modified file ", path, " ... skipped.");
writeln("", path, " is currently checked out or locked for editing by another user.");
log.fileOnly(path, " is currently checked out or locked for editing by another user.");
uploadFailed = true;
return;
} else {
// display what the error is
writeln("skipped.");
displayOneDriveErrorMessage(e.msg, getFunctionName!({}));
uploadFailed = true;
return;
}
} catch (FileException e) {
// display the error message
writeln("skipped.");
displayFileSystemErrorMessage(e.msg, getFunctionName!({}));
uploadFailed = true;
return;
}
// upload done without error
writeln("done."); writeln("done.");
// As the session.upload includes the last modified time, save the response // As the session.upload includes the last modified time, save the response
// Is the response a valid JSON object - validation checking done in saveItem // Is the response a valid JSON object - validation checking done in saveItem
saveItem(response); saveItem(response);
} else { } else {
// Due to https://github.com/OneDrive/onedrive-api-docs/issues/935 Microsoft modifies all PDF, MS Office & HTML files with added XML content. It is a 'feature' of SharePoint. // uploadFailed, return
// This means, as a session upload, on 'completion' the file is 'moved' and generates a 404 ......
writeln("skipped.");
log.fileOnly("Uploading modified file ", path, " ... skipped.");
log.vlog("Skip Reason: Microsoft Sharepoint 'enrichment' after upload issue");
log.vlog("See: https://github.com/OneDrive/onedrive-api-docs/issues/935 for further details");
// Delete record from the local database - file will be uploaded as a new file
itemdb.deleteById(item.driveId, item.id);
return; return;
} }
} }
@ -3732,6 +3736,68 @@ final class SyncEngine
} }
} }
} }
private JSONValue handleSharePointMetadataAdditionBug(const ref Item item, const(string) path)
{
// Explicit function for handling https://github.com/OneDrive/onedrive-api-docs/issues/935
JSONValue response;
// Handle certain file types differently
if ((extension(path) == ".txt") || (extension(path) == ".csv")) {
// .txt and .csv are unaffected by https://github.com/OneDrive/onedrive-api-docs/issues/935
// For logging consistency
writeln("");
try {
response = session.upload(path, item.driveId, item.parentId, baseName(path), item.eTag);
} catch (OneDriveException e) {
if (e.httpStatusCode == 401) {
// OneDrive returned a 'HTTP/1.1 401 Unauthorized Error' - file failed to be uploaded
writeln("skipped.");
log.vlog("OneDrive returned a 'HTTP 401 - Unauthorized' - gracefully handling error");
uploadFailed = true;
return response;
}
// Resolve https://github.com/abraunegg/onedrive/issues/36
if ((e.httpStatusCode == 409) || (e.httpStatusCode == 423)) {
// The file is currently checked out or locked for editing by another user
// We cant upload this file at this time
writeln("skipped.");
log.fileOnly("Uploading modified file ", path, " ... skipped.");
writeln("", path, " is currently checked out or locked for editing by another user.");
log.fileOnly(path, " is currently checked out or locked for editing by another user.");
uploadFailed = true;
return response;
} else {
// display what the error is
writeln("skipped.");
displayOneDriveErrorMessage(e.msg, getFunctionName!({}));
uploadFailed = true;
return response;
}
} catch (FileException e) {
// display the error message
writeln("skipped.");
displayFileSystemErrorMessage(e.msg, getFunctionName!({}));
uploadFailed = true;
return response;
}
// upload done without error
writeln("done.");
} else {
// Due to https://github.com/OneDrive/onedrive-api-docs/issues/935 Microsoft modifies all PDF, MS Office & HTML files with added XML content. It is a 'feature' of SharePoint.
// This means, as a session upload, on 'completion' the file is 'moved' and generates a 404 ......
writeln("skipped.");
log.fileOnly("Uploading modified file ", path, " ... skipped.");
log.vlog("Skip Reason: Microsoft Sharepoint 'enrichment' after upload issue");
log.vlog("See: https://github.com/OneDrive/onedrive-api-docs/issues/935 for further details");
// Delete record from the local database - file will be uploaded as a new file
itemdb.deleteById(item.driveId, item.id);
uploadFailed = true;
return response;
}
// return a JSON response so that it can be used and saved
return response;
}
// upload new items to OneDrive // upload new items to OneDrive
private void uploadNewItems(const(string) path) private void uploadNewItems(const(string) path)
@ -3973,18 +4039,15 @@ final class SyncEngine
if (!uploadFailed) { if (!uploadFailed) {
// Upload did not fail // Upload did not fail
// Issue #763 - Delete local files after sync handling // Issue #763 - Delete local files after sync handling
// are we in an --upload-only scenario? // are we in an --upload-only & --remove-source-files scenario?
if (uploadOnly) { if ((uploadOnly) && (localDeleteAfterUpload)) {
// are we in a delete local file after upload? // Log that we are deleting a local item
if (localDeleteAfterUpload) { log.log("Removing local file as --upload-only & --remove-source-files configured");
// Log that we are deleting a local item // are we in a --dry-run scenario?
log.log("Removing local file as --upload-only & --remove-source-files configured"); if (!dryRun) {
// are we in a --dry-run scenario? // No --dry-run ... process local file delete
if (!dryRun) { log.vdebug("Removing local file: ", path);
// No --dry-run ... process local file delete safeRemove(path);
log.vdebug("Removing local file: ", path);
safeRemove(path);
}
} }
} }
} }
@ -4551,9 +4614,20 @@ final class SyncEngine
// This has been seen with PNG / JPG files mainly, which then contributes to generating a 412 error when we attempt to update the metadata // This has been seen with PNG / JPG files mainly, which then contributes to generating a 412 error when we attempt to update the metadata
// Validate here that the file uploaded, at least in size, matches in the response to what the size is on disk // Validate here that the file uploaded, at least in size, matches in the response to what the size is on disk
if (thisFileSize != uploadFileSize){ if (thisFileSize != uploadFileSize){
if(disableUploadValidation){ // Upload size did not match local size
// Print a warning message // There are 2 scenarios where this happens:
// 1. Failed Transfer
// 2. Upload file is going to a SharePoint Site, where Microsoft enriches the file with additional metadata with no way to disable
// For this client:
// - If a SharePoint Library, disableUploadValidation gets flagged as True
// - If we are syncing a business shared folder, this folder could reside on a Users Path (there should be no upload issue) or SharePoint (upload issue)
if ((disableUploadValidation)|| (syncBusinessFolders && (parent.driveId != defaultDriveId))){
// Print a warning message - should only be triggered if:
// - disableUploadValidation gets flagged (documentLibrary account type)
// - syncBusinessFolders is being used & parent.driveId != defaultDriveId
log.log("WARNING: Uploaded file size does not match local file - skipping upload validation"); log.log("WARNING: Uploaded file size does not match local file - skipping upload validation");
log.vlog("WARNING: Due to Microsoft Sharepoint 'enrichment' of files, this file is now technically different to your local copy");
log.vlog("See: https://github.com/OneDrive/onedrive-api-docs/issues/935 for further details");
} else { } else {
// OK .. the uploaded file does not match and we did not disable this validation // OK .. the uploaded file does not match and we did not disable this validation
log.log("Uploaded file size does not match local file - upload failure - retrying"); log.log("Uploaded file size does not match local file - upload failure - retrying");
@ -4820,53 +4894,71 @@ final class SyncEngine
} else { } else {
// OneDrive Business account modified file upload handling // OneDrive Business account modified file upload handling
if (accountType == "business"){ if (accountType == "business"){
// OneDrive Business Account - always use a session to upload // OneDrive Business Account
writeln(""); if ((!syncBusinessFolders) || (parent.driveId == defaultDriveId)) {
try { // If we are not syncing Shared Business Folders, or this change is going to the 'users' default drive, handle normally
response = session.upload(path, parent.driveId, parent.id, baseName(path), fileDetailsFromOneDrive["eTag"].str); // For logging consistency
} catch (OneDriveException e) { writeln("");
log.vdebug("response = session.upload(path, parent.driveId, parent.id, baseName(path), fileDetailsFromOneDrive['eTag'].str); generated a OneDriveException"); try {
if (e.httpStatusCode == 401) { response = session.upload(path, parent.driveId, parent.id, baseName(path), fileDetailsFromOneDrive["eTag"].str);
// OneDrive returned a 'HTTP/1.1 401 Unauthorized Error' - file failed to be uploaded } catch (OneDriveException e) {
log.vdebug("response = session.upload(path, parent.driveId, parent.id, baseName(path), fileDetailsFromOneDrive['eTag'].str); generated a OneDriveException");
if (e.httpStatusCode == 401) {
// OneDrive returned a 'HTTP/1.1 401 Unauthorized Error' - file failed to be uploaded
writeln("skipped.");
log.vlog("OneDrive returned a 'HTTP 401 - Unauthorized' - gracefully handling error");
uploadFailed = true;
return;
}
if (e.httpStatusCode == 429) {
// HTTP request returned status code 429 (Too Many Requests). We need to leverage the response Retry-After HTTP header to ensure minimum delay until the throttle is removed.
handleOneDriveThrottleRequest();
// Retry original request by calling function again to avoid replicating any further error handling
log.vdebug("Retrying original request that generated the OneDrive HTTP 429 Response Code (Too Many Requests) - calling uploadNewFile(path);");
uploadNewFile(path);
// return back to original call
return;
}
if (e.httpStatusCode == 504) {
// HTTP request returned status code 504 (Gateway Timeout)
log.log("OneDrive returned a 'HTTP 504 - Gateway Timeout' - retrying upload request");
// Retry original request by calling function again to avoid replicating any further error handling
uploadNewFile(path);
// return back to original call
return;
} else {
// error uploading file
// display what the error is
writeln("skipped.");
displayOneDriveErrorMessage(e.msg, getFunctionName!({}));
uploadFailed = true;
return;
}
} catch (FileException e) {
// display the error message
writeln("skipped."); writeln("skipped.");
log.vlog("OneDrive returned a 'HTTP 401 - Unauthorized' - gracefully handling error"); displayFileSystemErrorMessage(e.msg, getFunctionName!({}));
uploadFailed = true; uploadFailed = true;
return; return;
} }
if (e.httpStatusCode == 429) { // upload complete
// HTTP request returned status code 429 (Too Many Requests). We need to leverage the response Retry-After HTTP header to ensure minimum delay until the throttle is removed. writeln("done.");
handleOneDriveThrottleRequest(); saveItem(response);
// Retry original request by calling function again to avoid replicating any further error handling } else {
log.vdebug("Retrying original request that generated the OneDrive HTTP 429 Response Code (Too Many Requests) - calling uploadNewFile(path);"); // If we are uploading to a shared business folder, there are a couple of corner cases here:
uploadNewFile(path); // 1. Shared Folder is a 'users' folder
// return back to original call // 2. Shared Folder is a 'SharePoint Library' folder, meaning we get hit by this stupidity: https://github.com/OneDrive/onedrive-api-docs/issues/935
return;
} // Need try{} & catch (OneDriveException e) { & catch (FileException e) { handler for this query
if (e.httpStatusCode == 504) { response = handleSharePointMetadataAdditionBugReplaceFile(fileDetailsFromOneDrive, parent, path);
// HTTP request returned status code 504 (Gateway Timeout) if (!uploadFailed){
log.log("OneDrive returned a 'HTTP 504 - Gateway Timeout' - retrying upload request"); // Is the response a valid JSON object - validation checking done in saveItem
// Retry original request by calling function again to avoid replicating any further error handling saveItem(response);
uploadNewFile(path);
// return back to original call
return;
} else { } else {
// error uploading file // uploadFailed, return
// display what the error is
writeln("skipped.");
displayOneDriveErrorMessage(e.msg, getFunctionName!({}));
uploadFailed = true;
return; return;
} }
} catch (FileException e) {
// display the error message
writeln("skipped.");
displayFileSystemErrorMessage(e.msg, getFunctionName!({}));
uploadFailed = true;
return;
} }
// upload complete
writeln("done.");
saveItem(response);
} }
// OneDrive SharePoint account modified file upload handling // OneDrive SharePoint account modified file upload handling
@ -4875,93 +4967,15 @@ final class SyncEngine
// as if too large, the following error will be generated by OneDrive: // as if too large, the following error will be generated by OneDrive:
// HTTP request returned status code 413 (Request Entity Too Large) // HTTP request returned status code 413 (Request Entity Too Large)
// We also cant use a session to upload the file, we have to use simpleUploadReplace // We also cant use a session to upload the file, we have to use simpleUploadReplace
// Calculate existing hash for this file // Need try{} & catch (OneDriveException e) { & catch (FileException e) { handler for this query
string existingFileHash = computeQuickXorHash(path); response = handleSharePointMetadataAdditionBugReplaceFile(fileDetailsFromOneDrive, parent, path);
if (!uploadFailed){
if (getSize(path) <= thresholdFileSize) { // Is the response a valid JSON object - validation checking done in saveItem
// Upload file via simpleUploadReplace as below threshold size saveItem(response);
try {
response = onedrive.simpleUploadReplace(path, fileDetailsFromOneDrive["parentReference"]["driveId"].str, fileDetailsFromOneDrive["id"].str, fileDetailsFromOneDrive["eTag"].str);
} catch (OneDriveException e) {
if (e.httpStatusCode == 401) {
// OneDrive returned a 'HTTP/1.1 401 Unauthorized Error' - file failed to be uploaded
writeln("skipped.");
log.vlog("OneDrive returned a 'HTTP 401 - Unauthorized' - gracefully handling error");
uploadFailed = true;
return;
} else {
// display what the error is
writeln("skipped.");
displayOneDriveErrorMessage(e.msg, getFunctionName!({}));
uploadFailed = true;
return;
}
} catch (FileException e) {
// display the error message
writeln("skipped.");
displayFileSystemErrorMessage(e.msg, getFunctionName!({}));
uploadFailed = true;
return;
}
} else { } else {
// Have to upload via a session, however we have to delete the file first otherwise this will generate a 404 error post session upload // uploadFailed, return
// Remove the existing file return;
onedrive.deleteById(fileDetailsFromOneDrive["parentReference"]["driveId"].str, fileDetailsFromOneDrive["id"].str, fileDetailsFromOneDrive["eTag"].str);
// Upload as a session, as a new file
writeln("");
try {
response = session.upload(path, parent.driveId, parent.id, baseName(path));
} catch (OneDriveException e) {
if (e.httpStatusCode == 401) {
// OneDrive returned a 'HTTP/1.1 401 Unauthorized Error' - file failed to be uploaded
writeln("skipped.");
log.vlog("OneDrive returned a 'HTTP 401 - Unauthorized' - gracefully handling error");
uploadFailed = true;
return;
} else {
// display what the error is
writeln("skipped.");
displayOneDriveErrorMessage(e.msg, getFunctionName!({}));
uploadFailed = true;
return;
}
} catch (FileException e) {
// display the error message
writeln("skipped.");
displayFileSystemErrorMessage(e.msg, getFunctionName!({}));
uploadFailed = true;
return;
}
}
writeln(" done.");
// Is the response a valid JSON object - validation checking done in saveItem
saveItem(response);
// Due to https://github.com/OneDrive/onedrive-api-docs/issues/935 Microsoft modifies all PDF, MS Office & HTML files with added XML content. It is a 'feature' of SharePoint.
// So - now the 'local' and 'remote' file is technically DIFFERENT ... thanks Microsoft .. NO way to disable this stupidity
string uploadNewFileHash;
if (hasQuickXorHash(response)) {
// use the response json hash detail to compare
uploadNewFileHash = response["file"]["hashes"]["quickXorHash"].str;
}
if (existingFileHash != uploadNewFileHash) {
// file was modified by Microsoft post upload to SharePoint site
log.vdebug("Existing Local File Hash: ", existingFileHash);
log.vdebug("New Remote File Hash: ", uploadNewFileHash);
if(!uploadOnly){
// Download the Microsoft 'modified' file so 'local' is now in sync
log.vlog("Due to Microsoft Sharepoint 'enrichment' of files, downloading 'enriched' file to ensure local file is in-sync");
log.vlog("See: https://github.com/OneDrive/onedrive-api-docs/issues/935 for further details");
auto fileSize = response["size"].integer;
onedrive.downloadById(response["parentReference"]["driveId"].str, response["id"].str, path, fileSize);
} else {
// we are not downloading a file, warn that file differences will exist
log.vlog("WARNING: Due to Microsoft Sharepoint 'enrichment' of files, this file is now technically different to your local copy");
log.vlog("See: https://github.com/OneDrive/onedrive-api-docs/issues/935 for further details");
}
} }
} }
} }
@ -5038,6 +5052,106 @@ final class SyncEngine
} }
} }
private JSONValue handleSharePointMetadataAdditionBugReplaceFile(JSONValue fileDetailsFromOneDrive, const ref Item parent, const(string) path)
{
// Explicit function for handling https://github.com/OneDrive/onedrive-api-docs/issues/935
// Replace existing file
JSONValue response;
// Depending on the file size, this will depend on how best to handle the modified local file
// as if too large, the following error will be generated by OneDrive:
// HTTP request returned status code 413 (Request Entity Too Large)
// We also cant use a session to upload the file, we have to use simpleUploadReplace
// Calculate existing hash for this file
string existingFileHash = computeQuickXorHash(path);
if (getSize(path) <= thresholdFileSize) {
// Upload file via simpleUploadReplace as below threshold size
try {
response = onedrive.simpleUploadReplace(path, fileDetailsFromOneDrive["parentReference"]["driveId"].str, fileDetailsFromOneDrive["id"].str, fileDetailsFromOneDrive["eTag"].str);
} catch (OneDriveException e) {
if (e.httpStatusCode == 401) {
// OneDrive returned a 'HTTP/1.1 401 Unauthorized Error' - file failed to be uploaded
writeln("skipped.");
log.vlog("OneDrive returned a 'HTTP 401 - Unauthorized' - gracefully handling error");
uploadFailed = true;
return response;
} else {
// display what the error is
writeln("skipped.");
displayOneDriveErrorMessage(e.msg, getFunctionName!({}));
uploadFailed = true;
return response;
}
} catch (FileException e) {
// display the error message
writeln("skipped.");
displayFileSystemErrorMessage(e.msg, getFunctionName!({}));
uploadFailed = true;
return response;
}
} else {
// Have to upload via a session, however we have to delete the file first otherwise this will generate a 404 error post session upload
// Remove the existing file
onedrive.deleteById(fileDetailsFromOneDrive["parentReference"]["driveId"].str, fileDetailsFromOneDrive["id"].str, fileDetailsFromOneDrive["eTag"].str);
// Upload as a session, as a new file
writeln("");
try {
response = session.upload(path, parent.driveId, parent.id, baseName(path));
} catch (OneDriveException e) {
if (e.httpStatusCode == 401) {
// OneDrive returned a 'HTTP/1.1 401 Unauthorized Error' - file failed to be uploaded
writeln("skipped.");
log.vlog("OneDrive returned a 'HTTP 401 - Unauthorized' - gracefully handling error");
uploadFailed = true;
return response;
} else {
// display what the error is
writeln("skipped.");
displayOneDriveErrorMessage(e.msg, getFunctionName!({}));
uploadFailed = true;
return response;
}
} catch (FileException e) {
// display the error message
writeln("skipped.");
displayFileSystemErrorMessage(e.msg, getFunctionName!({}));
uploadFailed = true;
return response;
}
}
writeln("done.");
// Due to https://github.com/OneDrive/onedrive-api-docs/issues/935 Microsoft modifies all PDF, MS Office & HTML files with added XML content. It is a 'feature' of SharePoint.
// So - now the 'local' and 'remote' file is technically DIFFERENT ... thanks Microsoft .. NO way to disable this stupidity
string uploadNewFileHash;
if (hasQuickXorHash(response)) {
// use the response json hash detail to compare
uploadNewFileHash = response["file"]["hashes"]["quickXorHash"].str;
}
if (existingFileHash != uploadNewFileHash) {
// file was modified by Microsoft post upload to SharePoint site
log.vdebug("Existing Local File Hash: ", existingFileHash);
log.vdebug("New Remote File Hash: ", uploadNewFileHash);
if(!uploadOnly){
// Download the Microsoft 'modified' file so 'local' is now in sync
log.vlog("Due to Microsoft Sharepoint 'enrichment' of files, downloading 'enriched' file to ensure local file is in-sync");
log.vlog("See: https://github.com/OneDrive/onedrive-api-docs/issues/935 for further details");
auto fileSize = response["size"].integer;
onedrive.downloadById(response["parentReference"]["driveId"].str, response["id"].str, path, fileSize);
} else {
// we are not downloading a file, warn that file differences will exist
log.vlog("WARNING: Due to Microsoft Sharepoint 'enrichment' of files, this file is now technically different to your local copy");
log.vlog("See: https://github.com/OneDrive/onedrive-api-docs/issues/935 for further details");
}
}
// return a JSON response so that it can be used and saved
return response;
}
// delete an item on OneDrive // delete an item on OneDrive
private void uploadDeleteItem(Item item, const(string) path) private void uploadDeleteItem(Item item, const(string) path)
{ {
@ -5212,11 +5326,18 @@ final class SyncEngine
if (jsonItem.type() == JSONType.object){ if (jsonItem.type() == JSONType.object){
// Check if the response JSON has an 'id', otherwise makeItem() fails with 'Key not found: id' // Check if the response JSON has an 'id', otherwise makeItem() fails with 'Key not found: id'
if (hasId(jsonItem)) { if (hasId(jsonItem)) {
// Takes a JSON input and formats to an item which can be used by the database // Are we in a --upload-only & --remove-source-files scenario?
Item item = makeItem(jsonItem); // We do not want to add the item to the database in this situation as there is no local reference to the file post file deletion
// Add to the local database if ((uploadOnly) && (localDeleteAfterUpload)) {
log.vdebug("Adding to database: ", item); // Log that we are deleting a local item
itemdb.upsert(item); log.vdebug("Skipping adding to database as --upload-only & --remove-source-files configured");
} else {
// Takes a JSON input and formats to an item which can be used by the database
Item item = makeItem(jsonItem);
// Add to the local database
log.vdebug("Adding to database: ", item);
itemdb.upsert(item);
}
} else { } else {
// log error // log error
log.error("ERROR: OneDrive response missing required 'id' element"); log.error("ERROR: OneDrive response missing required 'id' element");
@ -5356,9 +5477,19 @@ final class SyncEngine
void deleteByPath(const(string) path) void deleteByPath(const(string) path)
{ {
Item item; Item item;
if (!itemdb.selectByPath(path, defaultDriveId, item)) { // Need to check all driveid's we know about, not just the defaultDriveId
bool itemInDB = false;
foreach (searchDriveId; driveIDsArray) {
if (itemdb.selectByPath(path, searchDriveId, item)) {
// item was found in the DB
itemInDB = true;
break;
}
}
if (!itemInDB) {
throw new SyncException("The item to delete is not in the local database"); throw new SyncException("The item to delete is not in the local database");
} }
if (item.parentId == null) { if (item.parentId == null) {
// the item is a remote folder, need to do the operation on the parent // the item is a remote folder, need to do the operation on the parent
enforce(itemdb.selectByPathWithoutRemote(path, defaultDriveId, item)); enforce(itemdb.selectByPathWithoutRemote(path, defaultDriveId, item));
@ -5418,93 +5549,158 @@ final class SyncEngine
string site_id; string site_id;
string drive_id; string drive_id;
string webUrl;
bool found = false; bool found = false;
JSONValue siteQuery; JSONValue siteQuery;
string nextLink;
string[] siteSearchResults;
log.log("Office 365 Library Name Query: ", o365SharedLibraryName); log.log("Office 365 Library Name Query: ", o365SharedLibraryName);
try { for (;;) {
siteQuery = onedrive.o365SiteSearch(); try {
} catch (OneDriveException e) { siteQuery = onedrive.o365SiteSearch(nextLink);
log.error("ERROR: Query of OneDrive for Office 365 Library Name failed"); } catch (OneDriveException e) {
if (e.httpStatusCode == 403) { log.error("ERROR: Query of OneDrive for Office 365 Library Name failed");
// Forbidden - most likely authentication scope needs to be updated if (e.httpStatusCode == 403) {
log.error("ERROR: Authentication scope needs to be updated. Use --logout and re-authenticate client."); // Forbidden - most likely authentication scope needs to be updated
return; log.error("ERROR: Authentication scope needs to be updated. Use --logout and re-authenticate client.");
} else { return;
// display what the error is }
displayOneDriveErrorMessage(e.msg, getFunctionName!({})); // HTTP request returned status code 429 (Too Many Requests)
return; if (e.httpStatusCode == 429) {
} // HTTP request returned status code 429 (Too Many Requests). We need to leverage the response Retry-After HTTP header to ensure minimum delay until the throttle is removed.
} handleOneDriveThrottleRequest();
log.vdebug("Retrying original request that generated the OneDrive HTTP 429 Response Code (Too Many Requests) - attempting to query OneDrive drive children");
// is siteQuery a valid JSON object & contain data we can use? }
if ((siteQuery.type() == JSONType.object) && ("value" in siteQuery)) { // HTTP request returned status code 504 (Gateway Timeout) or 429 retry
// valid JSON object if ((e.httpStatusCode == 429) || (e.httpStatusCode == 504)) {
log.vdebug("O365 Query Response: ", siteQuery); // re-try the specific changes queries
if (e.httpStatusCode == 504) {
foreach (searchResult; siteQuery["value"].array) { log.log("OneDrive returned a 'HTTP 504 - Gateway Timeout' when attempting to query Sharepoint Sites - retrying applicable request");
// Need an 'exclusive' match here with o365SharedLibraryName as entered log.vdebug("siteQuery = onedrive.o365SiteSearch(nextLink) previously threw an error - retrying");
log.vdebug("Found O365 Site: ", searchResult); // The server, while acting as a proxy, did not receive a timely response from the upstream server it needed to access in attempting to complete the request.
log.vdebug("Thread sleeping for 30 seconds as the server did not receive a timely response from the upstream server it needed to access in attempting to complete the request");
// 'displayName', 'id' and 'webUrl' have to be present in the search result record Thread.sleep(dur!"seconds"(30));
if (("displayName" in searchResult) && ("id" in searchResult) && ("webUrl" in searchResult)) { }
if (o365SharedLibraryName == searchResult["displayName"].str){ // re-try original request - retried for 429 and 504
// 'displayName' matches search request try {
site_id = searchResult["id"].str; log.vdebug("Retrying Query: siteQuery = onedrive.o365SiteSearch(nextLink)");
webUrl = searchResult["webUrl"].str; siteQuery = onedrive.o365SiteSearch(nextLink);
JSONValue siteDriveQuery; log.vdebug("Query 'siteQuery = onedrive.o365SiteSearch(nextLink)' performed successfully on re-try");
} catch (OneDriveException e) {
try { // display what the error is
siteDriveQuery = onedrive.o365SiteDrives(site_id); log.vdebug("Query Error: siteQuery = onedrive.o365SiteSearch(nextLink) on re-try after delay");
} catch (OneDriveException e) { // error was not a 504 this time
log.error("ERROR: Query of OneDrive for Office Site ID failed"); displayOneDriveErrorMessage(e.msg, getFunctionName!({}));
// display what the error is return;
displayOneDriveErrorMessage(e.msg, getFunctionName!({}));
return;
}
// is siteDriveQuery a valid JSON object & contain data we can use?
if ((siteDriveQuery.type() == JSONType.object) && ("value" in siteDriveQuery)) {
// valid JSON object
foreach (driveResult; siteDriveQuery["value"].array) {
// Display results
found = true;
writeln("SiteName: ", searchResult["displayName"].str);
writeln("drive_id: ", driveResult["id"].str);
writeln("URL: ", webUrl);
}
} else {
// not a valid JSON object
log.error("ERROR: There was an error performing this operation on OneDrive");
log.error("ERROR: Increase logging verbosity to assist determining why.");
return;
}
} }
} else { } else {
// 'displayName' not present in JSON results // display what the error is
log.error("ERROR: The results returned from OneDrive API do not contain the required items to match. Please check your permissions with your site administrator."); displayOneDriveErrorMessage(e.msg, getFunctionName!({}));
log.error("ERROR: Your site security settings is preventing the following details from being accessed: 'displayName', 'id' and 'webUrl'");
log.error("ERROR: To debug this further, please use --verbose --verbose to provide insight as to what details are actually returned.");
return; return;
} }
} }
if(!found) { // is siteQuery a valid JSON object & contain data we can use?
log.error("ERROR: The requested SharePoint site could not be found. Please check it's name and your permissions to access the site."); if ((siteQuery.type() == JSONType.object) && ("value" in siteQuery)) {
// List all sites returned to assist user // valid JSON object
log.log("\nThe following SharePoint site names were returned:"); log.vdebug("O365 Query Response: ", siteQuery);
foreach (searchResult; siteQuery["value"].array) { foreach (searchResult; siteQuery["value"].array) {
// list the display name that we use to match against the user query // Need an 'exclusive' match here with o365SharedLibraryName as entered
log.log(" * ", searchResult["displayName"].str); log.vdebug("Found O365 Site: ", searchResult);
// 'displayName' and 'id' have to be present in the search result record in order to query the site
if (("displayName" in searchResult) && ("id" in searchResult)) {
if (o365SharedLibraryName == searchResult["displayName"].str){
// 'displayName' matches search request
site_id = searchResult["id"].str;
JSONValue siteDriveQuery;
try {
siteDriveQuery = onedrive.o365SiteDrives(site_id);
} catch (OneDriveException e) {
log.error("ERROR: Query of OneDrive for Office Site ID failed");
// display what the error is
displayOneDriveErrorMessage(e.msg, getFunctionName!({}));
return;
}
// is siteDriveQuery a valid JSON object & contain data we can use?
if ((siteDriveQuery.type() == JSONType.object) && ("value" in siteDriveQuery)) {
// valid JSON object
foreach (driveResult; siteDriveQuery["value"].array) {
// Display results
writeln("-----------------------------------------------");
log.vdebug("Site Details: ", driveResult);
found = true;
writeln("Site Name: ", searchResult["displayName"].str);
writeln("Library Name: ", driveResult["name"].str);
writeln("drive_id: ", driveResult["id"].str);
writeln("Library URL: ", driveResult["webUrl"].str);
}
// closeout
writeln("-----------------------------------------------");
} else {
// not a valid JSON object
log.error("ERROR: There was an error performing this operation on OneDrive");
log.error("ERROR: Increase logging verbosity to assist determining why.");
return;
}
}
} else {
// 'displayName', 'id' or ''webUrl' not present in JSON results for a specific site
string siteNameAvailable = "Site 'name' was restricted by OneDrive API permissions";
bool displayNameAvailable = false;
bool idAvailable = false;
if ("name" in searchResult) siteNameAvailable = searchResult["name"].str;
if ("displayName" in searchResult) displayNameAvailable = true;
if ("id" in searchResult) idAvailable = true;
// Display error details for this site data
log.error("\nERROR: SharePoint Site details not provided for: ", siteNameAvailable);
log.error("ERROR: The SharePoint Site results returned from OneDrive API do not contain the required items to match. Please check your permissions with your site administrator.");
log.error("ERROR: Your site security settings is preventing the following details from being accessed: 'displayName' or 'id'");
log.vlog(" - Is 'displayName' available = ", displayNameAvailable);
log.vlog(" - Is 'id' available = ", idAvailable);
log.error("ERROR: To debug this further, please increase verbosity (--verbose or --verbose --verbose) to provide further insight as to what details are actually being returned.");
}
} }
if(!found) {
// The SharePoint site we are searching for was not found in this bundle set
// Add to siteSearchResults so we can display what we did find
string siteSearchResultsEntry;
foreach (searchResult; siteQuery["value"].array) {
siteSearchResultsEntry = " * " ~ searchResult["displayName"].str;
siteSearchResults ~= siteSearchResultsEntry;
}
}
} else {
// not a valid JSON object
log.error("ERROR: There was an error performing this operation on OneDrive");
log.error("ERROR: Increase logging verbosity to assist determining why.");
return;
}
// If a collection exceeds the default page size (200 items), the @odata.nextLink property is returned in the response
// to indicate more items are available and provide the request URL for the next page of items.
if ("@odata.nextLink" in siteQuery) {
// Update nextLink to next set of SharePoint library names
nextLink = siteQuery["@odata.nextLink"].str;
log.vdebug("Setting nextLink to (@odata.nextLink): ", nextLink);
} else break;
}
// Was the intended target found?
if(!found) {
log.error("\nERROR: The requested SharePoint site could not be found. Please check it's name and your permissions to access the site.");
// List all sites returned to assist user
log.log("\nThe following SharePoint site names were returned:");
foreach (searchResultEntry; siteSearchResults) {
// list the display name that we use to match against the user query
log.log(searchResultEntry);
} }
} else {
// not a valid JSON object
log.error("ERROR: There was an error performing this operation on OneDrive");
log.error("ERROR: Increase logging verbosity to assist determining why.");
return;
} }
} }
@ -6209,8 +6405,8 @@ final class SyncEngine
// to indicate more items are available and provide the request URL for the next page of items. // to indicate more items are available and provide the request URL for the next page of items.
if ("@odata.nextLink" in thisLevelChildren) { if ("@odata.nextLink" in thisLevelChildren) {
// Update nextLink to next changeSet bundle // Update nextLink to next changeSet bundle
log.vdebug("Setting nextLink to (@odata.nextLink): ", nextLink);
nextLink = thisLevelChildren["@odata.nextLink"].str; nextLink = thisLevelChildren["@odata.nextLink"].str;
log.vdebug("Setting nextLink to (@odata.nextLink): ", nextLink);
} else break; } else break;
} }