deblan-mirror/abraunegg-onedrive
deblan-mirror/abraunegg-onedrive
1
0
Fork 0
mirror of https://github.com/abraunegg/onedrive synced 2024-06-29 10:50:30 +02:00
Code Issues Projects Releases Wiki Activity

Merge branch 'master' into fix-issue-966

Browse source
This commit is contained in:
abraunegg 2020-07-07 19:51:23 +10:00 committed by GitHub
parent f853e86536 25fd9a1b00
commit 75de6491f7
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
5 changed files with 214 additions and 89 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

31
docs/Docker.md
View file

@ -34,11 +34,12 @@ The second docker volume is for your data folder and is created in the next step
### 3. First run
Onedrive needs to be authorized with your Microsoft account. This is achieved by running docker in interactive mode. Run the docker image with the two commands below and **make sure to change `onedriveDir` to the onedrive data directory on your filesystem (e.g. `"/home/abraunegg/OneDrive"`)**
Onedrive needs to be authorized with your Microsoft account. This is achieved by running docker in interactive mode. Run the docker image with the two commands below and **make sure to change `onedriveDir` to the onedrive data directory on your filesystem (e.g. `"/home/abraunegg/OneDrive"`)**.
Additionally, the user id and group id should be added to remove any potential user conflicts, denoted by the environment variables `${ONEDRIVE_UID}` and `${ONEDRIVE_GID}`.
```bash
onedriveDir="${HOME}/OneDrive"
docker run -it --restart unless-stopped --name onedrive -v onedrive_conf:/onedrive/conf -v "${onedriveDir}:/onedrive/data" driveone/onedrive
docker run -it --name onedrive -v onedrive_conf:/onedrive/conf -v "${onedriveDir}:/onedrive/data" -e "ONEDRIVE_UID:${ONEDRIVE_UID}" -e "ONEDRIVE_GID:${ONEDRIVE_GID}" driveone/onedrive
```
- You will be asked to open a specific link using your web browser
@ -81,7 +82,29 @@ docker rm -f onedrive
```
## Advanced Setup
### 5. Edit the config
### 5. Docker-compose
Also supports docker-compose schemas > 3.
In the following example it is assumed you have a `onedriveDir` environment variable and a `onedrive_conf` volume.
However, you can also use bind mounts for the configuration folder, e.g. `export ONEDRIVE_CONF="${HOME}/OneDriveConfig"`.
```
version: "3"
services:
onedrive:
image: driveone/onedrive
restart: unless-stopped
environment:
- ONEDRIVE_UID=${PUID}
- ONEDRIVE_GID=${PGID}
volumes:
- onedrive_conf:/onedrive/conf
- ${onedriveDir}:/onedrive/data
```
Note that you still have to perform step 3: First Run.
### 6. Edit the config
Onedrive should run in default configuration, however you can change your configuration by placing a custom config file in the `onedrive_conf` docker volume. First download the default config from [here](https://raw.githubusercontent.com/abraunegg/onedrive/master/config)
Then put it into your onedrive_conf volume path, which can be found with:
@ -94,7 +117,7 @@ Or you can map your own config folder to the config volume. Make sure to copy al
The detailed document for the config can be found here: [additional-configuration](https://github.com/abraunegg/onedrive#additional-configuration)
### 6. Sync multiple accounts
### 7. Sync multiple accounts
There are many ways to do this, the easiest is probably to
1. Create a second docker config volume (replace `Work` with your desired name): `docker volume create onedrive_conf_Work`

2
docs/INSTALL.md
View file

@ -6,7 +6,7 @@ This project has been packaged for the following Linux distributions:
* Arch Linux, available from AUR as [onedrive-abraunegg](https://aur.archlinux.org/packages/onedrive-abraunegg/)
* Debian, available from the package repository as [onedrive](https://packages.debian.org/sid/net/onedrive)
* Fedora, available via package repositories as [onedrive](https://koji.fedoraproject.org/koji/packageinfo?packageID=26044)
* NixOS, available on unstable channel (and stable since 20.03). Use package `onedrive` either by adding it to `configuration.nix` or by using the command `nix-env -iA <channel name>.onedrive`. This does not install a service. There is a [PR](https://github.com/NixOS/nixpkgs/pull/77734) (which needs work) which has code which can be used to install a service. See documentation in the same PR.
* NixOS, use package `onedrive` either by adding it to `configuration.nix` or by using the command `nix-env -iA <channel name>.onedrive`. This does not install a service. To install a service, use unstable channel (will stabilize in 20.09) and add `services.onedrive.enable=true` in `configuration.nix`. You can also add a custom package using the `services.onedrive.package` option (recommended since package lags upstream). Enabling the service installs a default package too (based on the channel). You can also add multiple onedrive accounts trivially, see [documentation](https://github.com/NixOS/nixpkgs/pull/77734#issuecomment-575874225)`.
* openSUSE, available for Tumbleweed as [onedrive](https://software.opensuse.org/package/onedrive) - just install using: `zypper in onedrive`
* Slackware, available from the slackbuilds.org repository as [onedrive](https://slackbuilds.org/repository/14.2/network/onedrive/)
* Solus, available from the package repository as [onedrive](https://dev.getsol.us/search/query/FB7PIf1jG9Z9/#R)

47
src/monitor.d
View file

@ -84,19 +84,27 @@ final class Monitor
return;
}
// skip monitoring any filtered items
// Skip the monitoring of any user filtered items
if (dirname != ".") {
// is the directory name a match to a skip_dir entry?
if (selectiveSync.isDirNameExcluded(dirname.strip('.').strip('/'))) {
// dont add a watch for this item
log.vdebug("Skipping monitoring due to skip_dir match: ", dirname);
return;
// Is the directory name a match to a skip_dir entry?
// The path that needs to be checked needs to include the '/'
// This due to if the user has specified in skip_dir an exclusive path: '/path' - that is what must be matched
if (isDir(dirname)) {
if (selectiveSync.isDirNameExcluded(dirname.strip('.'))) {
// dont add a watch for this item
log.vdebug("Skipping monitoring due to skip_dir match: ", dirname);
return;
}
}
// is the filename a match to a skip_file entry?
if (selectiveSync.isFileNameExcluded(baseName(dirname))) {
// dont add a watch for this item
log.vdebug("Skipping monitoring due to skip_file match: ", dirname);
return;
if (isFile(dirname)) {
// Is the filename a match to a skip_file entry?
// The path that needs to be checked needs to include the '/'
// This due to if the user has specified in skip_file an exclusive path: '/path/file' - that is what must be matched
if (selectiveSync.isFileNameExcluded(dirname.strip('.'))) {
// dont add a watch for this item
log.vdebug("Skipping monitoring due to skip_file match: ", dirname);
return;
}
}
// is the path exluded by sync_list?
if (selectiveSync.isPathExcludedViaSyncList(buildNormalizedPath(dirname))) {
@ -272,13 +280,20 @@ final class Monitor
// if the event is not to be ignored, obtain path
path = getPath(event);
// skip events that should be excluded based on application configuration
if (selectiveSync.isDirNameExcluded(path.strip('.').strip('/'))) {
goto skip;
if (isDir(path)) {
// The path that needs to be checked needs to include the '/'
// This due to if the user has specified in skip_dir an exclusive path: '/path' - that is what must be matched
if (selectiveSync.isDirNameExcluded(path.strip('.'))) {
goto skip;
}
}
if (selectiveSync.isFileNameExcluded(path.strip('.').strip('/'))) {
goto skip;
if (isFile(path)) {
// The path that needs to be checked needs to include the '/'
// This due to if the user has specified in skip_file an exclusive path: '/path/file' - that is what must be matched
if (selectiveSync.isFileNameExcluded(path.strip('.'))) {
goto skip;
}
}
if (selectiveSync.isPathExcludedViaSyncList(path)) {
goto skip;

7
src/selective.d
View file

@ -5,6 +5,7 @@ import std.path;
import std.regex;
import std.stdio;
import util;
import log;
final class SelectiveSync
{
@ -83,6 +84,7 @@ final class SelectiveSync
// Does the directory name match skip_dir config entry?
// Returns true if the name matches a skip_dir config entry
// Returns false if no match
log.vdebug("skip_dir evaluation for: ", name);
// Try full path match first
if (!name.matchFirst(dirmask).empty) {
@ -91,8 +93,8 @@ final class SelectiveSync
// Do we check the base name as well?
if (!skipDirStrictMatch) {
// check just the basename in the path
string filename = baseName(name);
if(!filename.matchFirst(dirmask).empty) {
string parent = baseName(name);
if(!parent.matchFirst(dirmask).empty) {
return true;
}
}
@ -107,6 +109,7 @@ final class SelectiveSync
// Does the file name match skip_file config entry?
// Returns true if the name matches a skip_file config entry
// Returns false if no match
log.vdebug("skip_file evaluation for: ", name);
// Try full path match first
if (!name.matchFirst(mask).empty) {

216
src/sync.d
View file

@ -1752,6 +1752,9 @@ final class SyncEngine
// Reset the downloadFailed flag for this item
downloadFailed = false;
// Path we will be using
string path = "";
if(isItemDeleted(driveItem)){
// Change is to delete an item
log.vdebug("Remote deleted item");
@ -1831,6 +1834,18 @@ final class SyncEngine
simplePathToCheck = driveItem["name"].str;
}
// If 'simplePathToCheck' or 'complexPathToCheck' is of the following format: root:/folder
// then isDirNameExcluded matching will not work
// Clean up 'root:' if present
if (startsWith(simplePathToCheck, "root:")){
log.vdebug("Updating simplePathToCheck to remove 'root:'");
simplePathToCheck = strip(simplePathToCheck, "root:");
}
if (startsWith(complexPathToCheck, "root:")){
log.vdebug("Updating complexPathToCheck to remove 'root:'");
complexPathToCheck = strip(complexPathToCheck, "root:");
}
// OK .. what checks are we doing?
if ((simplePathToCheck != "") && (complexPathToCheck == "")) {
// just a simple check
@ -1860,15 +1875,31 @@ final class SyncEngine
if (!unwanted) {
// Is the item a file and not a deleted item?
if ((isItemFile(driveItem)) && (!isItemDeleted(driveItem))) {
log.vdebug("skip_file item to check: ", item.name);
unwanted = selectiveSync.isFileNameExcluded(item.name);
// skip_file can contain 4 types of entries:
// - wildcard - *.txt
// - text + wildcard - name*.txt
// - full path + combination of any above two - /path/name*.txt
// - full path to file - /path/to/file.txt
// need to compute the full path for this file
path = itemdb.computePath(item.driveId, item.parentId) ~ "/" ~ item.name;
// The path that needs to be checked needs to include the '/'
// This due to if the user has specified in skip_file an exclusive path: '/path/file' - that is what must be matched
if (!startsWith(path, "/")){
// Add '/' to the path
path = '/' ~ path;
}
log.vdebug("skip_file item to check: ", path);
unwanted = selectiveSync.isFileNameExcluded(path);
log.vdebug("Result: ", unwanted);
if (unwanted) log.vlog("Skipping item - excluded by skip_file config: ", item.name);
}
}
// check the item type
string path = "";
if (!unwanted) {
if (isItemFile(driveItem)) {
log.vdebug("The item we are syncing is a file");
@ -1879,7 +1910,9 @@ final class SyncEngine
assert(isItemFolder(driveItem["remoteItem"]), "The remote item is not a folder");
} else {
// Why was this unwanted?
path = itemdb.computePath(item.driveId, item.parentId) ~ "/" ~ item.name;
if (path.empty) {
path = itemdb.computePath(item.driveId, item.parentId) ~ "/" ~ item.name;
}
// Microsoft OneNote container objects present as neither folder or file but has file size
if ((!isItemFile(driveItem)) && (!isItemFolder(driveItem)) && (hasFileSize(driveItem))) {
// Log that this was skipped as this was a Microsoft OneNote item and unsupported
@ -2171,8 +2204,40 @@ final class SyncEngine
}
}
} else {
// path does not exist locally - this will be a new file download or folder creation
// Should this 'download' be skipped?
// Path does not exist locally - this will be a new file download or folder creation
// Should this 'download' be skipped due to 'skip_dir' directive
if (cfg.getValueString("skip_dir") != "") {
string pathToCheck;
// does the path start with '/'?
if (!startsWith(path, "/")){
// path does not start with '/', but we need to check skip_dir entries with and without '/'
// so always make sure we are checking a path with '/'
// If this is a file, we need to check the parent path
if (item.type == ItemType.file) {
// use parent path and add '/'
pathToCheck = '/' ~ dirName(path);
} else {
// use path and add '/'
pathToCheck = '/' ~ path;
}
}
// perform the check
if (selectiveSync.isDirNameExcluded(pathToCheck)) {
// this path should be skipped
if (item.type == ItemType.file) {
log.vlog("Skipping item - file path is excluded by skip_dir config: ", path);
} else {
log.vlog("Skipping item - excluded by skip_dir config: ", path);
}
// flag that this download failed, otherwise the 'item' is added to the database - then, as not present on the local disk, would get deleted from OneDrive
downloadFailed = true;
return;
}
}
// Should this 'download' be skipped due to nosync directive?
// Do we need to check for .nosync? Only if --check-for-nosync was passed in
if (cfg.getValueBool("check_nosync")) {
// need the parent path for this object
@ -2733,21 +2798,29 @@ final class SyncEngine
}
}
log.vlog("Processing ", item.name);
bool unwanted = false;
string path;
// Is the path excluded?
unwanted = selectiveSync.isDirNameExcluded(item.name);
// Compute this item path early as we we use this path often
path = itemdb.computePath(item.driveId, item.id);
log.vlog("Processing ", buildNormalizedPath(path));
// If the path is not excluded, is the filename excluded?
if (!unwanted) {
// What type of DB item are we processing
// Is this item excluded by user configuration of skip_dir or skip_file?
// Is this item a directory or 'remote' type? A 'remote' type is a folder DB tie so should be compared as directory for exclusion
if ((item.type == ItemType.dir)||(item.type == ItemType.remote)) {
// Is the path excluded?
unwanted = selectiveSync.isDirNameExcluded(item.name);
}
// Is this item a file?
if (item.type == ItemType.file) {
// Is the filename excluded?
unwanted = selectiveSync.isFileNameExcluded(item.name);
}
// If path or filename does not exclude, is this excluded due to use of selective sync?
if (!unwanted) {
path = itemdb.computePath(item.driveId, item.id);
// Is the path excluded via sync_list?
unwanted = selectiveSync.isPathExcludedViaSyncList(path);
}
@ -3307,9 +3380,8 @@ final class SyncEngine
return;
}
// Is the path length is less than maxPathLength
if(pathWalkLength < maxPathLength){
// path length is less than maxPathLength
// skip dot files if configured
if (cfg.getValueBool("skip_dotfiles")) {
if (isDotFile(path)) {
@ -3326,6 +3398,7 @@ final class SyncEngine
}
}
// Is the path a symbolic link
if (isSymlink(path)) {
// if config says so we skip all symlinked items
if (cfg.getValueBool("skip_symlinks")) {
@ -3386,7 +3459,9 @@ final class SyncEngine
log.vdebug("Checking local path: ", path);
// Only check path if config is != ""
if (cfg.getValueString("skip_dir") != "") {
if (selectiveSync.isDirNameExcluded(path.strip('.').strip('/'))) {
// The path that needs to be checked needs to include the '/'
// This due to if the user has specified in skip_dir an exclusive path: '/path' - that is what must be matched
if (selectiveSync.isDirNameExcluded(path.strip('.'))) {
log.vlog("Skipping item - excluded by skip_dir config: ", path);
return;
}
@ -3398,6 +3473,7 @@ final class SyncEngine
// is added again after this sync
if ((exists(cfg.businessSharedFolderFilePath)) && (!syncBusinessFolders)){
// business_shared_folders file exists, but we are not using / syncing them
// The file contents can only contain 'folder' names, so we need to strip './' from any path we are checking
if(selectiveSync.isSharedFolderMatched(strip(path,"./"))){
// path detected as a 'new item' is matched as a path in business_shared_folders
log.vlog("Skipping item - excluded as included in business_shared_folders config: ", path);
@ -3406,13 +3482,17 @@ final class SyncEngine
}
}
}
if (isFile(path)) {
log.vdebug("Checking file: ", path);
if (selectiveSync.isFileNameExcluded(path.strip('.').strip('/'))) {
// The path that needs to be checked needs to include the '/'
// This due to if the user has specified in skip_file an exclusive path: '/path/file' - that is what must be matched
if (selectiveSync.isFileNameExcluded(path.strip('.'))) {
log.vlog("Skipping item - excluded by skip_file config: ", path);
return;
}
}
if (selectiveSync.isPathExcludedViaSyncList(path)) {
if ((isFile(path)) && (cfg.getValueBool("sync_root_files")) && (rootName(path.strip('.').strip('/')) == "")) {
log.vdebug("Not skipping path due to sync_root_files inclusion: ", path);
@ -3469,55 +3549,64 @@ final class SyncEngine
return;
}
} else {
bool fileFoundInDB = false;
// This item is a file
long fileSize = getSize(path);
// Can we upload this file - is there enough free space? - https://github.com/skilion/onedrive/issues/73
// However if the OneDrive account does not provide the quota details, we have no idea how much free space is available
if ((!quotaAvailable) || ((remainingFreeSpace - fileSize) > 0)){
if (!quotaAvailable) {
log.vlog("Ignoring OneDrive account quota details to upload file - this may fail if not enough space on OneDrive ..");
}
Item item;
foreach (driveId; driveIDsArray) {
if (itemdb.selectByPath(path, driveId, item)) {
fileFoundInDB = true;
// path is not a directory, is it a valid file?
// pipes - whilst technically valid files, are not valid for this client
// prw-rw-r--. 1 user user 0 Jul 7 05:55 my_pipe
if (isFile(path)) {
// Path is a valid file
bool fileFoundInDB = false;
// This item is a file
long fileSize = getSize(path);
// Can we upload this file - is there enough free space? - https://github.com/skilion/onedrive/issues/73
// However if the OneDrive account does not provide the quota details, we have no idea how much free space is available
if ((!quotaAvailable) || ((remainingFreeSpace - fileSize) > 0)){
if (!quotaAvailable) {
log.vlog("Ignoring OneDrive account quota details to upload file - this may fail if not enough space on OneDrive ..");
}
Item item;
foreach (driveId; driveIDsArray) {
if (itemdb.selectByPath(path, driveId, item)) {
fileFoundInDB = true;
}
}
}
// Was the file found in the database?
if (!fileFoundInDB) {
// File not found in database when searching all drive id's, upload as new file
uploadNewFile(path);
// did the upload fail?
if (!uploadFailed) {
// upload did not fail
// Issue #763 - Delete local files after sync handling
// are we in an --upload-only scenario?
if (uploadOnly) {
// are we in a delete local file after upload?
if (localDeleteAfterUpload) {
// Log that we are deleting a local item
log.log("Removing local file as --upload-only & --remove-source-files configured");
// are we in a --dry-run scenario?
if (!dryRun) {
// No --dry-run ... process local file delete
log.vdebug("Removing local file: ", path);
safeRemove(path);
// Was the file found in the database?
if (!fileFoundInDB) {
// File not found in database when searching all drive id's, upload as new file
uploadNewFile(path);
// did the upload fail?
if (!uploadFailed) {
// upload did not fail
// Issue #763 - Delete local files after sync handling
// are we in an --upload-only scenario?
if (uploadOnly) {
// are we in a delete local file after upload?
if (localDeleteAfterUpload) {
// Log that we are deleting a local item
log.log("Removing local file as --upload-only & --remove-source-files configured");
// are we in a --dry-run scenario?
if (!dryRun) {
// No --dry-run ... process local file delete
log.vdebug("Removing local file: ", path);
safeRemove(path);
}
}
}
// how much space is left on OneDrive after upload?
remainingFreeSpace = (remainingFreeSpace - fileSize);
log.vlog("Remaining free space on OneDrive: ", remainingFreeSpace);
}
// how much space is left on OneDrive after upload?
remainingFreeSpace = (remainingFreeSpace - fileSize);
log.vlog("Remaining free space on OneDrive: ", remainingFreeSpace);
}
} else {
// Not enough free space
log.log("Skipping item '", path, "' due to insufficient free space available on OneDrive");
}
} else {
// Not enough free space
log.log("Skipping item '", path, "' due to insufficient free space available on OneDrive");
}
// path is not a valid file
log.log("Skipping item - item is not a valid file: ", path);
}
}
} else {
// This path was skipped - why?
@ -3760,14 +3849,9 @@ final class SyncEngine
// If performing a dry-run or parent path is found in the database
if ((dryRun) || (parentPathFoundInDB)) {
// Maximum file size upload
// https://support.microsoft.com/en-au/help/3125202/restrictions-and-limitations-when-you-sync-files-and-folders
// 1. OneDrive Business say's 15GB
// 2. Another article updated April 2018 says 20GB:
// https://answers.microsoft.com/en-us/onedrive/forum/odoptions-oddesktop-sdwin10/personal-onedrive-file-upload-size-max/a3621fc9-b766-4a99-99f8-bcc01ccb025f
// Use smaller size for now
auto maxUploadFileSize = 16106127360; // 15GB
//auto maxUploadFileSize = 21474836480; // 20GB
// https://support.microsoft.com/en-us/office/invalid-file-names-and-file-types-in-onedrive-and-sharepoint-64883a5d-228e-48f5-b3d2-eb39e07630fa?ui=en-us&rs=en-us&ad=us
// July 2020, maximum file size for all accounts is 100GB
auto maxUploadFileSize = 107374182400; // 100GB
auto thisFileSize = getSize(path);
// To avoid a 409 Conflict error - does the file actually exist on OneDrive already?
JSONValue fileDetailsFromOneDrive;