deblan-mirror/abraunegg-onedrive
deblan-mirror/abraunegg-onedrive
1
0
Fork 0
mirror of https://github.com/abraunegg/onedrive synced 2024-04-28 12:12:51 +02:00
Code Issues Projects Releases Wiki Activity

Implement OneDrive Business Shared Folders Support (Issue #459) (#473)

Browse source 
* Implement OneDrive Business Shared Folders Support
This commit is contained in:
abraunegg 2020-06-27 19:10:37 +10:00 committed by GitHub
parent 650cb97d3f
commit 9cc72c2396
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
11 changed files with 1233 additions and 340 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
Makefile.in
View file

@ -54,7 +54,7 @@ endif
system_unit_files = contrib/systemd/onedrive@.service
user_unit_files = contrib/systemd/onedrive.service
DOCFILES = README.md config LICENSE CHANGELOG.md docs/Docker.md docs/INSTALL.md docs/Office365.md docs/USAGE.md
DOCFILES = README.md config LICENSE CHANGELOG.md docs/Docker.md docs/INSTALL.md docs/Office365.md docs/USAGE.md docs/BusinessSharedFolders.md
ifneq ("$(wildcard /etc/redhat-release)","")
RHEL = $(shell cat /etc/redhat-release | grep -E "(Red Hat Enterprise Linux Server|CentOS)" | wc -l)

7
README.md
View file

@ -17,8 +17,8 @@ This client is a 'fork' of the [skilion](https://github.com/skilion/onedrive) cl
* File upload / download validation to ensure data integrity
* Resumable uploads
* Support OneDrive for Business (part of Office 365)
* Shared folders (OneDrive Personal)
* SharePoint / Office 365 Shared Libraries
* Shared Folder support for OneDrive Personal and OneDrive Business accounts
* SharePoint / Office365 Shared Libraries
* Desktop notifications via libnotify
* Dry-run capability to test configuration changes
* Prevent major OneDrive accidental data deletion after configuration change
@ -37,6 +37,9 @@ See [docs/USAGE.md](https://github.com/abraunegg/onedrive/blob/master/docs/USAGE
## Docker support
See [docs/Docker.md](https://github.com/abraunegg/onedrive/blob/master/docs/Docker.md)
## OneDrive Business Shared Folders
See [docs/BusinessSharedFolders.md](https://github.com/abraunegg/onedrive/blob/master/docs/docs/BusinessSharedFolders.md)
## SharePoint / Office 365 Shared Libraries (Business or Education)
See [docs/Office365.md](https://github.com/abraunegg/onedrive/blob/master/docs/Office365.md)

2
config
View file

@ -37,3 +37,5 @@
# application_id = ""
# resync = "false"
# bypass_data_preservation = "false"
# azure_ad_endpoint = ""
# sync_business_shared_folders = "false"

187
docs/BusinessSharedFolders.md Normal file
View file

@ -0,0 +1,187 @@
# How to configure OneDrive Business Shared Folder Sync
Syncing OneDrive Business Shared Folders requires additional configuration for your 'onedrive' client:
1. List available shared folders to determine which folder you wish to sync & to validate that you have access to that folder
2. Create a new file called 'business_shared_folders' in your config directory which contains a list of the shared folders you wish to sync
3. Perform a sync
## Listing available OneDrive Business Shared Folders
List the available OneDrive Business Shared folders with the following command:
```text
onedrive --list-shared-folders
```
This will return a listing of all OneDrive Business Shared folders which have been shared with you and by whom. This is important for conflict resolution:
```text
Initializing the Synchronization Engine ...
Listing available OneDrive Business Shared Folders:
---------------------------------------
Shared Folder: SharedFolder0
Shared By: Firstname Lastname
---------------------------------------
Shared Folder: SharedFolder1
Shared By: Firstname Lastname
---------------------------------------
Shared Folder: SharedFolder2
Shared By: Firstname Lastname
---------------------------------------
Shared Folder: SharedFolder0
Shared By: Firstname Lastname (user@domain)
---------------------------------------
Shared Folder: SharedFolder1
Shared By: Firstname Lastname (user@domain)
---------------------------------------
Shared Folder: SharedFolder2
Shared By: Firstname Lastname (user@domain)
...
```
## Configuring OneDrive Business Shared Folders
1. Create a new file called 'business_shared_folders' in your config directory
2. On each new line, list the OneDrive Business Shared Folder you wish to sync
```text
[alex@centos7full onedrive]$ cat ~/.config/onedrive/business_shared_folders
# comment
Child Shared Folder
# Another comment
Top Level to Share
[alex@centos7full onedrive]$
```
3. Validate your configuration with `onedrive --display-config`:
```text
Configuration file successfully loaded
onedrive version = v2.4.3
Config path = /home/alex/.config/onedrive-business/
Config file found in config path = true
Config option 'check_nosync' = false
Config option 'sync_dir' = /home/alex/OneDriveBusiness
Config option 'skip_dir' =
Config option 'skip_file' = ~*|.~*|*.tmp
Config option 'skip_dotfiles' = false
Config option 'skip_symlinks' = false
Config option 'monitor_interval' = 300
Config option 'min_notify_changes' = 5
Config option 'log_dir' = /var/log/onedrive/
Config option 'classify_as_big_delete' = 1000
Config option 'sync_root_files' = false
Selective sync 'sync_list' configured = false
Business Shared Folders configured = true
business_shared_folders contents:
# comment
Child Shared Folder
# Another comment
Top Level to Share
```
## Performing a sync of OneDrive Business Shared Folders
Perform a standalone sync using the following command: `onedrive --synchronize --sync-shared-folders --verbose`:
```text
onedrive --synchronize --sync-shared-folders --verbose
Using 'user' Config Dir: /home/alex/.config/onedrive-business/
Using 'system' Config Dir:
Configuration file successfully loaded
Initializing the OneDrive API ...
Configuring Global Azure AD Endpoints
Opening the item database ...
All operations will be performed in: /home/alex/OneDriveBusiness
Application version: v2.4.3
Account Type: business
Default Drive ID: b!bO8V7s9SSk6r7mWHpIjURotN33W1W2tEv3OXV_oFIdQimEdOHR-1So7CqeT1MfHA
Default Root ID: 01WIXGO5V6Y2GOVW7725BZO354PWSELRRZ
Remaining Free Space: 1098316220277
Fetching details for OneDrive Root
OneDrive Root exists in the database
Initializing the Synchronization Engine ...
Syncing changes from OneDrive ...
Applying changes of Path ID: 01WIXGO5V6Y2GOVW7725BZO354PWSELRRZ
Number of items from OneDrive to process: 0
Attempting to sync OneDrive Business Shared Folders
Syncing this OneDrive Business Shared Folder: Child Shared Folder
OneDrive Business Shared Folder - Shared By: test user
Applying changes of Path ID: 01JRXHEZMREEB3EJVHNVHKNN454Q7DFXPR
Adding OneDrive root details for processing
Adding OneDrive folder details for processing
Adding 4 OneDrive items for processing from OneDrive folder
Adding 2 OneDrive items for processing from /Child Shared Folder/Cisco VDI Whitepaper
Adding 2 OneDrive items for processing from /Child Shared Folder/SMPP_Shared
Processing 11 OneDrive items to ensure consistent local state
Syncing this OneDrive Business Shared Folder: Top Level to Share
OneDrive Business Shared Folder - Shared By: test user (testuser@mynasau3.onmicrosoft.com)
Applying changes of Path ID: 01JRXHEZLRMXHKBYZNOBF3TQOPBXS3VZMA
Adding OneDrive root details for processing
Adding OneDrive folder details for processing
Adding 4 OneDrive items for processing from OneDrive folder
Adding 3 OneDrive items for processing from /Top Level to Share/10-Files
Adding 2 OneDrive items for processing from /Top Level to Share/10-Files/Cisco VDI Whitepaper
Adding 2 OneDrive items for processing from /Top Level to Share/10-Files/Images
Adding 8 OneDrive items for processing from /Top Level to Share/10-Files/Images/JPG
Adding 8 OneDrive items for processing from /Top Level to Share/10-Files/Images/PNG
Adding 2 OneDrive items for processing from /Top Level to Share/10-Files/SMPP
Processing 31 OneDrive items to ensure consistent local state
Uploading differences of ~/OneDriveBusiness
Processing root
The directory has not changed
Processing SMPP_Local
The directory has not changed
Processing SMPP-IF-SPEC_v3_3-24858.pdf
The file has not changed
Processing SMPP_v3_4_Issue1_2-24857.pdf
The file has not changed
Processing new_local_file.txt
The file has not changed
Processing root
The directory has not changed
...
The directory has not changed
Processing week02-03-Combinational_Logic-v1.pptx
The file has not changed
Uploading new items of ~/OneDriveBusiness
Applying changes of Path ID: 01WIXGO5V6Y2GOVW7725BZO354PWSELRRZ
Number of items from OneDrive to process: 0
Attempting to sync OneDrive Business Shared Folders
Syncing this OneDrive Business Shared Folder: Child Shared Folder
OneDrive Business Shared Folder - Shared By: test user
Applying changes of Path ID: 01JRXHEZMREEB3EJVHNVHKNN454Q7DFXPR
Adding OneDrive root details for processing
Adding OneDrive folder details for processing
Adding 4 OneDrive items for processing from OneDrive folder
Adding 2 OneDrive items for processing from /Child Shared Folder/Cisco VDI Whitepaper
Adding 2 OneDrive items for processing from /Child Shared Folder/SMPP_Shared
Processing 11 OneDrive items to ensure consistent local state
Syncing this OneDrive Business Shared Folder: Top Level to Share
OneDrive Business Shared Folder - Shared By: test user (testuser@mynasau3.onmicrosoft.com)
Applying changes of Path ID: 01JRXHEZLRMXHKBYZNOBF3TQOPBXS3VZMA
Adding OneDrive root details for processing
Adding OneDrive folder details for processing
Adding 4 OneDrive items for processing from OneDrive folder
Adding 3 OneDrive items for processing from /Top Level to Share/10-Files
Adding 2 OneDrive items for processing from /Top Level to Share/10-Files/Cisco VDI Whitepaper
Adding 2 OneDrive items for processing from /Top Level to Share/10-Files/Images
Adding 8 OneDrive items for processing from /Top Level to Share/10-Files/Images/JPG
Adding 8 OneDrive items for processing from /Top Level to Share/10-Files/Images/PNG
Adding 2 OneDrive items for processing from /Top Level to Share/10-Files/SMPP
Processing 31 OneDrive items to ensure consistent local state
```
**Note:** Whenever you modify the `business_shared_folders` file you must perform a `--resync` of your database to clean up stale entries due to changes in your configuration.
## Enable / Disable syncing of OneDrive Business Shared Folders
Performing a sync of the configured OneDrive Business Shared Folders can be enabled / disabled via adding the following to your configuration file.
### Enable syncing of OneDrive Business Shared Folders via config file
```text
sync_business_shared_folders = "true"
```
### Disable syncing of OneDrive Business Shared Folders via config file
```text
sync_business_shared_folders = "false"
```
## Known Issues
Shared folders, shared with you from people outside of your 'organisation' are unable to be synced. This is due to the Microsoft Graph API not presenting these folders.
Shared folders that match this scenario, when you view 'Shared' via OneDrive online, will have a 'world' symbol as per below:
![shared_with_me](./images/shared_with_me.jpg)
This issue is being tracked by: [#966](https://github.com/abraunegg/onedrive/issues/966)

BIN
docs/images/shared_with_me.JPG Normal file
View file

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 45 KiB

22
src/config.d
View file

@ -23,6 +23,7 @@ final class Config
public string configFileSyncDir = "";
public string configFileSkipFile = "";
public string configFileSkipDir = "";
public string businessSharedFolderFilePath = "";
private string userConfigFilePath = "";
private string systemConfigFilePath = "";
// was the application just authorised - paste of response uri
@ -33,8 +34,9 @@ final class Config
private string[string] stringValues;
private bool[string] boolValues;
private long[string] longValues;
// Compile time regex - this does not change
public auto configRegex = ctRegex!(`^(\w+)\s*=\s*"(.*)"\s*$`);
this(string confdirOption)
{
// default configuration - entries in config file ~/.config/onedrive/config
@ -100,6 +102,8 @@ final class Config
// AD Endpoint: https://login.chinacloudapi.cn
// Graph Endpoint: https://microsoftgraph.chinacloudapi.cn
stringValues["azure_ad_endpoint"] = "";
// Allow enable / disable of the syncing of OneDrive Business Shared Folders via configuration file
boolValues["sync_business_shared_folders"] = false;
// DEVELOPER OPTIONS
// display_memory = true | false
@ -189,6 +193,7 @@ final class Config
userConfigFilePath = buildNormalizedPath(configDirName ~ "/config");
syncListFilePath = buildNormalizedPath(configDirName ~ "/sync_list");
systemConfigFilePath = buildNormalizedPath(systemConfigDirName ~ "/config");
businessSharedFolderFilePath = buildNormalizedPath(configDirName ~ "/business_shared_folders");
// Debug Output for application set variables based on configDirName
log.vdebug("refreshTokenFilePath = ", refreshTokenFilePath);
@ -199,6 +204,7 @@ final class Config
log.vdebug("userConfigFilePath = ", userConfigFilePath);
log.vdebug("syncListFilePath = ", syncListFilePath);
log.vdebug("systemConfigFilePath = ", systemConfigFilePath);
log.vdebug("businessSharedFolderFilePath = ", businessSharedFolderFilePath);
}
bool initialize()
@ -259,13 +265,16 @@ final class Config
boolValues["force"] = false;
boolValues["remove_source_files"] = false;
boolValues["skip_dir_strict_match"] = false;
boolValues["list_business_shared_folders"] = false;
// Application Startup option validation
try {
string tmpStr;
bool tmpBol;
long tmpVerb;
// duplicated from main.d to get full help output!
auto opt = getopt(
args,
std.getopt.config.bundling,
std.getopt.config.caseSensitive,
@ -404,7 +413,6 @@ final class Config
"user-agent",
"Specify a User Agent string to the http client",
&stringValues["user_agent"],
// duplicated from main.d to get full help output!
"confdir",
"Set the directory used to store the configuration files",
&tmpStr,
@ -413,7 +421,13 @@ final class Config
&tmpVerb,
"version",
"Print the version and exit",
&tmpBol
&tmpBol,
"list-shared-folders",
"List OneDrive Business Shared Folders",
&boolValues["list_business_shared_folders"],
"sync-shared-folders",
"Sync OneDrive Business Shared Folders",
&boolValues["sync_business_shared_folders"]
);
if (opt.helpWanted) {
outputLongHelp(opt.options);

43
src/itemdb.d
View file

@ -2,6 +2,8 @@ import std.datetime;
import std.exception;
import std.path;
import std.string;
import std.stdio;
import std.algorithm.searching;
import core.stdc.stdlib;
import sqlite;
static import log;
@ -368,9 +370,14 @@ final class ItemDatabase
if (r2.empty) {
// root reached
assert(path.length >= 4);
// remove "root"
if (path.length >= 5) path = path[5 .. $];
else path = path[4 .. $];
// remove "root/" from path string if it exists
if (path.length >= 5) {
if (canFind(path, "root/")){
path = path[5 .. $];
}
} else {
path = path[4 .. $];
}
// special case of computing the path of the root itself
if (path.length == 0) path = ".";
break;
@ -427,17 +434,39 @@ final class ItemDatabase
// As we query /children to get all children from OneDrive, update anything in the database
// to be flagged as not-in-sync, thus, we can use that flag to determing what was previously
// in-sync, but now deleted on OneDrive
void downgradeSyncStatusFlag()
void downgradeSyncStatusFlag(const(char)[] driveId, const(char)[] id)
{
db.exec("UPDATE item SET syncStatus = 'N'");
assert(driveId);
auto stmt = db.prepare("UPDATE item SET syncStatus = 'N' WHERE driveId = ?1 AND id = ?2");
stmt.bind(1, driveId);
stmt.bind(2, id);
stmt.exec();
}
// National Cloud Deployments (US and DE) do not support /delta as a query
// Select items that have a out-of-sync flag set
Item[] selectOutOfSyncItems()
Item[] selectOutOfSyncItems(const(char)[] driveId)
{
assert(driveId);
Item[] items;
auto stmt = db.prepare("SELECT * FROM item WHERE syncStatus = 'N'");
auto stmt = db.prepare("SELECT * FROM item WHERE syncStatus = 'N' AND driveId = ?1");
stmt.bind(1, driveId);
auto res = stmt.exec();
while (!res.empty) {
items ~= buildItem(res);
res.step();
}
return items;
}
// OneDrive Business Folders are stored in the database potentially without a root | parentRoot link
// Select items associated with the provided driveId
Item[] selectByDriveId(const(char)[] driveId)
{
assert(driveId);
Item[] items;
auto stmt = db.prepare("SELECT * FROM item WHERE driveId = ?1 AND parentId IS NULL");
stmt.bind(1, driveId);
auto res = stmt.exec();
while (!res.empty) {
items ~= buildItem(res);

121
src/main.d
View file

@ -26,6 +26,7 @@ int main(string[] args)
string configFilePath;
string syncListFilePath;
string databaseFilePath;
string businessSharedFolderFilePath;
string currentConfigHash;
string currentSyncListHash;
string previousConfigHash;
@ -35,7 +36,11 @@ int main(string[] args)
string configBackupFile;
string syncDir;
string logOutputMessage;
string currentBusinessSharedFoldersHash;
string previousBusinessSharedFoldersHash;
string businessSharedFoldersHashFile;
bool configOptionsDifferent = false;
bool businessSharedFoldersDifferent = false;
bool syncListConfigured = false;
bool syncListDifferent = false;
bool syncDirDifferent = false;
@ -144,6 +149,7 @@ int main(string[] args)
configFilePath = buildNormalizedPath(cfg.configDirName ~ "/config");
syncListFilePath = buildNormalizedPath(cfg.configDirName ~ "/sync_list");
databaseFilePath = buildNormalizedPath(cfg.configDirName ~ "/items.db");
businessSharedFolderFilePath = buildNormalizedPath(cfg.configDirName ~ "/business_shared_folders");
// Has any of our configuration that would require a --resync been changed?
// 1. sync_list file modification
@ -152,6 +158,7 @@ int main(string[] args)
configHashFile = buildNormalizedPath(cfg.configDirName ~ "/.config.hash");
syncListHashFile = buildNormalizedPath(cfg.configDirName ~ "/.sync_list.hash");
configBackupFile = buildNormalizedPath(cfg.configDirName ~ "/.config.backup");
businessSharedFoldersHashFile = buildNormalizedPath(cfg.configDirName ~ "/.business_shared_folders.hash");
// Does a config file exist with a valid hash file
if ((exists(configFilePath)) && (!exists(configHashFile))) {
@ -165,6 +172,12 @@ int main(string[] args)
std.file.write(syncListHashFile, computeQuickXorHash(syncListFilePath));
}
// check if business_shared_folders & business_shared_folders hash exists
if ((exists(businessSharedFolderFilePath)) && (!exists(businessSharedFoldersHashFile))) {
// Hash of business_shared_folders file needs to be created
std.file.write(businessSharedFoldersHashFile, computeQuickXorHash(businessSharedFolderFilePath));
}
// If hash files exist, but config files do not ... remove the hash, but only if --resync was issued as now the application will use 'defaults' which 'may' be different
if ((!exists(configFilePath)) && (exists(configHashFile))) {
// if --resync safe remove config.hash and config.backup
@ -180,11 +193,18 @@ int main(string[] args)
if (cfg.getValueBool("resync")) safeRemove(syncListHashFile);
}
if ((!exists(businessSharedFolderFilePath)) && (exists(businessSharedFoldersHashFile))) {
// if --resync safe remove business_shared_folders.hash
if (cfg.getValueBool("resync")) safeRemove(businessSharedFoldersHashFile);
}
// Read config hashes if they exist
if (exists(configFilePath)) currentConfigHash = computeQuickXorHash(configFilePath);
if (exists(syncListFilePath)) currentSyncListHash = computeQuickXorHash(syncListFilePath);
if (exists(businessSharedFolderFilePath)) currentBusinessSharedFoldersHash = computeQuickXorHash(businessSharedFolderFilePath);
if (exists(configHashFile)) previousConfigHash = readText(configHashFile);
if (exists(syncListHashFile)) previousSyncListHash = readText(syncListHashFile);
if (exists(businessSharedFoldersHashFile)) previousBusinessSharedFoldersHash = readText(businessSharedFoldersHashFile);
// Was sync_list file updated?
if (currentSyncListHash != previousSyncListHash) {
@ -193,6 +213,13 @@ int main(string[] args)
syncListDifferent = true;
}
// Was business_shared_folders updated?
if (currentBusinessSharedFoldersHash != previousBusinessSharedFoldersHash) {
// Debugging output to assist what changed
log.vdebug("business_shared_folders file has been updated, --resync needed");
businessSharedFoldersDifferent = true;
}
// Was config file updated between last execution ang this execution?
if (currentConfigHash != previousConfigHash) {
// config file was updated, however we only want to trigger a --resync requirement if sync_dir, skip_dir, skip_file or drive_id was modified
@ -315,7 +342,7 @@ int main(string[] args)
}
// Has anything triggered a --resync requirement?
if (configOptionsDifferent || syncListDifferent || syncDirDifferent || skipFileDifferent || skipDirDifferent) {
if (configOptionsDifferent || syncListDifferent || syncDirDifferent || skipFileDifferent || skipDirDifferent || businessSharedFoldersDifferent) {
// --resync needed, is the user just testing configuration changes?
if (!cfg.getValueBool("display_config")){
// not testing configuration changes
@ -340,6 +367,11 @@ int main(string[] args)
log.vdebug("updating sync_list hash as --resync issued");
std.file.write(syncListHashFile, computeQuickXorHash(syncListFilePath));
}
if (exists(businessSharedFolderFilePath)) {
// update business_shared_folders hash
log.vdebug("updating business_shared_folders hash as --resync issued");
std.file.write(businessSharedFoldersHashFile, computeQuickXorHash(businessSharedFolderFilePath));
}
}
}
}
@ -438,10 +470,8 @@ int main(string[] args)
if (cfg.getValueBool("display_config")){
// Display application version
writeln("onedrive version = ", strip(import("version")));
// Display all of the pertinent configuration options
writeln("Config path = ", cfg.configDirName);
// Does a config file exist or are we using application defaults
writeln("Config file found in config path = ", exists(configFilePath));
@ -465,7 +495,7 @@ int main(string[] args)
// Is sync_list configured?
if (exists(syncListFilePath)){
writeln("Config option 'sync_root_files' = ", cfg.getValueBool("sync_root_files"));
writeln("Selective sync configured = true");
writeln("Selective sync 'sync_list' configured = true");
writeln("sync_list contents:");
// Output the sync_list contents
auto syncListFile = File(syncListFilePath);
@ -476,10 +506,25 @@ int main(string[] args)
}
} else {
writeln("Config option 'sync_root_files' = ", cfg.getValueBool("sync_root_files"));
writeln("Selective sync configured = false");
writeln("Selective sync 'sync_list' configured = false");
}
// exit
// Is business_shared_folders configured
if (exists(businessSharedFolderFilePath)){
writeln("Business Shared Folders configured = true");
writeln("business_shared_folders contents:");
// Output the business_shared_folders contents
auto businessSharedFolderFileList = File(businessSharedFolderFilePath);
auto range = businessSharedFolderFileList.byLine();
foreach (line; range)
{
writeln(line);
}
} else {
writeln("Business Shared Folders configured = false");
}
// Exit
return EXIT_SUCCESS;
}
@ -518,9 +563,9 @@ int main(string[] args)
performSyncOK = true;
}
// create-directory, remove-directory, source-directory, destination-directory
// create-directory, remove-directory, source-directory, destination-directory
// these are activities that dont perform a sync, so to not generate an error message for these items either
if (((cfg.getValueString("create_directory") != "") || (cfg.getValueString("remove_directory") != "")) || ((cfg.getValueString("source_directory") != "") && (cfg.getValueString("destination_directory") != "")) || (cfg.getValueString("get_file_link") != "") || (cfg.getValueString("get_o365_drive_id") != "") || cfg.getValueBool("display_sync_status")) {
if (((cfg.getValueString("create_directory") != "") || (cfg.getValueString("remove_directory") != "")) || ((cfg.getValueString("source_directory") != "") && (cfg.getValueString("destination_directory") != "")) || (cfg.getValueString("get_file_link") != "") || (cfg.getValueString("get_o365_drive_id") != "") || cfg.getValueBool("display_sync_status") || cfg.getValueBool("list_business_shared_folders")) {
performSyncOK = true;
}
@ -580,11 +625,13 @@ int main(string[] args)
// Configure selective sync by parsing and getting a regex for skip_file config component
auto selectiveSync = new SelectiveSync();
if (exists(cfg.syncListFilePath)){
// load sync_list if it exists
if (exists(syncListFilePath)){
log.vdebug("Loading user configured sync_list file ...");
syncListConfigured = true;
// list what will be synced
auto syncListFile = File(cfg.syncListFilePath);
auto syncListFile = File(syncListFilePath);
auto range = syncListFile.byLine();
foreach (line; range)
{
@ -596,7 +643,20 @@ int main(string[] args)
syncListFile.close();
}
}
selectiveSync.load(cfg.syncListFilePath);
selectiveSync.load(syncListFilePath);
// load business_shared_folders if it exists
if (exists(businessSharedFolderFilePath)){
log.vdebug("Loading user configured business_shared_folders file ...");
// list what will be synced
auto businessSharedFolderFileList = File(businessSharedFolderFilePath);
auto range = businessSharedFolderFileList.byLine();
foreach (line; range)
{
log.vdebug("business_shared_folders: ", line);
}
}
selectiveSync.loadSharedFolders(businessSharedFolderFilePath);
// Configure skip_dir, skip_file, skip-dir-strict-match & skip_dotfiles from config entries
// Handle skip_dir configuration in config file
@ -725,11 +785,42 @@ int main(string[] args)
// Are we obtaining the Office 365 Drive ID for a given Office 365 SharePoint Shared Library?
if (cfg.getValueString("get_o365_drive_id") != "") {
sync.querySiteCollectionForDriveID(cfg.getValueString("get_o365_drive_id"));
// Exit application
// Use exit scopes to shutdown API
return EXIT_SUCCESS;
}
// Are we obtaining the URL path for a synced file?
if (cfg.getValueString("get_file_link") != "") {
sync.queryOneDriveForFileURL(cfg.getValueString("get_file_link"), syncDir);
// Exit application
// Use exit scopes to shutdown API
return EXIT_SUCCESS;
}
// Are we listing OneDrive Business Shared Folders
if (cfg.getValueBool("list_business_shared_folders")) {
// Is this a business account type?
if (sync.getAccountType() == "business"){
// List OneDrive Business Shared Folders
sync.listOneDriveBusinessSharedFolders();
} else {
log.error("ERROR: Unsupported account type for listing OneDrive Business Shared Folders");
}
// Exit application
// Use exit scopes to shutdown API
return EXIT_SUCCESS;
}
// Are we going to sync OneDrive Business Shared Folders
if (cfg.getValueBool("sync_business_shared_folders")) {
// Is this a business account type?
if (sync.getAccountType() == "business"){
// Configure flag to sync business folders
sync.setSyncBusinessFolders();
} else {
log.error("ERROR: Unsupported account type for syncing OneDrive Business Shared Folders");
}
}
// Are we displaying the sync status of the client?
@ -752,9 +843,9 @@ int main(string[] args)
if (cfg.getValueBool("synchronize")) {
if (online) {
// Check user entry for local path - the above chdir means we are already in ~/OneDrive/ thus singleDirectory is local to this path
if (cfg.getValueString("single_directory") != ""){
if (cfg.getValueString("single_directory") != "") {
// Does the directory we want to sync actually exist?
if (!exists(cfg.getValueString("single_directory"))){
if (!exists(cfg.getValueString("single_directory"))) {
// the requested directory does not exist ..
log.logAndNotify("ERROR: The requested local directory does not exist. Please check ~/OneDrive/ for requested path");
// Use exit scopes to shutdown API
@ -938,7 +1029,7 @@ int main(string[] args)
}
try {
// perform a --monitor sync
log.vlog("Starting a sync with OneDrive");
if (logMonitorCounter == logInterval) log.log("Starting a sync with OneDrive");
performSync(sync, cfg.getValueString("single_directory"), cfg.getValueBool("download_only"), cfg.getValueBool("local_first"), cfg.getValueBool("upload_only"), (logMonitorCounter == logInterval ? MONITOR_LOG_QUIET : MONITOR_LOG_SILENT), fullScanRequired, syncListConfiguredFullScanOverride, displaySyncOptions, cfg.getValueBool("monitor"), m);
if (!cfg.getValueBool("download_only")) {
// discard all events that may have been generated by the sync that have not already been handled
@ -949,7 +1040,7 @@ int main(string[] args)
log.error("ERROR: The following inotify error was generated: ", e.msg);
}
}
log.vlog("Sync with OneDrive is complete");
if (logMonitorCounter == logInterval) log.log("Sync with OneDrive is complete");
} catch (CurlException e) {
// we already tried three times in the performSync routine
// if we still have problems, then the sync handle might have

56
src/onedrive.d
View file

@ -60,6 +60,9 @@ private {
string driveUrl = globalGraphEndpoint ~ "/v1.0/me/drive";
string driveByIdUrl = globalGraphEndpoint ~ "/v1.0/drives/";
// What is 'shared with me' Query
string sharedWithMe = globalGraphEndpoint ~ "/v1.0/me/drive/sharedWithMe";
// Item Queries
string itemByIdUrl = globalGraphEndpoint ~ "/v1.0/me/drive/items/";
string itemByPathUrl = globalGraphEndpoint ~ "/v1.0/me/drive/root:/";
@ -156,6 +159,8 @@ final class OneDriveApi
// Office 365 / SharePoint Queries
siteSearchUrl = usl4GraphEndpoint ~ "/v1.0/sites?search";
siteDriveUrl = usl4GraphEndpoint ~ "/v1.0/sites/";
// Shared With Me
sharedWithMe = usl4GraphEndpoint ~ "/v1.0/me/drive/sharedWithMe";
break;
case "USL5":
log.log("Configuring Azure AD for US Government Endpoints (DOD)");
@ -172,6 +177,8 @@ final class OneDriveApi
// Office 365 / SharePoint Queries
siteSearchUrl = usl5GraphEndpoint ~ "/v1.0/sites?search";
siteDriveUrl = usl5GraphEndpoint ~ "/v1.0/sites/";
// Shared With Me
sharedWithMe = usl5GraphEndpoint ~ "/v1.0/me/drive/sharedWithMe";
break;
case "DE":
log.log("Configuring Azure AD Germany");
@ -188,6 +195,8 @@ final class OneDriveApi
// Office 365 / SharePoint Queries
siteSearchUrl = deGraphEndpoint ~ "/v1.0/sites?search";
siteDriveUrl = deGraphEndpoint ~ "/v1.0/sites/";
// Shared With Me
sharedWithMe = deGraphEndpoint ~ "/v1.0/me/drive/sharedWithMe";
break;
case "CN":
log.log("Configuring AD China operated by 21Vianet");
@ -204,6 +213,8 @@ final class OneDriveApi
// Office 365 / SharePoint Queries
siteSearchUrl = cnGraphEndpoint ~ "/v1.0/sites?search";
siteDriveUrl = cnGraphEndpoint ~ "/v1.0/sites/";
// Shared With Me
sharedWithMe = cnGraphEndpoint ~ "/v1.0/me/drive/sharedWithMe";
break;
// Default - all other entries
default:
@ -392,9 +403,25 @@ final class OneDriveApi
url = driveUrl ~ "/root";
return get(url);
}
// https://docs.microsoft.com/en-us/onedrive/developer/rest-api/api/driveitem_get
JSONValue getDriveIdRoot(const(char)[] driveId)
{
checkAccessTokenExpired();
const(char)[] url;
url = driveByIdUrl ~ driveId ~ "/root";
return get(url);
}
// https://docs.microsoft.com/en-us/graph/api/drive-sharedwithme
JSONValue getSharedWithMe()
{
checkAccessTokenExpired();
return get(sharedWithMe);
}
// https://docs.microsoft.com/en-us/onedrive/developer/rest-api/api/driveitem_delta
JSONValue viewChangesById(const(char)[] driveId, const(char)[] id, const(char)[] deltaLink)
JSONValue viewChangesByItemId(const(char)[] driveId, const(char)[] id, const(char)[] deltaLink)
{
checkAccessTokenExpired();
const(char)[] url;
@ -408,6 +435,18 @@ final class OneDriveApi
return get(url);
}
// https://docs.microsoft.com/en-us/onedrive/developer/rest-api/api/driveitem_delta
JSONValue viewChangesByDriveId(const(char)[] driveId, const(char)[] deltaLink)
{
checkAccessTokenExpired();
const(char)[] url = deltaLink;
if (url == null) {
url = driveByIdUrl ~ driveId ~ "/root/delta";
url ~= "?select=id,name,eTag,cTag,deleted,file,folder,root,fileSystemInfo,remoteItem,parentReference,size";
}
return get(url);
}
// https://docs.microsoft.com/en-us/onedrive/developer/rest-api/api/driveitem_list_children
JSONValue listChildren(const(char)[] driveId, const(char)[] id, const(char)[] nextLink)
{
@ -422,7 +461,7 @@ final class OneDriveApi
}
return get(url);
}
// https://docs.microsoft.com/en-us/onedrive/developer/rest-api/api/driveitem_get_content
void downloadById(const(char)[] driveId, const(char)[] id, string saveToPath, long fileSize)
{
@ -506,6 +545,19 @@ final class OneDriveApi
return get(url);
}
// Return the requested details of the specified path on the specified drive id
JSONValue getPathDetailsByDriveId(const(char)[] driveId, const(string) path)
{
checkAccessTokenExpired();
const(char)[] url;
// string driveByIdUrl = "https://graph.microsoft.com/v1.0/drives/";
// Required format: /drives/{drive-id}/root:/{item-path}
url = driveByIdUrl ~ driveId ~ "/root:/" ~ encodeComponent(path);
url ~= "?select=id,name,eTag,cTag,deleted,file,folder,root,fileSystemInfo,remoteItem,parentReference,size";
return get(url);
}
// Return the requested details of the specified id
// https://docs.microsoft.com/en-us/onedrive/developer/rest-api/api/driveitem_get
JSONValue getFileDetails(const(char)[] driveId, const(char)[] id)

425
src/selective.d
View file

@ -1,185 +1,240 @@
import std.algorithm;
import std.array;
import std.file;
import std.path;
import std.regex;
import std.stdio;
import util;
final class SelectiveSync
{
private string[] paths;
private Regex!char mask;
private Regex!char dirmask;
private bool skipDirStrictMatch = false;
private bool skipDotfiles = false;
void load(string filepath)
{
if (exists(filepath)) {
// open file as read only
auto file = File(filepath, "r");
auto range = file.byLine();
foreach (line; range) {
// Skip comments in file
if (line.length == 0 || line[0] == ';' || line[0] == '#') continue;
paths ~= buildNormalizedPath(line);
}
file.close();
}
}
// Configure skipDirStrictMatch if function is called
// By default, skipDirStrictMatch = false;
void setSkipDirStrictMatch()
{
skipDirStrictMatch = true;
}
void setFileMask(const(char)[] mask)
{
this.mask = wild2regex(mask);
}
void setDirMask(const(char)[] dirmask)
{
this.dirmask = wild2regex(dirmask);
}
// Configure skipDotfiles if function is called
// By default, skipDotfiles = false;
void setSkipDotfiles()
{
skipDotfiles = true;
}
// return value of skipDotfiles
bool getSkipDotfiles()
{
return skipDotfiles;
}
// config file skip_dir parameter
bool isDirNameExcluded(string name)
{
// Does the directory name match skip_dir config entry?
// Returns true if the name matches a skip_dir config entry
// Returns false if no match
// Try full path match first
if (!name.matchFirst(dirmask).empty) {
return true;
} else {
// Do we check the base name as well?
if (!skipDirStrictMatch) {
// check just the basename in the path
string filename = baseName(name);
if(!filename.matchFirst(dirmask).empty) {
return true;
}
}
}
// no match
return false;
}
// config file skip_file parameter
bool isFileNameExcluded(string name)
{
// Does the file name match skip_file config entry?
// Returns true if the name matches a skip_file config entry
// Returns false if no match
// Try full path match first
if (!name.matchFirst(mask).empty) {
return true;
} else {
// check just the file name
string filename = baseName(name);
if(!filename.matchFirst(mask).empty) {
return true;
}
}
// no match
return false;
}
// Match against sync_list only
bool isPathExcludedViaSyncList(string path)
{
return .isPathExcluded(path, paths);
}
// Match against skip_dir, skip_file & sync_list entries
bool isPathExcludedMatchAll(string path)
{
return .isPathExcluded(path, paths) || .isPathMatched(path, mask) || .isPathMatched(path, dirmask);
}
// is the path a dotfile?
bool isDotFile(string path)
{
// always allow the root
if (path == ".") return false;
path = buildNormalizedPath(path);
auto paths = pathSplitter(path);
foreach(base; paths) {
if (startsWith(base, ".")){
return true;
}
}
return false;
}
}
// test if the given path is not included in the allowed paths
// if there are no allowed paths always return false
private bool isPathExcluded(string path, string[] allowedPaths)
{
// always allow the root
if (path == ".") return false;
// if there are no allowed paths always return false
if (allowedPaths.empty) return false;
path = buildNormalizedPath(path);
foreach (allowed; allowedPaths) {
auto comm = commonPrefix(path, allowed);
if (comm.length == path.length) {
// the given path is contained in an allowed path
return false;
}
if (comm.length == allowed.length && path[comm.length] == '/') {
// the given path is a subitem of an allowed path
return false;
}
}
return true;
}
// test if the given path is matched by the regex expression.
// recursively test up the tree.
private bool isPathMatched(string path, Regex!char mask) {
path = buildNormalizedPath(path);
auto paths = pathSplitter(path);
string prefix = "";
foreach(base; paths) {
prefix ~= base;
if (!path.matchFirst(mask).empty) {
// the given path matches something which we should skip
return true;
}
prefix ~= dirSeparator;
}
return false;
}
unittest
{
assert(isPathExcluded("Documents2", ["Documents"]));
assert(!isPathExcluded("Documents", ["Documents"]));
assert(!isPathExcluded("Documents/a.txt", ["Documents"]));
assert(isPathExcluded("Hello/World", ["Hello/John"]));
assert(!isPathExcluded(".", ["Documents"]));
}
import std.algorithm;
import std.array;
import std.file;
import std.path;
import std.regex;
import std.stdio;
import util;
final class SelectiveSync
{
private string[] paths;
private string[] businessSharedFoldersList;
private Regex!char mask;
private Regex!char dirmask;
private bool skipDirStrictMatch = false;
private bool skipDotfiles = false;
// load sync_list file
void load(string filepath)
{
if (exists(filepath)) {
// open file as read only
auto file = File(filepath, "r");
auto range = file.byLine();
foreach (line; range) {
// Skip comments in file
if (line.length == 0 || line[0] == ';' || line[0] == '#') continue;
paths ~= buildNormalizedPath(line);
}
file.close();
}
}
// Configure skipDirStrictMatch if function is called
// By default, skipDirStrictMatch = false;
void setSkipDirStrictMatch()
{
skipDirStrictMatch = true;
}
// load business_shared_folders file
void loadSharedFolders(string filepath)
{
if (exists(filepath)) {
// open file as read only
auto file = File(filepath, "r");
auto range = file.byLine();
foreach (line; range) {
// Skip comments in file
if (line.length == 0 || line[0] == ';' || line[0] == '#') continue;
businessSharedFoldersList ~= buildNormalizedPath(line);
}
file.close();
}
}
void setFileMask(const(char)[] mask)
{
this.mask = wild2regex(mask);
}
void setDirMask(const(char)[] dirmask)
{
this.dirmask = wild2regex(dirmask);
}
// Configure skipDotfiles if function is called
// By default, skipDotfiles = false;
void setSkipDotfiles()
{
skipDotfiles = true;
}
// return value of skipDotfiles
bool getSkipDotfiles()
{
return skipDotfiles;
}
// config file skip_dir parameter
bool isDirNameExcluded(string name)
{
// Does the directory name match skip_dir config entry?
// Returns true if the name matches a skip_dir config entry
// Returns false if no match
// Try full path match first
if (!name.matchFirst(dirmask).empty) {
return true;
} else {
// Do we check the base name as well?
if (!skipDirStrictMatch) {
// check just the basename in the path
string filename = baseName(name);
if(!filename.matchFirst(dirmask).empty) {
return true;
}
}
}
// no match
return false;
}
// config file skip_file parameter
bool isFileNameExcluded(string name)
{
// Does the file name match skip_file config entry?
// Returns true if the name matches a skip_file config entry
// Returns false if no match
// Try full path match first
if (!name.matchFirst(mask).empty) {
return true;
} else {
// check just the file name
string filename = baseName(name);
if(!filename.matchFirst(mask).empty) {
return true;
}
}
// no match
return false;
}
// Match against sync_list only
bool isPathExcludedViaSyncList(string path)
{
return .isPathExcluded(path, paths);
}
// Match against skip_dir, skip_file & sync_list entries
bool isPathExcludedMatchAll(string path)
{
return .isPathExcluded(path, paths) || .isPathMatched(path, mask) || .isPathMatched(path, dirmask);
}
// is the path a dotfile?
bool isDotFile(string path)
{
// always allow the root
if (path == ".") return false;
path = buildNormalizedPath(path);
auto paths = pathSplitter(path);
foreach(base; paths) {
if (startsWith(base, ".")){
return true;
}
}
return false;
}
// is business shared folder matched
bool isSharedFolderMatched(string name)
{
// if there are no shared folder always return false
if (businessSharedFoldersList.empty) return false;
if (!name.matchFirst(businessSharedFoldersList).empty) {
return true;
} else {
return false;
}
}
// is business shared folder included
bool isPathIncluded(string path, string[] allowedPaths)
{
// always allow the root
if (path == ".") return true;
// if there are no allowed paths always return true
if (allowedPaths.empty) return true;
path = buildNormalizedPath(path);
foreach (allowed; allowedPaths) {
auto comm = commonPrefix(path, allowed);
if (comm.length == path.length) {
// the given path is contained in an allowed path
return true;
}
if (comm.length == allowed.length && path[comm.length] == '/') {
// the given path is a subitem of an allowed path
return true;
}
}
return false;
}
}
// test if the given path is not included in the allowed paths
// if there are no allowed paths always return false
private bool isPathExcluded(string path, string[] allowedPaths)
{
// always allow the root
if (path == ".") return false;
// if there are no allowed paths always return false
if (allowedPaths.empty) return false;
path = buildNormalizedPath(path);
foreach (allowed; allowedPaths) {
auto comm = commonPrefix(path, allowed);
if (comm.length == path.length) {
// the given path is contained in an allowed path
return false;
}
if (comm.length == allowed.length && path[comm.length] == '/') {
// the given path is a subitem of an allowed path
return false;
}
}
return true;
}
// test if the given path is matched by the regex expression.
// recursively test up the tree.
private bool isPathMatched(string path, Regex!char mask) {
path = buildNormalizedPath(path);
auto paths = pathSplitter(path);
string prefix = "";
foreach(base; paths) {
prefix ~= base;
if (!path.matchFirst(mask).empty) {
// the given path matches something which we should skip
return true;
}
prefix ~= dirSeparator;
}
return false;
}
// unit tests
unittest
{
assert(isPathExcluded("Documents2", ["Documents"]));
assert(!isPathExcluded("Documents", ["Documents"]));
assert(!isPathExcluded("Documents/a.txt", ["Documents"]));
assert(isPathExcluded("Hello/World", ["Hello/John"]));
assert(!isPathExcluded(".", ["Documents"]));
}

708
src/sync.d
View file

File diff suppressed because it is too large Load diff