Commit graph

418 commits

Author SHA1 Message Date
abraunegg 8d272b0f77 Update sync.d
* Update logging output
2023-12-21 07:33:32 +11:00
abraunegg cbe55cd4d2 Update sync.d
* Add warning message about HTML characters in JSON element
2023-12-21 06:56:02 +11:00
abraunegg 06061da419 Update sync.d
Change the handling of selfBuiltPath so correct path is used when attempting to test against 'sync_list'
2023-12-20 18:49:25 +11:00
abraunegg c2e98ff639 Update sync.d
* Potentially fix crash identified by #2562
2023-12-18 18:55:54 +11:00
abraunegg 5fbfc49e32 Handle --upload-only & --remove-source-files for upload session resumption
* Handle --upload-only & --remove-source-files for upload session resumption
2023-11-09 13:37:45 +11:00
abraunegg bcc77b79eb Add resumable session uploads
* Check the system for any session files, that indicate that a session upload did not complete
* Change from using a CRC32 of the file, to a random 16 character alphanumeric string to use as the session filename extension as it is not computationally valid to do a CRC32 on large files
2023-11-09 12:20:11 +11:00
abraunegg 423bf07b77 Replace safeRename() with safeBackup()
* Finally replace safeRename() with safeBackup() to ensure consistent messaging and process when a local file is being renamed.
2023-11-08 15:30:31 +11:00
abraunegg 16595efbdb Update FR #2359
Update --display-quota output to:

Deleted:   8.19 GB (8793627814 bytes)
Remaining: 25580.69 GB (27467060611614 bytes)
State:     normal
Total:     25600.00 GB (27487790694400 bytes)
Used:      11.12 GB (11936454972 bytes)
2023-10-27 05:27:38 +11:00
JC-comp 1d29ca0489
Fix curl reusing socket (#2526)
* Fix curl reusing socket
2023-10-25 07:20:16 +11:00
abraunegg 77c294ba0c Implement --display-quota
* Implement --display-quota
2023-10-20 08:53:59 +11:00
abraunegg d7ea79773f Update sync.d
* Add a check when using --download-only to test if the file already exists locally, and if it does, has the local file been modified since the file was last downloaded. If it has, then rename the existing file to preserve it to prevent local data loss
2023-10-20 05:45:33 +11:00
abraunegg 2257f0ba92 Update PR
* Fix onedrive -s --download-only
* Fix onedrive -s --download-only --local-first
2023-10-19 21:27:03 +11:00
abraunegg d653ff33a5 Update sync.d
* Fix that the '.' were not being printed in --verbose mode for fetching the /delta response
2023-10-19 10:05:36 +11:00
abraunegg 35edb4c1b0 Uplift to v2.5.0-alpha-3
* Uplift to v2.5.0-alpha-3
2023-10-19 05:39:33 +11:00
abraunegg 4253318835 Squashed commit of the following:
commit 1eff2d7d67
Author: abraunegg <alex.braunegg@gmail.com>
Date:   Wed Oct 18 19:15:27 2023 +1100

    Update PR

    * Add  --source-directory 'path/as/source/' --destination-directory 'path/as/destination' functionality

commit ad3ddee0ec
Author: abraunegg <alex.braunegg@gmail.com>
Date:   Wed Oct 18 17:32:24 2023 +1100

    Update PR

    * Add --create-directory
    * Add --remove-directory

commit 7dfe6b65b7
Author: abraunegg <alex.braunegg@gmail.com>
Date:   Wed Oct 18 12:27:03 2023 +1100

    Update PR

    * Update PR

commit 75c071e56f
Author: abraunegg <alex.braunegg@gmail.com>
Date:   Wed Oct 18 10:12:05 2023 +1100

    Update PR

    * Update PR

commit 6db484cdad
Author: abraunegg <alex.braunegg@gmail.com>
Date:   Mon Oct 16 17:01:25 2023 +1100

    Update PR

    * Update PR

commit d893ea5460
Author: abraunegg <alex.braunegg@gmail.com>
Date:   Wed Oct 11 10:43:50 2023 +1100

    Update PR

    * Update PR

commit 82bd593bf4
Author: abraunegg <alex.braunegg@gmail.com>
Date:   Wed Oct 11 09:14:17 2023 +1100

    Update PR

    * Validate and document --auth-files operation

commit c551203f4c
Author: abraunegg <alex.braunegg@gmail.com>
Date:   Wed Oct 11 05:48:22 2023 +1100

    Update PR

    * Add --create-share-link

commit fbf63999ff
Author: abraunegg <alex.braunegg@gmail.com>
Date:   Tue Oct 10 18:39:21 2023 +1100

    Update PR

    * Update PR

commit 72a4680035
Author: abraunegg <alex.braunegg@gmail.com>
Date:   Tue Oct 10 17:43:15 2023 +1100

    Update PR

    * Add --get-file-link
    * Add --modified-by

commit 0d3fc3ebf2
Author: abraunegg <alex.braunegg@gmail.com>
Date:   Tue Oct 10 14:28:10 2023 +1100

    Add --display-sync-status

    * Add --display-sync-status

commit 1f183ca03e
Author: abraunegg <alex.braunegg@gmail.com>
Date:   Mon Oct 9 08:18:13 2023 +1100

    Update PR

    * Update PR with doc updates

commit b0628d7099
Author: abraunegg <alex.braunegg@gmail.com>
Date:   Sun Oct 8 10:52:52 2023 +1100

    Update PR

    * Update PR

commit 7e3df956ce
Author: abraunegg <alex.braunegg@gmail.com>
Date:   Sat Oct 7 05:31:26 2023 +1100

    Update PR

    * Update PR

commit c69f2abc4b
Author: abraunegg <alex.braunegg@gmail.com>
Date:   Sat Oct 7 05:28:28 2023 +1100

    Update PR

    * Update PR

commit ea1ca33374
Author: abraunegg <alex.braunegg@gmail.com>
Date:   Fri Oct 6 14:57:51 2023 +1100

    Update PR

    * Update PR

commit 1503f969df
Author: abraunegg <alex.braunegg@gmail.com>
Date:   Fri Oct 6 09:19:04 2023 +1100

    Update PR

    * Update PR

commit 5127464f2c
Author: abraunegg <alex.braunegg@gmail.com>
Date:   Fri Oct 6 06:48:20 2023 +1100

    Change when the integrity check is performed

    * Change when the integrity check is performed

commit c7cc45d95c
Author: abraunegg <alex.braunegg@gmail.com>
Date:   Thu Oct 5 19:40:05 2023 +1100

    Update maxInotifyWatches location

    * Update maxInotifyWatches location

commit c44ad963a6
Author: abraunegg <alex.braunegg@gmail.com>
Date:   Thu Oct 5 17:41:31 2023 +1100

    Update main.d

    * Fix --version segfault

commit 51f0ffcb1f
Author: abraunegg <alex.braunegg@gmail.com>
Date:   Thu Oct 5 17:24:30 2023 +1100

    Uplift to v2.5.0-alpha-2

    * Uplift to v2.5.0-alpha-2

commit cbe3e6ea84
Author: abraunegg <alex.braunegg@gmail.com>
Date:   Thu Oct 5 17:17:26 2023 +1100

    Clean up before onedrive-v2.5.0-alpha-2

    * Clean up before onedrive-v2.5.0-alpha-2
2023-10-19 05:31:50 +11:00
abraunegg 43b0bed4cb
Fix debugging log output when reporting file size and hash mismatch (#2473)
* Fix debug logging output with correct hash values as these were switched
* Only calculate local file size post download once and reuse value
2023-08-24 18:08:21 +10:00
abraunegg 268dcf9120
Fix that zero-byte files do not have a hash as reported by the OneDrive API thus should not generate an error message (#2436)
* The OneDrive API does not present a hash for all files, most notably those that are zero byte in size (this may be fixed by the OneDrive API in the future). Add a wrapper to the existing makeItem function to test the file size before working out if this is a going to be a problem and if it is, then print out an error message if the file size is greater than 0 including either the full filename path or the items id.
2023-06-21 13:21:05 +10:00
abraunegg 06420c9a0a
Remove sha1 use and cleanup defunct remaining crc32 use (#2424)
* Remove sha1 from being used by the client as this is being depreciated by Microsoft in July 2023 - https://devblogs.microsoft.com/microsoft365dev/deprecation-of-sha1hash-on-onedrive-personal/
* Complete the removal of crc32 as this is also no longer present for a long time, but some code elements still existed
* Only compute quickXorHash, not quickXorHash and sha256Hash as computing sha256Hash is CPU expensive
* Update cache database stored items to only store quickXorHash and sha256Hash values (remove crc32 and sha1)
2023-06-20 06:55:00 +10:00
abraunegg 00ce7eed0e
Fix that folders are renamed despite using --dry-run (#2343)
* Fix that folders are renamed despite using --dry-run
2023-03-27 17:22:48 +11:00
abraunegg cfb4933778
Fix that a HTTP 504 - Gateway Timeout causes local files to be deleted when using --download-only & --cleanup-local-files mode (#2342)
* Fix that a HTTP 504 - Gateway Timeout causes local files to be deleted when using --download-only & --cleanup-local-files mode
2023-03-19 09:54:58 +11:00
abraunegg 678add91f8
Update --get-O365-drive-id error handling (#2266)
* Ensure a 'Personal' account type is not being used
* If the /sites?search API cannot be found, display a more appropriate error message
2022-12-16 06:08:46 +11:00
abraunegg 0b89a1fea0
Change order of Microsoft OneDrive restrictions and limitations check and skip_file|skip_dir (#2238)
* Perform check against Microsoft OneDrive restriction and limitations after user configured items to skip has been performed
2022-11-26 05:42:38 +11:00
abraunegg 5a7da61cbb
Fix issue of silent exit when unable to delete local files when using --cleanup-local-files (#2227)
* Add try blocks for performing local deletes to catch in function permission issues preventing file|folder local deletion.
2022-11-19 06:18:56 +11:00
abraunegg 0ec1c95e4a
Only call isPathExcludedViaSyncList if 'sync_list' is enabled where possible (#2213)
* Only check 'sync_list' if this has been enabled and configured, otherwise do not undertake a check as it is computationally redundant where possible
2022-11-10 16:58:54 +11:00
abraunegg 941e1e215d
Ensure path does not start with '/' before sync_list check (#2215)
* Ensure path does not start with '/' before sync_list check due to changes from 2211 and 2212
2022-11-10 14:28:20 +11:00
abraunegg 361c3cf0a1
Update Fix 2211 and 2212 (#2214)
* Update fixes for 2211 and 2212 due to undesirable path issue and output when application processing is actually correct
2022-11-10 13:43:36 +11:00
abraunegg 4a6c78a7a6
Only calculate path with computeItemPath if path is empty (#2212)
* Only calculate path with computeItemPath if path is empty, avoiding a double calculation and DB operations
2022-11-10 07:03:58 +11:00
abraunegg c73c0aa608
Fix application crash when local file is changed to a symbolic link with non-existent target (#2211)
* Add a try & catch block for testing if the file exists locally to catch any filesystem error that may be generated
* Test path to be valid if a symbolic link
2022-11-10 06:50:37 +11:00
abraunegg 7daf8797f8
Add performance output for key functions (#2180)
* Add developer option 'display_processing_time' to control if performance timing is outputted or not
* If option is enabled, print performance data around how long key functions are taking to process data to assist with understanding any performance related questions
2022-10-12 13:34:42 +11:00
abraunegg 514df625b9
Add option to create a read-write shareable link (#2154)
* Implement Feature Request 2110 to add option to create a read-write shareable link
2022-09-26 17:56:42 +10:00
abraunegg e2ed2d2b58
Update notification handling for specific skipped scenarios (#2153)
* Update skip notification handling for the following scenarios:
  * Invalid Name (Microsoft Naming Convention)
  * Invalid Name (Contains an invalid whitespace item)
  * Invalid Name (Contains HTML ASCII Code)
  * Invalid Item (Invalid symbolic link)
2022-09-26 07:40:22 +10:00
abraunegg 738be2d150
Add option to cleanup local files regardless of sync state when using --download-only (#2113)
* Add option to cleanup local files regardless of sync state when using --download-only
2022-08-31 06:41:52 +10:00
abraunegg d169dfc642
Improve application logging output for error messages (#2100)
* When enabling system logging to a log file, the actual ERROR line is forced to a new line in the application log. The reason for this is the \n prefix in the error message, which was in place so that when performing CLI logging or systemd logging, the error message would be displayed clearly. This change removes the \n from the actual error message, but inserts a newline before the error message is displayed (and also in some cases post error message) - thus keeping the application runtime look and feel, but improving the application log output.
2022-08-18 09:14:13 +10:00
abraunegg 9b8a25f34c
Fix that logging output for skipped uploaded files is missing (#2094)
* Add missing logfile output when enabling logging so that when uploaded new & modified files are skipped, this is correctly reflected in logfile output
2022-08-17 05:48:11 +10:00
abraunegg c82e90a140
Ensure that any active http instance is shutdown before any application exit (#2035)
* When the client needs to exit due to an issue, ensure that the curl http instance is shutdown before the exit is performed. This also potentially solves some segmentation faults seen on Ubuntu|Debian platforms due to issues in the shared library libphobos2-ldc-shared.so.X
2022-07-07 09:37:45 +10:00
Sebastien Badia 0ab2955bd7
Various little spelling fixes (check with lintian during Debian packaging) (#2027)
* contrib: remove bash hashbang from completion

This file starts with the #! sequence that marks interpreted scripts, but
it is a bash completion script that is merely intended to be sourced.

* src: spelling error (Attemtping => Attempting)

* src: spelling error (reponse => response)

* src: spelling error (sucessfully => successfully)
2022-07-02 05:14:50 +10:00
abraunegg ca984eba70
Update OneDrive API response handling for National Cloud Deployments (#2023)
* Update OneDrive API response handling for National Cloud Deployments
* Add developer option to allow easy switch between /children and /delta to query OneDrive for changes
2022-06-30 07:08:29 +10:00
abraunegg 042949f1c1
Fix 'foreign key constraint failed' when using OneDrive Business Shared Folders (#2017)
In some OneDrive Business scenarios, the shared folder /delta response lacks the 'root' drive details. When this occurs, this creates the following error: A database statement execution error occurred: foreign key constraint failed. Ensure we query independently the root details for this shared folder and ensure that it is added before we process the /delta response
2022-06-24 14:54:32 +10:00
abraunegg 1935def140
Update when DB is updated by OneDrive API data and update when file hash is required to be generated (#2004)
* Only update the database if the item modified time is different, otherwise we are needlessly updating the database with data that is the same
* Generating a file hash, during the file integrity check is computationally expensive. Only generate a file hash if the modified time of the file is different, otherwise it is pointless to generate the file hash during each integrity check
* Only flush SHM and WAL post integrity check
2022-06-15 09:16:06 +10:00
abraunegg 4e1deec933
Update Business Shared Folders to use a /delta query (#1989)
* Change Business Shared Folders to use a /delta query now that the OneDrive API appears to support such query
2022-06-05 14:52:24 +10:00
abraunegg 0fffb8fbc0
Implement a check to validate local filesystem available space before attempting file download (#1971)
* Implement a check to validate local filesystem available space before attempting file download
* Implement 'space_reservation' config option with a default value of 50 MB
* Prevent the original run-away logging error message 'Failed initialization on handle XXXX' from occurring if the system is out of space
* Update documentation and man page
2022-06-01 05:57:05 +10:00
Philipp 446be3743c
Fix logging output when download failed (#1894)
* Fix logging output when download failed
2022-04-29 16:54:55 +10:00
abraunegg 37bce6f82d
Update local folder creation timestamp with timestamp from OneDrive (#1934)
* When a new folder is created locally, use the lastModifiedDateTime as provided by OneDrive for the folders timestamp instead of the time when the client created the directory

Note: This only will impact new folder that are created, will not touch or modify existing folders
2022-04-26 08:23:18 +10:00
abraunegg 29b0ef0b98
Fix crash when trying to sync broken symbolic link (#1917)
* Check the response for directory checks as to if it generates a file exception error
2022-04-08 05:50:47 +10:00
abraunegg 3124b2dcf2
Implement feature request --modified-by (#1869)
* Implement --modified-by to display who last modified a file and when the modification was done
2022-03-10 07:00:07 +11:00
abraunegg d68c27fb4f
Add --reauth to allow easy re-authentication of the client (#1867)
* Add --reauth to allow easy re-authentication of the client
2022-03-08 05:35:00 +11:00
abraunegg c0e40e857e
Split children query into a separate function which can be self-referred (#1828)
* Split children query into a separate function which can be self-referred in case of multiple 429 or 504 responses in a row
2022-02-03 13:27:57 +11:00
abraunegg 8870603939
Update fixing edge case with OneDrive Personal Shared Folders and --resync --upload-only (#1762)
* Expand on the edge case solution fix for #1586 to drop --single-directory qualifier in this very specific scenario, and add the DB tie record in all --resync --upload-only scenarios where the driveId is actually remote
2021-12-31 06:18:32 +11:00
abraunegg f4c0a79425
Fix unable to upload to OneDrive Business Shared Folders due to API restricting quota information (#1725)
* In fixing #1712  which was due to a OneDrive API change, the flagging that quota is being restricted was missed, thus, if this is truly being restricted may cause files to OneDrive Business Shared Folders be unable to be uploaded despite space being available.
2021-11-25 06:48:50 +11:00
abraunegg 2f47beab60
Fix overwriting local files with older versions from OneDrive when items.sqlite3 does not exist and --resync is not used (#1717)
* Resolve issue where --resync is not used and items.sqlite3 is not available at application startup, local files, whilst potentially newer, will not be preserved thus leading to a potential data loss scenario
2021-11-24 06:44:45 +11:00