general: enhance multi-thread downloader with cancelable context, immediately stop all stream processes when canceled;
feat(crypt): improve stream closing;
general: fix the bug of downloading files becomes previewing stream on modern browsers;
Co-authored-by: Sean He <866155+seanhe26@users.noreply.github.com>
Co-authored-by: Andy Hsu <i@nn.ci>
this PR has several enhancements, fixes, and features:
- [x] Crypt: a transparent encryption driver. Anyone can easily, and safely store encrypted data on the remote storage provider. Consider your data is safely stored in the safe, and the storage provider can only see the safe, but not your data.
- [x] Optional: compatible with [Rclone Crypt](https://rclone.org/crypt/). More ways to manipulate the encrypted data.
- [x] directory and filename encryption
- [x] server-side encryption mode (server encrypts & decrypts all data, all data flows thru the server)
- [x] obfuscate sensitive information internally
- [x] introduced a server memory-cached multi-thread downloader.
- [x] Driver: **Quark** enabled this feature, faster load in any single thread scenario. e.g. media player directly playing from the link, now it's faster.
- [x] general improvement on HTTP/WebDAV stream processing & header handling & response handling
- [x] Driver: **Mega** driver support ranged http header
- [x] Driver: **Quark** fix bug of not closing HTTP request to Quark server while user end has closed connection to alist
## Crypt, a transparent Encrypt/Decrypt Driver. (Rclone Crypt compatible)
e.g.
Crypt mount path -> /vault
Crypt remote path -> /ali/encrypted
Aliyun mount paht -> /ali
when the user uploads a.jpg to /vault, the data will be encrypted and saved to /ali/encrypted/xxxxx. And when the user wants to access a.jpg, it's automatically decrypted, and the user can do anything with it.
Since it's Rclone Crypt compatible, users can download /ali/encrypted/xxxxx and decrypt it with rclone crypt tool. Or the user can mount this folder using rclone, then mount the decrypted folder in Linux...
NB. Some breaking changes is made to make it follow global standard, e.g. processing the HTTP header properly.
close#4679close#4827
Co-authored-by: Sean He <866155+seanhe26@users.noreply.github.com>
Co-authored-by: Andy Hsu <i@nn.ci>
* Determine whether the URL requires Sign
* Add File and Mem based KV
NOT TESTED: TokenKV Function
* Change Token KV func to common func.
Add File based KV func
* Remove KV, Remove Token
I found that the original Sign function is enough to complete the link signature, and only need to add simple configuration items to meet the requirements.
* Add IsStorageSigned func to judge if Signing is enabled in the storage settings.
It should be working now.
* Add a SIGN button to the management panel.
* Add enable_sign to the basic storage struct.
Can enable sign for every driver now.
Bug: When sign enabled, in download page, Copy link doesn't contain a sign.
(Not done yet)
* Fix a bug from commit 8f6c25f.
Response of fsread function does not contain sign.
* Optimize code and follow advices.
- Add back public/dist/README.md
- Enable sign when DownProxyUrl is enabled
- Merge needSign() to isEncrypt() in fsread.go
* simplify code
---------
Co-authored-by: Andy Hsu <i@nn.ci>
* feat(qbittorrent): authorization and logging in support
* feat(qbittorrent/client): support `AddFromLink`
* refactor(qbittorrent/client): check authorization when getting a new client
* feat(qbittorrent/client): support `GetInfo`
* test(qbittorrent/client): update test cases
* feat(qbittorrent): init qbittorrent client on bootstrap
* feat(qbittorrent): support setting webui url via gin
* feat(qbittorrent/client): support deleting
* feat(qbittorrent/client): parse `TorrentStatus` enum when unmarshalling json in `GetInfo()`
* feat(qbittorrent/client): support getting files by id
* feat(qbittorrent): support adding qbittorrent tasks via gin
* refactor(qbittorrent/client): return a `Client` interface in `New()` instead of `*client`
* refactor: task handle
* chore: fix typo
* chore: change path
---------
Co-authored-by: Andy Hsu <i@nn.ci>
* refactor:separate the setting method from the db package to the op package and add the cache
* refactor:separate the meta method from the db package to the op package
* fix:setting not load database data
* refactor:separate the user method from the db package to the op package
* refactor:remove user JoinPath error
* fix:op package user cache
* refactor:fs package list method
* fix:tile virtual paths (close#2743)
* Revert "refactor:remove user JoinPath error"
This reverts commit 4e20daaf9e.
* clean path directly may lead to unknown behavior
* fix: The path of the meta passed in must be prefix of reqPath
* chore: rename all virtualPath to mountPath
* fix: `getStoragesByPath` and `GetStorageVirtualFilesByPath`
is_sub_path:
/a/b isn't subpath of /a/bc
* fix: don't save setting if hook error
Co-authored-by: Noah Hsu <i@nn.ci>
* refactor:Prepare to remove the get interface
* feat:add obj Unwarp interface
* refactor:obj name mapping and program internal path processing
* chore: fix typo
* feat: unwrap get
* fix: no use op.Get to get parent id
* fix: set the path uniformly
Co-authored-by: Noah Hsu <i@nn.ci>
* feat: remove index on `SearchNode.Name`
As we do not use s% on name column, index there does not work
* fix: init index after init data
Or on the first run, it will log 'init index error: readObjectStart: expect { or n, but found , error found in #0 byte of ...||..., bigger context ...||...'
* fix: match parent more precisely
It will match `/a/bc` if we search in `/a/b` originally.
But it is not backward compatible by adding a suffix `/`
to all the data in parent field
* feat: ignore AList storage on indexing
* fix: remove unused err in `walkFn`
* chore(ci): fix auto_lang trigger and run it
* feat: batch index
* feat: quit index & init index
* feat: set DocType for bleve data
* fix: build index cleanup check origin err
Andy Hsu