With -c option, aria2 can continue download after the existing file
position. If it is not completed, then after completion aria2 runs
checksum checking if available. But if existing file has already been
completed, then CreateRequestCommand exits without issuing checksum
checking. And aria2 treats it download error because it needs checksum
verification but it has not been done. This change fixes this by
properly checking download state and issue checksum checking before
CreateRequestCommand.
This change replaces the current 64 bit sequential GID with 64 bits
random bytes GID in an attempt to support persistent GID. Internally,
the GID is stored as uint64_t. For human representation and RPC
interface, GID is represented as 16 bytes hex string. For console
readout, 16 bytes are too long, so it is abbreviated to first 6 bytes.
When querying GID in RPC calls, user can speicfy the prefix of GID as
long as the prefix is shared by more than 1 GID entries.
This option enables disk cache. If SIZE is 0, the disk cache is
disabled. This feature caches the downloaded data in memory, which
grows to at most SIZE bytes. The cache storage is created for aria2
instance and shared by all downloads. The one advantage of the disk
cache is reduce the disk seek time because the data is written in
larger unit and it is reordered by the offset of the file. If the
underlying file is heavily fragmented it is not the case.
The old implementation calculates download/upload statistics for a
RequestGroup by summing up all PeerStat objects. For global
statistics, those are summed together. This clearly incurs runtime
penalty and we introduced some kind of caching which updates
statistics every 250ms but it did not work right.
This change removes all these aggregation code, and instead makes
RequestGroup and RequestGroupMan objects hold NetStat object and
download/upload bytes are directly calculated by thier own NetStat.
This is far more simplar than the old way and less runtime penalty and
brings more accuracy.
DownloadHandlerConstants was simplified. MIME type handling in Accept
header was also reworked. DownloadContext's metalinkServerContacted_
is replaced with acceptMetalink_ and its boolean value is reverted.
RequestGroup and HttpRequest now do not hold vector of accepting
types. HttpRequest has the flag acceptMetalink_ which will be set by
the same value of DownloadContext::accpetMetalink_ and if it is true,
Metalink MIME types are added to Accept header field.
The only exception is zero-length file. If server tells file is
zero-length and --checksum option is given, aria2 now correctly checks
its checksum. There is one known issue: If downloaded file is
zero-length file and .aria2 file exists, it will not be deleted on
successful verification, because .aria2 file is not loaded.
RequestGroup holds a poitner to btRuntime_ and peerStorage_. After
removing them from BtRegistry, we failed to set 0 to them. When
program access them, it goes undefined world, such as random crash.
We found this bug when pasuing download and valgrind warned memory
corruption.
We introduced SocketRecvBuffer which buffers received bytes. Since
HTTP response header and response body are divided with \r\n, we have
to buffer up several bytes to find this delimiter. We use
SocketRecvBuffer to hold these bytes and only consumes header and
passes SocketRecvBuffer, which may contain head of response body, to
next Command. Since FTPConnection doesn't use SocketCore::peekData(),
we left it as is.
Removed SharedHandle::isNull(). Instead we added operator* and
operator unspecified_bool_type. Removed use of WeakHandle and
replaced with raw pointer.
Use RFC1123 date format in debug log of MDTM response and
last-modified time and creation date of .torrent file printed
using -S.
* src/FtpNegotiationCommand.cc
* src/RequestGroup.cc
* src/bittorrent_helper.cc
Fixed the bug that a file gets overwritten if -V is given and no
hash is provided. Fixed the bug that --dry-run leads download
error. Added RequestGroup::createCheckIntegrityEntry() which
correctly creates CheckIntegrityEntry objects and open files based
on -V option and the existence of control file.
* src/AbstractCommand.cc
* src/AbstractCommand.h
* src/ChecksumCheckIntegrityEntry.cc
* src/DownloadContext.cc
* src/DownloadContext.h
* src/FtpNegotiationCommand.cc
* src/HttpResponseCommand.cc
* src/PieceHashCheckIntegrityEntry.cc
* src/RequestGroup.cc
* src/RequestGroup.h
* src/RequestGroupEntry.cc
* src/RequestGroupEntry.h
Check hash(hash for entire file, not piece hash) if
--check-integrity option is given and file is
downloaded(determined by file length). If it fails, re-download
file.
* src/AbstractCommand.cc
* src/ChecksumCheckIntegrityEntry.cc
* src/ChecksumCheckIntegrityEntry.h
* src/FtpNegotiationCommand.cc
* src/HttpResponseCommand.cc
* src/RequestGroup.cc
Fixed the bug that aria2 reports error and exits with non-zero
status when file is already downloaded.
* src/FtpNegotiationCommand.cc
* src/HttpResponseCommand.cc
* src/RequestGroup.cc
Don't listen on IPv6 socket for BitTorrent, DHT, XML-RPC if
--disable-ipv6 is true.
* src/BtSetup.cc
* src/DownloadEngineFactory.cc
* src/RequestGroup.cc
Count the number of command used in HTTP(S)/FTP and the number of
connections in HTTP(S)/FTP independently. The former is used to
determin whether additional command is needed. The latter is used
to report user to how many connections are used in a download.
* src/AbstractCommand.cc
* src/AbstractCommand.h
* src/CreateRequestCommand.cc
* src/RequestGroup.cc
* src/RequestGroup.h
Added --min-split-size=SIZE option. aria2 does not split less
than 2*SIZE byte range. For example, let's consider downloading
20MiB file. If SIZE is 10M, aria2 can split file into 2 range
[0-10MiB) and [10MiB-20MiB) and download it using 2 sources(if
--split >= 2, of course). If SIZE is 15M, since 2*15M > 20MiB,
aria2 does not split file and download it using 1 source.
* src/BitfieldMan.cc
* src/BitfieldMan.h
* src/DefaultPieceStorage.cc
* src/DefaultPieceStorage.h
* src/OptionHandlerFactory.cc
* src/RequestGroup.cc
* src/prefs.cc
* src/prefs.h
* src/usage_text.h
* test/BitfieldManTest.cc
* test/SegmentManTest.cc
Fixed the bug that DownloadContext::resetDownloadStartTime() never
be called in downloads whose total length is unknown. This causes
avg speed in Download Results is always 0 for these downloads.
* src/RequestGroup.cc
* src/BtFileAllocationEntry.cc
* src/StreamFileAllocationEntry.cc
Run whole checksum check when download file name and size are
determined and download is already completed. Related to r2221.
Guarded ENABLE_MESSAGE_DIGEST where necessary.
* src/AbstractCommand.cc
* src/RequestGroup.cc
* src/message.h
Fixed that bug that when aria2 is stopped before whole hash
check(not piece hash check) is done, aria2 reports its file is
downloaded successfully. It should report the download is
in-progress and next invocation of aria2 should check hash.
* src/ChecksumCheckIntegrityEntry.cc
* src/DownloadCommand.cc
* src/DownloadContext.cc
* src/DownloadContext.h
* src/RequestGroup.cc
* src/RequestGroupMan.cc
Hold reference to PieceStorage when paused in order to provide
progress information via XML-RPC.
* src/RequestGroup.cc
* src/RequestGroup.h
* src/RequestGroupMan.cc
Added aria2.pause and aria2.unpause XML-RPC method. aria2.pause
pauses the download denoted by gid. gid is of type string. The
status of paused download becomes "paused" and the download is
placed on the first position of waiting queue. As long as the
status is "paused", the download is not started. To change status
to "waiting", use aria2.unpause method. This method returns GID of
paused download. aria2.unpause changes the status of the download
denoted by gid from "paused" to "waiting". This makes the download
eligible to restart. gid is of type string. This method returns
GID of unpaused download.
* doc/aria2c.1.txt
* src/DownloadContext.cc
* src/FileEntry.cc
* src/OptionHandlerFactory.cc
* src/RequestGroup.cc
* src/RequestGroup.h
* src/RequestGroupMan.cc
* src/XmlRpcMethodFactory.cc
* src/XmlRpcMethodImpl.cc
* src/XmlRpcMethodImpl.h
* src/prefs.cc
* src/prefs.h
* src/usage_text.h
Added --always-resume and --max-resume-failure-tries option. If
--always-resume=false is given, when all given URIs does not
support resume or aria2 encounters N URIs which does not support
resume
(N is the value specified using --max-resume-failure-tries
option), aria2 download file from scratch. The default behavior
is --always-resume=true, which means if all URIs do not support
resume, download fails. I think this is OK because user normally
don't like to see that partially downloaded file is
overwritten(this is particularly true if file size is big). This
option is useful when aria2 is used as download backend and
graceful falling back to overwritten behavior is preferable.
Added exit status value 8, which means download failed because
server did not support resume.
* src/AbstractCommand.cc
* src/DefaultPieceStorage.cc
* src/DownloadCommand.cc
* src/DownloadResultCode.h
* src/FileEntry.h
* src/FtpNegotiationCommand.cc
* src/HttpResponse.cc
* src/HttpResponseCommand.cc
* src/OptionHandlerFactory.cc
* src/RequestGroup.cc
* src/RequestGroup.h
* src/SegmentMan.cc
* src/SegmentMan.h
* src/prefs.cc
* src/prefs.h
* src/usage_text.h
* test/DefaultPieceStorageTest.cc
* test/SegmentManTest.cc
Added --remove-control-file option. This option removes control
file(*.aria2 file) before download. Using with
--allow-overwrite=true, download always starts from scratch. This
will be useful for users behind proxy server which disables
resume. For such proxy user, -C1 is also recommended for Metalink
downloads to avoid establishing unnecessary connections.
* doc/aria2c.1.txt
* src/OptionHandlerFactory.cc
* src/RequestGroup.cc
* src/prefs.cc
* src/prefs.h
* src/usage_text.h
Create CreateRequestCommand to the number of -s/-C when URIs are
added by XML-RPC.
* src/RequestGroup.cc
* src/RequestGroup.h
* src/XmlRpcMethodImpl.cc
Only execute RequestGroupMan::fillRequestGroupFromReserver() when
queue maintenance is requested by RequestGroup to avoid to call
the function unnecessarily.
* src/FillRequestGroupCommand.cc
* src/RequestGroup.cc
* src/RequestGroup.h
* src/RequestGroupMan.cc
* src/RequestGroupMan.h
Avoid to calculate end() iterator in each loop. std::deque is
particularly slow. Make sure that recalculate end iterator if
element is erased during loop.
Removed RequestGroup::initSegmentMan(). Guaranteed that either
both _pieceStorage and _segmentMan are initialized or they are
not.
* src/RequestGroup.cc
* src/RequestGroup.h
* test/BtDependencyTest.cc
Fixed the bug that causes segmentaiton fault when aria2 sees '404
not found' in the N times(N is where --max-file-not-found=N, N>0)
in a row without single '200 OK' response.
* src/RequestGroup.cc.
Fixed memory leak. Commands stored in std::deque<Command*> are not
deleted when exception is thrown. Make sure that when passing
std::deque<Command*> to function to store Commands, handle
exception and delete these Commands.
* src/AbstractCommand.cc
* src/CheckIntegrityCommand.cc
* src/FileAllocationCommand.cc
* src/RequestGroup.cc
* src/TrackerWatcherCommand.cc
Tatsuhiro Tsujikawa