The old implementation calculates download/upload statistics for a
RequestGroup by summing up all PeerStat objects. For global
statistics, those are summed together. This clearly incurs runtime
penalty and we introduced some kind of caching which updates
statistics every 250ms but it did not work right.
This change removes all these aggregation code, and instead makes
RequestGroup and RequestGroupMan objects hold NetStat object and
download/upload bytes are directly calculated by thier own NetStat.
This is far more simplar than the old way and less runtime penalty and
brings more accuracy.
Previous implementation does not store information which Command cuid
uses which Piece. Mark Piece acquired by SegmentMan by setting
Piece::setUsedBySegment(true).
Added .cc file for classes/structs that only provided by header
file. Defined non-POD classes' ctor, dtor in .cc file. Moved
implementation code in header file to .cc file for major
classes/strucsts.
Fixed the bug that corrups file if segment returned from
SegmetnMan::getCleanSegmentIfOwnerIsIdle() has writtenLength > 0.
* src/DownloadCommand.cc
* src/SegmentMan.cc
* src/SegmentMan.h
* test/SegmentManTest.cc
Added --always-resume and --max-resume-failure-tries option. If
--always-resume=false is given, when all given URIs does not
support resume or aria2 encounters N URIs which does not support
resume
(N is the value specified using --max-resume-failure-tries
option), aria2 download file from scratch. The default behavior
is --always-resume=true, which means if all URIs do not support
resume, download fails. I think this is OK because user normally
don't like to see that partially downloaded file is
overwritten(this is particularly true if file size is big). This
option is useful when aria2 is used as download backend and
graceful falling back to overwritten behavior is preferable.
Added exit status value 8, which means download failed because
server did not support resume.
* src/AbstractCommand.cc
* src/DefaultPieceStorage.cc
* src/DownloadCommand.cc
* src/DownloadResultCode.h
* src/FileEntry.h
* src/FtpNegotiationCommand.cc
* src/HttpResponse.cc
* src/HttpResponseCommand.cc
* src/OptionHandlerFactory.cc
* src/RequestGroup.cc
* src/RequestGroup.h
* src/SegmentMan.cc
* src/SegmentMan.h
* src/prefs.cc
* src/prefs.h
* src/usage_text.h
* test/DefaultPieceStorageTest.cc
* test/SegmentManTest.cc
Don't call prepareForRetry(1) if all segments are ignored in
SegmentMan.
* src/AbstractCommand.cc
* src/BitfieldMan.cc
* src/BitfieldMan.h
* src/SegmentMan.cc
* src/SegmentMan.h
Rewritten PeerStats handling in SegmentMan.cc. Now we have 2 list
of PeerStat in SegmentMan. peerStats is used for calculating
download speed. Therefore all active PeerStats should be in
there. Another one is _fastestPeerStats and it only contains
fastest PeerStat for each hostname/protocol pair. They are used
for updating ServerStat.
* src/DownloadCommand.cc
* src/Request.h
* src/RequestGroupMan.cc
* src/SegmentMan.cc
* src/SegmentMan.h
* test/SegmentManTest.cc
Rewritten PeerStat handling. In the previous implementation,
faster command tries to find slower command. In this new
implementation, slower command tries to find faster command.
PeerStat is now created in each HTTP/FTP request to get download
rate correctly. If the download range is small, the download rate
tends to small and this will occur at the later stage of download.
* src/AbstractCommand.cc
* src/DownloadCommand.cc
* src/FileEntry.cc
* src/FileEntry.h
* src/Request.cc
* src/Request.h
* src/RequestGroup.h
* src/RequestGroupMan.cc
* src/SegmentMan.cc
* src/SegmentMan.h
* test/SegmentManTest.cc
Cached TransferStat/download speed to avoid to sum them up in
each call.
* src/DefaultPeerStorage.cc
* src/DefaultPeerStorage.h
* src/SegmentMan.cc
* src/SegmentMan.h
Reduced the number of calls to PeerStorage::calculateStat() and
SegmentMan::calculateDownloadSpeed() to lower CPU usage on
higher transfer rate.
* src/DefaultBtInteractive.cc
* src/DefaultBtMessageDispatcher.cc
* src/DefaultPeerStorage.cc
* src/DefaultPeerStorage.h
* src/DownloadCommand.cc
* src/DownloadEngine.cc
* src/PeerStorage.h
* src/SegmentMan.cc
* src/SegmentMan.h
* test/MockPeerStorage.h
Added --max-file-not-found=NUM option. If aria2 receives `file
not
found' error from HTTP/FTP server NUM times without getting
single
bytes of data, then force the download to fail.
Basically aria2 tris all URIs available. So if long list of URIs
are
specified in Metalink, then aria2 will take a long time to fail
eventually if all URIs are outdated. This option eases this
situation
to limit the number of servers to try.
* src/FtpNegotiationCommand.cc
* src/HelpItemFactory.cc
* src/HttpResponseCommand.cc
* src/OptionHandlerFactory.cc
* src/PeerStat.h
* src/RequestGroup.cc
* src/RequestGroup.h
* src/SegmentMan.cc
* src/SegmentMan.h
* src/option_processing.cc
* src/prefs.cc
* src/prefs.h
* src/usage_text.h
Implemented download speed based URI selection algorithm.
Introduced new option --uri-selector.
If --uri-selector=feedback is given, aria2 uses download speed
observed
in the previous downloads and chooses fastest server in the URI
list.
Currently at most 10 URIs are considered to introduce
randomeness for
finding better servers. The speed is average download speed in
the
downloads.
On the other hand, if --uri-selector=inorder is given, which is
default,
URI is tried in order in URI list.
The usage text for the new option has not been written yet.
* src/AbstractCommand.cc
* src/DownloadCommand.cc
* src/DownloadEngine.cc
* src/DownloadEngineFactory.cc
* src/InOrderURISelector.cc
* src/InOrderURISelector.h
* src/OptionHandlerFactory.cc
* src/PeerStat.h
* src/RequestGroup.cc
* src/RequestGroup.h
* src/RequestGroupMan.cc
* src/RequestGroupMan.h
* src/SegmentMan.cc
* src/SegmentMan.h
* src/ServerStat.cc
* src/ServerStat.h
* src/ServerStatMan.cc
* src/ServerStatMan.h
* src/ServerStatURISelector.cc
* src/ServerStatURISelector.h
* src/URISelector.h
* src/option_processing.cc
* src/prefs.cc
* src/prefs.h
* test/InOrderURISelectorTest.cc
* test/RequestGroupManTest.cc
* test/ServerStatManTest.cc
* test/ServerStatURISelectorTest.cc
Now SegmentMan::findSlowerSegmentEntry() picks up the segment
for which
the transfer has not yet started.
Rewritten SegmentMan::registerPeerStat() and
SegmentMan::getPeerStat().
* src/SegmentMan.cc
* src/SegmentMan.h
* test/SegmentManTest.cc
Added StringFormat class, which internally calls vasprintf.
operator<< is defined for this class, so it can be used with iostream
classes nicely. SimpleLogger and following functions are rewritten
using StringFormat class.
Besides, now Logger class's methods are non-const, many classes
that has a const Logger* as a member variable are modified to remove
const qualifier from the variable declaration.
* src/HelpItemFactory.cc
* src/Request.cc
* src/SimpleLogger.cc
* src/StringFormat.cc
* src/StringFormat.h
* src/Util.cc
* src/option_processing.cc
* src/version_usage.cc
* test/StringFormatTest.cc
* src/*.h: The classes that has const Logger* as a member variable.
Rewritten SharedHandle. Now copy constructor taking raw pointer
has
keyword explicit and SharedHandle's default constructor
initializes
its internal obj to null, old implementation initializes it
using
obj's default constructor.
To assign null, write SharedHandle<T> x(...); x.reset();
TODO: test/SharedHandleTest.cc needs more tests.
* src/SharedHandle.h
Fixed the bug that always first found Segment is removed from
usedSegmentEntries. Removed unused functions.
* src/SegmentMan.{h, cc}
* test/SegmentManTest.cc
Fixed the bug that ServerHost is not removed.
* src/RequestGroup.cc
Implemented BitTorrent/http/ftp integrated download.
I've rewritten lots of files and now some headers have forward
class declarations to reduce compile time.
The implementation is extremely alpha stage, I recommend to use this
for testing purpose only.
To add the ability to resume downloading a partially downloaded
file
which is downloaded from the beginning:
* src/FileAllocator.h: Made abstract class. New
DefaultFileAllocator
takes this role.
* src/main.cc: Added -c option.
* src/BitfieldMan.h, src/BitfieldMan.cc (setBitRange): New
function.
* src/DiskWriter.h (openExistingFile): Added totalLength
argument.
* src/prefs.h (PREF_CONTINUE): New definition.
* src/SegmentMan.h, src/SegmentMan.cc (markPieceDone): New
function.
* src/DefaultDiskWriter.cc
(createNewDiskWriter): Add GlowFileAllocator to the new object.
* src/AbstractDiskWriter.h
(glowFileAllocator): New variable.
* src/AbstractDiskWriter.cc
(openExistingFile): Now preallocate file space from the end of
the
existing file if totalLength argument is specified and grater
than 0.
* src/UrlRequestInfo.cc: If -c option is specified and the file
to download exists in local, continue the download of the file.
--allow-overwrite=true is assumed in this context.
* src/DefaultFileAllocator.h, src/DefaultFileAllocator.cc: New
class.
* src/GlowFileAllocator.h, src/GlowFileAllocator.cc: New class.
Throw exception if --check-integrity=true is specified but chunk
checksums are not provided:
* src/UrlRequestInfo.cc
Nils Maier