Applied AdaptiveURISelector patch from Aurelien Lefebvre. This
patch adds AdaptiveURISelector, which selects one of the bests
mirrors for first and reserved connections. For supplementary
ones, it returns mirrors which has not been tested yet, and if
each of them already tested, returns mirrors which has to be
tested again. Otherwise, it doesn't return anymore mirrors.
* src/AdaptiveURISelector.cc
* src/AdaptiveURISelector.h
* src/FtpNegotiationCommand.cc
* src/HttpResponseCommand.cc
* src/Makefile.am
* src/Makefile.in
* src/OptionHandlerFactory.cc
* src/RequestGroup.cc
* src/RequestGroup.h
* src/RequestGroupMan.cc
* src/ServerStat.cc
* src/ServerStat.h
* src/ServerStatMan.cc
* src/SpeedCalc.cc
* src/URISelector.h
* src/prefs.cc
* src/prefs.h
* test/RequestGroupManTest.cc
* test/ServerStatManTest.cc
* test/ServerStatTest.cc
Fixed the bug that causes corrupted downloads if HTTP pipelining
is enabled and the server doesn't support keep-alive.
* src/AbstractCommand.cc
* src/DownloadCommand.cc
* src/HttpDownloadCommand.cc
* src/HttpDownloadCommand.h
* src/HttpResponseCommand.cc
* src/Request.cc
* src/Request.h
* test/RequestTest.cc
Use HEAD method to get file size from HTTP server for segmented
downloading.
The request to the BitTorrent Tracker always uses GET method
because the response of the tracker is small and it doesn't need
segmented download.
* src/FtpNegotiationCommand.cc
* src/HttpRequest.cc
* src/HttpResponseCommand.cc
* src/HttpSkipResponseCommand.cc
* src/RequestGroup.cc
* src/RequestGroup.h
* src/TrackerWatcherCommand.cc
* test/HttpRequestTest.cc
Fixed the bug that causes time out when redirection occurs with
content-length 0.
* src/HttpResponseCommand.cc
* src/HttpSkipResponseCommand.cc
* src/HttpSkipResponseCommand.h
Don't set Request::supportsPersistentConnection() in
HttpConnection
since this code is also used in AbstractProxyResponseCommand.
Moved those code to HttpResponse.
* src/HttpResponse.cc
* src/HttpResponse.h
* src/HttpConnection.cc
* src/HttpConnection.h
* src/HttpResponseCommand.cc
* test/HttpResponseTest.cc
Added --max-file-not-found=NUM option. If aria2 receives `file
not
found' error from HTTP/FTP server NUM times without getting
single
bytes of data, then force the download to fail.
Basically aria2 tris all URIs available. So if long list of URIs
are
specified in Metalink, then aria2 will take a long time to fail
eventually if all URIs are outdated. This option eases this
situation
to limit the number of servers to try.
* src/FtpNegotiationCommand.cc
* src/HelpItemFactory.cc
* src/HttpResponseCommand.cc
* src/OptionHandlerFactory.cc
* src/PeerStat.h
* src/RequestGroup.cc
* src/RequestGroup.h
* src/SegmentMan.cc
* src/SegmentMan.h
* src/option_processing.cc
* src/prefs.cc
* src/prefs.h
* src/usage_text.h
Implemented the ability to get timestamp from remote HTTP server
and
apply it to local file. To enable this feature, --remote-time
option
is added. No usage text has been written yet.
If several servers returns difference timestamp, then aria2 uses
latest
one.
* src/CopyDiskAdaptor.cc
* src/CopyDiskAdaptor.h
* src/DirectDiskAdaptor.cc
* src/DirectDiskAdaptor.h
* src/DiskAdaptor.h
* src/File.cc
* src/File.h
* src/HttpHeader.cc
* src/HttpHeader.h
* src/HttpResponse.cc
* src/HttpResponse.h
* src/HttpResponseCommand.cc
* src/HttpResponseCommand.h
* src/MultiDiskAdaptor.cc
* src/MultiDiskAdaptor.h
* src/OptionHandlerFactory.cc
* src/RequestGroup.cc
* src/RequestGroup.h
* src/RequestGroupMan.cc
* src/option_processing.cc
* src/prefs.cc
* src/prefs.h
* test/CopyDiskAdaptorTest.cc
* test/FileTest.cc
* test/Makefile.am
* test/Makefile.in
* test/MultiDiskAdaptorTest.cc
* test/TestUtil.cc
Made files whose name ends with ".gz", ".tgz" not inflated by
Content
Encoding Decoder. Removed size threshold for turning off on the
fly
inflation because resulting file may or may not be inflated
depending
on the file size and I think it is not expected by users.
This change fixes segmentation fault when Metalink file contains
gzipped
file and its filesize is provided.
* src/HttpResponseCommand.cc
* src/HttpResponseCommand.h
Fixed the bug that causes segmentaion fault when resuming
download
using metalink without size tag. Reproducible only using HTTP
URI.
* src/HttpResponseCommand.cc
* src/FtpNegotiationCommand.cc
Fixed broken gzip inflation.
Turn off segmented downloading if gzip content is smaller than
or equal
to 1MiB and inflate the data on the fly, because HTTP response
header
doesn't contain the length of inflated file we can't determin
where
the chunk of data should be written.
On the other hand, if gzip content is larger than 1MB, then turn
off
on the fly inflation, because some servers returns
"content-type: gzip"
for *.tgz, *.gz files.
* src/DownloadCommand.cc
* src/HttpResponseCommand.cc
* src/HttpResponseCommand.h
Added gzip, deflate decoding support in HTTP using libz. If
compiled
with this feature, aria2 sends "Accept-Encoding: deflate, gzip"
header
to a HTTP server. If a server returns "Content-Encoding: gzip"
or
"Content-Encoding: deflate" then, aria2 decodes the response
body on the
fly and writes decoded data to a local disk.
* README
* README.html
* configure.ac
* m4/aria2_arg.m4: Added ARIA2_ARG_WITH and ARIA2_ARG_ENABLE,
they are
wrapper function for AC_ARG_WITH and AC_ARG_ENABLE respectively.
* m4/libz.m4
* src/Decoder.h
* src/DownloadCommand.cc
* src/DownloadCommand.h
* src/Exception.h
* src/GZipDecoder.cc
* src/GZipDecoder.h
* src/HttpHeader.cc
* src/HttpHeader.h
* src/HttpRequest.cc
* src/HttpRequest.h
* src/HttpResponse.cc
* src/HttpResponse.h
* src/HttpResponseCommand.cc
* src/Makefile.am
* test/GZipDecoderTest.cc
* test/HttpRequestTest.cc
* test/HttpResponseTest.cc
* test/Makefile.am
* test/Makefile.in
* test/gzip_decode_test.gz
Receive content body of 302 response so that the connection can
be
reused later when http keep-alive is enabled.
* src/HttpNullDownloadCommand.cc
* src/HttpNullDownloadCommand.h
* src/HttpResponseCommand.cc
* src/HttpResponseCommand.h
Rewritten Exception class. Throw exception object, not its pointer and
catch by reference, so that remove problematic delete operator for
catched exception.
* src/Exception.cc
* src/Exception.h
* test/ExceptionTest.cc
* src/*: All files throwing/catching exception.
* test/*: All files throwing/catching exception.
Reuse socket if the origin server supports HTTP keep-alive and
--enable-http-keep-alive or --enable-http-pipelining is given.
The current implementation doesn't reuse the connections
established
against proxy server.
* src/AbstractCommand.cc
* src/DownloadEngine.cc
* src/DownloadEngine.h
* src/HttpDownloadCommand.cc
* src/HttpInitiateConnectionCommand.cc
* src/HttpResponseCommand.cc
Eliminates the time lag between sequential downloads and
commands in
the same RequestGroup.
In old implementation, aria2 occasionally waits 1 seconds before
executing next command or RequestGroup.
This is really a waste of time, and new implementation
eliminates
this unnecessary time lag.
* src/AbstractCommand.cc
* src/AutoSaveCommand.cc
* src/CheckIntegrityCommand.cc
* src/Command.{cc, h}
* src/DownloadEngine.{cc, h}
* src/DownloadEngineFactory.cc
* src/FileAllocationCommand.cc
* src/FileAllocationDispatcherCommand.cc
* src/FillRequestGroupCommand.cc
* src/FtpInitiateConnectionCommand.cc
* src/HaveEraseCommand.cc
* src/HttpInitiateConnectionCommand.cc
* src/HttpResponseCommand.cc
* src/RealtimeCommand.cc
* src/RequestGroup.cc
* src/RequestGroupMan.cc
* src/StreamFileAllocationEntry.cc
* src/TimeBasedCommand.{cc, h}
* src/TimedHaltCommand.cc
Rewritten SharedHandle. Now copy constructor taking raw pointer
has
keyword explicit and SharedHandle's default constructor
initializes
its internal obj to null, old implementation initializes it
using
obj's default constructor.
To assign null, write SharedHandle<T> x(...); x.reset();
TODO: test/SharedHandleTest.cc needs more tests.
* src/SharedHandle.h
Fixed the bug that causes aria2 not to finish download.
BUG#1855875.
I could reproduce this bug in following procedure:
1. Stop the download at the very beginning(1% or 100KB
downloaded).
2. Restart aria2.
3. You see the download stopped around 99%.
* src/HttpResponseCommand.cc (handleDefaultEncoding)
* src/StreamFileAllocationEntry.cc: Removed the timeout
handling.
If timeout is reached, then _nextCommand is unused and it may
contains
segments and they won't be canceled. Actually, timeout is not
needed
here because if the server dropped connection, then retry is
made.
If several protocols are available for a mirror, aria2 now use
one of
them. --metalink-preferred-protocol option was added to specify
the
preference of protocol.
* src/AbstractCommand.cc
* src/OptionHandlerFactory.cc
* src/ServerHost.{h, cc}
* src/Metalink2RequestGroup.cc
* src/RequestGroup.{h, cc}
* test/RequestGroupTest.cc
* src/option_processing.cc
* src/prefs.h
* src/HttpResponseCommand.cc
* src/MetalinkResource.{h, cc}
* src/FtpNegotiationCommand.cc
* src/MetalinkEntry.{h, cc}
* src/MetalinkEntryTest.cc
Fixed the bug#1841757: aria2 will connect to server to only get
file size.
* src/StreamFileAllocationEntry.cc
* src/option_processing.cc
* src/HttpResponseCommand.cc: Now reuse connection in segmented
downloads.
* src/FtpNegotiationCommand.cc
Suppressed wrong message when finding PreDownloadHandler
* src/RequestGroup.cc
Set Content-Type to SingleFileDownloadContext when http response
is
received.
* src/HttpResponseCommand.cc
* src/HttpResponse.{h, cc}
* test/HttpResponseTest.cc
Rewritten to add content-type support.
* src/DownloadHandler.{h, cc}
* src/BtPostDownloadHandler.{h, cc}
* test/BtPostDownloadHandlerTest.cc
* src/MetalinkPostDownloadHandler.{h, cc}
* test/MetalinkPostDownloadHandlerTest.cc
* src/PostDownloadHandler.{h, cc}
* src/DownloadHandlerConstants.{h, cc}
* src/RequestGroup.cc
* src/HttpResponseCommand.cc
* src/FtpNegotiationCommand.cc
* src/SingleFileDownloadContext.{h, cc}
* src/RequestGroup.h
* src/RequestGroupCriteria.h
* src/ContentTypeRequestGroupCriteria.h
Added 'mem' option value for --follow-metalink,
--follow-torrent.
If it is give, metalink/torrent file is not written to the disk,
but
just is kept in memory. Parsing is occurred on memory.
* src/MetalinkHelper.{h, cc}
* src/MetalinkProcessor.h
* src/Xml2MetalinkProcessor.{h, cc}
* test/Xml2MetalinkProcessorTest.cc
* src/DownloadHandlerFactory.{h, cc}
* test/DownloadHandlerFactoryTest.cc
* src/PreDownloadHandler.{h, cc}
* src/OptionHandlerFactory.cc
* src/DefaultBtContext.{h, cc}
* test/DefaultBtContextTest.cc
* src/version_usage.cc
* src/Metalink2RequestGroup.{h, cc}
* src/RequestGroup.{h, cc}
* src/a2functional.h
* test/a2functionalTest.cc
* src/MemoryBufferPreDownloadHandler.{h, cc}
* src/OptionHandlerImpl.h
* src/prefs.h
* src/Util.{h, cc}
* test/UtilTest.cc
Keep DownloadResult rather than RequestGroup after downloads to
reduce
memory usage.
* src/RequestGroupMan.{h, cc}
* src/DownloadEngine.cc
* src/BtDependency.{h, cc}: Changed the type of dependee from
WeakHandle to SharedHandle because WeakHandle could be null.
* src/RequestGroup.{h, cc}
* src/DownloadEngineFactory.cc
* src/DownloadResult.h
Set totalLength after download finished
* src/UnknownLengthPieceStorage.{h, cc}
Keep torrent file specified in metalink in memory.
* src/Metalink2RequestGroup.cc
* src/BtDependency.cc
* src/TrueRequestGroupCriteria.h
Fixed the bug: seekg is used where seekp should be used.
* src/ByteArrayDiskWriter.cc
* test/ByteArraydiskWriterTest.cc
* src/version_usage.cc
(showVersion): Reworked.
(showUsage): Added examples.
Removed Hint for total length and file name.
* src/Metalink2RequestGroup.cc
* src/RequestGroup.{h, cc}
* src/HttpResponseCommand.cc
* src/FtpNegotiationCommand.cc
Now --follow-torrent and --follow-metalink option work properly.
* src/RequestGroup.cc
Updated man page
* doc/aria2c.1.txt
* doc/aria2c.1
Added the ability to detect duplicate download entry which is
about to
download the same file other RequestGroup is downloading.
* src/RequestGroup.cc
* src/HttpResponseCommand.cc
* src/FtpNegotiationCommand.cc
Reflect the download length of in-flight pieces.
It makes the download length readout more precise.
* src/DefaultPieceStorage.{h, cc}
* test/DefaultPieceStorageTest.cc
* src/a2functional.h
* test/a2functionalTest.cc
Lower CPU load when --max-download-limit is used.
There is up and down in speed indicator when enabling
http-pipelining but a download goes well. I think the problem is
that
because http-pipelining is enabled, DownloadCommand is created
for
each segment and in its constructor, PeerStat::downloadStart()
is
called. In PeerStat::downloadStart(), speed calculation object
is
reseted, which makes download speed zero.
* src/DownloadCommand.cc
Rewritten using accumulate.
* src/RequestGroupMan.cc (calculateStat)
Code clearnup.
* src/FtpNegotiationCommand.cc
* src/HttpResponseCommand.cc
Now a file is stored in the directory specified in .metalnk file
(file[@name]).
* src/Metalink2RequestGroup.cc
Create the directory structure when opening the file if it
doesn't
exist.
* src/AbstractDiskWriter.cc
* src/Util.{h, cc}
* src/File.h
* test/UtilTest.cc
Removed file name comparison
* src/Metalink2RequestGroup.cc
* src/HttpResponseCommand.cc
Rewritten using Util::mkdirs()
* src/FileEntry.cc (setupDir)
* test/FileEntryTest.cc
Updated doc
* src/SingleFileDownloadContext.h
Implemented BitTorrent/http/ftp integrated download.
I've rewritten lots of files and now some headers have forward
class declarations to reduce compile time.
The implementation is extremely alpha stage, I recommend to use this
for testing purpose only.
Merged Ross's win32 patch(manually)
In the course of merging, following files are added.
* src/strptime.{h,c}
* src/libgen.{h,c}
* src/inet_aton.{h,c}
* src/Platform.{h,cc}
* src/localtime_r.{h,c}
* src/getaddrinfo.{h,c}
I've gethered network related things, and put them to following
file:
* src/a2netcompat.h
Also io related things are put to following file:
* src/a2io.h
Changed %lld to %s because mingw32 doesn't recognize %lld.
* src/message.h
(MSG_ALLOCATION_COMPLETED)
(EX_TOO_LARGE_FILE)
(EX_SIZE_MISMATCH)
(EX_FILE_OFFSET_OUT_OF_RANGE)
(EX_INVALID_CHUNK_CHECKSUM)
(EX_INVALID_RANGE_HEADER)
* src/FileAllocationCommand.cc
* src/HttpResponse.cc
* src/RequestGroup.cc
* src/MultiDiskAdaptor.cc
* src/OptionHandlerImpl.h
* src/HttpResponseCommand.cc
* src/FtpNegotiateCommand.cc
* src/IteratableChecksumValidator.cc
* src/SegmentMan.cc
* src/ChunkChecksumValidator.cc
Added Randomizer::getRandomNumber(long int)
* src/Randomizer.h (getRandomNumber)
* src/SimpleRandomizer.h (getRandomNumber)
* src/BitfieldMan.cc (getMissingIndexRandomly): Use this new
function.
* src/Util.cc (randomAlpha): Use this new function.
Fixed the bug that prevents filename in content-disposition from
being retrieved when filename is not quoted.
* src/Util.cc (getContentDispositionFilename)
Fixed the bug that causes infinate loop and memory leak when
file open
operation failed.
* src/HttpResponseCommand.cc (handleDefaultEncoding)
RequestGroup::getNextCommand() was renamed to
createNextCommand().
Added its overloaded method.
* src/RequestGroup.h
(_numConcurrentCommand): New variable.
(setNumConcurrentCommand): New function.
* src/RequestGroup.cc
Abort download if same file is being downloaded concurrently.
* src/RequestGroup.h, src/RequestGroupMan.cc
(isSameFileBeingDownloaded): New function.
* src/HttpResponseCommand.cc (executeInternal)
* src/FtpNegotiateCommand.cc (recvSize)
* src/message.h (EX_DUPLICATE_FILE_DOWNLOAD): New definition.
* main.cc: Added help message for -i option.
Tatsuhiro Tsujikawa