github
/
aria2
mirror of https://github.com/aria2/aria2
1
0
Fork 0
Code Issues Releases Wiki Activity
4,579 Commits
9 Branches
152 Tags
64 MiB
Commit Graph

162 Commits (d0ad7033c3e11549529b44990ef022a655877601)

Author SHA1 Message Date
Tatsuhiro Tsujikawa d0ad7033c3 sftp: basic download works 
This commits implements basic downloading feature for sftp URI.  This
includes segmented downloads, and connection (sftp session) pooling.
Download via HTTP proxy has not been implemented yet.
 2015-05-12 00:31:57 +09:00
Tatsuhiro Tsujikawa 692a674fe0 SocketRecvBuffer: Eliminate memmove 
Since we don't try to read into buffer if buffer is not empty, we
don't have to memmove things.  This commit mostly rewrites
SocketRecvBuffer.
 2015-04-14 00:25:23 +09:00
Dan Fandrich 0538aa40da Fixed some typos in code comments 2015-01-16 01:17:35 +01:00
Nils Maier 010131161e Remove ENABLE_MESSAGE_DIGEST, since we got the internal md, always 2014-04-17 18:02:42 +02:00
Nils Maier c694a05797 Fix some warnings 2014-04-13 18:51:05 +02:00
Nils Maier 2db0c81fc8 Stuff memory holes 
Padding changes on x86_64:
- RequestGroup: 29 -> 5 bytes
- DownloadContext:  25 -> 1 bytes
- Cookie: 20 -> 4 bytes
- Command: 8 -> 0 bytes (affects many derived as well)
- HttpRequest: 20 -> 4 bytes
- DownloadCommand: 18 -> 2 bytes
- TimeBasedCommand: 14 -> 6 bytes
- AbstractCommand: 13 -> 5 bytes
- HttpSkipResponseCommand: 12 -> 4 bytes
- FileEntry: 10 -> 2 bytes
- DownloadResult: 15 -> 7 bytes
- UriStruct: 12 -> 4 bytes
- Piece: 11 -> 3 bytes
- BitfieldMan -> 11 -> 3 bytes
 2013-12-09 15:41:43 +01:00
Tatsuhiro Tsujikawa 6b397c8125 Use std::unique_ptr for CheckIntegrityEntry and FileAllocationEntry 2013-07-06 15:56:08 +09:00
Tatsuhiro Tsujikawa 57f1902ee1 Use std::unique_ptr for StreamFilter instead of std::shared_ptr 2013-07-04 00:39:11 +09:00
Tatsuhiro Tsujikawa fa9f3fb5a3 Wrap Command object in std::unique_ptr 2013-06-23 21:55:52 +09:00
Tatsuhiro Tsujikawa 07d270c87e Require -std=c++11 and use std::shared_ptr instead of SharedHandle 2013-06-22 01:10:38 +09:00
Tatsuhiro Tsujikawa 2c5e77f422 Flush cached data on checkout/cancel Segment 
This change prevents the cached data from overlap because of BT peers.
 2012-12-06 01:16:28 +09:00
Tatsuhiro Tsujikawa 56c498bcbf Clear write cache on Segment::clear and Piece::clearAllBlock 2012-12-06 00:39:14 +09:00
Tatsuhiro Tsujikawa f314719618 Added --disk-cache option 
This option enables disk cache. If SIZE is 0, the disk cache is
disabled. This feature caches the downloaded data in memory, which
grows to at most SIZE bytes. The cache storage is created for aria2
instance and shared by all downloads. The one advantage of the disk
cache is reduce the disk seek time because the data is written in
larger unit and it is reordered by the offset of the file. If the
underlying file is heavily fragmented it is not the case.
 2012-12-03 01:39:10 +09:00
Tatsuhiro Tsujikawa 0ecfa19925 Reworked download/upload statistics calculation 
The old implementation calculates download/upload statistics for a
RequestGroup by summing up all PeerStat objects. For global
statistics, those are summed together. This clearly incurs runtime
penalty and we introduced some kind of caching which updates
statistics every 250ms but it did not work right.

This change removes all these aggregation code, and instead makes
RequestGroup and RequestGroupMan objects hold NetStat object and
download/upload bytes are directly calculated by thier own NetStat.
This is far more simplar than the old way and less runtime penalty and
brings more accuracy.
 2012-10-25 23:33:45 +09:00
Tatsuhiro Tsujikawa 46bdaf0e8d Made HTTP/FTP download speed limiter more accurate 
We have to do the same thing for BitTorrent.
 2012-10-24 01:05:41 +09:00
Tatsuhiro Tsujikawa 3258614033 Removed *Handle typedef 2012-09-28 23:27:46 +09:00
Tatsuhiro Tsujikawa e2340efe27 Use std::vector instead of std::map for ContextAttribute objects 
Now key for ContextAttribute object store is int instead of string.
 2012-09-22 17:37:30 +09:00
Tatsuhiro Tsujikawa 16d395779a Removed static_cast<int64>()s where they are unnecessary 2012-06-25 23:44:52 +09:00
Tatsuhiro Tsujikawa 1b874268a0 Use int64_t instead of off_t 
Some classes such as DiskAdaptor, DiskWriter, FileAllocationIterator
and ChecksumValidator are left unchanged.
 2012-06-25 23:35:24 +09:00
Tatsuhiro Tsujikawa 4b03634f36 Use PRId64 for int64_t format specifier 2012-06-25 22:43:33 +09:00
Tatsuhiro Tsujikawa b6fd4366fe Use short form of util::startsWith and util::endsWith 2012-01-11 01:17:51 +09:00
Tatsuhiro Tsujikawa f0bcfa822e Use int instead of unsigned int where unsigned int is not needed. 2011-12-09 21:39:43 +09:00
Tatsuhiro Tsujikawa 9d15d732ed Use int32_t for piece length instead of size_t 2011-12-09 21:39:43 +09:00
Tatsuhiro Tsujikawa 12988e5282 Replaced uint64_t with off_t or int64_t. 
Since off_t is int64_t with LFS, we cannot take advantage of extra
capacity of uint64_t.
 2011-12-09 21:39:43 +09:00
Tatsuhiro Tsujikawa 1687741303 Use fmt instead of util::itos 2011-11-13 20:55:06 +09:00
Tatsuhiro Tsujikawa 79876af88f Removed util::endsWith(a, b). Added util::iendsWith() 2011-11-05 00:25:24 +09:00
Tatsuhiro Tsujikawa 310e58bc62 Stored hash value as raw byte string, instead of hex digest. 2011-10-13 21:40:07 +09:00
Tatsuhiro Tsujikawa e8e3a6f259 wallclock is now retrieved using global::wallclock() call. 
This is necessary to avoid global variable initialization order
problem.
 2011-08-10 01:17:28 +09:00
Tatsuhiro Tsujikawa 03f0774482 Consistent naming scheme for hash type and digest for DownloadContext. 2011-07-27 22:44:30 +09:00
Tatsuhiro Tsujikawa 2666bc8170 Renamed MessageDigestHelper as message_digest_helper 
Static member variables in MessageDigestHelper class are now functions
under message_digest namespace.
 2011-02-05 23:38:51 +09:00
Tatsuhiro Tsujikawa 4824b09237 Reverted comment which was removed unintentionally. Moved 2 
DownloadEngine related lines outside of #ifdef ENABLE_MESSAGE_DIGEST.
 2011-01-16 22:35:30 +09:00
Tatsuhiro Tsujikawa ea1b4b3ee5 Eliminated SocketCore::peekData() form HTTP/FTP downloads. 
We introduced SocketRecvBuffer which buffers received bytes. Since
HTTP response header and response body are divided with \r\n, we have
to buffer up several bytes to find this delimiter. We use
SocketRecvBuffer to hold these bytes and only consumes header and
passes SocketRecvBuffer, which may contain head of response body, to
next Command.  Since FTPConnection doesn't use SocketCore::peekData(),
we left it as is.
 2011-01-16 16:55:41 +09:00
Tatsuhiro Tsujikawa 235b5a0848 Allocate buffer used by DownloadCommand in stack. 2010-12-08 20:41:12 +09:00
Tatsuhiro Tsujikawa 947967fc63 Renamed DownloadErrorCode.h as error_code.h. Renamed 
downloaderrorcode::Value as error_code::Value.
 2010-11-28 16:52:02 +09:00
Tatsuhiro Tsujikawa 03417e94b4 2010-11-20 Tatsuhiro Tsujikawa <t-tujikawa@users.sourceforge.net> 
Use format specifier %lld for cuid_t. Defined cuid_t as long long
	int instead of int64_t, since g++ complains int64_t is not
	suitable for %lld.
 2010-11-20 12:12:06 +00:00
Tatsuhiro Tsujikawa 694fb307aa 2010-11-20 Tatsuhiro Tsujikawa <t-tujikawa@users.sourceforge.net> 
Replaced StringFormat with fmt
 2010-11-20 09:36:14 +00:00
Tatsuhiro Tsujikawa 580098eb49 2010-11-20 Tatsuhiro Tsujikawa <t-tujikawa@users.sourceforge.net> 
Rewritten Logger interface. Logger now writes source file name and
	line number in log file.
 2010-11-20 08:21:36 +00:00
Tatsuhiro Tsujikawa 1372ac51ad 2010-11-14 Tatsuhiro Tsujikawa <t-tujikawa@users.sourceforge.net> 
Added .cc file for classes/structs that only provided by header
	file. Defined non-POD classes' ctor, dtor in .cc file.  Moved
	implementation code in header file to .cc file for major
	classes/strucsts.
 2010-11-14 07:17:55 +00:00
Tatsuhiro Tsujikawa 93e5dbed32 2010-11-12 Tatsuhiro Tsujikawa <t-tujikawa@users.sourceforge.net> 
Removed SharedHandle::isNull(). Instead we added operator* and
	operator unspecified_bool_type. Removed use of WeakHandle and
	replaced with raw pointer.
 2010-11-12 12:48:48 +00:00
Tatsuhiro Tsujikawa 89f997ec0d 2010-11-11 Tatsuhiro Tsujikawa <t-tujikawa@users.sourceforge.net> 
Replaced MessageDigestContext with MessageDigest.  Cleaned up
	unnecessary functions in MessageDigestHelper.
	* src/BtPieceMessage.cc
	* src/Checksum.h
	* src/DHTTokenTracker.cc
	* src/DownloadCommand.cc
	* src/DownloadCommand.h
	* src/HashFuncEntry.h
	* src/IteratableChecksumValidator.cc
	* src/IteratableChecksumValidator.h
	* src/IteratableChunkChecksumValidator.cc
	* src/IteratableChunkChecksumValidator.h
	* src/LibgcryptMessageDigestImpl.cc
	* src/LibgcryptMessageDigestImpl.h
	* src/LibsslMessageDigestImpl.cc
	* src/LibsslMessageDigestImpl.h
	* src/MSEHandshake.cc
	* src/MSEHandshake.h
	* src/Makefile.am
	* src/MessageDigest.cc
	* src/MessageDigest.h
	* src/MessageDigestHelper.cc
	* src/MessageDigestHelper.h
	* src/MessageDigestImpl.h
	* src/MetalinkParserController.cc
	* src/Piece.cc
	* src/Piece.h
	* src/UTMetadataDataExtensionMessage.cc
	* src/bittorrent_helper.cc
	* src/messageDigest.cc: Removed
	* src/messageDigest.h: Removed
	* src/util.cc
	* src/version_usage.cc
	* test/BittorrentHelperTest.cc
	* test/GZipDecoderTest.cc
	* test/GZipDecodingStreamFilterTest.cc
	* test/IteratableChecksumValidatorTest.cc
	* test/IteratableChunkChecksumValidatorTest.cc
	* test/Makefile.am
	* test/MessageDigestHelperTest.cc
	* test/MessageDigestTest.cc
	* test/Metalink2RequestGroupTest.cc
	* test/MetalinkProcessorTest.cc
	* test/PieceTest.cc
	* test/TestUtil.cc
	* test/TestUtil.h
	* test/UTMetadataDataExtensionMessageTest.cc
	* test/UTMetadataPostDownloadHandlerTest.cc
 2010-11-11 02:56:24 +00:00
Tatsuhiro Tsujikawa 236e64cb25 2010-10-30 Tatsuhiro Tsujikawa <t-tujikawa@users.sourceforge.net> 
Use unnamed namespace instead of static keyword.
	* src/AbstractCommand.cc
	* src/AdaptiveURISelector.cc
	* src/Base64.cc
	* src/BitfieldMan.cc
	* src/BtDependency.cc
	* src/ConsoleStatCalc.cc
	* src/ContentTypeRequestGroupCriteria.cc
	* src/CookieStorage.cc
	* src/DHTMessageFactoryImpl.cc
	* src/DHTRoutingTableDeserializer.cc
	* src/DefaultBtAnnounce.cc
	* src/DefaultBtProgressInfoFile.cc
	* src/DefaultPeerStorage.cc
	* src/DefaultPieceStorage.cc
	* src/DownloadCommand.cc
	* src/DownloadEngine.cc
	* src/EpollEventPoll.cc
	* src/ExpatMetalinkProcessor.cc
	* src/ExpatXmlRpcRequestProcessor.cc
	* src/FileEntry.cc
	* src/HttpRequest.cc
	* src/HttpRequestCommand.cc
	* src/HttpResponseCommand.cc
	* src/KqueueEventPoll.cc
	* src/LongestSequencePieceSelector.cc
	* src/MetalinkParserStateV3Impl.cc
	* src/MetalinkParserStateV4Impl.cc
	* src/MultiDiskAdaptor.cc
	* src/MultiUrlRequestInfo.cc
	* src/OptionParser.cc
	* src/PeerSessionResource.cc
	* src/PortEventPoll.cc
	* src/Request.cc
	* src/RequestGroupMan.cc
	* src/SelectEventPoll.cc
	* src/SessionSerializer.cc
	* src/SimpleLogFormatter.cc
	* src/Sqlite3CookieParser.cc
	* src/TrackerWatcherCommand.cc
	* src/XML2SAXMetalinkProcessor.cc
	* src/Xml2XmlRpcRequestProcessor.cc
	* src/XmlRpcMethod.cc
	* src/XmlRpcMethodImpl.cc
	* src/XmlRpcResponse.cc
	* src/base32.cc
	* src/bencode2.cc
	* src/bittorrent_helper.cc
	* src/download_helper.cc
	* src/main.cc
	* src/messageDigest.cc
	* src/option_processing.cc
	* src/util.cc
	* test/AnnounceListTest.cc
	* test/BtRegistryTest.cc
	* test/DHTBucketTest.cc
	* test/DHTRoutingTableTest.cc
	* test/DefaultBtAnnounceTest.cc
	* test/FileEntryTest.cc
	* test/FtpConnectionTest.cc
	* test/MSEHandshakeTest.cc
	* test/MagnetTest.cc
	* test/XmlRpcMethodTest.cc
	* test/array_funTest.cc
 2010-10-30 16:02:15 +00:00
Tatsuhiro Tsujikawa b7df39b7ae 2010-10-28 Tatsuhiro Tsujikawa <t-tujikawa@users.sourceforge.net> 
Fixed compiler warning about printf format.
	* src/AdaptiveURISelector.cc
	* src/BtPieceMessage.cc
	* src/DHTAbstractNodeLookupTask.h
	* src/DHTMessageFactoryImpl.cc
	* src/DHTPeerLookupTask.cc
	* src/DHTReplaceNodeTask.cc
	* src/DefaultBtAnnounce.cc
	* src/DefaultBtInteractive.cc
	* src/DefaultBtMessageDispatcher.cc
	* src/DefaultBtProgressInfoFile.cc
	* src/DefaultBtRequestFactory.cc
	* src/DefaultExtensionMessageFactory.cc
	* src/DefaultPieceStorage.cc
	* src/DownloadCommand.cc
	* src/FileAllocationCommand.cc
	* src/FileEntry.cc
	* src/HandshakeExtensionMessage.cc
	* src/IteratableChunkChecksumValidator.cc
	* src/LibgcryptDHKeyExchange.h
	* src/LibsslDHKeyExchange.h
	* src/LpdDispatchMessageCommand.cc
	* src/MSEHandshake.cc
	* src/MessageDigestHelper.cc
	* src/MultiDiskAdaptor.cc
	* src/PeerConnection.cc
	* src/SegmentMan.cc
	* src/TimedHaltCommand.cc
	* src/UTMetadataRequestExtensionMessage.cc
	* src/UTPexExtensionMessage.cc
	* src/bencode2.cc
	* src/bittorrent_helper.cc
	* src/message.h
 2010-10-28 14:19:29 +00:00
Tatsuhiro Tsujikawa ac7d8dc483 2010-09-23 Tatsuhiro Tsujikawa <t-tujikawa@users.sourceforge.net> 
Made small optimization.
	* src/DownloadCommand.cc
 2010-09-23 11:49:37 +00:00
Tatsuhiro Tsujikawa 5bf8a53b3f 2010-09-12 Tatsuhiro Tsujikawa <t-tujikawa@users.sourceforge.net> 
Fixed compile error without OpenSSL/GnuTLS.
	* src/DownloadCommand.cc
 2010-09-11 15:57:41 +00:00
Tatsuhiro Tsujikawa 70ebb788b2 2010-09-12 Tatsuhiro Tsujikawa <t-tujikawa@users.sourceforge.net> 
Fixed compile error without BitTorrent support and OpenSSL/GnuTLS.
	* src/DownloadCommand.cc
	* src/DownloadCommand.h
	* src/RequestGroup.cc
 2010-09-11 15:54:53 +00:00
Tatsuhiro Tsujikawa efbfe4c006 2010-09-06 Tatsuhiro Tsujikawa <t-tujikawa@users.sourceforge.net> 
Data from remote server in HTTP/FTP download are now written to
	the disk(or memory) through StreamFilter. Decoding chunked and
	gziped streams are done cascading StreamFilter.
	Removed inefficient 1byte read code.
	* src/ChunkedDecodingStreamFilter.cc
	* src/ChunkedDecodingStreamFilter.h
	* src/DownloadCommand.cc
	* src/DownloadCommand.h
	* src/GZipDecodingStreamFilter.cc
	* src/GZipDecodingStreamFilter.h
	* src/HttpConnection.cc
	* src/HttpDownloadCommand.cc
	* src/HttpResponse.cc
	* src/HttpResponse.h
	* src/HttpResponseCommand.cc
	* src/HttpResponseCommand.h
	* src/HttpSkipResponseCommand.cc
	* src/HttpSkipResponseCommand.h
	* src/Makefile.am
	* src/NullSinkStreamFilter.cc
	* src/NullSinkStreamFilter.h
	* src/RequestGroup.cc
	* src/SinkStreamFilter.cc
	* src/SinkStreamFilter.h
	* src/StreamFilter.cc
	* src/StreamFilter.h
	* test/ChunkedDecodingStreamFilterTest.cc
	* test/GZipDecodingStreamFilterTest.cc
	* test/HttpResponseTest.cc
	* test/Makefile.am
	* test/MockSegment.h
 2010-09-06 14:29:36 +00:00
Tatsuhiro Tsujikawa dfdf2d8e56 2010-08-04 Tatsuhiro Tsujikawa <t-tujikawa@users.sourceforge.net> 
Don't use pre-calculate hash value when end-game mode.  Throw
	exception if bad piece is received.
	* src/BtPieceMessage.cc
	* src/DownloadCommand.cc
 2010-08-04 14:15:29 +00:00
Tatsuhiro Tsujikawa cd13647abe 2010-07-17 Tatsuhiro Tsujikawa <t-tujikawa@users.sourceforge.net> 
Pass maxSplitSize as an argument, instead of member variable of
	DefaultPieceStorage. SegmentMan::getSegment(cuid_t,size_t index)
	was renamed to SegmentMan::getSegmentWithIndex(...)
	* src/AbstractCommand.cc
	* src/AbstractCommand.h
	* src/DefaultPieceStorage.cc
	* src/DefaultPieceStorage.h
	* src/DownloadCommand.cc
	* src/FtpNegotiationCommand.cc
	* src/HttpResponseCommand.cc
	* src/PieceStorage.h
	* src/RequestGroup.cc
	* src/SegmentMan.cc
	* src/SegmentMan.h
	* src/UnknownLengthPieceStorage.cc
	* src/UnknownLengthPieceStorage.h
	* test/MockPieceStorage.h
	* test/SegmentManTest.cc
 2010-07-17 14:36:18 +00:00
Tatsuhiro Tsujikawa 8274432f14 2010-07-12 Tatsuhiro Tsujikawa <t-tujikawa@users.sourceforge.net> 
Set end byte in Range header if start byte > 0 to get more chance
	to pool socket.
	* src/DefaultPieceStorage.cc
	* src/DefaultPieceStorage.h
	* src/DownloadCommand.cc
	* src/HttpDownloadCommand.cc
	* src/HttpRequest.cc
	* src/HttpRequest.h
	* src/HttpRequestCommand.cc
	* src/PieceStorage.h
	* src/UnknownLengthPieceStorage.h
	* test/DefaultPieceStorageTest.cc
	* test/HttpRequestTest.cc
	* test/MockPieceStorage.h
 2010-07-12 11:55:23 +00:00
Tatsuhiro Tsujikawa f3b097b5af 2010-07-11 Tatsuhiro Tsujikawa <t-tujikawa@users.sourceforge.net> 
Fixed the bug that segments are not filled to
	Request::getMaxPipelinedRequest().
	Make sure that trailing data of transfer encoding is read propery,
	after file data is received.
	* src/AbstractCommand.cc
	* src/DownloadCommand.cc
	* src/DownloadCommand.h
	* src/FtpDownloadCommand.cc
	* src/FtpDownloadCommand.h
	* src/HttpDownloadCommand.cc
	* src/HttpDownloadCommand.h
 2010-07-11 14:21:17 +00:00