DropDeck 1.0.0 Smart Download Manager - batch / manifest edition ================================================================================ WHAT IT IS DropDeck downloads a whole list of URLs in one shot, politely. You hand it a manifest (a plain text file, one URL per line) and it fetches everything into a directory you choose, while holding each host to a bandwidth budget and a maximum number of simultaneous connections. This is the tool for grabbing a big media playlist, a dataset split across dozens of files, or a mirror listing, WITHOUT hammering the server hard enough to get yourself throttled, tar-pitted, or IP-banned. HOW IT DIFFERS FROM ITS SIBLINGS GrabFlow Simple, guided, one download at a time. Great for a single file when you want hand-holding. DownloadPilot A persistent queue with resume and retry. Great for long, interrupted, restartable jobs. DropDeck Batch manifest fetching with a REAL per-host token-bucket rate limiter and per-host concurrency caps. Neither GrabFlow nor DownloadPilot does rate limiting. That is the entire point of DropDeck. If you do not need to throttle yourself, one of the siblings is probably a better fit. INSTALL Pre-built binaries are in dist/. Download the one for your platform, make it executable, and put it on your PATH: chmod +x dropdeck-linux-amd64 mv dropdeck-linux-amd64 /usr/local/bin/dropdeck Building from source needs nothing but a Go 1.24 toolchain. There are no third-party dependencies, so no network access is required to build: go build -o dropdeck . USAGE dropdeck fetch --out [options] dropdeck plan [--json] dropdeck help | -h | --help dropdeck version FETCH OPTIONS --out Destination directory. Required. --rate Rate limit PER HOST, in bytes per second. 0 means unlimited, which is the default. Accepts k / m / g suffixes (1024-based), so --rate 500000, --rate 512k and --rate 2m all work. --per-host-concurrency Maximum simultaneous downloads per host. Default 2. --timeout Response-header timeout per request. Default 30. --apply Actually download. See SAFETY below. PLAN OPTIONS --json Emit machine-readable JSON instead of a table. Flags may be written before or after the manifest argument. These are the same: dropdeck fetch list.txt --out ./media --rate 512k --apply dropdeck fetch --apply --rate 512k --out ./media list.txt SAFETY: DRY RUN BY DEFAULT `fetch` writes nothing unless you pass --apply. Run without --apply and DropDeck prints exactly what it would do: every URL, the file it would be written to, which files are already present locally, and the per-host grouping and limits. The destination directory is not even created. Read the plan, then re-run with --apply. MANIFEST FORMAT One URL per line. For example: # comments start with a hash and are ignored https://example.com/media/track01.mp3 https://example.com/media/track02.mp3 # blank lines are fine too https://cdn.example.net/art/cover.jpg Blank lines are skipped. Lines whose first non-space character is '#' are comments. Leading and trailing whitespace is trimmed. Only http and https URLs are accepted; anything else is reported as an unusable line and the rest of the manifest still runs. An empty manifest, or one containing nothing but comments, prints a short "nothing to do" message and exits 0. That is not an error. HOW THE RATE LIMITER ACTUALLY WORKS Each host gets its own token bucket. One token is one byte, tokens accrue at the rate you set, and the bucket holds up to 100 milliseconds' worth of burst. The bucket is consumed INSIDE the read loop: the HTTP response body is wrapped in a limiting reader that blocks until tokens are available before every single read. It is not a sleep between files, and it is not a post-hoc average. A single 2 MB file at --rate 500000 takes about 4.2 seconds, start to finish. Because the budget is per host, a manifest spanning two hosts saturates both independently: two hosts at --rate 500000 move roughly 1 MB/s in total, not 500 KB/s. Concurrency is per host too, so a slow host cannot starve a fast one. Within a single host, the rate limit is SHARED across that host's concurrent transfers. Four files from one host at --rate 500000 will finish in about the same total time whether you allow 1 or 4 at a time; concurrency changes the interleaving, not the host's total bandwidth. This is deliberate: --rate is a promise about how hard you hit a server, and raising concurrency must never quietly break it. HOW FILES ARE WRITTEN Each download goes to ".part" and is renamed to "" only after the transfer completes and the byte count matches what the server advertised. A download that fails, is truncated, or is interrupted leaves NO .part file behind and never leaves a half-file under the real name. File names come from the last path segment of the URL. A URL with no useful path segment is saved as "index". If two URLs would produce the same name, the later ones get -2, -3 and so on, so nothing is silently clobbered. DropDeck prints the first 12 hex characters of each file's SHA-256 as it finishes, so you can spot-check integrity without a second pass over the data. RE-RUNNING IS CHEAP AND SAFE Before downloading, DropDeck issues a HEAD request and compares the size the server reports against the file already on disk. If they match, the file is skipped and its bytes are never transferred again. So re-running the same fetch over a mostly-complete directory costs a handful of HEAD requests. If the local file exists but the size does not match, it is re-downloaded and replaced. That makes a re-run a cheap way to repair a directory that was damaged or partially populated. ONE BAD URL DOES NOT SINK THE BATCH A 404, a refused connection, a dead host, a truncated response - each is reported against its own URL, and every other download in the manifest still completes. At the end you get a summary and an itemised failure list. EXIT STATUS 0 Success, or an explicit help request, or a dry run, or a manifest with nothing to download. 1 Bad invocation, unreadable manifest, or at least one download failed. Because a partial batch exits 1, scripts can branch on it. The successfully downloaded files are still on disk. EXAMPLES # See what a manifest contains, grouped by host dropdeck plan playlist.txt # Same, as JSON, for scripting dropdeck plan playlist.txt --json # Dry run: show the plan, download nothing dropdeck fetch playlist.txt --out ./media # Do it, at 500 KB/s per host, 2 connections per host dropdeck fetch playlist.txt --out ./media --rate 500000 --apply # Be gentler: 128 KB/s per host, one connection at a time dropdeck fetch playlist.txt --out ./media --rate 128k \ --per-host-concurrency 1 --apply # Go fast, no limit dropdeck fetch playlist.txt --out ./media --per-host-concurrency 8 --apply ================================================================================ WHAT IS IMPLEMENTED ================================================================================ * Batch downloading from a plain-text manifest, with # comments and blank lines. * A genuine per-host token-bucket rate limiter, enforced inside the read loop on the response body. * Per-host concurrency caps, applied independently per host. * Dry run by default; --apply required before anything is written. * .part staging with atomic rename on success, and cleanup on any failure. * Truncated-response detection by comparing bytes read to Content-Length. * Idempotent re-runs via a HEAD size check; matching files are skipped. * Automatic re-download when a local file's size does not match the server. * Fault isolation: one failing URL does not stop the rest of the batch. * SHA-256 prefix printed for every completed file. * Filename collision handling and safe name derivation (nothing can escape the --out directory). * `plan` with a human-readable table and a --json mode. * http and https, with system proxy settings honoured. * Graceful Ctrl-C: in-flight transfers stop and their .part files are removed. ================================================================================ WHAT IS NOT IMPLEMENTED (ROADMAP) ================================================================================ These do not exist yet. Do not expect them to work: * Browser integration. There is no extension, no click-to-capture, and no hook into a browser's download list. You write the manifest yourself. * Torrent support and segmented multi-connection downloads. DropDeck opens exactly one connection per file. It does not split a single file across several ranged requests to speed it up, and it speaks no peer-to-peer protocol. * Bandwidth scheduling by time of day. You cannot say "full speed after midnight, 100 KB/s during business hours". The --rate you pass applies for the whole run. Also absent, and worth knowing about: * No resume of a partial transfer. An interrupted file restarts from zero on the next run. (DownloadPilot is the sibling that does resume.) * No automatic retry or backoff. A failed URL is reported, not retried. * No authentication, cookies, or custom headers. * No checksum manifest verification. DropDeck reports SHA-256 values but does not compare them against an expected list. * The idempotency check trusts the server's Content-Length. A server that reports the wrong size, or none at all, will cause a re-download.