FETCHFORGE 1.0.0 Smart Download Manager - Team variant ================================================================================ WHAT IT IS FetchForge is a SHARED download queue for a team. One machine runs the FetchForge server. That server holds ONE queue. Everybody else on the team submits URLs to it from their own laptop, and everybody sees the same queue and the same progress. A pool of worker goroutines inside the server drains that queue concurrently into one shared output directory. Items move queued -> downloading -> done or failed, and the whole queue is written to disk so it survives a restart of the server. That shared, multi-machine part is the entire point. A download manager whose queue lives in a file on your own laptop cannot do this, no matter how good it is: your teammate has no way to add to your queue, and you have no way to see what they are pulling. FetchForge exists for the case where the queue itself is the thing the team needs to share. Where it sits next to its siblings: GrabFlow simple guided one-at-a-time downloads DownloadPilot a local persistent queue with resume, single user DropDeck batch fetch with per-host rate limiting, single user FetchForge ONE queue on a server, many people submitting to it <- this -------------------------------------------------------------------------------- INSTALL Pre-built binaries are in dist/: dist/fetchforge-linux-amd64 dist/fetchforge-darwin-arm64 dist/fetchforge-darwin-amd64 dist/fetchforge-windows-amd64.exe Copy the one for your platform somewhere on your PATH and mark it executable (chmod +x on Linux and macOS). There is nothing else to install: no runtime, no libraries, no config file. To build from source you need Go 1.24 or newer and nothing else: go build -o fetchforge . The tool uses the Go standard library only. There are no third-party dependencies, so the build works with no network access. -------------------------------------------------------------------------------- QUICK START 1. On the machine that will hold the queue (a build box, a NAS, anything the team can reach): fetchforge serve \ --queue /srv/fetchforge/team-queue.json \ --out /srv/fetchforge/downloads \ --token hunter2 \ --port 8080 \ --workers 4 This blocks and keeps running. It prints one line per submission and one line per finished download. 2. On each teammate's machine: fetchforge submit https://example.com/dataset.tar.gz \ --server http://buildbox:8080 --token hunter2 --as dana fetchforge submit https://example.com/model.safetensors \ --server http://buildbox:8080 --token hunter2 --as raj 3. Anyone can watch the whole team's queue: fetchforge status --server http://buildbox:8080 --token hunter2 team queue at http://buildbox:8080 out /srv/fetchforge/downloads workers 4 ID STATE BY FILE SIZE URL 1 done dana dataset.tar.gz 293.0 KiB https://example.com/... 2 downloading raj - - https://example.com/... 2 total: 1 done, 0 failed, 1 downloading, 0 queued -------------------------------------------------------------------------------- COMMANDS fetchforge serve --queue --out --token SECRET --port N [--workers 4] [--host 0.0.0.0] fetchforge submit --server http://host:port --token SECRET [--as NAME] fetchforge status --server http://host:port --token SECRET [--json] fetchforge help | -h | --help fetchforge --version serve flags --queue where the shared queue is persisted. Created if missing. Reloaded on startup, so restarting the server keeps the team's history. --out directory downloads land in. Created if missing. --token SECRET the shared secret. Every client must send it. --port N TCP port to listen on. --workers N how many downloads run at once. Default 4. --host ADDR interface to bind. Default 0.0.0.0 (all interfaces), so teammates on the network can reach it. Use 127.0.0.1 to keep it local. submit / status flags --server URL base URL of the team server. A bare host:port works too; http:// is assumed. --token SECRET must match the server's --token. --as NAME the teammate name recorded against the item (submit only). Defaults to "anonymous". --json print the server's raw status JSON instead of the table (status only). Suitable for piping into jq or a script. Flags may appear before or after positional arguments. These are equivalent: fetchforge submit https://example.com/f.bin --server http://box:8080 --token t fetchforge submit --server http://box:8080 --token t https://example.com/f.bin Exit codes: 0 on success. 1 for a usage error, an authentication failure, or a network failure reaching the server. -------------------------------------------------------------------------------- HTTP API The server speaks plain HTTP and JSON, so a teammate without the binary can use curl. Every endpoint requires the header "X-Fetch-Token: SECRET". Without it, or with the wrong value, the server answers 401 and does nothing. POST /submit body is either {"url":"https://...","as":"dana"} or just a bare URL as plain text. Appends one item to the shared queue. Answers 200 with the created item. GET /status the whole queue plus per-state counts, the output directory, and the worker count. GET /queue the raw persisted queue document, exactly as it is written to the --queue file. Example: curl -H 'X-Fetch-Token: hunter2' \ -d 'https://example.com/f.bin' \ http://buildbox:8080/submit curl -H 'X-Fetch-Token: hunter2' http://buildbox:8080/status -------------------------------------------------------------------------------- HOW THE DOWNLOADS BEHAVE Concurrency --workers goroutines pull from the single shared queue. All queue mutation happens under one mutex; workers block on a condition variable when the queue is empty and wake as soon as anything is submitted. With --workers 4 and 8 downloads that each take a second, the batch finishes in about two seconds rather than eight. Partial files Each download is written to ".part" and renamed into place only after the body has been fully read and the file closed. A file that appears in the output directory without a .part suffix is complete. A crash mid-download leaves at most a .part file, never a truncated file under the real name. Naming The file name comes from the last path segment of the URL. Path separators and control characters are replaced, and names longer than 180 characters are truncated. If that name is already taken, on disk or by another in-flight download, the item id is appended: alpha.bin becomes alpha-9.bin. Two workers can never pick the same output name. Failures are isolated A 404, a refused connection, a DNS failure, or a disk error marks that one item "failed" and records the reason, which then shows up in status. Every other item in the queue keeps going. One bad URL never stops the batch. Restarts The queue is rewritten atomically (temp file plus rename) on every state change, and again on shutdown. On startup the server reloads it, so completed and failed history is preserved and the item id sequence continues. Anything that was mid-flight when the server died is reset from "downloading" back to "queued" and picked up again. Shutdown SIGINT or SIGTERM stops the listener, lets the workers finish their current item, saves the queue, and exits. The port is released cleanly. -------------------------------------------------------------------------------- WHAT IS IMPLEMENTED Everything described above is implemented and works: - a real blocking HTTP server holding one shared queue - /submit, /status and /queue, all gated on the X-Fetch-Token header - a worker pool of --workers goroutines draining the queue concurrently - per-item states: queued, downloading, done, failed, with failure reasons - .part write plus atomic rename, and collision-free output names - the queue persisted to --queue and reloaded across restarts, with interrupted items requeued - submit and status clients that talk to the server over the network - status as a human table or as raw JSON - graceful shutdown on SIGINT and SIGTERM -------------------------------------------------------------------------------- WHAT IS NOT IMPLEMENTED Please read this before deploying FetchForge anywhere that matters. These are real limitations, not hypotheticals. NO TLS. The server speaks plain HTTP only. The shared token is sent as a cleartext header on every request, and downloaded content crosses the network in the clear. Run FetchForge on a trusted network, or put it behind a reverse proxy that terminates TLS. Do not expose it to the public internet as it stands. NO USER ACCOUNTS AND NO RBAC. There is exactly one shared token for the whole team. Everyone who has it can submit, read the full queue, and see every teammate's URLs. The --as name is a label the client chooses, not an identity: it is not verified and anyone can claim any name. There is no way to give one person read-only access, and no way to revoke one person without rotating the token for everybody. Token comparison is a plain string comparison, not a constant-time one. NO PER-USER QUOTAS OR RATE LIMITING. Nothing caps how many items one person may submit, how much total data the queue may pull, or how fast the server hits any given origin host. One teammate can fill the queue and monopolize all the workers. There is no disk-space check before a download starts, so a large download can fill the output volume. NO WEB DASHBOARD. Status is a CLI table or JSON. There is no browser UI, no live progress bars, and no push notification when a download finishes; you poll with fetchforge status. NO RESUME OF PARTIAL DOWNLOADS. An interrupted download is retried from byte zero; the .part file is discarded rather than continued with a Range request. There are also no automatic retries: a failed item stays failed until someone submits the URL again. Sibling tool DownloadPilot is the one that does resume for a single user's local queue. NO CHECKSUM VERIFICATION. FetchForge does not verify a hash of what it downloaded. Verify it yourself if the content matters. OTHER GAPS. No proxy support and no custom request headers or cookies, so authenticated origins are out of reach. No item removal, cancellation, or reprioritization once submitted. No queue size limit, so the queue file grows without bound. The queue is held entirely in memory and rewritten in full on every state change, which is fine for thousands of items and not for millions. -------------------------------------------------------------------------------- ROADMAP - TLS, with certificate configuration on the server - real user accounts and role-based access control, replacing the single shared token - per-user quotas and per-host rate limiting - a web dashboard with live progress - resumable downloads via HTTP Range, and automatic retry with backoff - optional checksum verification of completed downloads -------------------------------------------------------------------------------- SUPPORT Part of the Techlosoft Smart Download Manager line.