=============================================================================== SectorPilot -- read-only block-level surface scan and health map Techlosoft "Disk Operations Suite" =============================================================================== SectorPilot reads an image (or any file) block by block, times every single read, and draws a map of the whole surface: which blocks read cleanly, which were SLOW, and which FAILED outright. It saves the scan so you can re-run it later and see whether the trouble is SPREADING. Where the rest of the suite answers different questions: DiskOps / DriveForge measure THROUGHPUT -- how fast is this drive overall. PartitionGuard parses STRUCTURE -- what does the partition table say. SectorPilot localises TROUBLE -- which byte offset is struggling, and is it getting worse. An aggregate megabytes-per-second number tells a technician that a drive feels slow. SectorPilot tells them that blocks 8,690 to 8,695 -- byte offsets 569,507,840 to 569,835,520 -- took thirty times longer than the median, that they were fine on Monday, and that four of their neighbours have joined them since. That is the difference between a benchmark and a triage tool. SectorPilot is strictly READ-ONLY on the thing it scans. It opens the image with os.Open (O_RDONLY) and only ever calls ReadAt on that handle. It contains no code that can write to, resize, repair or erase a target. ------------------------------------------------------------------------------- READ THIS FIRST -- THE THREE THINGS SECTORPILOT IS NOT ------------------------------------------------------------------------------- 1. IT SCANS FILES AND IMAGES, NOT RAW BLOCK DEVICES. SectorPilot works on regular files: disk images, forensic images, virtual machine disks, big archives, or any file at all. It deliberately REFUSES anything that is not a regular file, and will tell you so: sectorpilot: /dev/sda is not a regular file; sectorpilot scans files and images, not raw block devices or special files Scanning a raw device is a different job. It needs elevated privileges, it needs platform-specific size discovery (BLKGETSIZE64 on Linux, DKIOCGETB on macOS, IOCTL_DISK_GET_LENGTH_INFO on Windows), it needs sector-aligned unbuffered reads to mean anything, and getting it wrong on a live disk is how people lose data. SectorPilot does not pretend to do it. Image the device first (dd, ddrescue, your imaging tool of choice), then scan the image. 2. IT CANNOT READ SMART ATTRIBUTES. No reallocated sector count, no pending sector count, no power-on hours, no drive self-test log, no vendor attributes. SMART requires device-level ATA or NVMe pass-through commands, which need raw device access and privileges SectorPilot does not take. If you want SMART, run smartctl alongside this tool. SectorPilot measures what the read path actually delivers; SMART reports what the drive says about itself. They are complementary, and this tool only does the first. 3. A SLOW BLOCK IS A HINT, NOT A DIAGNOSIS. This is the most important sentence in this file. SectorPilot measures the wall-clock time of a read as seen by this process. That number includes the filesystem, the page cache, the I/O scheduler, the device queue, the virtualisation layer if there is one, and whatever else the machine happened to be doing at that microsecond. A block flagged SLOW may be: - a genuinely weak sector that the drive had to re-read internally, or - a perfectly healthy block that was unlucky enough to be read while something else hammered the disk or stole the CPU. The tool cannot tell those apart from a single scan, and it does not claim to. That is exactly why 'compare' and 'verify' exist: a block that is slow on an idle machine, twice, in the same place, is interesting. A block that is slow once during a backup is noise. Re-run on a quiet machine before you conclude anything, and never condemn a drive on one scan. Likewise, a FAILED block means "this read did not return the data it should have". On real media that usually means an unrecoverable read error. On a file it can also mean the file was truncated or is sparse in a way the scan did not expect. SectorPilot reports the read error verbatim so you can tell which. ------------------------------------------------------------------------------- USAGE ------------------------------------------------------------------------------- sectorpilot scan [--block 64KB] [--slow-ms N] [--save scan.json] [--top N] [--json] sectorpilot compare --before a.json --after b.json [--json] sectorpilot verify --scan scan.json [--json] sectorpilot help COMMANDS scan Read the whole image in --block chunks, timing each read, and classify every block OK / SLOW / FAILED. Reports the block count, counts per class, median and p95 read time, the slowest N blocks with their byte offsets, and an ASCII map of the surface. compare Diff two saved scans of the same image. Reports blocks that were OK and are now SLOW or FAILED, blocks that recovered, blocks that are bad in both, and a verdict on whether damage is spreading. verify Re-read only the blocks a saved scan flagged. Much faster than a full rescan, and it answers the only question that matters after a bad scan: are those blocks still bad? help Show usage. Also -h and --help. OPTIONS --block Block size for scan. Accepts a plain byte count (4096) or a suffix: B, KB/KiB/K, MB/MiB/M, GB/GiB/G. All suffixes are powers of 1024, so 64KB means 65536 bytes exactly. Default 64KB. Must be between 512 bytes and 64 MiB. --slow-ms Fixed slow threshold in milliseconds. Fractions are allowed: --slow-ms 0.05 is valid and useful on fast media. When omitted (or 0) SectorPilot uses an ADAPTIVE threshold instead -- see "How SLOW is decided" below. --save Write the scan to a JSON file for later compare/verify. SectorPilot refuses to point this at the image it is scanning. --top How many of the slowest blocks to list. Default 10, maximum 1000. --before compare: the earlier scan file. --after compare: the later scan file. --scan verify: the saved scan whose flagged blocks to re-read. --json Emit machine-readable JSON instead of text. -h, --help Show usage. Options may be given before or after the image path; both orders behave identically, and --flag=value works as well as --flag value. EXIT STATUS 0 success, nothing failed 1 usage error, or the image or scan file could not be read 2 scan: at least one block FAILED compare: at least one block that did not fail before now fails verify: at least one rechecked block still FAILED EXAMPLES sectorpilot scan disk.img --save monday.json sectorpilot scan disk.img --block 1MB --slow-ms 50 --top 25 sectorpilot compare --before monday.json --after friday.json sectorpilot verify disk.img --scan monday.json --json ------------------------------------------------------------------------------- HOW "SLOW" IS DECIDED ------------------------------------------------------------------------------- Every block gets exactly one timed read. The clock starts immediately before the read call and stops immediately after it. Nothing else is inside the measurement. There are two threshold modes, and SectorPilot always prints which one it used and what the resulting number was: FIXED (you passed --slow-ms N) A block is SLOW when its measured read time is strictly greater than N milliseconds. Nothing else is considered. ADAPTIVE (default, when --slow-ms is absent or 0) threshold = max( 8 x median read time , 1.0 ms ) The multiple of the median makes the threshold scale with the medium: an NVMe drive and a USB 2.0 stick have wildly different "normal", and a fixed millisecond number that is sensible for one is nonsense for the other. The 1.0 ms floor stops the tool crying wolf on fast or cached media, where 8 x a 0.01 ms median would flag ordinary scheduler jitter. The median and p95 are computed over the blocks that read successfully; failed blocks contribute no timing. The full line SectorPilot prints looks like: slow threshold 1.0000 ms [adaptive: a block is SLOW when its read takes longer than 1.000 ms = max(8 x median 0.011 ms, 1.0 ms floor)] CACHING. SectorPilot measures the read path as your operating system actually serves it. If the file is already in the page cache, reads are memory copies and will time in the microseconds; the same file read cold off the device on the same machine times two orders of magnitude slower. That is not a defect, it is the honest answer to "how long did this read take", but it means you should compare like with like: scan under the same conditions each time, and treat a scan of a freshly written file with suspicion because it is warm. SectorPilot does not (and in portable Go cannot) bypass the page cache. ------------------------------------------------------------------------------- THE SURFACE MAP ------------------------------------------------------------------------------- The map is a compressed picture of the whole image, so a damage PATTERN is visible at a glance -- a bad head shows up as regular stripes, a scratched platter as a contiguous band, a dying flash block as one cluster. cells = min(total blocks, 256) layout = rows of 64 characters (so at most 4 rows) each cell = one contiguous run of blocks, total/cells of them each row is labelled with the byte offset of its first block '.' every block in that cell read OK '!' that cell contains at least one SLOW block 'X' that cell contains at least one FAILED block (X wins over ! wins over .) A cell is pessimistic on purpose: one bad block in a run of 500 paints the whole cell. The map is for spotting the shape of the problem; the "Flagged blocks" list and the saved JSON carry the exact offsets. Example, an image whose second half became unreadable: 0x0000000000 ....!...!.......!.....................................!..!.....! 0x0007D00000 .....!.......!......!........................................... 0x000FA00000 XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 0x0017700000 XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX ------------------------------------------------------------------------------- THE SAVED SCAN FILE ------------------------------------------------------------------------------- --save writes one JSON object. It records the image path, its size, a cheap fingerprint (SHA-256 over the size plus the first and last 4 KiB), the block size, the block count, the threshold that was used, the class counts, the timing statistics, the surface map, the slowest blocks, and every FLAGGED block with its index, byte offset, length, measured time and error text. Blocks that read OK are NOT listed individually. A scan of a 1 GiB image with 16,384 blocks and 36 slow ones is a small file, not a 16,384-entry dump. Any offset absent from the flagged list read OK, and both compare and verify rely on that. The consequence is visible in compare output: where a block was OK in one scan there is no recorded time for it, and compare prints "n/a" rather than inventing a zero. verify checks the fingerprint before it re-reads anything and refuses a scan file that plainly belongs to a different image: sectorpilot: monday.json was taken on a different image: scan fingerprint sha256:4050a1f7..., other.img is sha256:d0e7db4c... (5000003 bytes vs 8388608 bytes recorded) ------------------------------------------------------------------------------- WHAT IS IMPLEMENTED ------------------------------------------------------------------------------- * Block-by-block sequential scan of any regular file, with the block size under your control from 512 bytes to 64 MiB. * One individually timed read per block, measured around the read call only. * Correct handling of a final short block when the file size is not a multiple of the block size: the last block is read at its true length and the reported byte total equals the exact file size. * Classification into OK / SLOW / FAILED, with both a fixed and an adaptive slow threshold, and the mode and effective number always printed. * Read-time statistics: min, median, p95, max, mean, and the total elapsed time and throughput of the scan. * The slowest N blocks with block index, byte offset, block length, measured time and class. * A full list of flagged blocks with the verbatim read error where there was one. * The ASCII surface map described above, in both text and JSON output. * --save / compare / verify: saved scans, a diff between two of them naming the changed offsets, a spreading verdict, and a fast re-check of only the flagged blocks. * --json on all three commands, stable field names, parses with any JSON library. * Refusal to scan a directory, an empty file, or a non-regular file, and refusal to --save over the image being scanned. * Go standard library only. No third-party dependencies at all. WHAT IS NOT IMPLEMENTED * Raw block device scanning. Regular files only, by design. See above. * SMART, self-test logs, reallocated/pending sector counts, drive temperature, power-on hours. None of it. No device pass-through. * Any kind of write: no surface repair, no sector reallocation, no forced rewrite to "refresh" a weak block, no zero-fill, no erase, no wipe. * Data recovery. SectorPilot tells you where the trouble is; it does not extract anything. Use an imaging tool for that. * Bypassing the page cache (no O_DIRECT / F_NOCACHE / POSIX_FADV_DONTNEED). Portable Go has no way to do it; see the CACHING note above. * Parallel or random-order scanning. Reads are strictly sequential and single-threaded, so the timings are not distorted by the tool's own queueing. * Filesystem awareness. SectorPilot does not know or care what is stored at a given offset; it reports byte offsets, not filenames. * Any judgement about the health of the drive as a whole. It reports measurements and changes between measurements. The conclusion is yours. ------------------------------------------------------------------------------- READ-ONLY GUARANTEE ------------------------------------------------------------------------------- The image is opened exactly once, with os.Open, which is O_RDONLY. The only operations performed on that handle are Stat and ReadAt. The source contains zero occurrences of os.Create, os.OpenFile, .Write(, .WriteAt(, .Truncate(, os.Remove, os.Rename, any O_WRONLY / O_RDWR / O_TRUNC / O_APPEND flag, any syscall.* call and any use of unsafe. The single write the program is capable of is os.WriteFile to the path you pass to --save, and saveScan refuses that path if it resolves to the image being scanned (both by cleaned absolute path and by os.SameFile). If you never pass --save, SectorPilot creates and modifies nothing at all. ------------------------------------------------------------------------------- ROADMAP ------------------------------------------------------------------------------- * RAW DEVICE SCANNING. Per-platform device size discovery and aligned unbuffered reads so a physical disk can be surfaced-scanned directly, behind an explicit opt-in flag and with a privilege check, still strictly read-only. * SMART CORRELATION. Read the drive's own reallocated and pending sector counts and line them up against the measured slow and failed offsets, so "block 8,690 is slow" can become "block 8,690 is slow AND the drive has 3 pending sectors it did not have on Monday" -- which turns a hint into something much closer to a diagnosis. * NON-DESTRUCTIVE WRITE TESTING. The classic read-verify-restore cycle: read a block, keep it, write a pattern, read it back, write the original contents back, verify. This catches weak blocks that read fine but no longer hold data. It is genuinely dangerous if interrupted, so it will be opt-in, refuse to run on mounted or in-use targets, and be kept firmly outside the default read-only path this tool ships with today.