ClipStudio 1.0.0 Clip library indexing, secret scanning and redaction Techlosoft - Screen Studio Lite ================================================================== WHAT IT IS ========== ClipStudio is the clip library for a team that shares terminal recordings. SessionForge records a session to a cast file. That cast file is plaintext: if somebody exported an AWS key on camera, the key is sitting in the file. It is in the transcript, and it is often in the recorded command line too. ClipStudio is what you run before that recording leaves the building. It does three jobs, and only these three: INDEX Walk a directory of casts, parse every header, hash every file, and build a library index you can list and search. Search matches the recording's text content, not just filenames. SCAN and Reconstruct each recording's text stream and find the credentials REDACT in it, then write a NEW cast with every finding replaced by a same-length mask. Event timing is preserved byte for byte and the original file is never modified. RETAIN Report which clips a retention policy would archive, and move them into an archive directory when you say so. Nothing is ever deleted, the dry run is the default, and every run is written to an append-only audit ledger. The sibling tools cover the other angles: ScreenFlow renders clips to GIF, RecordDeck edits timelines, CaptureStudio pulls stills, SessionForge does the recording and replay. ClipStudio is the only one that reads what is IN the recording and decides whether it is safe to share. INSTALL ======= Pre-built binaries are in dist/. There is nothing to install - copy the one for your platform anywhere on your PATH and run it. dist/clipstudio-linux-amd64 Linux, x86-64 dist/clipstudio-darwin-arm64 macOS, Apple Silicon dist/clipstudio-darwin-amd64 macOS, Intel dist/clipstudio-windows-amd64.exe Windows, x86-64 On macOS and Linux you may need to mark it executable: chmod +x clipstudio-linux-amd64 To build from source you need Go 1.24 or newer. The Go standard library is the only dependency - no third-party modules, no cgo, no network access at build time or at run time: go build -o clipstudio . go test ./... COMMANDS ======== clipstudio index --lib [--index ] [--json] clipstudio list (--index | --lib ) [--json] clipstudio search (--index | --lib ) [--regexp] [--json] clipstudio scan ( ... | --lib ) [--json] clipstudio redact --out [--force] [--json] clipstudio redact --lib --out-dir [--force] [--json] clipstudio retain --lib --older-than --ledger [--archive ] [--apply] [--json] clipstudio help | -h | --help clipstudio version index Walks the library, parses each cast header (duration, dimensions, recorded-at, title, recorded command), computes the SHA-256 of every file and writes a library index. Files that fail to parse are still indexed, with their error recorded, so a broken clip is visible rather than silently missing. Only *.jsonl and *.cast are considered; the _archive directory and dot-directories are skipped. list Prints the indexed clips, sorted by path. Reads a saved index, or builds one on the fly from --lib. search Finds clips by term. It matches the reconstructed TRANSCRIPT TEXT as well as the filename, title and recorded command, and reports the timestamp inside the recording where each match occurs. --regexp treats the term as a Go regular expression. Search runs over the REDACTED transcript. Detected credentials are masked before matching, so searching for a credential returns nothing - you cannot use search to pull a secret back out of the library. scan Reconstructs each recording's text stream and reports the credentials found in it: detector name, severity, Shannon entropy, the timestamp in the recording, the event and stream, and a MASKED excerpt. The credential itself is never printed, in any output mode. redact Writes a NEW cast with every finding replaced by a same-length run of '*'. Then it reloads what it wrote and checks two things before reporting success: that every event timestamp token is byte-identical to the original, and that a fresh scan of the output returns zero findings. The input is opened read-only. Writing over the input is refused, and an existing output is not overwritten without --force. retain Applies a retention policy by age. Dry run by default: it lists what WOULD be archived and touches nothing. With --apply it MOVES the selected clips into the archive directory. Every run - dry or applied - appends one JSON line to --ledger, which is why --ledger is mandatory. FLAGS --lib Clip library directory. --index Library index to write (index) or read (list, search). --out Redacted output for a single input cast. --out-dir Redacted output directory for --lib or several inputs. --force Overwrite an existing redaction output. --regexp Treat the search term as a regular expression. --older-than Retention age: 90d, 12h, 6w, 1y, 30m, 45s. --archive Archive destination. Default /_archive. --ledger Append-only JSON-lines audit ledger. Required by retain, on dry runs as well as on --apply. --apply Perform the archive moves. --json Machine-readable output, on every reporting subcommand. Secrets are masked in JSON too. Short forms -l, -i and -o are accepted for --lib, --index and --out. Flags may appear before or after positional arguments; either order works, so "search deploy --lib ./clips" and "search --lib ./clips deploy" are the same command. AGE STRINGS =========== 90d 90 days 12h 12 hours 6w 6 weeks 1y 365 days 30m 30 MINUTES (not months) 45s 45 seconds A unit is mandatory. A bare number like "90" is rejected rather than guessed at, because guessing wrong here moves the wrong files. "m" means minutes; there is no month unit - use 30d or 4w. QUICK START =========== # index the team's clip library clipstudio index --lib ./clips --index clips.json # what is in here? clipstudio list --index clips.json clipstudio search "restore" --index clips.json # what would leak if we published these? clipstudio scan --lib ./clips # produce shareable copies clipstudio redact --lib ./clips --out-dir ./share # what does the 90-day policy want to archive? (changes nothing) clipstudio retain --lib ./clips --older-than 90d --ledger retention.jsonl # do it clipstudio retain --lib ./clips --older-than 90d --ledger retention.jsonl \ --apply EXAMPLE OUTPUT ============== $ clipstudio scan --lib clips ClipStudio secret scan library : clips scanned : 5 clips, 2.2 KiB of reconstructed text findings : 9 in 3 clips clips/debug-latency.jsonl clean (491 B of text) clips/prod-deploy.jsonl sha256 : 15577c263a3f700c7f4f3b67668c7e6393a3873c33bb280beaba8be193bd8320 findings : 5 [1] aws-access-key-id at 1.522s (event 2, stream o, line 3) severity : critical entropy : 3.684 bits/char over 20 bytes masked : AKIA**************** finger : sha256:1a5d44a2dca1 excerpt : + export AWS_ACCESS_KEY_ID=AKIA****************\r [4] connection-string-password at 4.880s (event 6, stream o, line 7) severity : critical entropy : 3.507 bits/char over 15 bytes masked : *************** finger : sha256:31013ade6422 excerpt : + psql postgres://deployer:***************@db-prod.internal... $ clipstudio redact clips/prod-deploy.jsonl --out share/prod-deploy.jsonl output : share/prod-deploy.jsonl input sha256 : 15577c263a3f700c7f4f3b67668c7e6393a3873c33bb280beaba8be1... output sha256 : 3b7ae736bc190600aaa8e264ac1f8666374d169fd2fab66107d2cad6... findings masked : 5 (150 bytes replaced) timing : preserved exactly (15 events, identical timestamps) re-scan : clean (0 findings remain) $ clipstudio retain --lib clips --older-than 90d --ledger retention.jsonl mode : DRY RUN - nothing was moved CLIP SIZE AGE(d) ACTION RECORDED 2026-q1/db-restore.jsonl 527 B 146.01 would-archive 2026-03-18... api-smoketest.jsonl 458 B 0.22 keep 2026-08-11... onboarding-2026-04.jsonl 1.0 KiB 130.83 would-archive 2026-04-02... scanned 5, keep 3, selected 2 (1.5 KiB) HOW DETECTION WORKS =================== Everything runs over the RECONSTRUCTED TEXT of a cast: every event payload concatenated in recorded order, plus the header's title and each argument of the recorded command line. Reconstructing the stream first is what lets a credential be found even when it was split across two events by the pipe buffer, and what gives every finding a timestamp inside the recording. STAGE 1 - NAMED PATTERN DETECTORS Each detector knows the shape of one credential family, and each declares how many leading bytes are a PUBLIC marker that may stay visible after masking. private-key-pem PEM private key block, BEGIN marker to END marker. The whole block is masked, markers included. aws-access-key-id AKIA or ASIA + 16 upper-case alphanumerics. "AKIA" stays visible. github-token ghp_/gho_/ghs_/ghu_/ghr_ + 36 characters, or github_pat_ + 22-82. The prefix stays visible. google-api-key AIza + 35 URL-safe characters. slack-token xox[baprs]- + 10 or more token characters. jwt Three base64url segments whose FIRST segment actually base64url-decodes to a JSON object carrying "alg" or "typ". Shape alone matches far too much; the decodable header is what makes it a JWT. aws-secret-access-key 40 base64 characters, but only when an aws-secret context word appears within 80 bytes in front of it and the string itself carries at least 3.5 bits/char. The bare shape is far too common to report on its own. connection-string-password The password in scheme://user:pass@host. bearer-token Bearer + 12 or more token characters. secret-assignment PASSWORD= / SECRET= / TOKEN= / API_KEY= and friends, quoted or bare. When two detectors claim overlapping bytes, the more specific one wins: "Authorization: Bearer ghp_..." is reported once, as a github-token. STAGE 2 - SHANNON ENTROPY For credentials no pattern names, candidate tokens matching [A-Za-z0-9+/_-] (with '=' accepted only as trailing base64 padding) are scored with Shannon entropy in bits per character: H = -sum(p_i * log2(p_i)) over the distinct bytes of the token A token is reported when EITHER length >= 20 AND H >= 4.0 AND it contains lower-case, upper-case and digits length >= 32 AND H >= 4.5 (the looser character-class evidence is compensated for by a higher entropy bar) Why those numbers. The maximum entropy of a string is log2(number of distinct characters), so a 20-character token cannot exceed 4.32 bits/char - a threshold of 4.0 at the length floor demands roughly 17 distinct characters out of 20. For reference: English prose runs about 3.4-4.2, lower-case hex cannot exceed 4.0, and random base64 lands around 5.5-6.0 out of a maximum of 6.0. DE-NOISING Without this pass the entropy detector buries the real findings under log noise. A candidate is discarded when it is: - already mostly mask characters (something already redacted it) - a UUID - pure hex at a standard digest width (32/40/56/64/96/128), or any pure hex within 48 bytes of a word like sha256, md5, commit, digest, etag - all digits (timestamps, ids, byte counts) - base64 that DECODES to mostly-printable text - i.e. output piped through base64, rather than base64 of random key material, which decodes to bytes that are mostly unprintable - fewer than 6 distinct characters Assignment values are separately filtered: $VAR and ${VAR} references, values that are filesystem paths, , true/false/null/none, and short all-numeric values are not credentials. WHAT IS REPORTED Never the credential. Each finding carries the detector name, severity, measured entropy, byte length, the timestamp and event index in the recording, a masked rendering, an excerpt cut from the ALREADY-MASKED text, and a fingerprint. The fingerprint is the first 12 hex characters of the SHA-256 of the credential. It exists so the same credential can be recognised across clips and across runs without being printed. It is a truncated digest, not the secret - but be aware that a short or low-entropy secret could in principle be brute-forced from it, so treat a scan report as sensitive-ish, not public. HOW REDACTION WORKS =================== Every finding is replaced by a run of '*' of exactly the same BYTE length, except for the public marker a detector declared ("AKIA", "ghp_", "xoxb-"), which stays readable so the reader can tell what kind of credential was there. Because the replacement is the same length, nothing about the file's structure moves. Concretely, ClipStudio copies through verbatim: - the header line, unless the header itself held a credential - the timestamp TOKEN of every event, character for character, so no float formatting can perturb the timing by even one ulp - the footer line - the payload encoding (a base64 "b64" event stays a "b64" event) Then it reloads the file it just wrote and verifies, before reporting success, that the event count, every timestamp token, every stream tag and every payload length are identical, and that a fresh scan of the output returns zero findings. If either check fails it says so and exits 1. The mask character '*' is deliberately outside the token alphabet of every pattern detector, so masked output cannot re-trigger them. The two detectors whose value grammar does accept '*' - secret-assignment and connection-string-password - are handled by a rule that ignores any candidate which is already at least half mask characters. The original file is opened read-only and is never written to. Output is written to a temporary file and renamed into place, so an interrupted redaction cannot leave a half-written "safe" copy lying around. HOW RETENTION WORKS =================== A clip's age comes from the cast header's started_at. If that is missing or unparseable, the file's mtime is used instead, and every report says which source was used for every clip - a copied file has a new mtime and an old recording, and you should be able to see that. Clips older than --older-than are SELECTED. A dry run stops there. With --apply, each selected clip is MOVED into the archive directory, mirroring its path within the library. The move is an os.Rename when the archive is on the same filesystem: atomic, and the file is never in zero places. When rename fails because the archive is on a different filesystem, ClipStudio copies the file, fsyncs it, verifies the destination's SHA-256 against the source's, and only then unlinks the source. If the checksums do not match, the original is left exactly where it was and the run reports an error. An existing archived clip is never overwritten: a colliding name gets a -1, -2, ... suffix. The archive directory is skipped when walking the library, so running retain twice is a no-op rather than an archive of the archive. Every run appends one JSON line to --ledger containing the timestamp, the policy, the cutoff, whether it was applied, the counts, and every clip with its SHA-256, age, action and destination. MEASURED DETECTOR ACCURACY ========================== These are real numbers from `go test -v -run Accuracy ./...`, not estimates. Two corpora are used, and the difference between them is the point. CORPUS A - written alongside the detectors (20 positives, 30 negatives) false negatives : 0 recall 1.000 false positives : 0 fp-rate 0.000 This number is nearly worthless on its own. The detectors were developed against this corpus, so scoring 100% on it demonstrates only that the code does what it was written to do. CORPUS B - HELD OUT (20 positives, 30 negatives) Deliberately built from credential families ClipStudio has NO named rule for, and from real terminal output that looks credential-shaped. The detectors were not tuned against it, and it has not been adjusted to flatter them. false negatives : 7 of 20 recall 0.650 MISS RATE 0.350 false positives : 3 of 30 fp-rate 0.100 ClipStudio misses roughly ONE IN THREE credentials of a kind it has no rule for. Read that number again before you rely on a redacted clip. The seven misses, precisely: Authorization: Basic YWRtaW46c3VwZXJzZWNyZXQxMjM= HTTP Basic credentials. The de-noising pass discards base64 that decodes to printable text, and "admin:supersecret123" is printable text. This is the de-noising rule working as designed and costing us a real credential. mysql -u root -phunter2Sw0rdf1sh Password glued to a single-letter flag. No rule covers this. curl -u admin:s3cr3tp4ssw0rd https://... The -u user:pass form. The connection-string rule requires a scheme://, so it does not fire. export DB_PASS=Qw3rty2026Zx "PASS" is not in the assignment keyword list; "PASSWORD", "PASSWD" and "PASSPHRASE" are. heroku config:set SECRET_KEY_BASE=8f14e45fceea167a5a36dedd4bea2543 "SECRET" is followed by "_", so the word-boundary check in the assignment rule fails, and the value is 32 hex characters, which the digest de-noising rule discards. redis-cli -a Sup3rS3cr3tR3d1s Password after a bare -a flag. htpasswd -bc .htpasswd admin Str0ngP@ssw0rd2026 Password as a bare positional argument, with no keyword anywhere. The three false positives: ssh-rsa AAAAB3NzaC1yc2E... a PUBLIC key, reported as -----BEGIN PUBLIC KEY----- body high-entropy-string npm integrity sha512-Xk8vQm3Tp... a base64 digest; the hash-context de-noising rule only covers hex A false positive costs you a needlessly masked line in a shared clip. A false negative costs you a leaked credential. The thresholds above are set to favour the first failure over the second, and the numbers show the tool still fails the second way one time in three on unfamiliar credential shapes. SCOPE / WHAT THIS DOES NOT DO ============================= Read this part. It is the honest boundary of the tool. DETECTION IS HEURISTIC AND WILL MISS THINGS. Measured miss rate on the held-out corpus is 35% - roughly one in three credentials of a kind it has no named rule for. It is a pattern matcher and an entropy score, not an oracle. A redacted clip is SAFER than the original. It is not certified clean, and ClipStudio will never tell you it is. Watch a redacted recording before you publish it. In particular it does not understand: passwords passed as bare positional arguments or glued to single-letter flags (-phunter2, -a secret), HTTP Basic credentials, credentials your team invented a private format for, or anything a program printed in a form no rule anticipated. NO VIDEO OR IMAGE REDACTION. ClipStudio reads text. It cannot see a credential in a screen recording, a screenshot, a webcam frame or a GIF. If a key was visible on screen but never present in the captured byte stream - because it was in a window ClipStudio's input does not contain, or was rendered as pixels - ClipStudio has no idea it exists. ScreenFlow and CaptureStudio produce those artefacts and neither they nor this tool scan them. NO TERMINAL EMULATION. The transcript is the raw byte stream, not a rendered screen. A credential typed and then erased with backspaces or overwritten with cursor movement is still in the byte stream, so ClipStudio will find it - good. But a credential ASSEMBLED on screen from fragments written at different cursor positions never exists as a contiguous string in the stream, and no detector can match it. NO CLOUD UPLOAD, NO SHARING, NO NETWORK. ClipStudio never opens a socket. There is no upload, no share link, no "publish to the team", no telemetry, no update check, no license call-home. It reads and writes local files, and that is the whole of its I/O. Moving a redacted clip to wherever your team watches clips is your job. NO ACCESS CONTROL, NO USER ACCOUNTS, NO ENCRYPTION. There are no users, roles, permissions, sign-in or audit identity. The ledger records what was done and when, not by whom - it has no way to know. Anyone who can read the library can read every clip in it, including the unredacted originals. Cast files and index files are plaintext on disk and ClipStudio does not encrypt them. File-system permissions are your only access control. THE ARCHIVE IS A LOCAL MOVE, NOT A BACKUP SYSTEM. "Archiving" means renaming a file into a subdirectory, usually on the same disk, in the same failure domain, with no second copy, no versioning, no off-site replication, no integrity re-check over time and no restore command. If that disk dies, the archive dies with it. Point a real backup at the archive directory. ClipStudio is not one. ClipStudio never deletes a clip. There is no delete command and no code path that removes a file without the content already existing elsewhere. Pruning the archive, if you ever want to, is a decision you make with rm. FURTHER DELIBERATE LIMITS - Redaction is one-way. There is no un-redact, no key escrow, no way to recover the original from the redacted copy. Keep the originals. - Scanning is whole-file and single-threaded, and reconstructs the entire transcript in memory. A multi-gigabyte cast will use multi-gigabyte memory. Individual cast lines above 8 MiB are rejected. - Only SessionForge cast version 1 is understood. Real asciinema v2 files are similar but not identical and are not claimed to work - see the RELATIONSHIP TO ASCIINEMA section of SessionForge's README. - There is no configuration file and no way to add a detector, change a threshold or suppress a finding without editing the source. There are no allow-lists, no inline "clipstudio:ignore" comments and no baseline file. - The library index is a point-in-time snapshot. It is not watched, not incremental and not invalidated when a clip changes; re-run index. - No scheduling and no daemon. Run retain from cron, a systemd timer, launchd or Task Scheduler. - Exit status does not encode "findings were present". scan exits 0 whether it found ten credentials or none; parse --json if you want a CI gate. - Search matches bytes. There is no stemming, ranking, fuzzy matching or index-accelerated lookup - every search re-reads and re-scans every clip. NOT IMPLEMENTED - ROADMAP ========================= Honestly absent, not stubbed: - Detectors for the miss classes measured above: -p/-a style flag passwords, HTTP Basic, positional-argument passwords, and a keyword list that covers PASS/KEY_BASE style names without drowning in false positives. - Verification of found credentials against the issuing service, so a revoked key can be reported as harmless and a live one escalated. That needs network access, which this tool does not have. - A configurable detector set: custom patterns, per-repo thresholds, allow-lists and a baseline file for accepted findings. - A CI mode with a meaningful exit code and a diff-only scan. - Terminal emulation, so the SCREEN as rendered can be scanned rather than the byte stream, catching credentials assembled by cursor movement. - Streaming scan for casts too large to hold in memory. - Real archive management: compression, off-site copy, periodic integrity verification and a restore command. EXIT CODES ========== 0 Success. A scan that found nine credentials exits 0 - finding them is the job working, not a failure. A retention dry run exits 0. 1 Bad invocation (unknown command, missing or unparseable flag), an I/O error, a cast that will not parse, a redaction whose output failed its own timing or re-scan check, or an archive move that failed. Explicit help exits 0. Every error message goes to stderr and starts with "clipstudio: ". Usage goes to stdout for -h, --help and help, and to stderr for a bad invocation. Errors never quote file content. A malformed cast is reported by line number and error kind only, because the content on that line is exactly the thing that might be a credential. FILE FORMATS ============ CAST FILES (read, and written by redact) SessionForge JSON Lines. A header object, then one array per event, then an optional footer object: {"version":1,"command":["bash","-c","./deploy.sh"], "started_at":"2026-08-09T09:14:02Z","shell":"/bin/bash", "width":120,"height":34,"title":"prod deploy"} [1.522153, "o", "+ export AWS_ACCESS_KEY_ID=AKIA****************\r\n"] [2.610487, "e", "warning: ...\r\n"] {"exit_code":0,"duration":41.38} Stream tags are "o" for stdout and "e" for stderr. A payload that is not valid UTF-8 is stored base64 with a fourth field, "b64". See SessionForge's README.txt for the authoritative description. LIBRARY INDEX (--index) A single pretty-printed JSON object: tool, index_version, library, the time it was generated, and one entry per clip with path, abs_path, size, SHA-256, mtime, title, command, started_at, dimensions, event count, duration, byte counts by stream, exit code and any parse error. The title and command stored in the index are REDACTED, because a recorded command line is a common hiding place for a credential and an index file is the sort of thing people commit to a repository. header_fields_redacted counts how many header strings were masked. RETENTION LEDGER (--ledger) Append-only JSON Lines, one object per run, never rewritten: {"tool":"clipstudio","ts":"2026-08-11T07:20:19Z","action":"retain", "library":"/srv/clips","archive":"/srv/clips/_archive", "older_than":"90d","cutoff_utc":"2026-05-13T07:20:19Z", "applied":true,"clips_scanned":5,"clips_kept":3,"clips_selected":2, "clips_moved":2,"errors":0,"bytes_moved":1579, "clips":[{"path":"onboarding-2026-04.jsonl","sha256":"ad7a28cd...", "age_days":130.83,"action":"archived","method":"rename", "dest":"/srv/clips/_archive/onboarding-2026-04.jsonl"}]} The format is deliberately boring. Grep it, split it, or read it with one line of jq.