Compare commits
2 Commits
3ee7d7a3ac
...
665f846dc9
| Author | SHA1 | Date | |
|---|---|---|---|
| 665f846dc9 | |||
| 5e6b4428bb |
@@ -12,6 +12,16 @@ A single-file web tool that decodes a base64 string and saves it back out as a r
|
|||||||
xdg-open base64-decoder/base64-decoder.html
|
xdg-open base64-decoder/base64-decoder.html
|
||||||
```
|
```
|
||||||
|
|
||||||
|
### [base64 for Windows](./base64-rs/)
|
||||||
|
|
||||||
|
A real `base64.exe` for Windows, which otherwise has no `base64` command — only `certutil`, which is file-only, adds PEM markers and can't be piped. Matches GNU coreutils flag for flag (`-d`, `-i`, `-w`, stdin/stdout, exit codes, error messages), verified against coreutils 9.7 with a 153-case differential test. Zero dependencies, single static binary, streams its input.
|
||||||
|
|
||||||
|
```powershell
|
||||||
|
cargo build --release # then put target\release\base64.exe on your PATH
|
||||||
|
base64 image.png > image.b64
|
||||||
|
base64 -d image.b64 > image.png
|
||||||
|
```
|
||||||
|
|
||||||
### [Fake Media](./fake-media/)
|
### [Fake Media](./fake-media/)
|
||||||
|
|
||||||
Generates large collections of placeholder PNG images for testing without downloading real files. Supports a terminal UI and a non-interactive CLI mode. Configurable image sizes, folder structure, file counts, and naming prefixes — uses only Python's standard library.
|
Generates large collections of placeholder PNG images for testing without downloading real files. Supports a terminal UI and a non-interactive CLI mode. Configurable image sizes, folder structure, file counts, and naming prefixes — uses only Python's standard library.
|
||||||
|
|||||||
@@ -0,0 +1 @@
|
|||||||
|
/target
|
||||||
Generated
+7
@@ -0,0 +1,7 @@
|
|||||||
|
# This file is automatically @generated by Cargo.
|
||||||
|
# It is not intended for manual editing.
|
||||||
|
version = 4
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "base64"
|
||||||
|
version = "1.0.0"
|
||||||
@@ -0,0 +1,16 @@
|
|||||||
|
[package]
|
||||||
|
name = "base64"
|
||||||
|
version = "1.0.0"
|
||||||
|
edition = "2021"
|
||||||
|
|
||||||
|
[dependencies]
|
||||||
|
|
||||||
|
[[bin]]
|
||||||
|
name = "base64"
|
||||||
|
path = "src/main.rs"
|
||||||
|
|
||||||
|
[profile.release]
|
||||||
|
opt-level = 3
|
||||||
|
lto = true
|
||||||
|
strip = true
|
||||||
|
panic = "abort"
|
||||||
@@ -0,0 +1,81 @@
|
|||||||
|
# base64 for Windows
|
||||||
|
|
||||||
|
Windows has no `base64` command. `certutil -encode` is the usual substitute, but it only works on files, wraps output in `-----BEGIN CERTIFICATE-----` markers, and can't be piped. This is a real `base64.exe` that behaves like the one on a Linux box, so the commands and scripts you already know carry over.
|
||||||
|
|
||||||
|
```powershell
|
||||||
|
base64 image.png > image.b64
|
||||||
|
base64 -d image.b64 > image.png
|
||||||
|
type secrets.json | base64 -w0
|
||||||
|
base64 -di messy.txt > payload.bin
|
||||||
|
```
|
||||||
|
|
||||||
|
## Build and install
|
||||||
|
|
||||||
|
```powershell
|
||||||
|
cargo build --release
|
||||||
|
```
|
||||||
|
|
||||||
|
Copy `target\release\base64.exe` somewhere on your PATH — e.g. `%LOCALAPPDATA%\Microsoft\WindowsApps`, or a `C:\tools\bin` folder you've added yourself:
|
||||||
|
|
||||||
|
```powershell
|
||||||
|
mkdir C:\tools\bin
|
||||||
|
copy target\release\base64.exe C:\tools\bin\
|
||||||
|
[Environment]::SetEnvironmentVariable(
|
||||||
|
'Path',
|
||||||
|
[Environment]::GetEnvironmentVariable('Path','User') + ';C:\tools\bin',
|
||||||
|
'User')
|
||||||
|
```
|
||||||
|
|
||||||
|
Open a new terminal afterwards for the PATH change to take effect. No runtime dependencies — it's a single static binary, and the crate itself has zero dependencies, so the build works offline.
|
||||||
|
|
||||||
|
## Usage
|
||||||
|
|
||||||
|
```
|
||||||
|
Usage: base64 [OPTION]... [FILE]
|
||||||
|
Base64 encode or decode FILE, or standard input, to standard output.
|
||||||
|
|
||||||
|
With no FILE, or when FILE is -, read standard input.
|
||||||
|
|
||||||
|
-d, --decode decode data
|
||||||
|
-i, --ignore-garbage when decoding, ignore non-alphabet characters
|
||||||
|
-w, --wrap=COLS wrap encoded lines after COLS character (default 76).
|
||||||
|
Use 0 to disable line wrapping
|
||||||
|
--help display this help and exit
|
||||||
|
--version output version information and exit
|
||||||
|
```
|
||||||
|
|
||||||
|
Long options accept unambiguous abbreviations (`--dec`), short options cluster (`-di`), `-w` takes its value joined or separate (`-w0`, `-w 0`, `--wrap=0`), and `--` ends option parsing.
|
||||||
|
|
||||||
|
## Behaviour
|
||||||
|
|
||||||
|
Verified against GNU coreutils 9.7 with a 154-case differential test ([tests/parity.py](tests/parity.py)) covering encoding at every wrap width, decode round-trips, malformed padding, truncated input, garbage handling, all the usage errors and file operands. Output, exit codes and error messages match, including the fiddly parts:
|
||||||
|
|
||||||
|
- **Partial output on failure.** Bytes are emitted as soon as enough bits are available, so a stream that goes bad halfway still writes the valid prefix before the error — same as coreutils.
|
||||||
|
- **Unpadded input is accepted** when the encoding is canonical, i.e. the leftover bits are zero. `aGVsbG8` decodes to `hello`; `aGV` is rejected because `V` carries stray bits.
|
||||||
|
- **Concatenated streams** decode as one (`aGVsbG8=aGVsbG8=` → `hellohello`).
|
||||||
|
- **`-w0` emits no trailing newline**; wrapped output always ends with one. Empty input produces no output at all.
|
||||||
|
- **Exit codes**: `0` success, `1` on any error.
|
||||||
|
|
||||||
|
### Deliberate differences
|
||||||
|
|
||||||
|
| | Why |
|
||||||
|
|---|---|
|
||||||
|
| `\r` is skipped when decoding, as `\n` already is | GNU rejects CRLF input as invalid. On Windows a base64 file will routinely have CRLF line endings, and erroring on those would make the tool useless for its main job. Everything else is still rejected unless you pass `-i`. |
|
||||||
|
| `--help` omits GNU's support URLs | This isn't GNU coreutils; pointing at their bug tracker would be wrong. |
|
||||||
|
| Error messages quote with `'ASCII'` | GNU uses `'…'` in a UTF-8 locale. Windows consoles are frequently cp437/cp1252, where those bytes render as mojibake. |
|
||||||
|
|
||||||
|
## Running the tests
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo build --release
|
||||||
|
python3 tests/parity.py
|
||||||
|
```
|
||||||
|
|
||||||
|
Linux-only — it diffs against a real GNU `base64`, so it can't run on the Windows box this tool targets. Note that many distributions now ship **uutils** coreutils as `/usr/bin/base64`; it disagrees with GNU on most malformed-input cases, so the script hunts for a genuine GNU build and refuses to run against anything else. On Ubuntu with `coreutils-from-uutils` installed, the GNU one is `gnubase64`.
|
||||||
|
|
||||||
|
## Notes
|
||||||
|
|
||||||
|
- Roughly 2× slower than GNU on large inputs (100 MB encodes in ~0.35s vs ~0.18s). GNU uses hand-tuned routines; this is plain portable Rust and fast enough that the difference is invisible for normal use.
|
||||||
|
- Input is streamed, not buffered whole, so encoding a multi-gigabyte file doesn't blow up memory.
|
||||||
|
- Piping to a program that exits early (`base64 big.bin | head`) is handled quietly rather than reporting a broken pipe.
|
||||||
|
- Writing raw binary to a Windows console fails with a clear message telling you to redirect, because consoles take UTF-16 text and would corrupt the bytes. Redirecting to a file or piping is unaffected.
|
||||||
@@ -0,0 +1,450 @@
|
|||||||
|
//! A drop-in `base64` for Windows, matching GNU coreutils behaviour.
|
||||||
|
//!
|
||||||
|
//! Same flags, same output, same error messages and exit codes as the
|
||||||
|
//! `base64` on a Linux box, so muscle memory and scripts carry over.
|
||||||
|
|
||||||
|
use std::env;
|
||||||
|
use std::fs::File;
|
||||||
|
use std::io::{self, BufWriter, ErrorKind, Read, Write};
|
||||||
|
use std::process::ExitCode;
|
||||||
|
|
||||||
|
const PROGRAM: &str = "base64";
|
||||||
|
const VERSION: &str = concat!("base64 (jwtf tools) ", env!("CARGO_PKG_VERSION"));
|
||||||
|
const DEFAULT_WRAP: usize = 76;
|
||||||
|
|
||||||
|
const ALPHABET: &[u8; 64] = b"ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/";
|
||||||
|
const PAD: u8 = b'=';
|
||||||
|
|
||||||
|
/// Reverse lookup for the alphabet; -1 marks a byte that is not a base64 digit.
|
||||||
|
const DECODE: [i8; 256] = {
|
||||||
|
let mut table = [-1i8; 256];
|
||||||
|
let mut i = 0;
|
||||||
|
while i < 64 {
|
||||||
|
table[ALPHABET[i] as usize] = i as i8;
|
||||||
|
i += 1;
|
||||||
|
}
|
||||||
|
table
|
||||||
|
};
|
||||||
|
|
||||||
|
const HELP: &str = "\
|
||||||
|
Usage: base64 [OPTION]... [FILE]
|
||||||
|
Base64 encode or decode FILE, or standard input, to standard output.
|
||||||
|
|
||||||
|
With no FILE, or when FILE is -, read standard input.
|
||||||
|
|
||||||
|
Mandatory arguments to long options are mandatory for short options too.
|
||||||
|
-d, --decode decode data
|
||||||
|
-i, --ignore-garbage when decoding, ignore non-alphabet characters
|
||||||
|
-w, --wrap=COLS wrap encoded lines after COLS character (default 76).
|
||||||
|
Use 0 to disable line wrapping
|
||||||
|
--help display this help and exit
|
||||||
|
--version output version information and exit
|
||||||
|
|
||||||
|
The data are encoded as described for the base64 alphabet in RFC 4648.
|
||||||
|
When decoding, the input may contain newlines in addition to the bytes of
|
||||||
|
the formal base64 alphabet. Use --ignore-garbage to attempt to recover
|
||||||
|
from any other non-alphabet bytes in the encoded stream.";
|
||||||
|
|
||||||
|
const LONG_OPTS: [&str; 5] = ["decode", "ignore-garbage", "wrap", "help", "version"];
|
||||||
|
|
||||||
|
fn main() -> ExitCode {
|
||||||
|
match run() {
|
||||||
|
Ok(()) => ExitCode::SUCCESS,
|
||||||
|
Err(Fail::Quiet) => ExitCode::SUCCESS,
|
||||||
|
Err(Fail::Usage(msg)) => {
|
||||||
|
eprintln!("{PROGRAM}: {msg}");
|
||||||
|
eprintln!("Try '{PROGRAM} --help' for more information.");
|
||||||
|
ExitCode::FAILURE
|
||||||
|
}
|
||||||
|
Err(Fail::Error(msg)) => {
|
||||||
|
eprintln!("{PROGRAM}: {msg}");
|
||||||
|
ExitCode::FAILURE
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
enum Fail {
|
||||||
|
/// Usage problem: prints the "Try --help" hint after the message.
|
||||||
|
Usage(String),
|
||||||
|
/// Runtime problem: message only.
|
||||||
|
Error(String),
|
||||||
|
/// Downstream closed the pipe (`base64 f | head`); nothing to report.
|
||||||
|
Quiet,
|
||||||
|
}
|
||||||
|
|
||||||
|
struct Options {
|
||||||
|
decode: bool,
|
||||||
|
ignore_garbage: bool,
|
||||||
|
wrap: usize,
|
||||||
|
file: Option<String>,
|
||||||
|
}
|
||||||
|
|
||||||
|
fn run() -> Result<(), Fail> {
|
||||||
|
let opts = match parse_args(env::args().skip(1).collect())? {
|
||||||
|
Parsed::Run(o) => o,
|
||||||
|
Parsed::Exit(text) => {
|
||||||
|
println!("{text}");
|
||||||
|
return Ok(());
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
|
// Reading is buffered by the encode/decode loops, which pull in large
|
||||||
|
// chunks; stdout is wrapped so small writes do not hit the OS each time.
|
||||||
|
let stdout = io::stdout();
|
||||||
|
let mut out = BufWriter::with_capacity(64 * 1024, stdout.lock());
|
||||||
|
|
||||||
|
let result = match opts.file.as_deref() {
|
||||||
|
None | Some("-") => {
|
||||||
|
let stdin = io::stdin();
|
||||||
|
process(stdin.lock(), &mut out, &opts)
|
||||||
|
}
|
||||||
|
Some(path) => match File::open(path) {
|
||||||
|
Ok(f) => process(f, &mut out, &opts),
|
||||||
|
Err(e) => return Err(Fail::Error(format!("{path}: {}", describe(&e)))),
|
||||||
|
},
|
||||||
|
};
|
||||||
|
|
||||||
|
// Flush even on failure: a partial decode still writes its valid prefix,
|
||||||
|
// and it must land before main() prints the error to stderr.
|
||||||
|
let flushed = out.flush().map_err(write_failure);
|
||||||
|
result.and(flushed)
|
||||||
|
}
|
||||||
|
|
||||||
|
fn process<R: Read, W: Write>(input: R, out: &mut W, opts: &Options) -> Result<(), Fail> {
|
||||||
|
if opts.decode {
|
||||||
|
decode(input, out, opts.ignore_garbage)
|
||||||
|
} else {
|
||||||
|
encode(input, out, opts.wrap)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/* ── Argument parsing ────────────────────────────────────────────────── */
|
||||||
|
|
||||||
|
enum Parsed {
|
||||||
|
Run(Options),
|
||||||
|
/// --help / --version: print this and stop.
|
||||||
|
Exit(&'static str),
|
||||||
|
}
|
||||||
|
|
||||||
|
fn parse_args(args: Vec<String>) -> Result<Parsed, Fail> {
|
||||||
|
let mut opts = Options {
|
||||||
|
decode: false,
|
||||||
|
ignore_garbage: false,
|
||||||
|
wrap: DEFAULT_WRAP,
|
||||||
|
file: None,
|
||||||
|
};
|
||||||
|
let mut operands: Vec<String> = Vec::new();
|
||||||
|
let mut only_operands = false;
|
||||||
|
let mut it = args.into_iter().peekable();
|
||||||
|
|
||||||
|
while let Some(arg) = it.next() {
|
||||||
|
if only_operands {
|
||||||
|
operands.push(arg);
|
||||||
|
} else if arg == "--" {
|
||||||
|
only_operands = true;
|
||||||
|
} else if let Some(long) = arg.strip_prefix("--") {
|
||||||
|
// Split --wrap=76 into name and inline value.
|
||||||
|
let (name, inline) = match long.split_once('=') {
|
||||||
|
Some((n, v)) => (n, Some(v.to_string())),
|
||||||
|
None => (long, None),
|
||||||
|
};
|
||||||
|
match resolve_long(name)? {
|
||||||
|
"help" => return Ok(Parsed::Exit(HELP)),
|
||||||
|
"version" => return Ok(Parsed::Exit(VERSION)),
|
||||||
|
"decode" => opts.decode = true,
|
||||||
|
"ignore-garbage" => opts.ignore_garbage = true,
|
||||||
|
"wrap" => {
|
||||||
|
let value = match inline.or_else(|| it.next()) {
|
||||||
|
Some(v) => v,
|
||||||
|
None => {
|
||||||
|
return Err(Fail::Usage(format!("option '--{name}' requires an argument")))
|
||||||
|
}
|
||||||
|
};
|
||||||
|
opts.wrap = parse_wrap(&value)?;
|
||||||
|
}
|
||||||
|
_ => unreachable!(),
|
||||||
|
}
|
||||||
|
} else if arg.len() > 1 && arg.starts_with('-') {
|
||||||
|
// Short flags, clusterable: -di, -dw0, -w76
|
||||||
|
let chars: Vec<char> = arg.chars().collect();
|
||||||
|
let mut idx = 1;
|
||||||
|
while idx < chars.len() {
|
||||||
|
match chars[idx] {
|
||||||
|
'd' => opts.decode = true,
|
||||||
|
'i' => opts.ignore_garbage = true,
|
||||||
|
'w' => {
|
||||||
|
// Rest of this argument is the value, else the next one.
|
||||||
|
let rest: String = chars[idx + 1..].iter().collect();
|
||||||
|
let value = if rest.is_empty() {
|
||||||
|
match it.next() {
|
||||||
|
Some(v) => v,
|
||||||
|
None => {
|
||||||
|
return Err(Fail::Usage(
|
||||||
|
"option requires an argument -- 'w'".to_string(),
|
||||||
|
))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
rest
|
||||||
|
};
|
||||||
|
opts.wrap = parse_wrap(&value)?;
|
||||||
|
idx = chars.len();
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
c => return Err(Fail::Usage(format!("invalid option -- '{c}'"))),
|
||||||
|
}
|
||||||
|
idx += 1;
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
operands.push(arg);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if operands.len() > 1 {
|
||||||
|
return Err(Fail::Usage(format!("extra operand '{}'", operands[1])));
|
||||||
|
}
|
||||||
|
opts.file = operands.into_iter().next();
|
||||||
|
Ok(Parsed::Run(opts))
|
||||||
|
}
|
||||||
|
|
||||||
|
/// GNU accepts any unambiguous abbreviation of a long option.
|
||||||
|
fn resolve_long(name: &str) -> Result<&'static str, Fail> {
|
||||||
|
let hits: Vec<&'static str> = LONG_OPTS
|
||||||
|
.iter()
|
||||||
|
.copied()
|
||||||
|
.filter(|o| o.starts_with(name))
|
||||||
|
.collect();
|
||||||
|
match hits.len() {
|
||||||
|
1 => Ok(hits[0]),
|
||||||
|
0 => Err(Fail::Usage(format!("unrecognized option '--{name}'"))),
|
||||||
|
_ => Err(Fail::Usage(format!(
|
||||||
|
"option '--{name}' is ambiguous; possibilities: {}",
|
||||||
|
hits.iter()
|
||||||
|
.map(|o| format!("'--{o}'"))
|
||||||
|
.collect::<Vec<_>>()
|
||||||
|
.join(" ")
|
||||||
|
))),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Unlike the other usage errors, coreutils prints this one without the
|
||||||
|
/// "Try --help" hint.
|
||||||
|
fn parse_wrap(value: &str) -> Result<usize, Fail> {
|
||||||
|
value
|
||||||
|
.parse::<usize>()
|
||||||
|
.map_err(|_| Fail::Error(format!("invalid wrap size: '{value}'")))
|
||||||
|
}
|
||||||
|
|
||||||
|
/* ── Encode ──────────────────────────────────────────────────────────── */
|
||||||
|
|
||||||
|
fn encode<R: Read, W: Write>(mut input: R, out: &mut W, wrap: usize) -> Result<(), Fail> {
|
||||||
|
let mut inbuf = vec![0u8; 48 * 1024];
|
||||||
|
let mut pending: Vec<u8> = Vec::with_capacity(3);
|
||||||
|
let mut line: Vec<u8> = Vec::with_capacity(64 * 1024);
|
||||||
|
let mut col = 0usize;
|
||||||
|
let mut any = false;
|
||||||
|
|
||||||
|
loop {
|
||||||
|
let n = input.read(&mut inbuf).map_err(read_failure)?;
|
||||||
|
if n == 0 {
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
any = true;
|
||||||
|
pending.extend_from_slice(&inbuf[..n]);
|
||||||
|
|
||||||
|
let whole = pending.len() - pending.len() % 3;
|
||||||
|
for group in pending[..whole].chunks_exact(3) {
|
||||||
|
emit(&mut line, encode_triple(group[0], group[1], group[2]), wrap, &mut col);
|
||||||
|
}
|
||||||
|
pending.drain(..whole);
|
||||||
|
|
||||||
|
if line.len() >= 32 * 1024 {
|
||||||
|
out.write_all(&line).map_err(write_failure)?;
|
||||||
|
line.clear();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Tail of 1 or 2 bytes gets padded out to a full quad.
|
||||||
|
match pending.len() {
|
||||||
|
1 => {
|
||||||
|
let q = encode_triple(pending[0], 0, 0);
|
||||||
|
emit(&mut line, [q[0], q[1], PAD, PAD], wrap, &mut col);
|
||||||
|
}
|
||||||
|
2 => {
|
||||||
|
let q = encode_triple(pending[0], pending[1], 0);
|
||||||
|
emit(&mut line, [q[0], q[1], q[2], PAD], wrap, &mut col);
|
||||||
|
}
|
||||||
|
_ => {}
|
||||||
|
}
|
||||||
|
|
||||||
|
// The trailing newline is a line terminator, so it appears only when
|
||||||
|
// wrapping is on and something was actually written.
|
||||||
|
if any && col != 0 && wrap != 0 {
|
||||||
|
line.push(b'\n');
|
||||||
|
}
|
||||||
|
out.write_all(&line).map_err(write_failure)?;
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
|
fn encode_triple(a: u8, b: u8, c: u8) -> [u8; 4] {
|
||||||
|
let n = ((a as u32) << 16) | ((b as u32) << 8) | c as u32;
|
||||||
|
[
|
||||||
|
ALPHABET[(n >> 18) as usize & 63],
|
||||||
|
ALPHABET[(n >> 12) as usize & 63],
|
||||||
|
ALPHABET[(n >> 6) as usize & 63],
|
||||||
|
ALPHABET[n as usize & 63],
|
||||||
|
]
|
||||||
|
}
|
||||||
|
|
||||||
|
fn emit(line: &mut Vec<u8>, quad: [u8; 4], wrap: usize, col: &mut usize) {
|
||||||
|
for ch in quad {
|
||||||
|
if wrap != 0 && *col == wrap {
|
||||||
|
line.push(b'\n');
|
||||||
|
*col = 0;
|
||||||
|
}
|
||||||
|
line.push(ch);
|
||||||
|
*col += 1;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/* ── Decode ──────────────────────────────────────────────────────────── */
|
||||||
|
|
||||||
|
fn decode<R: Read, W: Write>(mut input: R, out: &mut W, ignore_garbage: bool) -> Result<(), Fail> {
|
||||||
|
let mut obuf: Vec<u8> = Vec::with_capacity(48 * 1024);
|
||||||
|
let outcome = decode_stream(&mut input, out, &mut obuf, ignore_garbage);
|
||||||
|
// Whatever decoded cleanly before the failure still gets written, which
|
||||||
|
// is what coreutils does — the error only stops further output.
|
||||||
|
let flushed = out.write_all(&obuf).map_err(write_failure);
|
||||||
|
outcome.and(flushed)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Bytes are emitted as soon as enough bits are in hand (one byte after two
|
||||||
|
/// digits, another after three, the last on a full quad), so a stream that
|
||||||
|
/// turns bad part-way still produces the valid prefix.
|
||||||
|
fn decode_stream<R: Read, W: Write>(
|
||||||
|
input: &mut R,
|
||||||
|
out: &mut W,
|
||||||
|
obuf: &mut Vec<u8>,
|
||||||
|
ignore_garbage: bool,
|
||||||
|
) -> Result<(), Fail> {
|
||||||
|
let mut inbuf = vec![0u8; 64 * 1024];
|
||||||
|
let mut d = [0u8; 3];
|
||||||
|
let mut have = 0usize;
|
||||||
|
// A quad holding two digits needs '=' twice; this tracks the second.
|
||||||
|
let mut need_pad = false;
|
||||||
|
|
||||||
|
loop {
|
||||||
|
let n = input.read(&mut inbuf).map_err(read_failure)?;
|
||||||
|
if n == 0 {
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
|
||||||
|
for &byte in &inbuf[..n] {
|
||||||
|
if byte == PAD {
|
||||||
|
if need_pad {
|
||||||
|
need_pad = false;
|
||||||
|
have = 0;
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
match have {
|
||||||
|
2 if d[1] & 0x0F == 0 => need_pad = true,
|
||||||
|
3 if d[2] & 0x03 == 0 => have = 0,
|
||||||
|
_ => return Err(invalid()),
|
||||||
|
}
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
let digit = DECODE[byte as usize];
|
||||||
|
if digit < 0 {
|
||||||
|
// Line endings are always skipped so CRLF files work; other
|
||||||
|
// junk is only tolerated under --ignore-garbage.
|
||||||
|
if byte == b'\n' || byte == b'\r' || ignore_garbage {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
return Err(invalid());
|
||||||
|
}
|
||||||
|
if need_pad {
|
||||||
|
return Err(invalid());
|
||||||
|
}
|
||||||
|
|
||||||
|
let v = digit as u8;
|
||||||
|
match have {
|
||||||
|
0 => {
|
||||||
|
d[0] = v;
|
||||||
|
have = 1;
|
||||||
|
}
|
||||||
|
1 => {
|
||||||
|
d[1] = v;
|
||||||
|
have = 2;
|
||||||
|
obuf.push((d[0] << 2) | (d[1] >> 4));
|
||||||
|
}
|
||||||
|
2 => {
|
||||||
|
d[2] = v;
|
||||||
|
have = 3;
|
||||||
|
obuf.push(((d[1] & 0x0F) << 4) | (d[2] >> 2));
|
||||||
|
}
|
||||||
|
_ => {
|
||||||
|
obuf.push(((d[2] & 0x03) << 6) | v);
|
||||||
|
have = 0;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if obuf.len() >= 32 * 1024 {
|
||||||
|
out.write_all(obuf).map_err(write_failure)?;
|
||||||
|
obuf.clear();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if need_pad {
|
||||||
|
return Err(invalid());
|
||||||
|
}
|
||||||
|
// An unpadded tail is only legal if the bits that would have been
|
||||||
|
// dropped are zero, i.e. the input was canonically encoded.
|
||||||
|
match have {
|
||||||
|
0 => Ok(()),
|
||||||
|
2 if d[1] & 0x0F == 0 => Ok(()),
|
||||||
|
3 if d[2] & 0x03 == 0 => Ok(()),
|
||||||
|
_ => Err(invalid()),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn invalid() -> Fail {
|
||||||
|
Fail::Error("invalid input".to_string())
|
||||||
|
}
|
||||||
|
|
||||||
|
/* ── I/O error mapping ───────────────────────────────────────────────── */
|
||||||
|
|
||||||
|
fn read_failure(e: io::Error) -> Fail {
|
||||||
|
Fail::Error(format!("read error: {}", describe(&e)))
|
||||||
|
}
|
||||||
|
|
||||||
|
fn write_failure(e: io::Error) -> Fail {
|
||||||
|
match e.kind() {
|
||||||
|
// `base64 big.bin | head` closes the pipe under us; exit quietly.
|
||||||
|
ErrorKind::BrokenPipe => Fail::Quiet,
|
||||||
|
// Windows consoles take UTF-16 text, so raw bytes cannot be printed
|
||||||
|
// to one. Redirecting to a file or piping sidesteps it entirely.
|
||||||
|
ErrorKind::InvalidData => Fail::Error(
|
||||||
|
"cannot write binary output to the console; redirect to a file (base64 -d in.txt > out.bin)"
|
||||||
|
.to_string(),
|
||||||
|
),
|
||||||
|
_ => Fail::Error(format!("write error: {}", describe(&e))),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Trims Rust's "(os error N)" suffix so messages read like coreutils'.
|
||||||
|
fn describe(e: &io::Error) -> String {
|
||||||
|
match e.kind() {
|
||||||
|
ErrorKind::NotFound => "No such file or directory".to_string(),
|
||||||
|
ErrorKind::PermissionDenied => "Permission denied".to_string(),
|
||||||
|
ErrorKind::IsADirectory => "Is a directory".to_string(),
|
||||||
|
_ => {
|
||||||
|
let s = e.to_string();
|
||||||
|
match s.find(" (os error ") {
|
||||||
|
Some(i) => s[..i].to_string(),
|
||||||
|
None => s,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,204 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Differential test: this base64 against GNU coreutils base64.
|
||||||
|
|
||||||
|
Runs both binaries over the same inputs and compares stdout, stderr and
|
||||||
|
exit code. Linux-only — it needs a real GNU base64 to diff against, so it
|
||||||
|
cannot run on the Windows machine the tool is actually built for.
|
||||||
|
|
||||||
|
cargo build --release && python3 tests/parity.py
|
||||||
|
|
||||||
|
Note that many distributions now ship uutils coreutils as /usr/bin/base64,
|
||||||
|
which differs from GNU on malformed input. This script looks for a genuine
|
||||||
|
GNU build and refuses to run against anything else.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import os
|
||||||
|
import random
|
||||||
|
import shutil
|
||||||
|
import subprocess
|
||||||
|
import sys
|
||||||
|
import tempfile
|
||||||
|
|
||||||
|
ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
|
||||||
|
MINE = os.path.join(ROOT, "target", "release", "base64")
|
||||||
|
|
||||||
|
# Differences that are intentional; see README.md.
|
||||||
|
EXPECTED_DIFF = {
|
||||||
|
# We skip '\r' alongside '\n' so CRLF base64 files work on Windows.
|
||||||
|
"decode CRLF wrapped",
|
||||||
|
# Our --help omits GNU's own support URLs.
|
||||||
|
"help",
|
||||||
|
"help after other flags",
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def find_gnu():
|
||||||
|
"""Locate a genuine GNU base64, ignoring uutils drop-in replacements."""
|
||||||
|
for name in ("gnubase64", "base64", "gbase64"):
|
||||||
|
path = shutil.which(name)
|
||||||
|
if not path:
|
||||||
|
continue
|
||||||
|
try:
|
||||||
|
banner = subprocess.run(
|
||||||
|
[path, "--version"], capture_output=True, text=True, timeout=10
|
||||||
|
).stdout
|
||||||
|
except OSError:
|
||||||
|
continue
|
||||||
|
if "GNU coreutils" in banner:
|
||||||
|
return path, banner.splitlines()[0]
|
||||||
|
return None, None
|
||||||
|
|
||||||
|
|
||||||
|
GNU, GNU_BANNER = find_gnu()
|
||||||
|
|
||||||
|
|
||||||
|
def norm(b, gnu_name):
|
||||||
|
"""Strip argv[0] and locale noise so the two are comparable."""
|
||||||
|
b = b.replace(gnu_name.encode(), b"base64")
|
||||||
|
b = b.replace(os.path.basename(gnu_name).encode(), b"base64")
|
||||||
|
# GNU quotes with U+2018/2019 outside the C locale; we always use ASCII.
|
||||||
|
b = b.replace("‘".encode(), b"'").replace("’".encode(), b"'")
|
||||||
|
return b
|
||||||
|
|
||||||
|
|
||||||
|
def run(exe, args, data):
|
||||||
|
# C locale: GNU otherwise uses en_GB spelling and Unicode quotes, neither
|
||||||
|
# of which a Windows console would produce.
|
||||||
|
env = dict(os.environ, LC_ALL="C")
|
||||||
|
p = subprocess.run([exe] + args, input=data, capture_output=True, env=env)
|
||||||
|
return p.returncode, norm(p.stdout, GNU), norm(p.stderr, GNU)
|
||||||
|
|
||||||
|
|
||||||
|
results = []
|
||||||
|
|
||||||
|
|
||||||
|
def t(name, args, data=b""):
|
||||||
|
mine = run(MINE, args, data)
|
||||||
|
gnu = run(GNU, args, data)
|
||||||
|
ok = mine == gnu
|
||||||
|
if not ok:
|
||||||
|
tag = "DIFF(expected)" if name in EXPECTED_DIFF else "FAIL"
|
||||||
|
print(f"[{tag}] {name} args={args}")
|
||||||
|
if mine[0] != gnu[0]:
|
||||||
|
print(f" exit: mine={mine[0]} gnu={gnu[0]}")
|
||||||
|
if mine[1] != gnu[1]:
|
||||||
|
print(f" out : mine={mine[1][:90]!r}")
|
||||||
|
print(f" gnu ={gnu[1][:90]!r}")
|
||||||
|
if mine[2] != gnu[2]:
|
||||||
|
print(f" err : mine={mine[2][:120]!r}")
|
||||||
|
print(f" gnu ={gnu[2][:120]!r}")
|
||||||
|
results.append((name, ok or name in EXPECTED_DIFF))
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
random.seed(7)
|
||||||
|
big = bytes(random.getrandbits(8) for _ in range(200_000))
|
||||||
|
|
||||||
|
# ── encoding ──────────────────────────────────────────────────────
|
||||||
|
t("encode empty", [], b"")
|
||||||
|
for n in (1, 2, 3, 4, 5, 56, 57, 58, 76, 1000):
|
||||||
|
t(f"encode {n} bytes", [], (bytes(range(256)) * 10)[:n])
|
||||||
|
t("encode all byte values", [], bytes(range(256)))
|
||||||
|
t("encode 200KB binary", [], big)
|
||||||
|
sample = b"hello world, this is a longer sample string for wrapping"
|
||||||
|
for w in ("0", "1", "2", "3", "4", "5", "76", "77", "1000"):
|
||||||
|
t(f"encode wrap {w}", ["-w", w], sample)
|
||||||
|
t(f"encode wrap {w} joined", [f"-w{w}"], sample)
|
||||||
|
t(f"encode wrap {w} long", ["--wrap=" + w], big[:5000])
|
||||||
|
|
||||||
|
# ── decoding ──────────────────────────────────────────────────────
|
||||||
|
for n in (0, 1, 2, 3, 4, 5, 100, 1000):
|
||||||
|
enc = subprocess.run([GNU], input=big[:n], capture_output=True).stdout
|
||||||
|
t(f"decode roundtrip {n}", ["-d"], enc)
|
||||||
|
t("decode 200KB roundtrip", ["-d"],
|
||||||
|
subprocess.run([GNU], input=big, capture_output=True).stdout)
|
||||||
|
t("decode unwrapped", ["-d"], b"aGVsbG8gd29ybGQ=")
|
||||||
|
t("decode no trailing newline", ["-d"], b"aGVsbG8=")
|
||||||
|
t("decode with newlines", ["-d"], b"aGVs\nbG8=\n")
|
||||||
|
t("decode CRLF wrapped", ["-d"], b"aGVs\r\nbG8=\r\n")
|
||||||
|
t("decode empty", ["-d"], b"")
|
||||||
|
t("decode only newline", ["-d"], b"\n")
|
||||||
|
t("decode spaces garbage", ["-d"], b"aGVs bG8=")
|
||||||
|
t("decode spaces garbage -i", ["-di"], b"aGVs bG8=")
|
||||||
|
t("decode junk", ["-d"], b"aG!!Vs*bG8=")
|
||||||
|
t("decode junk -i", ["-di"], b"aG!!Vs*bG8=")
|
||||||
|
t("decode junk only -i", ["-di"], b"!!!!")
|
||||||
|
t("decode nul bytes -i", ["-di"], b"aGVs\x00bG8=")
|
||||||
|
t("decode high bytes", ["-d"], b"aGVs\xffbG8=")
|
||||||
|
t("decode high bytes -i", ["-di"], b"aGVs\xffbG8=")
|
||||||
|
t("decode urlsafe chars", ["-d"], b"_-_-")
|
||||||
|
|
||||||
|
# Unpadded tails are legal only when the leftover bits are zero.
|
||||||
|
for s in (b"a", b"aA", b"aQ", b"ag", b"aG", b"aGA", b"aGE", b"aGV",
|
||||||
|
b"aB", b"aC", b"aGB", b"aGC", b"aGVs", b"aGVsb", b"aGVsbG",
|
||||||
|
b"aGVsbG8"):
|
||||||
|
t(f"decode tail {s.decode()}", ["-d"], s)
|
||||||
|
|
||||||
|
# Padding placement, and streams concatenated after padding.
|
||||||
|
for s in (b"=", b"====", b"=aGVsbG8=", b"aG=sbG8=", b"aGVsbG8===",
|
||||||
|
b"aA==", b"aQ==", b"aG==", b"aGA=", b"aGV=", b"aA=", b"aA=x",
|
||||||
|
b"aG=x", b"aGA=x", b"aA==x", b"aA==aGVs", b"aGVsbG8=x",
|
||||||
|
b"aGVsbG8=aGVs", b"aGVsbG8=\naGVs", b"aGVsbG8=aGVsbG8="):
|
||||||
|
t(f"decode pad {s.decode()}", ["-d"], s)
|
||||||
|
|
||||||
|
# Every prefix of a known-good encoding, strict and lenient.
|
||||||
|
enc = subprocess.run([GNU], input=b"hello world!", capture_output=True).stdout.strip()
|
||||||
|
for i in range(1, len(enc) + 1):
|
||||||
|
t(f"decode prefix {i}", ["-d"], enc[:i])
|
||||||
|
t(f"decode prefix {i} -i", ["-di"], enc[:i])
|
||||||
|
|
||||||
|
# ── options and usage errors ──────────────────────────────────────
|
||||||
|
t("help", ["--help"])
|
||||||
|
t("help after other flags", ["--decode", "--help"])
|
||||||
|
t("abbrev --dec", ["--dec"], b"aGVsbG8=")
|
||||||
|
t("abbrev --ign", ["--ign", "-d"], b"aGV s bG8=")
|
||||||
|
t("abbrev --w", ["--w", "4"], b"hello world")
|
||||||
|
t("bare --", ["--"], b"hello")
|
||||||
|
t("cluster -di", ["-di"], b"aG!Vs bG8=")
|
||||||
|
t("cluster -dw0", ["-dw0"], b"aGVsbG8=")
|
||||||
|
t("wrap missing arg", ["-w"])
|
||||||
|
t("wrap bad value", ["-w", "abc"])
|
||||||
|
t("wrap negative", ["-w", "-5"])
|
||||||
|
t("unknown short", ["-z"])
|
||||||
|
t("unknown long", ["--nope"])
|
||||||
|
t("ambiguous long", ["--i"])
|
||||||
|
t("extra operand", ["a.txt", "b.txt"])
|
||||||
|
t("missing file", ["/nonexistent/nope.txt"])
|
||||||
|
t("directory as file", ["/tmp"])
|
||||||
|
t("dash is stdin", ["-"], b"hello")
|
||||||
|
t("double dash then dash", ["--", "-"], b"hello")
|
||||||
|
|
||||||
|
# ── file operands ─────────────────────────────────────────────────
|
||||||
|
with tempfile.TemporaryDirectory() as tmp:
|
||||||
|
raw = os.path.join(tmp, "sample.bin")
|
||||||
|
with open(raw, "wb") as fh:
|
||||||
|
fh.write(big[:9999])
|
||||||
|
t("encode from file", [raw])
|
||||||
|
t("encode from file wrap 0", ["-w0", raw])
|
||||||
|
|
||||||
|
encoded = os.path.join(tmp, "sample.b64")
|
||||||
|
with open(encoded, "wb") as fh:
|
||||||
|
fh.write(subprocess.run([GNU], input=big[:9999], capture_output=True).stdout)
|
||||||
|
t("decode from file", ["-d", encoded])
|
||||||
|
|
||||||
|
passed = sum(1 for _, ok in results if ok)
|
||||||
|
print(f"\n{passed}/{len(results)} cases match GNU base64")
|
||||||
|
failed = [n for n, ok in results if not ok]
|
||||||
|
if failed:
|
||||||
|
print("failing:", ", ".join(failed))
|
||||||
|
return 1
|
||||||
|
print("all parity checks passed")
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
if not os.path.exists(MINE):
|
||||||
|
sys.exit(f"build it first: cargo build --release (looked for {MINE})")
|
||||||
|
if GNU is None:
|
||||||
|
sys.exit(
|
||||||
|
"no GNU coreutils base64 found. Distributions increasingly ship "
|
||||||
|
"uutils as /usr/bin/base64, which differs on malformed input; "
|
||||||
|
"install GNU coreutils to run this comparison."
|
||||||
|
)
|
||||||
|
print(f"comparing {MINE}\n against {GNU} ({GNU_BANNER})\n")
|
||||||
|
sys.exit(main())
|
||||||
Reference in New Issue
Block a user