The toolbox

Take itthe same thing, in the form your assistant reads

## Reading the web

Nothing in this project opens a socket. Every request goes through the `courteous` binary.

`courteous get <url>` prints a page or exits non-zero with an explanation. `courteous save <url>
<file>` does the same for something that is not text. `courteous state` shows what is owed to each
host.

It holds a global lock so only one request is in flight across every tool, waits twenty to forty
seconds between requests to a host, stops at sixty requests to that host per day, reads and obeys
robots.txt, and on a 429 or a 503 puts that host away for six hours.

A refusal is an answer, not an obstacle. If a site says no we stop — we do not slow down and come
back from a different angle, and we do not change the user agent.

When adding a tool that reads the web, call `courteous`. Do not add a fetch helper, a retry loop,
or a sleep constant of your own; those are what the binary exists to replace.

---
name: courteous
description: Build or audit a polite web fetcher. Use when a project needs to read public pages without behaving like a crawler, or when something has been rate-limited.
---

# courteous

A person reading a website is never told they are asking too often. If a rate limit comes back,
the thing asking was not behaving like a person, and a longer sleep does not fix it: the causes
are several callers with their own idea of polite, no memory between runs, and no ceiling on the
day. Put the manners in one small program everything else has to go through.

## Build it

One compiled binary, no runtime, roughly three hundred lines, reasoning in the comments.

Interface:
  courteous get <url>           prints the body, or exits non-zero with a plain explanation
  courteous save <url> <file>   the same, for something that is not text
  courteous state               what is currently owed to each host
  courteous log [n]             the last n requests

Rules it enforces rather than offers:
1. One request at a time across every caller. An exclusive lock file, waited for, broken only if
   more than five minutes stale.
2. Twenty to forty seconds between requests to one host, jittered. Next-allowed time stored per
   host on disk.
3. Sixty requests per host per day. At the ceiling that host is finished until tomorrow.
4. robots.txt read once per host per day, cached, obeyed. STRIP TRAILING COMMENTS FROM A RULE
   BEFORE MATCHING — a line like `Disallow: /works? # cruel but efficient` will otherwise never
   match and the program will silently not do the one thing it exists for.
5. On 429 or 503, put that host away for six hours and record it on disk. No retry, no second user
   agent, no other route in.
6. Log every request with host, status and timestamp.
7. A real user agent naming the project and a contact address.

## Then close the door

Add a check to the build that fails if any other file opens a socket directly. Manners that live in
one place are only real if nothing is allowed around them.

## Audit an existing project

Look for: several scripts fetching concurrently, per-script sleep constants, retry-on-429 loops,
a user agent copied from a browser, and any robots handling that does not strip comments. Each of
those is the same mistake wearing different clothes.

## Reading the web

Nothing in this project opens a socket. Every request goes through the `courteous` binary, which
holds a global lock, waits twenty to forty seconds between requests to a host, stops at sixty
requests to that host per day, obeys robots.txt, and puts a host away for six hours on a 429 or a
503 with no retry and no second user agent.

A refusal is an answer, not an obstacle. If a site says no, we stop — we do not slow down and come
back from a different angle.

---
description: All web access goes through the courteous binary
globs: ["**/*.py", "**/*.ts", "**/*.js", "**/*.rs", "**/*.go"]
alwaysApply: true
---

Nothing in this project opens a socket. Every request goes through the `courteous` binary:
`courteous get <url>`, `courteous save <url> <file>`, `courteous state`.

It holds a global lock across every caller, waits twenty to forty seconds between requests to one
host, stops at sixty per host per day, obeys robots.txt, and puts a host away for six hours after a
429 or a 503.

Never write a fetch helper, a retry-on-429 loop, or a per-script sleep constant. Those are the
things the binary replaces, and reintroducing one puts the manners back in the hands of whoever is
calling.

## Reading the web

Use the `courteous` binary for every network request: `courteous get <url>` for a page,
`courteous save <url> <file>` for anything that is not text.

Do not use fetch, requests, urllib, curl or an HTTP client directly, and do not write a retry loop
around a rate limit. One request at a time, twenty to forty seconds apart per host, sixty a day,
robots.txt obeyed, and a 429 stops that host for six hours.

A refusal is an answer. Do not route around it.

Build me a small command-line program called `courteous`. Its only job is to fetch web
pages politely, and every other tool I write will go through it instead of opening a socket itself.

Why: a person reading a website is never told they are asking too often. If a rate limit ever comes
back, the thing asking was not behaving like a person, and a longer sleep does not fix that. The
real causes are several scripts running at once with their own ideas of polite, no memory between
runs, and no ceiling on the day. Put the manners in one place that cannot be talked out of them.

Rules it must enforce, not merely offer:
 1. One request at a time across every caller. Take an exclusive lock file; wait for it; break it
    only if it is more than five minutes stale.
 2. Twenty to forty seconds between requests to the same host, jittered so the cadence is not a
    metronome. Store the next-allowed time on disk per host.
 3. A ceiling of sixty requests per host per day. When it is reached, that host is finished until
    tomorrow, whatever the caller still wants.
 4. Read robots.txt once per host per day, cache it, obey it. Strip trailing comments from a rule
    before matching — a line like `Disallow: /works? # cruel but efficient` will otherwise never
    match, and the one thing the program exists to do it will not be doing.
 5. On 429 or 503, put that host away for six hours and record it on disk. No retry, no second user
    agent, no other route in. A refusal is an answer.
 6. Log every request with host, status and timestamp, so behaviour is checkable rather than
    claimed.
 7. A real user agent naming the project and a contact address.

Interface: `courteous get <url>` prints the body or exits non-zero with a plain explanation;
`courteous save <url> <file>` does the same for something that is not text; `courteous state`
prints what is owed to each host; `courteous log [n]` prints the last n requests.

Write it in a compiled language with no runtime, keep it under about three hundred lines, and put
the reasoning in comments so the next person understands why each rule is there rather than only
what it does.

The source1 version, every one kept

//! courteous — one reader, slowly.
//!
//! Everything French Press fetches goes through here. See README.md for why; the short version is
//! that we were rate-limited once, a person never is, and the fix belonged in one place rather
//! than in every script's own idea of polite.

use std::collections::HashMap;
use std::fs::{self, File, OpenOptions};
use std::io::{Read, Write};
use std::path::PathBuf;
use std::time::{SystemTime, UNIX_EPOCH};

const UA: &str = "web:fp.caelivonstudios.com:v0.1 (one reader, slowly; [email protected])";
const GAP_MIN: f64 = 20.0;      // seconds between requests to one host
const GAP_MAX: f64 = 40.0;
const DAY_BUDGET: u32 = 60;     // requests per host per day, ours not theirs
const COOL: f64 = 6.0 * 3600.0; // a refusal puts a host away this long

fn now() -> f64 {
    SystemTime::now().duration_since(UNIX_EPOCH).unwrap().as_secs_f64()
}

/// Enough randomness to stop the cadence being a metronome. A person is not a clock.
fn jitter(lo: f64, hi: f64) -> f64 {
    let n = SystemTime::now().duration_since(UNIX_EPOCH).unwrap().subsec_nanos() as f64;
    lo + (n / 1_000_000_000.0) * (hi - lo)
}

fn home() -> PathBuf {
    let base = std::env::var("COURTEOUS_HOME")
        .unwrap_or_else(|_| std::env::var("HOME").unwrap_or_else(|_| ".".into()));
    let p = PathBuf::from(base).join(".courteous");
    let _ = fs::create_dir_all(&p);
    p
}

/// What we owe one host: how many today, when we may knock again, and whether it said no.
#[derive(Clone, Default)]
struct Host {
    day: String,
    count: u32,
    next: f64,
    cool_until: f64,
}

fn state_path() -> PathBuf { home().join("state.tsv") }
fn log_path() -> PathBuf { home().join("requests.log") }
fn lock_path() -> PathBuf { home().join("lock") }

fn today() -> String {
    // Days only need to be distinct and monotonic, so this counts them from the epoch rather
    // than pulling in a calendar library to print a date nobody reads.
    format!("d{}", (now() / 86400.0) as i64)
}

fn read_state() -> HashMap<String, Host> {
    let mut m = HashMap::new();
    if let Ok(s) = fs::read_to_string(state_path()) {
        for line in s.lines() {
            let f: Vec<&str> = line.split('\t').collect();
            if f.len() == 5 {
                m.insert(f[0].to_string(), Host {
                    day: f[1].into(),
                    count: f[2].parse().unwrap_or(0),
                    next: f[3].parse().unwrap_or(0.0),
                    cool_until: f[4].parse().unwrap_or(0.0),
                });
            }
        }
    }
    m
}

fn write_state(m: &HashMap<String, Host>) {
    let mut out = String::new();
    for (h, s) in m {
        out.push_str(&format!("{}\t{}\t{}\t{}\t{}\n", h, s.day, s.count, s.next, s.cool_until));
    }
    let _ = fs::write(state_path(), out);
}

/// One request at a time across every tool, however many are running. This is the part that was
/// actually missing: two of ours were reading the same site in the same minute.
struct Lock;
impl Lock {
    fn take() -> Result<Lock, String> {
        for _ in 0..600 {
            match OpenOptions::new().write(true).create_new(true).open(lock_path()) {
                Ok(mut f) => { let _ = write!(f, "{}", std::process::id()); return Ok(Lock); }
                Err(_) => {
                    if let Ok(md) = fs::metadata(lock_path()) {
                        if let Ok(age) = md.modified().and_then(|m| m.elapsed().map_err(|_| {
                            std::io::Error::new(std::io::ErrorKind::Other, "clock")
                        })) {
                            if age.as_secs() > 300 { let _ = fs::remove_file(lock_path()); continue; }
                        }
                    }
                    std::thread::sleep(std::time::Duration::from_secs(1));
                }
            }
        }
        Err("another reader has held the lock for ten minutes".into())
    }
}
impl Drop for Lock {
    fn drop(&mut self) { let _ = fs::remove_file(lock_path()); }
}

fn host_of(url: &str) -> String {
    url.split("://").nth(1).unwrap_or(url).split('/').next().unwrap_or("").to_string()
}

fn note(host: &str, code: u16, url: &str) {
    if let Ok(mut f) = OpenOptions::new().create(true).append(true).open(log_path()) {
        let _ = writeln!(f, "{}\t{}\t{}\t{}", now() as i64, host, code,
                         &url[..url.len().min(150)]);
    }
}

/// robots.txt, fetched once per host per day and obeyed. Cached so asking is itself polite.
fn robots_allows(host: &str, path: &str) -> bool {
    let cache = home().join(format!("robots-{}-{}.txt", host, today()));
    let body = if let Ok(s) = fs::read_to_string(&cache) {
        s
    } else {
        let got = ureq::get(&format!("https://{}/robots.txt", host))
            .set("User-Agent", UA)
            .timeout(std::time::Duration::from_secs(20))
            .call();
        let s = match got {
            Ok(r) => r.into_string().unwrap_or_default(),
            Err(_) => String::new(),          // no robots file is not a refusal
        };
        let _ = fs::write(&cache, &s);
        s
    };
    let mut applies = false;
    for line in body.lines() {
        let l = line.trim();
        let low = l.to_ascii_lowercase();
        if let Some(v) = low.strip_prefix("user-agent:") {
            applies = v.trim() == "*";
        } else if applies {
            if let Some(v) = low.strip_prefix("disallow:") {
                // A rule may carry a trailing comment. AO3's reads `Disallow: /works? # cruel but
                // efficient`, and taking the comment as part of the rule made the rule never
                // match — which meant the one thing this program exists to do, it was not doing.
                let rule = v.split('#').next().unwrap_or("").trim();
                if !rule.is_empty() && path.starts_with(rule.trim_end_matches('*')) {
                    return false;
                }
            }
        }
    }
    true
}

/// Same manners, but the body is written straight to a file. Pictures are not text and piping
/// them through a string is how a poster arrives corrupted.
fn save(url: &str, out: &str) -> Result<usize, String> {
    let bytes = fetch(url, true)?;
    let n = bytes.len();
    fs::write(out, bytes).map_err(|e| format!("{}", e))?;
    Ok(n)
}

fn get(url: &str) -> Result<String, String> {
    let b = fetch(url, false)?;
    Ok(String::from_utf8_lossy(&b).into_owned())
}

fn fetch(url: &str, _binary: bool) -> Result<Vec<u8>, String> {
    let host = host_of(url);
    let path = format!("/{}", url.split("://").nth(1).unwrap_or("")
        .splitn(2, '/').nth(1).unwrap_or(""));
    let mut st = read_state();
    let mut h = st.get(&host).cloned().unwrap_or_default();

    if h.cool_until > now() {
        return Err(format!("{} asked us to stop. {} minutes left of staying stopped.",
                           host, ((h.cool_until - now()) / 60.0) as i64));
    }
    if h.day != today() { h.day = today(); h.count = 0; }
    if h.count >= DAY_BUDGET {
        return Err(format!("{}: {} requests in a day is our own ceiling and we have reached it.",
                           host, DAY_BUDGET));
    }
    if !robots_allows(&host, &path) {
        return Err(format!("{} disallows this path in robots.txt.", host));
    }

    let _lock = Lock::take()?;
    let wait = h.next - now();
    if wait > 0.0 {
        std::thread::sleep(std::time::Duration::from_secs_f64(wait));
    }

    let res = ureq::get(url).set("User-Agent", UA)
        .timeout(std::time::Duration::from_secs(30)).call();

    h.count += 1;
    h.next = now() + jitter(GAP_MIN, GAP_MAX);

    let out = match res {
        Ok(r) => {
            note(&host, r.status(), url);
            let mut buf = Vec::new();
            r.into_reader().read_to_end(&mut buf).map_err(|e| format!("{}", e))?;
            Ok(buf)
        }
        Err(ureq::Error::Status(code, _)) => {
            note(&host, code, url);
            if code == 429 || code == 503 {
                h.cool_until = now() + COOL;
                Err(format!("{} answered {}. We stop for six hours. No retry, no second agent, \
                             no other route in.", host, code))
            } else {
                Err(format!("{} answered {}.", host, code))
            }
        }
        Err(e) => { note(&host, 0, url); Err(format!("{}", e)) }
    };
    st.insert(host, h);
    write_state(&st);
    out
}

fn main() {
    let a: Vec<String> = std::env::args().collect();
    match a.get(1).map(|s| s.as_str()) {
        Some("get") => match a.get(2) {
            Some(url) => match get(url) {
                // A closed pipe is the caller saying it has read enough, which is not an error.
                Ok(body) => {
                    use std::io::Write as _;
                    let out = std::io::stdout();
                    let _ = out.lock().write_all(body.as_bytes());
                }
                Err(why) => { eprintln!("{}", why); std::process::exit(3); }
            },
            None => { eprintln!("courteous get <url>"); std::process::exit(2); }
        },
        Some("save") => match (a.get(2), a.get(3)) {
            (Some(url), Some(out)) => match save(url, out) {
                Ok(n) => println!("{}", n),
                Err(why) => { eprintln!("{}", why); std::process::exit(3); }
            },
            _ => { eprintln!("courteous save <url> <file>"); std::process::exit(2); }
        },
        Some("state") => {
            for (host, s) in read_state() {
                let n = if s.day == today() { s.count } else { 0 };
                let cool = if s.cool_until > now() {
                    format!("   stopped for {} more minutes", ((s.cool_until - now()) / 60.0) as i64)
                } else { String::new() };
                println!("  {:<34} {:>3} today{}", host, n, cool);
            }
        }
        Some("log") => {
            let n: usize = a.get(2).and_then(|s| s.parse().ok()).unwrap_or(20);
            let mut s = String::new();
            if let Ok(mut f) = File::open(log_path()) { let _ = f.read_to_string(&mut s); }
            for line in s.lines().rev().take(n).collect::<Vec<_>>().into_iter().rev() {
                println!("  {}", line);
            }
        }
        _ => {
            eprintln!("courteous get <url>          one page, politely, or an explanation, exit 3");
            eprintln!("courteous save <url> <file>  the same, for something that is not text");
            eprintln!("courteous state       what we currently owe each host");
            eprintln!("courteous log [n]     the last n requests we made");
            std::process::exit(2);
        }
    }
}

What changednewest first