set up README for Max

README.md

@@ -1,77 +1,56 @@
 # UC Salary Pipeline
 
-Downloads University of California employee wage data and classifies every
+This script townloads University of California employee wage data and classifies every employee into an analytical category (Senate Faculty, Athletics, Senior Management, etc.), producing per-employee, aggregate, and top-earner CSVs.
-employee into one of 8 analytical categories (Senate Faculty, Athletics,
-Senior Management, etc.), producing per-employee, aggregate, and top-earner
-CSVs.
 
-## What it does
+## What the script does
 
-1. **Downloads** raw wage JSON for every requested campus/year from
+1. Downloads raw wage JSON for every requested campus/year from
    [ucinvestments/UC-wages](https://github.com/ucinvestments/UC-wages), a
    public dataset (not an API) sourced from
    [UC Annual Wage](https://ucannualwage.ucop.edu/wage/), covering 2010
    onward for all 10 campuses plus UCOP, ASUCLA, and UC SF Law.
-2. **Classifies** every employee record by job title (and, for non-academic
+2. Classifies every employee record by job title (and, for non-academic staff, gross pay) into one of 8 categories, and separately flags whether they work at a UC medical center. Categorization is based on job description, and probably needs some work and will never be complete for the medical centers really.
-   staff, gross pay) into one of 8 categories, and separately flags whether
+3. Outputs to CSV plus a formatted summary table printed to the terminal.
-   they work at a UC medical center.
-3. **Writes CSVs**: always a per-employee CSV, plus optional aggregate and
-   top-earners CSVs, plus a formatted summary table printed to the terminal.
 
 ## Architecture
 
-Three files, each with one job:
+the software has three files, each has one job:
 
-```
+- **`classifier.py`** — the classification engine. give it a job title string (and
-uc_wages_downloader.py  ──▶  classifier.py  ──▶  uc_salary_pipeline.py
-   (data acquisition)      (classification engine)   (orchestrator / CLI)
-```
-
-- **`classifier.py`** — the classification engine, and *only* the
-  classification engine. It has no knowledge of downloading, file formats,
-  or CSVs — it's a pure function library: give it a job title string (and
   optionally a gross pay figure and campus code), get back a category and/or
-  a medical-center flag. This is deliberate: it's the part of the tool that
+  a medical-center flag.  It's kept importable on its own —
-  encodes UC-specific domain knowledge (Academic Senate membership rules,
+  `python -c "import classifier"` works. See
-  which job codes mean what), and it's kept importable on its own —
-  `python -c "import classifier"` works with zero other dependencies. See
   [Classification logic](#classification-logic) below for how it works.
 
 - **`uc_wages_downloader.py`** — fetches raw per-campus/per-year JSON files
   from the UC-wages GitHub repo into a local cache directory
-  (`uc_wages_data/` by default). Knows nothing about classification. Also
+  (`uc_wages_data/` by default).
-  runnable standalone (`python uc_wages_downloader.py`) if you just want the
-  raw data.
 
-- **`uc_salary_pipeline.py`** — the orchestrator and CLI entry point. Calls
+- **`uc_salary_pipeline.py`** — main shebang. Calls the downloader, reads the cached JSON, maps UC-wages' field names onto what `classifier.py` expects, calls `classifier.classify_title()` /
-  the downloader, reads the cached JSON, maps UC-wages' field names onto
-  what `classifier.py` expects, calls `classifier.classify_title()` /
   `classifier.is_medical_center()` per record, and writes the output CSVs.
-  This is the script you actually run.
 
 ## Quick start
 
 ```bash
 # Download every campus, every available year, and classify everything
-# (~1.3GB download, multi-million-row output — see --years/--campuses to scope it down):
 python uc_salary_pipeline.py
 
-# A specific slice:
+# Get specific slice per year and campus:
 python uc_salary_pipeline.py --years 2023 2024 --campuses UCB UCLA
 
-# Re-classify already-downloaded data without hitting the network:
+# Skip downloading the data (script also skips existing data if it's there)
 python uc_salary_pipeline.py --skip-download
 
-# Full example with all optional outputs:
+# Full example
 python uc_salary_pipeline.py --years 2024 --campuses UCB UCLA UCSF \
     --aggregate-output agg.csv --group-by year campus medical_center \
     --top-earners-output top.csv --top-n 20
 ```
 
 Downloaded JSON is cached in `uc_wages_data/` (gitignored) and reused on
-every subsequent run — nothing is re-downloaded unless you delete it.
+every subsequent run
 
-## `uc_salary_pipeline.py` flags
+## Runtime flags for `uc_salary_pipeline.py`
 
 | Flag | Default | What it does |
 |---|---|---|
@@ -91,27 +70,7 @@ every subsequent run — nothing is re-downloaded unless you delete it.
 
 Category labels in every output (e.g. `"Senior Prof/Staff (>=$100k)"`)
 automatically reflect whatever `--senior-threshold`/`--support-threshold`
-you actually pass — they're not hardcoded strings.
+is passed.
-
-## `uc_wages_downloader.py` flags
-
-Only needed directly if you want to pre-fetch data without classifying it
-yet (`uc_salary_pipeline.py` calls this automatically unless you pass
-`--skip-download`).
-
-| Flag | Default | What it does |
-|---|---|---|
-| `--years Y [Y ...]` | `2010`–current year | Years to download. |
-| `--campuses C [C ...]` | all | Campuses to download (same choices as above). |
-| `--out-dir DIR` | `uc_wages_data` | Where to write the JSON files. |
-| `--dry-run` | off | Print the URLs that would be fetched; download nothing. |
-| `--timeout N` | `60` | HTTP timeout in seconds per request. |
-| `--retries N` | `3` | Retry attempts per file on transient failure. |
-
-Downloads go straight to `raw.githubusercontent.com` (no GitHub API calls,
-so no rate-limit risk). A campus/year combination that doesn't exist
-upstream (e.g. ASUCLA before 2018) returns a 404 and is logged and skipped,
-not treated as an error. Already-downloaded files are never re-fetched.
 
 ## Output files
 
@@ -155,16 +114,12 @@ All in `classifier.py`. Two independent things are computed per employee:
 
 ### 1. Category (`classify_title()`)
 
-Every employee falls into exactly one of 8 categories, organized around
+Every employee falls into exactly one of 8 categories, like so:
-whether a title carries UC Academic Senate membership (Regents Standing
-Order 105.1), tested in this order (earlier patterns win):
 
 1. `STUDENT_TRAINEE` — TAs, GSRs, postdocs, residents, interns, student workers
-2. `ATHLETICS` — coaches and athletics staff (split out so bonus-heavy
+2. `ATHLETICS` — coaches and athletics staff 
-   coaching pay doesn't distort management totals)
 3. `SENIOR_MANAGEMENT` — Chancellor/Provost/VP/Dean/Executive Director tier
-4. `SENATE_FACULTY` — ladder professors, instructors, and other
+4. `SENATE_FACULTY` — ladder professors and other Senate-membership titles
-   Senate-membership titles
 5. `NON_SENATE_ACADEMIC` — lecturers, adjuncts, HS clinical professors,
    project scientists, librarians, academic coordinators, etc.
 6. `SENIOR_PROF_STAFF` / `PROF_STAFF` / `SUPPORT_STAFF` — everyone else
@@ -178,37 +133,27 @@ thresholds are all defined at the top of `classifier.py`.
 
 ### 2. Medical center flag (`is_medical_center()`)
 
-Orthogonal to category — an employee can be `SENATE_FACULTY` *and*
+- (any campus): titles containing `MED CTR` (hospital
-`medical_center=True` (e.g. a med-school ladder professor on the HCOMP pay
-plan). Two-tier rule, built from actually surveying UC-wages title text
-rather than guessed:
-
-- **Always** (any campus): titles containing `MED CTR` (hospital
   facilities/exec titles), the HS Clinical Professor/Instructor series, or
   bare `-HCOMP` ladder titles.
-- **Only** at the five campuses that operate a UC-owned academic medical
+- Five campuses that operate a UC-owned academic medical
   center (`UCSF`, `UCLA`, `UCD`, `UCI`, `UCSD`): generic clinical-workforce
   titles — physicians, nurses, dentists, pharmacists, therapists, lab/rad
   techs, etc. These titles are identical at every campus (Student Health
   Services exists everywhere), so outside the 5 hospital campuses they
   can't be reliably told apart from student health staff by title text
   alone.
-- **Always excluded**: any title containing `SHS` (Student Health
+- Excluded any title containing `SHS` (Student Health
-  Services), even at UCSF — it's a distinct, campus-wide operation from the
+  Services), even at UCSF
-  hospital system.
 
-This is a title-text heuristic against a data source with no
+This won't catch things like a bare `DEAN` title that happens to be the Dean of Medicine.
-department/org-unit field, not a perfect org-chart lookup — it won't catch
-things like a bare `DEAN` title that happens to be the Dean of Medicine
-(indistinguishable from any other school's dean by title text alone).
 
 ## Data notes
 
 - UC-wages has no `benefits` field (unlike the old California State
   Controller GCC data this tool previously used), so `gross_pay` is the
-  only pay figure available — `regular_pay + overtime_pay + other_pay`.
+  only pay figure available — `regular_pay + overtime_pay + other_pay`... might be useful in the future if we want benefits data to go back to the GCC scraper.
 - Employee names are pre-redacted by the upstream source for some records
-  (`firstname`/`lastname` show as `*****`) — this is not something this
+  (`firstname`/`lastname` show as `*****`)
-  tool does.
 - The full dataset is ~1.3GB across ~170 files. `uc_wages_data/` is
-  gitignored; nothing here commits downloaded data to the repo.
+  gitignored; run the script to download the data first time