uc_salary_aggregator/README.md

# UC Salary Pipeline

Downloads University of California employee wage data and classifies every
employee into one of 8 analytical categories (Senate Faculty, Athletics,
Senior Management, etc.), producing per-employee, aggregate, and top-earner
CSVs.

## What it does

1. **Downloads** raw wage JSON for every requested campus/year from
   [ucinvestments/UC-wages](https://github.com/ucinvestments/UC-wages), a
   public dataset (not an API) sourced from
   [UC Annual Wage](https://ucannualwage.ucop.edu/wage/), covering 2010
   onward for all 10 campuses plus UCOP, ASUCLA, and UC SF Law.
2. **Classifies** every employee record by job title (and, for non-academic
   staff, gross pay) into one of 8 categories, and separately flags whether
   they work at a UC medical center.
3. **Writes CSVs**: always a per-employee CSV, plus optional aggregate and
   top-earners CSVs, plus a formatted summary table printed to the terminal.

## Architecture

Three files, each with one job:

```
uc_wages_downloader.py  ──▶  classifier.py  ──▶  uc_salary_pipeline.py
   (data acquisition)      (classification engine)   (orchestrator / CLI)
```

- **`classifier.py`** — the classification engine, and *only* the
  classification engine. It has no knowledge of downloading, file formats,
  or CSVs — it's a pure function library: give it a job title string (and
  optionally a gross pay figure and campus code), get back a category and/or
  a medical-center flag. This is deliberate: it's the part of the tool that
  encodes UC-specific domain knowledge (Academic Senate membership rules,
  which job codes mean what), and it's kept importable on its own —
  `python -c "import classifier"` works with zero other dependencies. See
  [Classification logic](#classification-logic) below for how it works.

- **`uc_wages_downloader.py`** — fetches raw per-campus/per-year JSON files
  from the UC-wages GitHub repo into a local cache directory
  (`uc_wages_data/` by default). Knows nothing about classification. Also
  runnable standalone (`python uc_wages_downloader.py`) if you just want the
  raw data.

- **`uc_salary_pipeline.py`** — the orchestrator and CLI entry point. Calls
  the downloader, reads the cached JSON, maps UC-wages' field names onto
  what `classifier.py` expects, calls `classifier.classify_title()` /
  `classifier.is_medical_center()` per record, and writes the output CSVs.
  This is the script you actually run.

## Quick start

```bash
# Download every campus, every available year, and classify everything
# (~1.3GB download, multi-million-row output — see --years/--campuses to scope it down):
python uc_salary_pipeline.py

# A specific slice:
python uc_salary_pipeline.py --years 2023 2024 --campuses UCB UCLA

# Re-classify already-downloaded data without hitting the network:
python uc_salary_pipeline.py --skip-download

# Full example with all optional outputs:
python uc_salary_pipeline.py --years 2024 --campuses UCB UCLA UCSF \
    --aggregate-output agg.csv --group-by year campus medical_center \
    --top-earners-output top.csv --top-n 20
```

Downloaded JSON is cached in `uc_wages_data/` (gitignored) and reused on
every subsequent run — nothing is re-downloaded unless you delete it.

## `uc_salary_pipeline.py` flags

| Flag | Default | What it does |
|---|---|---|
| `--years Y [Y ...]` | `2010`–current year | Years to process. |
| `--campuses C [C ...]` | all | Campuses to process. Choices: `ASUCLA UCB UCD UCI UCLA UCM UCR UCSD UCSF UCSB UCSC UCOP UC_SF_LAW`. |
| `--data-dir DIR` | `uc_wages_data` | Local cache directory for downloaded JSON. |
| `--output PATH` | `uc_salary_classified.csv` | Per-employee output CSV path. |
| `--aggregate-output PATH` | *(off)* | If given, also write a category-level rollup CSV (employee count, total/mean/median gross pay). |
| `--group-by [DIM ...]` | `year` | Extra breakdown dimensions for `--aggregate-output` and `--top-earners-output`, alongside category. Choices: `year`, `campus`, `medical_center`. Pass none to collapse to one row per category; pass all three for the finest breakdown. |
| `--top-earners-output PATH` | *(off)* | If given, also write the top `--top-n` gross-pay earners per category (respects `--group-by`). |
| `--top-n N` | `10` | How many top earners per group, for `--top-earners-output`. |
| `--skip-download` | off | Classify whatever's already cached in `--data-dir`; never hits the network. |
| `--quiet` | off | Suppress the formatted per-year summary table normally printed to stdout. |
| `--senior-threshold N` | `100000` | Gross pay at/above which non-academic staff are `SENIOR_PROF_STAFF` instead of `PROF_STAFF`. |
| `--support-threshold N` | `40000` | Gross pay below which non-academic staff are `SUPPORT_STAFF` instead of `PROF_STAFF`. |
| `-h`, `--help` | — | Full usage (argparse-generated). |

Category labels in every output (e.g. `"Senior Prof/Staff (>=$100k)"`)
automatically reflect whatever `--senior-threshold`/`--support-threshold`
you actually pass — they're not hardcoded strings.

## `uc_wages_downloader.py` flags

Only needed directly if you want to pre-fetch data without classifying it
yet (`uc_salary_pipeline.py` calls this automatically unless you pass
`--skip-download`).

| Flag | Default | What it does |
|---|---|---|
| `--years Y [Y ...]` | `2010`–current year | Years to download. |
| `--campuses C [C ...]` | all | Campuses to download (same choices as above). |
| `--out-dir DIR` | `uc_wages_data` | Where to write the JSON files. |
| `--dry-run` | off | Print the URLs that would be fetched; download nothing. |
| `--timeout N` | `60` | HTTP timeout in seconds per request. |
| `--retries N` | `3` | Retry attempts per file on transient failure. |

Downloads go straight to `raw.githubusercontent.com` (no GitHub API calls,
so no rate-limit risk). A campus/year combination that doesn't exist
upstream (e.g. ASUCLA before 2018) returns a 404 and is logged and skipped,
not treated as an error. Already-downloaded files are never re-fetched.

## Output files

### Per-employee CSV (`--output`, always written)

One row per employee per year:

```
year, campus, medical_center, title, category, category_label,
gross_pay, regular_pay, overtime_pay, other_pay, first_name, last_name
```

### Aggregate CSV (`--aggregate-output`, optional)

One row per category (further broken down by `--group-by`):

```
[year] [campus] [medical_center] category, category_label,
employee_count, total_gross_pay, mean_gross_pay, median_gross_pay
```

### Top-earners CSV (`--top-earners-output`, optional)

The top `--top-n` gross-pay earners per category (also broken down by
`--group-by`), ranked:

```
category, category_label, rank, year, campus, medical_center, title,
gross_pay, regular_pay, overtime_pay, other_pay, first_name, last_name
```

### Console summary (default; suppress with `--quiet`)

A formatted table printed to stdout, one per year processed, showing
employee count / total pay / median pay per category — a quick sanity
check without opening any CSV.

## Classification logic

All in `classifier.py`. Two independent things are computed per employee:

### 1. Category (`classify_title()`)

Every employee falls into exactly one of 8 categories, organized around
whether a title carries UC Academic Senate membership (Regents Standing
Order 105.1), tested in this order (earlier patterns win):

1. `STUDENT_TRAINEE` — TAs, GSRs, postdocs, residents, interns, student workers
2. `ATHLETICS` — coaches and athletics staff (split out so bonus-heavy
   coaching pay doesn't distort management totals)
3. `SENIOR_MANAGEMENT` — Chancellor/Provost/VP/Dean/Executive Director tier
4. `SENATE_FACULTY` — ladder professors, instructors, and other
   Senate-membership titles
5. `NON_SENATE_ACADEMIC` — lecturers, adjuncts, HS clinical professors,
   project scientists, librarians, academic coordinators, etc.
6. `SENIOR_PROF_STAFF` / `PROF_STAFF` / `SUPPORT_STAFF` — everyone else
   (non-academic staff), split purely by gross pay against
   `--senior-threshold` / `--support-threshold`

Titles are matched with regex against a normalized (uppercased,
whitespace-collapsed, abbreviation-expanded) form of the raw title string —
see `normalise()`. The category, its human-readable label, and the pay
thresholds are all defined at the top of `classifier.py`.

### 2. Medical center flag (`is_medical_center()`)

Orthogonal to category — an employee can be `SENATE_FACULTY` *and*
`medical_center=True` (e.g. a med-school ladder professor on the HCOMP pay
plan). Two-tier rule, built from actually surveying UC-wages title text
rather than guessed:

- **Always** (any campus): titles containing `MED CTR` (hospital
  facilities/exec titles), the HS Clinical Professor/Instructor series, or
  bare `-HCOMP` ladder titles.
- **Only** at the five campuses that operate a UC-owned academic medical
  center (`UCSF`, `UCLA`, `UCD`, `UCI`, `UCSD`): generic clinical-workforce
  titles — physicians, nurses, dentists, pharmacists, therapists, lab/rad
  techs, etc. These titles are identical at every campus (Student Health
  Services exists everywhere), so outside the 5 hospital campuses they
  can't be reliably told apart from student health staff by title text
  alone.
- **Always excluded**: any title containing `SHS` (Student Health
  Services), even at UCSF — it's a distinct, campus-wide operation from the
  hospital system.

This is a title-text heuristic against a data source with no
department/org-unit field, not a perfect org-chart lookup — it won't catch
things like a bare `DEAN` title that happens to be the Dean of Medicine
(indistinguishable from any other school's dean by title text alone).

## Data notes

- UC-wages has no `benefits` field (unlike the old California State
  Controller GCC data this tool previously used), so `gross_pay` is the
  only pay figure available — `regular_pay + overtime_pay + other_pay`.
- Employee names are pre-redacted by the upstream source for some records
  (`firstname`/`lastname` show as `*****`) — this is not something this
  tool does.
- The full dataset is ~1.3GB across ~170 files. `uc_wages_data/` is
  gitignored; nothing here commits downloaded data to the repo.