uc_salary_aggregator/README.md

UC Salary Pipeline

Downloads University of California employee wage data and classifies every employee into one of 8 analytical categories (Senate Faculty, Athletics, Senior Management, etc.), producing per-employee, aggregate, and top-earner CSVs.

What it does

1. Downloads raw wage JSON for every requested campus/year from ucinvestments/UC-wages, a public dataset (not an API) sourced from UC Annual Wage, covering 2010 onward for all 10 campuses plus UCOP, ASUCLA, and UC SF Law.
2. Classifies every employee record by job title (and, for non-academic staff, gross pay) into one of 8 categories, and separately flags whether they work at a UC medical center.
3. Writes CSVs: always a per-employee CSV, plus optional aggregate and top-earners CSVs, plus a formatted summary table printed to the terminal.

Architecture

Three files, each with one job:

uc_wages_downloader.py  ──▶  classifier.py  ──▶  uc_salary_pipeline.py
   (data acquisition)      (classification engine)   (orchestrator / CLI)

• classifier.py — the classification engine, and only the classification engine. It has no knowledge of downloading, file formats, or CSVs — it's a pure function library: give it a job title string (and optionally a gross pay figure and campus code), get back a category and/or a medical-center flag. This is deliberate: it's the part of the tool that encodes UC-specific domain knowledge (Academic Senate membership rules, which job codes mean what), and it's kept importable on its own — python -c "import classifier" works with zero other dependencies. See Classification logic below for how it works.

• uc_wages_downloader.py — fetches raw per-campus/per-year JSON files from the UC-wages GitHub repo into a local cache directory (uc_wages_data/ by default). Knows nothing about classification. Also runnable standalone (python uc_wages_downloader.py) if you just want the raw data.

• uc_salary_pipeline.py — the orchestrator and CLI entry point. Calls the downloader, reads the cached JSON, maps UC-wages' field names onto what classifier.py expects, calls classifier.classify_title() / classifier.is_medical_center() per record, and writes the output CSVs. This is the script you actually run.

Quick start

# Download every campus, every available year, and classify everything
# (~1.3GB download, multi-million-row output — see --years/--campuses to scope it down):
python uc_salary_pipeline.py

# A specific slice:
python uc_salary_pipeline.py --years 2023 2024 --campuses UCB UCLA

# Re-classify already-downloaded data without hitting the network:
python uc_salary_pipeline.py --skip-download

# Full example with all optional outputs:
python uc_salary_pipeline.py --years 2024 --campuses UCB UCLA UCSF \
    --aggregate-output agg.csv --group-by year campus medical_center \
    --top-earners-output top.csv --top-n 20

Downloaded JSON is cached in uc_wages_data/ (gitignored) and reused on every subsequent run — nothing is re-downloaded unless you delete it.

uc_salary_pipeline.py flags

Flag	Default	What it does
--years Y [Y ...]	2010–current year	Years to process.
--campuses C [C ...]	all	Campuses to process. Choices: ASUCLA UCB UCD UCI UCLA UCM UCR UCSD UCSF UCSB UCSC UCOP UC_SF_LAW.
--data-dir DIR	uc_wages_data	Local cache directory for downloaded JSON.
--output PATH	uc_salary_classified.csv	Per-employee output CSV path.
--aggregate-output PATH	(off)	If given, also write a category-level rollup CSV (employee count, total/mean/median gross pay).
--group-by [DIM ...]	year	Extra breakdown dimensions for --aggregate-output and --top-earners-output, alongside category. Choices: year, campus, medical_center. Pass none to collapse to one row per category; pass all three for the finest breakdown.
--top-earners-output PATH	(off)	If given, also write the top --top-n gross-pay earners per category (respects --group-by).
--top-n N	10	How many top earners per group, for --top-earners-output.
--skip-download	off	Classify whatever's already cached in --data-dir; never hits the network.
--quiet	off	Suppress the formatted per-year summary table normally printed to stdout.
--senior-threshold N	100000	Gross pay at/above which non-academic staff are SENIOR_PROF_STAFF instead of PROF_STAFF.
--support-threshold N	40000	Gross pay below which non-academic staff are SUPPORT_STAFF instead of PROF_STAFF.
-h, --help	—	Full usage (argparse-generated).

Category labels in every output (e.g. "Senior Prof/Staff (>=$100k)") automatically reflect whatever --senior-threshold/--support-threshold you actually pass — they're not hardcoded strings.

uc_wages_downloader.py flags

Only needed directly if you want to pre-fetch data without classifying it yet (uc_salary_pipeline.py calls this automatically unless you pass --skip-download).

Flag	Default	What it does
--years Y [Y ...]	2010–current year	Years to download.
--campuses C [C ...]	all	Campuses to download (same choices as above).
--out-dir DIR	uc_wages_data	Where to write the JSON files.
--dry-run	off	Print the URLs that would be fetched; download nothing.
--timeout N	60	HTTP timeout in seconds per request.
--retries N	3	Retry attempts per file on transient failure.

Downloads go straight to raw.githubusercontent.com (no GitHub API calls, so no rate-limit risk). A campus/year combination that doesn't exist upstream (e.g. ASUCLA before 2018) returns a 404 and is logged and skipped, not treated as an error. Already-downloaded files are never re-fetched.

Output files

Per-employee CSV (--output, always written)

One row per employee per year:

year, campus, medical_center, title, category, category_label,
gross_pay, regular_pay, overtime_pay, other_pay, first_name, last_name

Aggregate CSV (--aggregate-output, optional)

One row per category (further broken down by --group-by):

[year] [campus] [medical_center] category, category_label,
employee_count, total_gross_pay, mean_gross_pay, median_gross_pay

Top-earners CSV (--top-earners-output, optional)

The top --top-n gross-pay earners per category (also broken down by --group-by), ranked:

category, category_label, rank, year, campus, medical_center, title,
gross_pay, regular_pay, overtime_pay, other_pay, first_name, last_name

Console summary (default; suppress with --quiet)

A formatted table printed to stdout, one per year processed, showing employee count / total pay / median pay per category — a quick sanity check without opening any CSV.

Classification logic

All in classifier.py. Two independent things are computed per employee:

1. Category (classify_title())

Every employee falls into exactly one of 8 categories, organized around whether a title carries UC Academic Senate membership (Regents Standing Order 105.1), tested in this order (earlier patterns win):

1. STUDENT_TRAINEE — TAs, GSRs, postdocs, residents, interns, student workers
2. ATHLETICS — coaches and athletics staff (split out so bonus-heavy coaching pay doesn't distort management totals)
3. SENIOR_MANAGEMENT — Chancellor/Provost/VP/Dean/Executive Director tier
4. SENATE_FACULTY — ladder professors, instructors, and other Senate-membership titles
5. NON_SENATE_ACADEMIC — lecturers, adjuncts, HS clinical professors, project scientists, librarians, academic coordinators, etc.
6. SENIOR_PROF_STAFF / PROF_STAFF / SUPPORT_STAFF — everyone else (non-academic staff), split purely by gross pay against --senior-threshold / --support-threshold

Titles are matched with regex against a normalized (uppercased, whitespace-collapsed, abbreviation-expanded) form of the raw title string — see normalise(). The category, its human-readable label, and the pay thresholds are all defined at the top of classifier.py.

2. Medical center flag (is_medical_center())

Orthogonal to category — an employee can be SENATE_FACULTY and medical_center=True (e.g. a med-school ladder professor on the HCOMP pay plan). Two-tier rule, built from actually surveying UC-wages title text rather than guessed:

• Always (any campus): titles containing MED CTR (hospital facilities/exec titles), the HS Clinical Professor/Instructor series, or bare -HCOMP ladder titles.
• Only at the five campuses that operate a UC-owned academic medical center (UCSF, UCLA, UCD, UCI, UCSD): generic clinical-workforce titles — physicians, nurses, dentists, pharmacists, therapists, lab/rad techs, etc. These titles are identical at every campus (Student Health Services exists everywhere), so outside the 5 hospital campuses they can't be reliably told apart from student health staff by title text alone.
• Always excluded: any title containing SHS (Student Health Services), even at UCSF — it's a distinct, campus-wide operation from the hospital system.

This is a title-text heuristic against a data source with no department/org-unit field, not a perfect org-chart lookup — it won't catch things like a bare DEAN title that happens to be the Dean of Medicine (indistinguishable from any other school's dean by title text alone).

Data notes

• UC-wages has no benefits field (unlike the old California State Controller GCC data this tool previously used), so gross_pay is the only pay figure available — regular_pay + overtime_pay + other_pay.
• Employee names are pre-redacted by the upstream source for some records (firstname/lastname show as *****) — this is not something this tool does.
• The full dataset is ~1.3GB across ~170 files. uc_wages_data/ is gitignored; nothing here commits downloaded data to the repo.