add README documenting usage, all CLI flags, and the classifier/downloader/pipeline architecture
This commit is contained in:
Miloš Jovanović
parent
99d5108d76
commit
d81fcedfc2
1 changed files with 214 additions and 0 deletions
214
README.md
Normal file
214
README.md
Normal file
|
|
@ -0,0 +1,214 @@
|
|||
# UC Salary Pipeline
|
||||
|
||||
Downloads University of California employee wage data and classifies every
|
||||
employee into one of 8 analytical categories (Senate Faculty, Athletics,
|
||||
Senior Management, etc.), producing per-employee, aggregate, and top-earner
|
||||
CSVs.
|
||||
|
||||
## What it does
|
||||
|
||||
1. **Downloads** raw wage JSON for every requested campus/year from
|
||||
[ucinvestments/UC-wages](https://github.com/ucinvestments/UC-wages), a
|
||||
public dataset (not an API) sourced from
|
||||
[UC Annual Wage](https://ucannualwage.ucop.edu/wage/), covering 2010
|
||||
onward for all 10 campuses plus UCOP, ASUCLA, and UC SF Law.
|
||||
2. **Classifies** every employee record by job title (and, for non-academic
|
||||
staff, gross pay) into one of 8 categories, and separately flags whether
|
||||
they work at a UC medical center.
|
||||
3. **Writes CSVs**: always a per-employee CSV, plus optional aggregate and
|
||||
top-earners CSVs, plus a formatted summary table printed to the terminal.
|
||||
|
||||
## Architecture
|
||||
|
||||
Three files, each with one job:
|
||||
|
||||
```
|
||||
uc_wages_downloader.py ──▶ classifier.py ──▶ uc_salary_pipeline.py
|
||||
(data acquisition) (classification engine) (orchestrator / CLI)
|
||||
```
|
||||
|
||||
- **`classifier.py`** — the classification engine, and *only* the
|
||||
classification engine. It has no knowledge of downloading, file formats,
|
||||
or CSVs — it's a pure function library: give it a job title string (and
|
||||
optionally a gross pay figure and campus code), get back a category and/or
|
||||
a medical-center flag. This is deliberate: it's the part of the tool that
|
||||
encodes UC-specific domain knowledge (Academic Senate membership rules,
|
||||
which job codes mean what), and it's kept importable on its own —
|
||||
`python -c "import classifier"` works with zero other dependencies. See
|
||||
[Classification logic](#classification-logic) below for how it works.
|
||||
|
||||
- **`uc_wages_downloader.py`** — fetches raw per-campus/per-year JSON files
|
||||
from the UC-wages GitHub repo into a local cache directory
|
||||
(`uc_wages_data/` by default). Knows nothing about classification. Also
|
||||
runnable standalone (`python uc_wages_downloader.py`) if you just want the
|
||||
raw data.
|
||||
|
||||
- **`uc_salary_pipeline.py`** — the orchestrator and CLI entry point. Calls
|
||||
the downloader, reads the cached JSON, maps UC-wages' field names onto
|
||||
what `classifier.py` expects, calls `classifier.classify_title()` /
|
||||
`classifier.is_medical_center()` per record, and writes the output CSVs.
|
||||
This is the script you actually run.
|
||||
|
||||
## Quick start
|
||||
|
||||
```bash
|
||||
# Download every campus, every available year, and classify everything
|
||||
# (~1.3GB download, multi-million-row output — see --years/--campuses to scope it down):
|
||||
python uc_salary_pipeline.py
|
||||
|
||||
# A specific slice:
|
||||
python uc_salary_pipeline.py --years 2023 2024 --campuses UCB UCLA
|
||||
|
||||
# Re-classify already-downloaded data without hitting the network:
|
||||
python uc_salary_pipeline.py --skip-download
|
||||
|
||||
# Full example with all optional outputs:
|
||||
python uc_salary_pipeline.py --years 2024 --campuses UCB UCLA UCSF \
|
||||
--aggregate-output agg.csv --group-by year campus medical_center \
|
||||
--top-earners-output top.csv --top-n 20
|
||||
```
|
||||
|
||||
Downloaded JSON is cached in `uc_wages_data/` (gitignored) and reused on
|
||||
every subsequent run — nothing is re-downloaded unless you delete it.
|
||||
|
||||
## `uc_salary_pipeline.py` flags
|
||||
|
||||
| Flag | Default | What it does |
|
||||
|---|---|---|
|
||||
| `--years Y [Y ...]` | `2010`–current year | Years to process. |
|
||||
| `--campuses C [C ...]` | all | Campuses to process. Choices: `ASUCLA UCB UCD UCI UCLA UCM UCR UCSD UCSF UCSB UCSC UCOP UC_SF_LAW`. |
|
||||
| `--data-dir DIR` | `uc_wages_data` | Local cache directory for downloaded JSON. |
|
||||
| `--output PATH` | `uc_salary_classified.csv` | Per-employee output CSV path. |
|
||||
| `--aggregate-output PATH` | *(off)* | If given, also write a category-level rollup CSV (employee count, total/mean/median gross pay). |
|
||||
| `--group-by [DIM ...]` | `year` | Extra breakdown dimensions for `--aggregate-output` and `--top-earners-output`, alongside category. Choices: `year`, `campus`, `medical_center`. Pass none to collapse to one row per category; pass all three for the finest breakdown. |
|
||||
| `--top-earners-output PATH` | *(off)* | If given, also write the top `--top-n` gross-pay earners per category (respects `--group-by`). |
|
||||
| `--top-n N` | `10` | How many top earners per group, for `--top-earners-output`. |
|
||||
| `--skip-download` | off | Classify whatever's already cached in `--data-dir`; never hits the network. |
|
||||
| `--quiet` | off | Suppress the formatted per-year summary table normally printed to stdout. |
|
||||
| `--senior-threshold N` | `100000` | Gross pay at/above which non-academic staff are `SENIOR_PROF_STAFF` instead of `PROF_STAFF`. |
|
||||
| `--support-threshold N` | `40000` | Gross pay below which non-academic staff are `SUPPORT_STAFF` instead of `PROF_STAFF`. |
|
||||
| `-h`, `--help` | — | Full usage (argparse-generated). |
|
||||
|
||||
Category labels in every output (e.g. `"Senior Prof/Staff (>=$100k)"`)
|
||||
automatically reflect whatever `--senior-threshold`/`--support-threshold`
|
||||
you actually pass — they're not hardcoded strings.
|
||||
|
||||
## `uc_wages_downloader.py` flags
|
||||
|
||||
Only needed directly if you want to pre-fetch data without classifying it
|
||||
yet (`uc_salary_pipeline.py` calls this automatically unless you pass
|
||||
`--skip-download`).
|
||||
|
||||
| Flag | Default | What it does |
|
||||
|---|---|---|
|
||||
| `--years Y [Y ...]` | `2010`–current year | Years to download. |
|
||||
| `--campuses C [C ...]` | all | Campuses to download (same choices as above). |
|
||||
| `--out-dir DIR` | `uc_wages_data` | Where to write the JSON files. |
|
||||
| `--dry-run` | off | Print the URLs that would be fetched; download nothing. |
|
||||
| `--timeout N` | `60` | HTTP timeout in seconds per request. |
|
||||
| `--retries N` | `3` | Retry attempts per file on transient failure. |
|
||||
|
||||
Downloads go straight to `raw.githubusercontent.com` (no GitHub API calls,
|
||||
so no rate-limit risk). A campus/year combination that doesn't exist
|
||||
upstream (e.g. ASUCLA before 2018) returns a 404 and is logged and skipped,
|
||||
not treated as an error. Already-downloaded files are never re-fetched.
|
||||
|
||||
## Output files
|
||||
|
||||
### Per-employee CSV (`--output`, always written)
|
||||
|
||||
One row per employee per year:
|
||||
|
||||
```
|
||||
year, campus, medical_center, title, category, category_label,
|
||||
gross_pay, regular_pay, overtime_pay, other_pay, first_name, last_name
|
||||
```
|
||||
|
||||
### Aggregate CSV (`--aggregate-output`, optional)
|
||||
|
||||
One row per category (further broken down by `--group-by`):
|
||||
|
||||
```
|
||||
[year] [campus] [medical_center] category, category_label,
|
||||
employee_count, total_gross_pay, mean_gross_pay, median_gross_pay
|
||||
```
|
||||
|
||||
### Top-earners CSV (`--top-earners-output`, optional)
|
||||
|
||||
The top `--top-n` gross-pay earners per category (also broken down by
|
||||
`--group-by`), ranked:
|
||||
|
||||
```
|
||||
category, category_label, rank, year, campus, medical_center, title,
|
||||
gross_pay, regular_pay, overtime_pay, other_pay, first_name, last_name
|
||||
```
|
||||
|
||||
### Console summary (default; suppress with `--quiet`)
|
||||
|
||||
A formatted table printed to stdout, one per year processed, showing
|
||||
employee count / total pay / median pay per category — a quick sanity
|
||||
check without opening any CSV.
|
||||
|
||||
## Classification logic
|
||||
|
||||
All in `classifier.py`. Two independent things are computed per employee:
|
||||
|
||||
### 1. Category (`classify_title()`)
|
||||
|
||||
Every employee falls into exactly one of 8 categories, organized around
|
||||
whether a title carries UC Academic Senate membership (Regents Standing
|
||||
Order 105.1), tested in this order (earlier patterns win):
|
||||
|
||||
1. `STUDENT_TRAINEE` — TAs, GSRs, postdocs, residents, interns, student workers
|
||||
2. `ATHLETICS` — coaches and athletics staff (split out so bonus-heavy
|
||||
coaching pay doesn't distort management totals)
|
||||
3. `SENIOR_MANAGEMENT` — Chancellor/Provost/VP/Dean/Executive Director tier
|
||||
4. `SENATE_FACULTY` — ladder professors, instructors, and other
|
||||
Senate-membership titles
|
||||
5. `NON_SENATE_ACADEMIC` — lecturers, adjuncts, HS clinical professors,
|
||||
project scientists, librarians, academic coordinators, etc.
|
||||
6. `SENIOR_PROF_STAFF` / `PROF_STAFF` / `SUPPORT_STAFF` — everyone else
|
||||
(non-academic staff), split purely by gross pay against
|
||||
`--senior-threshold` / `--support-threshold`
|
||||
|
||||
Titles are matched with regex against a normalized (uppercased,
|
||||
whitespace-collapsed, abbreviation-expanded) form of the raw title string —
|
||||
see `normalise()`. The category, its human-readable label, and the pay
|
||||
thresholds are all defined at the top of `classifier.py`.
|
||||
|
||||
### 2. Medical center flag (`is_medical_center()`)
|
||||
|
||||
Orthogonal to category — an employee can be `SENATE_FACULTY` *and*
|
||||
`medical_center=True` (e.g. a med-school ladder professor on the HCOMP pay
|
||||
plan). Two-tier rule, built from actually surveying UC-wages title text
|
||||
rather than guessed:
|
||||
|
||||
- **Always** (any campus): titles containing `MED CTR` (hospital
|
||||
facilities/exec titles), the HS Clinical Professor/Instructor series, or
|
||||
bare `-HCOMP` ladder titles.
|
||||
- **Only** at the five campuses that operate a UC-owned academic medical
|
||||
center (`UCSF`, `UCLA`, `UCD`, `UCI`, `UCSD`): generic clinical-workforce
|
||||
titles — physicians, nurses, dentists, pharmacists, therapists, lab/rad
|
||||
techs, etc. These titles are identical at every campus (Student Health
|
||||
Services exists everywhere), so outside the 5 hospital campuses they
|
||||
can't be reliably told apart from student health staff by title text
|
||||
alone.
|
||||
- **Always excluded**: any title containing `SHS` (Student Health
|
||||
Services), even at UCSF — it's a distinct, campus-wide operation from the
|
||||
hospital system.
|
||||
|
||||
This is a title-text heuristic against a data source with no
|
||||
department/org-unit field, not a perfect org-chart lookup — it won't catch
|
||||
things like a bare `DEAN` title that happens to be the Dean of Medicine
|
||||
(indistinguishable from any other school's dean by title text alone).
|
||||
|
||||
## Data notes
|
||||
|
||||
- UC-wages has no `benefits` field (unlike the old California State
|
||||
Controller GCC data this tool previously used), so `gross_pay` is the
|
||||
only pay figure available — `regular_pay + overtime_pay + other_pay`.
|
||||
- Employee names are pre-redacted by the upstream source for some records
|
||||
(`firstname`/`lastname` show as `*****`) — this is not something this
|
||||
tool does.
|
||||
- The full dataset is ~1.3GB across ~170 files. `uc_wages_data/` is
|
||||
gitignored; nothing here commits downloaded data to the repo.
|
||||
Loading…
Add table
Add a link
Reference in a new issue