From a37e4558fa13943f84fea9a29d463dd63522b7bc Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Milo=C5=A1=20Jovanovi=C4=87?= Date: Wed, 5 Aug 2026 01:06:44 +0200 Subject: [PATCH] set up README for Max --- README.md | 111 ++++++++++++++---------------------------------------- 1 file changed, 28 insertions(+), 83 deletions(-) diff --git a/README.md b/README.md index 16de8ad..0c57ea5 100644 --- a/README.md +++ b/README.md @@ -1,77 +1,56 @@ # UC Salary Pipeline -Downloads University of California employee wage data and classifies every -employee into one of 8 analytical categories (Senate Faculty, Athletics, -Senior Management, etc.), producing per-employee, aggregate, and top-earner -CSVs. +This script townloads University of California employee wage data and classifies every employee into an analytical category (Senate Faculty, Athletics, Senior Management, etc.), producing per-employee, aggregate, and top-earner CSVs. -## What it does +## What the script does -1. **Downloads** raw wage JSON for every requested campus/year from +1. Downloads raw wage JSON for every requested campus/year from [ucinvestments/UC-wages](https://github.com/ucinvestments/UC-wages), a public dataset (not an API) sourced from [UC Annual Wage](https://ucannualwage.ucop.edu/wage/), covering 2010 onward for all 10 campuses plus UCOP, ASUCLA, and UC SF Law. -2. **Classifies** every employee record by job title (and, for non-academic - staff, gross pay) into one of 8 categories, and separately flags whether - they work at a UC medical center. -3. **Writes CSVs**: always a per-employee CSV, plus optional aggregate and - top-earners CSVs, plus a formatted summary table printed to the terminal. +2. Classifies every employee record by job title (and, for non-academic staff, gross pay) into one of 8 categories, and separately flags whether they work at a UC medical center. Categorization is based on job description, and probably needs some work and will never be complete for the medical centers really. +3. Outputs to CSV plus a formatted summary table printed to the terminal. ## Architecture -Three files, each with one job: +the software has three files, each has one job: -``` -uc_wages_downloader.py ──▶ classifier.py ──▶ uc_salary_pipeline.py - (data acquisition) (classification engine) (orchestrator / CLI) -``` - -- **`classifier.py`** — the classification engine, and *only* the - classification engine. It has no knowledge of downloading, file formats, - or CSVs — it's a pure function library: give it a job title string (and +- **`classifier.py`** — the classification engine. give it a job title string (and optionally a gross pay figure and campus code), get back a category and/or - a medical-center flag. This is deliberate: it's the part of the tool that - encodes UC-specific domain knowledge (Academic Senate membership rules, - which job codes mean what), and it's kept importable on its own — - `python -c "import classifier"` works with zero other dependencies. See + a medical-center flag. It's kept importable on its own — + `python -c "import classifier"` works. See [Classification logic](#classification-logic) below for how it works. - **`uc_wages_downloader.py`** — fetches raw per-campus/per-year JSON files from the UC-wages GitHub repo into a local cache directory - (`uc_wages_data/` by default). Knows nothing about classification. Also - runnable standalone (`python uc_wages_downloader.py`) if you just want the - raw data. + (`uc_wages_data/` by default). -- **`uc_salary_pipeline.py`** — the orchestrator and CLI entry point. Calls - the downloader, reads the cached JSON, maps UC-wages' field names onto - what `classifier.py` expects, calls `classifier.classify_title()` / +- **`uc_salary_pipeline.py`** — main shebang. Calls the downloader, reads the cached JSON, maps UC-wages' field names onto what `classifier.py` expects, calls `classifier.classify_title()` / `classifier.is_medical_center()` per record, and writes the output CSVs. - This is the script you actually run. ## Quick start ```bash # Download every campus, every available year, and classify everything -# (~1.3GB download, multi-million-row output — see --years/--campuses to scope it down): python uc_salary_pipeline.py -# A specific slice: +# Get specific slice per year and campus: python uc_salary_pipeline.py --years 2023 2024 --campuses UCB UCLA -# Re-classify already-downloaded data without hitting the network: +# Skip downloading the data (script also skips existing data if it's there) python uc_salary_pipeline.py --skip-download -# Full example with all optional outputs: +# Full example python uc_salary_pipeline.py --years 2024 --campuses UCB UCLA UCSF \ --aggregate-output agg.csv --group-by year campus medical_center \ --top-earners-output top.csv --top-n 20 ``` Downloaded JSON is cached in `uc_wages_data/` (gitignored) and reused on -every subsequent run — nothing is re-downloaded unless you delete it. +every subsequent run -## `uc_salary_pipeline.py` flags +## Runtime flags for `uc_salary_pipeline.py` | Flag | Default | What it does | |---|---|---| @@ -91,27 +70,7 @@ every subsequent run — nothing is re-downloaded unless you delete it. Category labels in every output (e.g. `"Senior Prof/Staff (>=$100k)"`) automatically reflect whatever `--senior-threshold`/`--support-threshold` -you actually pass — they're not hardcoded strings. - -## `uc_wages_downloader.py` flags - -Only needed directly if you want to pre-fetch data without classifying it -yet (`uc_salary_pipeline.py` calls this automatically unless you pass -`--skip-download`). - -| Flag | Default | What it does | -|---|---|---| -| `--years Y [Y ...]` | `2010`–current year | Years to download. | -| `--campuses C [C ...]` | all | Campuses to download (same choices as above). | -| `--out-dir DIR` | `uc_wages_data` | Where to write the JSON files. | -| `--dry-run` | off | Print the URLs that would be fetched; download nothing. | -| `--timeout N` | `60` | HTTP timeout in seconds per request. | -| `--retries N` | `3` | Retry attempts per file on transient failure. | - -Downloads go straight to `raw.githubusercontent.com` (no GitHub API calls, -so no rate-limit risk). A campus/year combination that doesn't exist -upstream (e.g. ASUCLA before 2018) returns a 404 and is logged and skipped, -not treated as an error. Already-downloaded files are never re-fetched. +is passed. ## Output files @@ -155,16 +114,12 @@ All in `classifier.py`. Two independent things are computed per employee: ### 1. Category (`classify_title()`) -Every employee falls into exactly one of 8 categories, organized around -whether a title carries UC Academic Senate membership (Regents Standing -Order 105.1), tested in this order (earlier patterns win): +Every employee falls into exactly one of 8 categories, like so: 1. `STUDENT_TRAINEE` — TAs, GSRs, postdocs, residents, interns, student workers -2. `ATHLETICS` — coaches and athletics staff (split out so bonus-heavy - coaching pay doesn't distort management totals) +2. `ATHLETICS` — coaches and athletics staff 3. `SENIOR_MANAGEMENT` — Chancellor/Provost/VP/Dean/Executive Director tier -4. `SENATE_FACULTY` — ladder professors, instructors, and other - Senate-membership titles +4. `SENATE_FACULTY` — ladder professors and other Senate-membership titles 5. `NON_SENATE_ACADEMIC` — lecturers, adjuncts, HS clinical professors, project scientists, librarians, academic coordinators, etc. 6. `SENIOR_PROF_STAFF` / `PROF_STAFF` / `SUPPORT_STAFF` — everyone else @@ -178,37 +133,27 @@ thresholds are all defined at the top of `classifier.py`. ### 2. Medical center flag (`is_medical_center()`) -Orthogonal to category — an employee can be `SENATE_FACULTY` *and* -`medical_center=True` (e.g. a med-school ladder professor on the HCOMP pay -plan). Two-tier rule, built from actually surveying UC-wages title text -rather than guessed: - -- **Always** (any campus): titles containing `MED CTR` (hospital +- (any campus): titles containing `MED CTR` (hospital facilities/exec titles), the HS Clinical Professor/Instructor series, or bare `-HCOMP` ladder titles. -- **Only** at the five campuses that operate a UC-owned academic medical +- Five campuses that operate a UC-owned academic medical center (`UCSF`, `UCLA`, `UCD`, `UCI`, `UCSD`): generic clinical-workforce titles — physicians, nurses, dentists, pharmacists, therapists, lab/rad techs, etc. These titles are identical at every campus (Student Health Services exists everywhere), so outside the 5 hospital campuses they can't be reliably told apart from student health staff by title text alone. -- **Always excluded**: any title containing `SHS` (Student Health - Services), even at UCSF — it's a distinct, campus-wide operation from the - hospital system. +- Excluded any title containing `SHS` (Student Health + Services), even at UCSF -This is a title-text heuristic against a data source with no -department/org-unit field, not a perfect org-chart lookup — it won't catch -things like a bare `DEAN` title that happens to be the Dean of Medicine -(indistinguishable from any other school's dean by title text alone). +This won't catch things like a bare `DEAN` title that happens to be the Dean of Medicine. ## Data notes - UC-wages has no `benefits` field (unlike the old California State Controller GCC data this tool previously used), so `gross_pay` is the - only pay figure available — `regular_pay + overtime_pay + other_pay`. + only pay figure available — `regular_pay + overtime_pay + other_pay`... might be useful in the future if we want benefits data to go back to the GCC scraper. - Employee names are pre-redacted by the upstream source for some records - (`firstname`/`lastname` show as `*****`) — this is not something this - tool does. + (`firstname`/`lastname` show as `*****`) - The full dataset is ~1.3GB across ~170 files. `uc_wages_data/` is - gitignored; nothing here commits downloaded data to the repo. + gitignored; run the script to download the data first time