hiffi-artist-intelligence/README.md

# HIFFI Hip-Hop 500 — Multi-Platform Artist Intelligence Pipeline

A Python-based pipeline for collecting, validating, scoring, and ranking hip-hop artists using YouTube, Spotify, and Instagram signals. It includes data freshness checks, identity verification, anomaly detection, publication eligibility rules, and regression auditing.

## Key Features

- Multi-platform artist metrics and scoring.
- Instagram identity matching with review-queue support.
- Missing-data handling and dynamic weight redistribution.
- Platform freshness and publication eligibility gates.
- Cross-platform anomaly detection.
- Regression auditing and rank-shift reports.
- Automated tests and GitHub Actions CI for Python 3.11 and 3.12.

## Pipeline Architecture: Telemetry Ingestion vs. Ranking Core

The project separates data collection from ranking so that platform data can be refreshed independently of score calculation.

### 1. Telemetry Ingestion Layer — `pipeline/batch_500_runner.py`

- Collects multi-platform telemetry through configured API calls and web scraping.
- Uses checkpoints in `data/checkpoints/` to support recovery after interrupted runs and reduce repeated data requests.
- Saves the refreshed master cohort dataset to `data/hiffi_multiplatform_500.csv`.

### 2. Ranking and Governance Core — `run_pipeline.py`

- Loads `data/hiffi_multiplatform_500.csv` and executes the ranking workflow independently of data collection.
- Handles missing metrics, redistributes weights across available signals, checks data freshness, and flags suspicious records.
- Applies publication eligibility rules and assigns published ranks.
- Generates an audit report at `reports/audit_report_latest.md`.

## Ranking Methodology

The project uses separate scores for artist production performance, multi-platform ranking, and breakout potential.

### HIFFI-XP Multi-Platform Score

| Signal | Base Weight |
|---|---:|
| YouTube production score | 50% |
| Spotify monthly listeners | 25% |
| Spotify daily stream velocity | 5% |
| Instagram followers | 20% |

When some signals are unavailable, the scoring logic redistributes their weights proportionally across the available signals. The effective weights are recorded in `effective_weights` for auditability.

### YouTube Production Score

The production score uses the following components:

| Component | Weight |
|---|---:|
| Recent momentum | 35% |
| View velocity | 30% |
| Audience reach | 20% |
| Consistency proxy | 15% |

Audience reach combines subscriber and lifetime-view signals. The score also uses confidence and eligibility checks where supported by the ranking implementation.

### Breakout Velocity Index (BVI)

The Breakout Velocity Index identifies artists showing growth potential based on available velocity signals. A 10,000-subscriber floor is used by the BVI logic to reduce the impact of very small accounts.

**Note:** The project's internal scoring methodology is not necessarily identical to the public HIFFI website's ranking formula.

## Identity Verification Gates

Only Instagram profiles that pass the configured verification rules qualify for multi-platform credit.

- **`AUTOMATED_VERIFIED`:** Confidence ≥ `0.55`, or confidence ≥ `0.40` with follower corroboration ≥ `10,000`.
- **`UNVERIFIED`:** Accounts that do not meet verification requirements.
- **`UNCERTAIN_NEEDS_REVIEW`:** Ambiguous account matches requiring human review.

Unverified accounts, low-follower outliers below `10,000`, and ambiguous handles are routed to `data/review_queue.csv` for review and excluded from multi-platform tier credit.

Follower count is a supporting signal, not proof of identity. The verification status and publication eligibility rules determine whether an account can contribute evidence.

## Data Freshness and Publication Eligibility

The ranking engine uses freshness windows to determine whether platform data qualifies as current evidence.

| Platform | Maximum Data Age |
|---|---:|
| YouTube | 7 days |
| Spotify | 7 days |
| Instagram | 14 days |

Missing, unparseable, or stale timestamps do not qualify for platform evidence credit. The engine records exclusion reasons in `evidence_exclusions`, such as `STALE_OR_MISSING_SPOTIFY_TIMESTAMP`.

A platform's evidence may be excluded from publication eligibility while its available metric can still contribute to a continuous score. Therefore, score calculation and publication eligibility are separate decisions.

## Missing-Data Handling

- Preserve missing values as `NaN` where supported by the data and scoring logic.
- Do not treat failed API requests as genuine zero-valued metrics.
- Redistribute weights across available signals according to the scoring implementation.
- Record effective weights so ranking results can be inspected and explained.

Some individual scoring functions may apply documented defaults. Review the relevant implementation before assuming every missing value remains `NaN` throughout all calculations.

## Anomaly Detection

Before publication, the pipeline flags suspicious cross-platform metric relationships for audit review.

| Flag | Condition |
|---|---|
| `SUSPICIOUS_IG_TO_YT_RATIO` | Instagram followers exceed YouTube lifetime views by more than 10×. |
| `EXTREME_SPOTIFY_VELOCITY_RATIO` | Daily stream growth exceeds 25% of monthly listeners. |

Flags are recorded in `anomaly_flags` so suspicious records can be investigated rather than silently ignored. A flag should trigger review; it does not, by itself, prove that the data is fraudulent.

## Regression Auditing

The regression auditor compares current ranking output against saved baselines.

- Baselines are stored in `data/snapshots/`.
- Rank movements greater than 100 positions (`|rank shift| > 100`) are identified for review.
- The latest report is written to `reports/audit_report_latest.md`.

Large ranking changes may reflect genuine changes in platform signals, data availability, scoring weights, or identity verification. They should be reviewed before publication.

## Project Structure

```text
hiffi-artist-intelligence/
├── collectors/
│   ├── instagram_verifier.py
│   ├── spotify_api_collector.py
│   └── spotify_fallback_collector.py
├── core/
│   ├── ranking_engine.py
│   └── regression_auditor.py
├── data/
│   ├── checkpoints/
│   ├── snapshots/
│   ├── review_queue.csv
│   └── hiffi_multiplatform_500.csv
├── pipeline/
│   └── batch_500_runner.py
├── reports/
│   └── audit_report_latest.md
├── tests/
├── run_pipeline.py
├── test_multiplatform_ranking.py
├── requirements.txt
├── .env.example
└── README.md
```

The tree shows the intended layout. Keep only files and folders that actually exist in the repository.

## Requirements

- Python 3.11 or 3.12
- Git
- API credentials for any integrations that require authentication

## Setup

### 1. Clone the repository

```bash
git clone <YOUR_REPOSITORY_URL>
cd hiffi-artist-intelligence
```

Replace `<YOUR_REPOSITORY_URL>` with your repository's clone URL.

### 2. Create a virtual environment

**Windows PowerShell**

```powershell
python -m venv .venv
.\.venv\Scripts\Activate.ps1
```

**macOS/Linux**

```bash
python3 -m venv .venv
source .venv/bin/activate
```

### 3. Install dependencies

```bash
python -m pip install --upgrade pip
pip install -r requirements.txt
```

### 4. Configure environment variables

Create a local `.env` file using `.env.example` as a guide. Add the credentials required by your enabled integrations, for example:

```dotenv
SPOTIFY_CLIENT_ID=your_spotify_client_id
SPOTIFY_CLIENT_SECRET=your_spotify_client_secret
```

Use the actual variable names expected by your collectors. Never commit `.env` files or live API credentials to Git.

## Running the Pipeline

### Refresh platform data

If `pipeline/batch_500_runner.py` is implemented as the ingestion entry point, run:

```bash
python -m pipeline.batch_500_runner
```

This stage should refresh the master dataset and use checkpoints where supported.

### Calculate rankings and generate the audit report

```bash
python run_pipeline.py
```

This stage consumes the available master dataset, applies scoring and governance rules, and produces ranking outputs and the audit report.

### Run tests

```bash
python -m pytest test_multiplatform_ranking.py -v
```

### Run the regression auditor

```bash
python -m core.regression_auditor
```

Use the exact commands and test paths defined by the current repository.

## Testing and Continuous Integration

The project reports 27 passing unit and integration tests, with GitHub Actions checks passing on Python 3.11 and 3.12 on Ubuntu.

The test suite is intended to cover ranking behavior, missing-data handling, freshness and eligibility gates, anomaly detection, and regression safeguards.

Test results should be confirmed against the latest commit before a release is declared complete.

## Important Output Fields

| Field | Purpose |
|---|---|
| `production_hiffi_score` | YouTube production-performance score. |
| `hiffi_xp_score` | Combined multi-platform score. |
| `hiffi_xp_rank` | Raw multi-platform ranking position. |
| `hiffi_xp_published_rank` | Published ranking position after eligibility rules. |
| `breakout_velocity_index` | Breakout potential score. |
| `breakout_rank` | Breakout ranking position. |
| `effective_weights` | Weights applied to the available signals. |
| `platform_evidence_tier` | Platform evidence tier used for eligibility. |
| `evidence_exclusions` | Reasons platform evidence was excluded. |
| `anomaly_flags` | Potentially suspicious metric relationships. |
| `rank_delta` | Change in ranking position relative to the comparison baseline. |

Actual output columns depend on the current implementation and input dataset.

## Security and Data Governance

- Keep API keys in environment variables, not source code.
- Keep `.env` and local credentials out of version control.
- Review uncertain artist identities before granting platform evidence credit.
- Keep data timestamps and exclusion reasons available for auditing.
- Investigate large rank shifts and anomaly flags before publication.
- Do not interpret a missing metric as a confirmed zero unless the relevant scoring rule explicitly defines that behavior.

## Limitations

- API access, quotas, and authentication can affect data availability.
- Platform metrics may have different refresh schedules and definitions.
- Similar artist names and unofficial social accounts can produce ambiguous identity matches.
- Anomaly rules identify unusual relationships, not necessarily fraudulent activity.
- Rankings depend on data quality, available signals, and the configured weighting methodology.
- Automated tests cannot guarantee that every live platform account or metric is correct.

## Future Improvements

- Expand identity verification using reliable official account evidence.
- Improve source-level data validation and stale-data monitoring.
- Add operational alerts for failed or incomplete ingestion runs.
- Strengthen checkpoint recovery and API retry behavior.
- Track ranking changes across releases and review significant shifts.
- Automate recurring regression reports and release checks.

## Summary

HIFFI Hip-Hop 500 separates platform data ingestion from ranking and governance. Its scoring, identity checks, freshness controls, anomaly flags, and regression reports are designed to make multi-platform rankings more consistent, explainable, and auditable.

For production use, verify that the documented entry points, thresholds, checkpoint behavior, and output paths match the implementation in the released repository.