hiffi-artist-intelligence/README.md

# HIFFI Artist Intelligence Pipeline

An automated data pipeline for analyzing and ranking 500 hip-hop artists using YouTube performance, Instagram audience reach, and cross-platform artist identity resolution.

The project maintains an existing YouTube-based production ranking alongside an experimental multi-platform ranking, with explicit handling of missing data and platform access limitations.

## Table of Contents

- [Overview](#overview)
- [Architecture](#architecture)
- [Project Structure](#project-structure)
- [Data Rules and Governance](#data-rules-and-governance)
- [Setup](#setup)
- [How to Run](#how-to-run)
- [Output Dataset](#output-dataset)
- [Limitations](#limitations)

## Overview

The pipeline is designed to:

1. **Collect YouTube metrics** — Process video views, recent view velocity, and available channel statistics.
2. **Resolve artist identities** — Use Wikidata and identifier matching to find candidate Spotify IDs and Instagram profiles.
3. **Collect Instagram metrics** — Extract publicly accessible follower information where available.
4. **Validate data quality** — Detect missing values, duplicate records, suspicious handles, and potential identity mismatches.
5. **Calculate artist rankings** — Maintain two separate ranking outputs:
   - **Production Rank:** The existing baseline based on validated YouTube performance data.
   - **HIFFI-XP Rank:** An experimental ranking combining normalized YouTube performance and Instagram audience reach.

## Architecture

```mermaid
flowchart TD
    A[Seed Artist Dataset - 500 Artists] --> B[Artist Identity Resolution]
    B --> C[Candidate Platform Profiles]
    A --> D[YouTube Metrics]
    C --> E[Instagram Metrics]
    C --> F[Spotify API Diagnostics]
    D --> G[Data Quality Validation]
    E --> G
    F --> G
    G --> H[Production YouTube Ranking]
    G --> I[Experimental HIFFI-XP Ranking]
    H --> J[Rankings and Reports]
    I --> J
    J --> K[CSV Output and Checkpoints]
```

## Project Structure

```text
hiffi-artist-intelligence/
├── collectors/
│   └── youtube_collector.py
├── config/
│   └── Pipeline and API configuration
├── core/
│   ├── confidence_scorer.py
│   ├── deduplication.py
│   ├── health_detector.py
│   └── ranking_engine.py
├── data/
│   ├── checkpoints/
│   ├── generate_seed_500.py
│   ├── handle_overrides.json
│   ├── hiffi_multiplatform_500.csv
│   ├── roster_loader.py
│   ├── seed_artists_50.py
│   └── seed_artists_500.csv
├── pipeline/
│   ├── audit_instagram_handles.py
│   ├── batch_500_runner.py
│   ├── generate_report.py
│   ├── multiplatform_scoring.py
│   └── patch_anomalies.py
├── storage/
├── verifiers/
├── app.py
├── main.py
├── enriched_artists.csv
├── requirements.txt
├── test_multiplatform_ranking.py
├── .env
└── .gitignore
```

*Directory descriptions are illustrative. Keep only files and directories that exist in the actual repository.*

## Data Rules and Governance

### 1. No Fabricated Metrics

- Missing metrics remain `NaN` or `NULL`, according to the storage format.
- Estimated values must not be presented as live platform measurements.
- Missing data must be handled consistently during normalization and scoring.

### 2. Spotify API Status

Spotify API access currently returns HTTP 403 for the configured request.

- Spotify IDs and Spotify performance metrics are treated separately.
- Unavailable Spotify metrics remain missing.
- The pipeline records the relevant error status for traceability.
- Spotify metrics must not be described as collected until a working, authorized data source supplies them.

Example status:

```text
spotify_status = BLOCKED_HTTP_403
```

Use `BLOCKED_HTTP_403_PREMIUM_REQUIRED` only if the actual diagnostic confirms that specific reason. An HTTP 403 alone does not establish its cause.

### 3. Artist Identity Validation

- Treat external identifiers as candidate matches until validated.
- Use official cross-links and other reliable evidence where available.
- Flag ambiguous accounts and suspicious follower counts for review.
- Record manual handle corrections in the appropriate override file.

### 4. Checkpoint Protection

The batch runner is designed to save intermediate progress so processing can resume after interruption. Checkpoint recovery depends on the runner's implementation and the availability of compatible saved checkpoint files.

## Setup

### Prerequisites

- Python 3.10 or later
- Git
- A virtual environment
- Valid credentials for any external APIs used

### 1. Clone the Repository

```bash
git clone <repository-url>
cd hiffi-artist-intelligence
```

Replace `<repository-url>` with the actual repository clone URL.

### 2. Create a Virtual Environment

**Windows PowerShell**

```powershell
python -m venv .venv
.venv\Scripts\Activate.ps1
```

**macOS / Linux**

```bash
python3 -m venv .venv
source .venv/bin/activate
```

### 3. Install Dependencies

```bash
python -m pip install --upgrade pip
pip install -r requirements.txt
playwright install chromium
```

## How to Run

Run the commands from the repository root with the virtual environment activated.

### 1. Run the 500-Artist Pipeline

```bash
python -m pipeline.batch_500_runner
```

This is the main batch-processing entry point. Depending on the implementation, it processes the configured artist dataset, collects available signals, validates records, and calculates ranking scores.

### 2. Audit Instagram Handles

```bash
python -m pipeline.audit_instagram_handles
```

Use the audit output to identify potential incorrect profiles, fan pages, or suspicious metric values.

### 3. Apply Targeted Corrections

```bash
python -m pipeline.patch_anomalies
```

Review proposed corrections before applying them. Preserve the original mappings and document changes.

### 4. Generate the Executive Report

```bash
python -m pipeline.generate_report
```

The report is intended to summarize ranking changes, leading gainers and fallers, and dataset-level metrics.

### 5. Run Ranking Tests

```bash
pytest -q test_multiplatform_ranking.py
```

Run the tests after modifying normalization, weighting, missing-value handling, or ranking logic.

## Output Dataset

The intended final dataset is:

`data/hiffi_multiplatform_500.csv`

Key columns may include the following, depending on the latest pipeline output.

| Column | Description |
|---|---|
| `name` | Artist stage name |
| `production_hiffi_rank` | Rank from the existing YouTube-based production baseline |
| `hiffi_xp_rank` | Rank from the experimental multi-platform model |
| `hiffi_xp_score` | Experimental composite score |
| `instagram_followers` | Collected Instagram follower count, when available |
| `spotify_id` | Resolved Spotify artist identifier, when available |
| `spotify_status` | Spotify collection or API status |

Check the generated CSV to confirm the exact column names and available values.

## Ranking Methodology

### Production Ranking

The production ranking uses the existing YouTube scoring implementation:

\[
S_{\text{production}} = S_{\text{YouTube}}
\]

### Experimental Ranking — HIFFI-XP

The experimental model combines normalized YouTube performance and Instagram audience reach.

\[
S_{\text{HIFFI-XP}} =
w_Y N(S_{\text{YouTube}})
+ w_I N(F_{\text{Instagram}})
\]

Where:

- \(N(S_{\text{YouTube}})\) is the normalized YouTube score.
- \(N(F_{\text{Instagram}})\) is the normalized Instagram follower count.
- \(w_Y\) and \(w_I\) are the configured weights, which should sum to 1.

The actual configured weights should be documented from the scoring code rather than assumed from this formula.

Missing Instagram metrics should not automatically be treated as zero. Document the chosen missing-data policy and report the number of artists eligible for each ranking.

Spotify remains a planned additional signal until live metrics are available through an authorized source.

## Limitations

- Spotify API access is currently blocked for the configured request.
- Instagram metadata may be unavailable, outdated, or affected by rate limits and website changes.
- Candidate profile links may not always identify the correct artist.
- Platform metrics measure different types of audience activity and require appropriate normalization.
- Experimental scores should be validated before being used as production rankings.
- Dataset coverage and ranking results may change between collection runs.

## Future Improvements

- Integrate an authorized source for Spotify performance metrics.
- Improve cross-platform identity matching and confidence checks.
- Add collection timestamps and source-level provenance to all metrics.
- Strengthen retry handling, rate-limit management, and checkpoint recovery.
- Expand automated tests for data quality and ranking stability.
- Generate repeatable weekly ranking and coverage reports.

---

**Project principle:** Keep observed metrics, unresolved identities, and unavailable platform data clearly separated. This ensures that rankings remain traceable, reproducible, and honest about data limitations.