284 lines
9.0 KiB
Markdown
284 lines
9.0 KiB
Markdown
# HIFFI Artist Intelligence Pipeline
|
|
|
|
An automated data pipeline for analyzing and ranking 500 hip-hop artists using YouTube performance, Instagram audience reach, and cross-platform artist identity resolution.
|
|
|
|
The project maintains an existing YouTube-based production ranking alongside an experimental multi-platform ranking, with explicit handling of missing data and platform access limitations.
|
|
|
|
## Table of Contents
|
|
|
|
- [Overview](#overview)
|
|
- [Architecture](#architecture)
|
|
- [Project Structure](#project-structure)
|
|
- [Data Rules and Governance](#data-rules-and-governance)
|
|
- [Setup](#setup)
|
|
- [How to Run](#how-to-run)
|
|
- [Output Dataset](#output-dataset)
|
|
- [Limitations](#limitations)
|
|
|
|
## Overview
|
|
|
|
The pipeline is designed to:
|
|
|
|
1. **Collect YouTube metrics** — Process video views, recent view velocity, and available channel statistics.
|
|
2. **Resolve artist identities** — Use Wikidata and identifier matching to find candidate Spotify IDs and Instagram profiles.
|
|
3. **Collect Instagram metrics** — Extract publicly accessible follower information where available.
|
|
4. **Validate data quality** — Detect missing values, duplicate records, suspicious handles, and potential identity mismatches.
|
|
5. **Calculate artist rankings** — Maintain two separate ranking outputs:
|
|
- **Production Rank:** The existing baseline based on validated YouTube performance data.
|
|
- **HIFFI-XP Rank:** An experimental ranking combining normalized YouTube performance and Instagram audience reach.
|
|
|
|
## Architecture
|
|
|
|
```mermaid
|
|
flowchart TD
|
|
A[Seed Artist Dataset - 500 Artists] --> B[Artist Identity Resolution]
|
|
B --> C[Candidate Platform Profiles]
|
|
A --> D[YouTube Metrics]
|
|
C --> E[Instagram Metrics]
|
|
C --> F[Spotify API Diagnostics]
|
|
D --> G[Data Quality Validation]
|
|
E --> G
|
|
F --> G
|
|
G --> H[Production YouTube Ranking]
|
|
G --> I[Experimental HIFFI-XP Ranking]
|
|
H --> J[Rankings and Reports]
|
|
I --> J
|
|
J --> K[CSV Output and Checkpoints]
|
|
```
|
|
|
|
## Project Structure
|
|
|
|
```text
|
|
hiffi-artist-intelligence/
|
|
├── collectors/
|
|
│ └── youtube_collector.py
|
|
├── config/
|
|
│ └── Pipeline and API configuration
|
|
├── core/
|
|
│ ├── confidence_scorer.py
|
|
│ ├── deduplication.py
|
|
│ ├── health_detector.py
|
|
│ └── ranking_engine.py
|
|
├── data/
|
|
│ ├── checkpoints/
|
|
│ ├── generate_seed_500.py
|
|
│ ├── handle_overrides.json
|
|
│ ├── hiffi_multiplatform_500.csv
|
|
│ ├── roster_loader.py
|
|
│ ├── seed_artists_50.py
|
|
│ └── seed_artists_500.csv
|
|
├── pipeline/
|
|
│ ├── audit_instagram_handles.py
|
|
│ ├── batch_500_runner.py
|
|
│ ├── generate_report.py
|
|
│ ├── multiplatform_scoring.py
|
|
│ └── patch_anomalies.py
|
|
├── storage/
|
|
├── verifiers/
|
|
├── app.py
|
|
├── main.py
|
|
├── enriched_artists.csv
|
|
├── requirements.txt
|
|
├── test_multiplatform_ranking.py
|
|
├── .env
|
|
└── .gitignore
|
|
```
|
|
|
|
*Directory descriptions are illustrative. Keep only files and directories that exist in the actual repository.*
|
|
|
|
## Data Rules and Governance
|
|
|
|
### 1. No Fabricated Metrics
|
|
|
|
- Missing metrics remain `NaN` or `NULL`, according to the storage format.
|
|
- Estimated values must not be presented as live platform measurements.
|
|
- Missing data must be handled consistently during normalization and scoring.
|
|
|
|
### 2. Spotify API Status
|
|
|
|
Spotify API access currently returns HTTP 403 for the configured request.
|
|
|
|
- Spotify IDs and Spotify performance metrics are treated separately.
|
|
- Unavailable Spotify metrics remain missing.
|
|
- The pipeline records the relevant error status for traceability.
|
|
- Spotify metrics must not be described as collected until a working, authorized data source supplies them.
|
|
|
|
Example status:
|
|
|
|
```text
|
|
spotify_status = BLOCKED_HTTP_403
|
|
```
|
|
|
|
Use `BLOCKED_HTTP_403_PREMIUM_REQUIRED` only if the actual diagnostic confirms that specific reason. An HTTP 403 alone does not establish its cause.
|
|
|
|
### 3. Artist Identity Validation
|
|
|
|
- Treat external identifiers as candidate matches until validated.
|
|
- Use official cross-links and other reliable evidence where available.
|
|
- Flag ambiguous accounts and suspicious follower counts for review.
|
|
- Record manual handle corrections in the appropriate override file.
|
|
|
|
### 4. Checkpoint Protection
|
|
|
|
The batch runner is designed to save intermediate progress so processing can resume after interruption. Checkpoint recovery depends on the runner's implementation and the availability of compatible saved checkpoint files.
|
|
|
|
## Setup
|
|
|
|
### Prerequisites
|
|
|
|
- Python 3.10 or later
|
|
- Git
|
|
- A virtual environment
|
|
- Valid credentials for any external APIs used
|
|
|
|
### 1. Clone the Repository
|
|
|
|
```bash
|
|
git clone <repository-url>
|
|
cd hiffi-artist-intelligence
|
|
```
|
|
|
|
Replace `<repository-url>` with the actual repository clone URL.
|
|
|
|
### 2. Create a Virtual Environment
|
|
|
|
**Windows PowerShell**
|
|
|
|
```powershell
|
|
python -m venv .venv
|
|
.venv\Scripts\Activate.ps1
|
|
```
|
|
|
|
**macOS / Linux**
|
|
|
|
```bash
|
|
python3 -m venv .venv
|
|
source .venv/bin/activate
|
|
```
|
|
|
|
### 3. Install Dependencies
|
|
|
|
```bash
|
|
python -m pip install --upgrade pip
|
|
pip install -r requirements.txt
|
|
playwright install chromium
|
|
```
|
|
|
|
## How to Run
|
|
|
|
Run the commands from the repository root with the virtual environment activated.
|
|
|
|
### 1. Run the 500-Artist Pipeline
|
|
|
|
```bash
|
|
python -m pipeline.batch_500_runner
|
|
```
|
|
|
|
This is the main batch-processing entry point. Depending on the implementation, it processes the configured artist dataset, collects available signals, validates records, and calculates ranking scores.
|
|
|
|
### 2. Audit Instagram Handles
|
|
|
|
```bash
|
|
python -m pipeline.audit_instagram_handles
|
|
```
|
|
|
|
Use the audit output to identify potential incorrect profiles, fan pages, or suspicious metric values.
|
|
|
|
### 3. Apply Targeted Corrections
|
|
|
|
```bash
|
|
python -m pipeline.patch_anomalies
|
|
```
|
|
|
|
Review proposed corrections before applying them. Preserve the original mappings and document changes.
|
|
|
|
### 4. Generate the Executive Report
|
|
|
|
```bash
|
|
python -m pipeline.generate_report
|
|
```
|
|
|
|
The report is intended to summarize ranking changes, leading gainers and fallers, and dataset-level metrics.
|
|
|
|
### 5. Run Ranking Tests
|
|
|
|
```bash
|
|
pytest -q test_multiplatform_ranking.py
|
|
```
|
|
|
|
Run the tests after modifying normalization, weighting, missing-value handling, or ranking logic.
|
|
|
|
## Output Dataset
|
|
|
|
The intended final dataset is:
|
|
|
|
`data/hiffi_multiplatform_500.csv`
|
|
|
|
Key columns may include the following, depending on the latest pipeline output.
|
|
|
|
| Column | Description |
|
|
|---|---|
|
|
| `name` | Artist stage name |
|
|
| `production_hiffi_rank` | Rank from the existing YouTube-based production baseline |
|
|
| `hiffi_xp_rank` | Rank from the experimental multi-platform model |
|
|
| `hiffi_xp_score` | Experimental composite score |
|
|
| `instagram_followers` | Collected Instagram follower count, when available |
|
|
| `spotify_id` | Resolved Spotify artist identifier, when available |
|
|
| `spotify_status` | Spotify collection or API status |
|
|
|
|
Check the generated CSV to confirm the exact column names and available values.
|
|
|
|
## Ranking Methodology
|
|
|
|
### Production Ranking
|
|
|
|
The production ranking uses the existing YouTube scoring implementation:
|
|
|
|
\[
|
|
S_{\text{production}} = S_{\text{YouTube}}
|
|
\]
|
|
|
|
### Experimental Ranking — HIFFI-XP
|
|
|
|
The experimental model combines normalized YouTube performance and Instagram audience reach.
|
|
|
|
\[
|
|
S_{\text{HIFFI-XP}} =
|
|
w_Y N(S_{\text{YouTube}})
|
|
+ w_I N(F_{\text{Instagram}})
|
|
\]
|
|
|
|
Where:
|
|
|
|
- \(N(S_{\text{YouTube}})\) is the normalized YouTube score.
|
|
- \(N(F_{\text{Instagram}})\) is the normalized Instagram follower count.
|
|
- \(w_Y\) and \(w_I\) are the configured weights, which should sum to 1.
|
|
|
|
The actual configured weights should be documented from the scoring code rather than assumed from this formula.
|
|
|
|
Missing Instagram metrics should not automatically be treated as zero. Document the chosen missing-data policy and report the number of artists eligible for each ranking.
|
|
|
|
Spotify remains a planned additional signal until live metrics are available through an authorized source.
|
|
|
|
## Limitations
|
|
|
|
- Spotify API access is currently blocked for the configured request.
|
|
- Instagram metadata may be unavailable, outdated, or affected by rate limits and website changes.
|
|
- Candidate profile links may not always identify the correct artist.
|
|
- Platform metrics measure different types of audience activity and require appropriate normalization.
|
|
- Experimental scores should be validated before being used as production rankings.
|
|
- Dataset coverage and ranking results may change between collection runs.
|
|
|
|
## Future Improvements
|
|
|
|
- Integrate an authorized source for Spotify performance metrics.
|
|
- Improve cross-platform identity matching and confidence checks.
|
|
- Add collection timestamps and source-level provenance to all metrics.
|
|
- Strengthen retry handling, rate-limit management, and checkpoint recovery.
|
|
- Expand automated tests for data quality and ranking stability.
|
|
- Generate repeatable weekly ranking and coverage reports.
|
|
|
|
---
|
|
|
|
**Project principle:** Keep observed metrics, unresolved identities, and unavailable platform data clearly separated. This ensures that rankings remain traceable, reproducible, and honest about data limitations.
|