feat(pipeline): finalize audited multi-platform dataset with clean provenance and entity resolution
This commit is contained in:
Srilekha committed
1 parent
ef0c0ca258
commit
4aa24d68c7
14 files changed
+2299
-169
No files matched your search
@@ -1,237 +1,283 @@
|
||||
# HIFFI Hip-Hop 500 — Artist Intelligence Engine
|
||||
# HIFFI Artist Intelligence Pipeline
|
||||
|
||||
An end-to-end data pipeline and interactive analytics dashboard designed to collect, validate, analyze, and rank 500 hip-hop artists based on their digital performance.
|
||||
An automated data pipeline for analyzing and ranking 500 hip-hop artists using YouTube performance, Instagram audience reach, and cross-platform artist identity resolution.
|
||||
|
||||
The system combines artist identity verification, performance metrics, weighted ranking formulas, and breakout talent analysis to support reliable, data-driven artist discovery.
|
||||
The project maintains an existing YouTube-based production ranking alongside an experimental multi-platform ranking, with explicit handling of missing data and platform access limitations.
|
||||
|
||||
## 📌 Project Overview
|
||||
## Table of Contents
|
||||
|
||||
Music industry datasets can contain incorrect artist links, unofficial fan accounts, missing metrics, and temporary viral spikes that distort rankings.
|
||||
- [Overview](#overview)
|
||||
- [Architecture](#architecture)
|
||||
- [Project Structure](#project-structure)
|
||||
- [Data Rules and Governance](#data-rules-and-governance)
|
||||
- [Setup](#setup)
|
||||
- [How to Run](#how-to-run)
|
||||
- [Output Dataset](#output-dataset)
|
||||
- [Limitations](#limitations)
|
||||
|
||||
The HIFFI Hip-Hop 500 Intelligence Engine addresses these challenges through four key capabilities:
|
||||
## Overview
|
||||
|
||||
1. **Artist Identity Verification** — Cross-references artist information using sources such as MusicBrainz and Wikidata to help identify incorrect account matches.
|
||||
2. **Channel Health Classification** — Categorizes artist channels as `ACTIVE_HEALTHY`, `CATALOG_EVERGREEN`, or `DORMANT_CATALOG` based on defined activity and performance criteria.
|
||||
3. **Multi-Factor Ranking** — Calculates a composite HIFFI score using four weighted factors: recent momentum, view velocity, audience reach, and consistency.
|
||||
4. **Breakout Artist Discovery** — Uses a Breakout Velocity Index (BVI) to highlight emerging artists with strong growth potential.
|
||||
5. **Interactive Analytics Dashboard** — Provides rankings, performance visualizations, artist-level insights, filtering, and CSV/Excel exports through Streamlit.
|
||||
The pipeline is designed to:
|
||||
|
||||
## ✨ Key Features
|
||||
1. **Collect YouTube metrics** — Process video views, recent view velocity, and available channel statistics.
|
||||
2. **Resolve artist identities** — Use Wikidata and identifier matching to find candidate Spotify IDs and Instagram profiles.
|
||||
3. **Collect Instagram metrics** — Extract publicly accessible follower information where available.
|
||||
4. **Validate data quality** — Detect missing values, duplicate records, suspicious handles, and potential identity mismatches.
|
||||
5. **Calculate artist rankings** — Maintain two separate ranking outputs:
|
||||
- **Production Rank:** The existing baseline based on validated YouTube performance data.
|
||||
- **HIFFI-XP Rank:** An experimental ranking combining normalized YouTube performance and Instagram audience reach.
|
||||
|
||||
- **Data Cleaning and Validation:** Identifies duplicate records, missing values, inconsistent data types, and potential outliers.
|
||||
- **Artist Identity Matching:** Uses available artist metadata and external knowledge sources to support identity verification.
|
||||
- **YouTube Metrics Collection:** Integrates YouTube Data API v3 to retrieve supported channel statistics and video information.
|
||||
- **Historical Performance Tracking:** Uses SQLite to store metric snapshots and support growth and velocity calculations.
|
||||
- **Weighted Ranking Model:** Combines four performance indicators into a composite score.
|
||||
- **Channel Health Assessment:** Distinguishes active channels, evergreen catalogs, and potentially dormant channels.
|
||||
- **Breakout Velocity Analysis:** Helps identify artists gaining attention relative to their existing audience size.
|
||||
- **Interactive Dashboard:** Displays key performance indicators, leaderboards, charts, filters, and artist details.
|
||||
- **Exportable Reports:** Generates enriched artist datasets and identity verification audit reports.
|
||||
## Architecture
|
||||
|
||||
## 📊 HIFFI Ranking Methodology
|
||||
|
||||
The HIFFI composite score uses four weighted performance factors:
|
||||
|
||||
| Ranking Factor | Weight | Description |
|
||||
|---|---:|---|
|
||||
| Recent Momentum | 35% | Measures recent audience or performance growth. |
|
||||
| View Velocity | 30% | Measures how quickly an artist's content gains views. |
|
||||
| Audience Reach | 20% | Represents the size of the artist's audience. |
|
||||
| Consistency | 15% | Measures performance stability over time. |
|
||||
| **Total** | **100%** | |
|
||||
|
||||
The factors are normalized to a common scale before being combined into the raw score.
|
||||
|
||||
### Confidence-Adjusted Score
|
||||
|
||||
The system can adjust the raw score according to the confidence of the artist identity match:
|
||||
|
||||
`Final Score = Raw Score × (0.80 + 0.20 × Confidence Score)`
|
||||
|
||||
The confidence score ranges from `0.00` to `1.00`.
|
||||
|
||||
This adjustment reduces the influence of uncertain identity matches. Additional eligibility rules may be applied to prevent unverified accounts from entering the official leaderboard.
|
||||
|
||||
### Breakout Velocity Index (BVI)
|
||||
|
||||
The BVI provides a separate perspective on artist performance, focusing on growth and view velocity relative to the artist's existing audience size.
|
||||
|
||||
It complements the overall HIFFI ranking by helping surface emerging artists who may not yet have a large established audience.
|
||||
|
||||
*Note: The exact BVI calculation depends on the implementation in the scoring module.*
|
||||
|
||||
## 🏗️ System Architecture
|
||||
|
||||
The pipeline follows this workflow:
|
||||
|
||||
```text
|
||||
Artist Source Data / Soundcharts Export
|
||||
|
|
||||
v
|
||||
Data Cleaning & Validation
|
||||
|
|
||||
v
|
||||
Artist Identity Verification
|
||||
(MusicBrainz / Wikidata)
|
||||
|
|
||||
v
|
||||
YouTube Metrics Collection
|
||||
|
|
||||
v
|
||||
Historical Snapshot Storage
|
||||
(SQLite)
|
||||
|
|
||||
v
|
||||
Scoring & Ranking Engine
|
||||
| |
|
||||
v v
|
||||
HIFFI Composite Breakout Velocity
|
||||
Score Index
|
||||
| |
|
||||
+--------+--------+
|
||||
|
|
||||
v
|
||||
Enriched Artist Dataset
|
||||
|
|
||||
v
|
||||
Streamlit Analytics Dashboard
|
||||
|
|
||||
v
|
||||
CSV / Excel Report Exports
|
||||
```mermaid
|
||||
flowchart TD
|
||||
A[Seed Artist Dataset - 500 Artists] --> B[Artist Identity Resolution]
|
||||
B --> C[Candidate Platform Profiles]
|
||||
A --> D[YouTube Metrics]
|
||||
C --> E[Instagram Metrics]
|
||||
C --> F[Spotify API Diagnostics]
|
||||
D --> G[Data Quality Validation]
|
||||
E --> G
|
||||
F --> G
|
||||
G --> H[Production YouTube Ranking]
|
||||
G --> I[Experimental HIFFI-XP Ranking]
|
||||
H --> J[Rankings and Reports]
|
||||
I --> J
|
||||
J --> K[CSV Output and Checkpoints]
|
||||
```
|
||||
|
||||
## 📁 Repository Structure
|
||||
## Project Structure
|
||||
|
||||
```text
|
||||
hiffi-hip-hop-500/
|
||||
├── collectors/ # Data collection modules
|
||||
├── verifiers/ # Artist identity verification
|
||||
├── core/ # Data processing and scoring logic
|
||||
├── data/ # Input datasets and reference data
|
||||
├── app.py # Streamlit analytics dashboard
|
||||
├── main.py # Pipeline orchestration
|
||||
├── enriched_artists.csv # Enriched artist metrics and rankings
|
||||
├── identity_verification_audit.xlsx # Identity audit and review flags
|
||||
├── requirements.txt # Python dependencies
|
||||
└── README.md # Project documentation
|
||||
hiffi-artist-intelligence/
|
||||
├── collectors/
|
||||
│ └── youtube_collector.py
|
||||
├── config/
|
||||
│ └── Pipeline and API configuration
|
||||
├── core/
|
||||
│ ├── confidence_scorer.py
|
||||
│ ├── deduplication.py
|
||||
│ ├── health_detector.py
|
||||
│ └── ranking_engine.py
|
||||
├── data/
|
||||
│ ├── checkpoints/
|
||||
│ ├── generate_seed_500.py
|
||||
│ ├── handle_overrides.json
|
||||
│ ├── hiffi_multiplatform_500.csv
|
||||
│ ├── roster_loader.py
|
||||
│ ├── seed_artists_50.py
|
||||
│ └── seed_artists_500.csv
|
||||
├── pipeline/
|
||||
│ ├── audit_instagram_handles.py
|
||||
│ ├── batch_500_runner.py
|
||||
│ ├── generate_report.py
|
||||
│ ├── multiplatform_scoring.py
|
||||
│ └── patch_anomalies.py
|
||||
├── storage/
|
||||
├── verifiers/
|
||||
├── app.py
|
||||
├── main.py
|
||||
├── enriched_artists.csv
|
||||
├── requirements.txt
|
||||
├── test_multiplatform_ranking.py
|
||||
├── .env
|
||||
└── .gitignore
|
||||
```
|
||||
|
||||
*The structure above describes the expected project layout. File names and generated artifacts should match the actual repository.*
|
||||
*Directory descriptions are illustrative. Keep only files and directories that exist in the actual repository.*
|
||||
|
||||
## 📦 Key Deliverables
|
||||
## Data Rules and Governance
|
||||
|
||||
| Deliverable | Purpose |
|
||||
|---|---|
|
||||
| `enriched_artists.csv` | Stores enriched artist records, performance metrics, scores, rankings, and health classifications. |
|
||||
| `identity_verification_audit.xlsx` | Records identity matching results, confidence scores, verification flags, and review requirements. |
|
||||
| `app.py` | Provides the interactive Streamlit dashboard for exploring artist performance. |
|
||||
| SQLite database | Stores historical metric snapshots used for time-based performance analysis. |
|
||||
### 1. No Fabricated Metrics
|
||||
|
||||
## 🛠️ Technology Stack
|
||||
- Missing metrics remain `NaN` or `NULL`, according to the storage format.
|
||||
- Estimated values must not be presented as live platform measurements.
|
||||
- Missing data must be handled consistently during normalization and scoring.
|
||||
|
||||
- **Language:** Python
|
||||
- **Data Processing:** pandas, NumPy
|
||||
- **Data Sources:** Soundcharts exports and YouTube Data API v3
|
||||
- **Identity Verification:** MusicBrainz and Wikidata, where available
|
||||
- **Database:** SQLite
|
||||
- **Dashboard:** Streamlit
|
||||
- **Visualizations:** Plotly
|
||||
- **Exports:** CSV and Excel
|
||||
### 2. Spotify API Status
|
||||
|
||||
## 🚀 Getting Started
|
||||
Spotify API access currently returns HTTP 403 for the configured request.
|
||||
|
||||
- Spotify IDs and Spotify performance metrics are treated separately.
|
||||
- Unavailable Spotify metrics remain missing.
|
||||
- The pipeline records the relevant error status for traceability.
|
||||
- Spotify metrics must not be described as collected until a working, authorized data source supplies them.
|
||||
|
||||
Example status:
|
||||
|
||||
```text
|
||||
spotify_status = BLOCKED_HTTP_403
|
||||
```
|
||||
|
||||
Use `BLOCKED_HTTP_403_PREMIUM_REQUIRED` only if the actual diagnostic confirms that specific reason. An HTTP 403 alone does not establish its cause.
|
||||
|
||||
### 3. Artist Identity Validation
|
||||
|
||||
- Treat external identifiers as candidate matches until validated.
|
||||
- Use official cross-links and other reliable evidence where available.
|
||||
- Flag ambiguous accounts and suspicious follower counts for review.
|
||||
- Record manual handle corrections in the appropriate override file.
|
||||
|
||||
### 4. Checkpoint Protection
|
||||
|
||||
The batch runner is designed to save intermediate progress so processing can resume after interruption. Checkpoint recovery depends on the runner's implementation and the availability of compatible saved checkpoint files.
|
||||
|
||||
## Setup
|
||||
|
||||
### Prerequisites
|
||||
|
||||
- Python 3.10 or later
|
||||
- Git
|
||||
- A virtual environment
|
||||
- Valid credentials for any external APIs used
|
||||
|
||||
### 1. Clone the Repository
|
||||
|
||||
```bash
|
||||
git clone <repository-url>
|
||||
cd hiffi-hip-hop-500
|
||||
cd hiffi-artist-intelligence
|
||||
```
|
||||
|
||||
Replace `<repository-url>` with the actual repository URL.
|
||||
Replace `<repository-url>` with the actual repository clone URL.
|
||||
|
||||
### 2. Create a Virtual Environment
|
||||
|
||||
```bash
|
||||
**Windows PowerShell**
|
||||
|
||||
```powershell
|
||||
python -m venv .venv
|
||||
```
|
||||
|
||||
Activate it:
|
||||
|
||||
**Windows**
|
||||
```bash
|
||||
.venv\Scripts\activate
|
||||
.venv\Scripts\Activate.ps1
|
||||
```
|
||||
|
||||
**macOS / Linux**
|
||||
|
||||
```bash
|
||||
python3 -m venv .venv
|
||||
source .venv/bin/activate
|
||||
```
|
||||
|
||||
### 3. Install Dependencies
|
||||
|
||||
```bash
|
||||
python -m pip install --upgrade pip
|
||||
pip install -r requirements.txt
|
||||
playwright install chromium
|
||||
```
|
||||
|
||||
### 4. Configure API Credentials
|
||||
## How to Run
|
||||
|
||||
Set up the required API credentials, including the YouTube Data API v3 key, if the collection pipeline requires it.
|
||||
Run the commands from the repository root with the virtual environment activated.
|
||||
|
||||
Store credentials in environment variables or a local `.env` file. Do not commit API keys, passwords, or other secrets to the repository.
|
||||
|
||||
### 5. Run the Data Pipeline
|
||||
### 1. Run the 500-Artist Pipeline
|
||||
|
||||
```bash
|
||||
python main.py
|
||||
python -m pipeline.batch_500_runner
|
||||
```
|
||||
|
||||
This runs the pipeline orchestrator, subject to the project's configured data sources, credentials, and implementation.
|
||||
This is the main batch-processing entry point. Depending on the implementation, it processes the configured artist dataset, collects available signals, validates records, and calculates ranking scores.
|
||||
|
||||
### 6. Launch the Dashboard
|
||||
### 2. Audit Instagram Handles
|
||||
|
||||
```bash
|
||||
streamlit run app.py
|
||||
python -m pipeline.audit_instagram_handles
|
||||
```
|
||||
|
||||
Open the local URL displayed in the terminal to access the dashboard.
|
||||
Use the audit output to identify potential incorrect profiles, fan pages, or suspicious metric values.
|
||||
|
||||
*Run commands should be verified against the actual repository before use.*
|
||||
### 3. Apply Targeted Corrections
|
||||
|
||||
## 📈 Dashboard Capabilities
|
||||
```bash
|
||||
python -m pipeline.patch_anomalies
|
||||
```
|
||||
|
||||
The dashboard is designed to provide:
|
||||
Review proposed corrections before applying them. Preserve the original mappings and document changes.
|
||||
|
||||
- **KPI Overview:** Summary counts for monitored artists and channel health categories.
|
||||
- **Top Artist Leaderboard:** Artists ranked by their HIFFI composite scores.
|
||||
- **Breakout Leaderboard:** Artists highlighted by the BVI.
|
||||
- **Identity Audit View:** Confidence scores, verification flags, and eligibility information.
|
||||
- **Performance Charts:** Comparisons of view velocity and recent growth.
|
||||
- **Category Filters:** Explore artist groups and subgenres.
|
||||
- **Artist Details:** Review available channel information and verification results.
|
||||
- **Data Exports:** Download filtered results in CSV or Excel format.
|
||||
### 4. Generate the Executive Report
|
||||
|
||||
## 🔍 Data Quality and Reliability
|
||||
```bash
|
||||
python -m pipeline.generate_report
|
||||
```
|
||||
|
||||
The system is designed to improve the reliability of artist rankings by:
|
||||
The report is intended to summarize ranking changes, leading gainers and fallers, and dataset-level metrics.
|
||||
|
||||
- Avoiding identity matches based only on similar names.
|
||||
- Flagging uncertain or conflicting account information.
|
||||
- Distinguishing unavailable metrics from genuine zero values.
|
||||
- Considering both recent growth and established audience reach.
|
||||
- Tracking historical snapshots to support time-based calculations.
|
||||
- Maintaining audit records for review and transparency.
|
||||
### 5. Run Ranking Tests
|
||||
|
||||
Automated identity matching supports verification but does not guarantee that every account is official. Uncertain matches should be reviewed before being treated as verified.
|
||||
```bash
|
||||
pytest -q test_multiplatform_ranking.py
|
||||
```
|
||||
|
||||
## 🔐 Security and Responsible Data Handling
|
||||
Run the tests after modifying normalization, weighting, missing-value handling, or ranking logic.
|
||||
|
||||
- Keep API credentials outside source control.
|
||||
- Use environment variables or approved secret-management tools.
|
||||
- Respect API quotas, platform terms, and applicable data-use policies.
|
||||
- Validate source data before including it in published rankings.
|
||||
## Output Dataset
|
||||
|
||||
## 🎯 Project Objective
|
||||
The intended final dataset is:
|
||||
|
||||
The goal of the HIFFI Hip-Hop 500 Intelligence Engine is to create a transparent, repeatable, and data-driven approach to hip-hop artist analysis.
|
||||
`data/hiffi_multiplatform_500.csv`
|
||||
|
||||
By combining data quality checks, identity verification, historical performance tracking, weighted ranking, and interactive analytics, the platform supports more informed artist discovery and performance evaluation.
|
||||
Key columns may include the following, depending on the latest pipeline output.
|
||||
|
||||
| Column | Description |
|
||||
|---|---|
|
||||
| `name` | Artist stage name |
|
||||
| `production_hiffi_rank` | Rank from the existing YouTube-based production baseline |
|
||||
| `hiffi_xp_rank` | Rank from the experimental multi-platform model |
|
||||
| `hiffi_xp_score` | Experimental composite score |
|
||||
| `instagram_followers` | Collected Instagram follower count, when available |
|
||||
| `spotify_id` | Resolved Spotify artist identifier, when available |
|
||||
| `spotify_status` | Spotify collection or API status |
|
||||
|
||||
Check the generated CSV to confirm the exact column names and available values.
|
||||
|
||||
## Ranking Methodology
|
||||
|
||||
### Production Ranking
|
||||
|
||||
The production ranking uses the existing YouTube scoring implementation:
|
||||
|
||||
\[
|
||||
S_{\text{production}} = S_{\text{YouTube}}
|
||||
\]
|
||||
|
||||
### Experimental Ranking — HIFFI-XP
|
||||
|
||||
The experimental model combines normalized YouTube performance and Instagram audience reach.
|
||||
|
||||
\[
|
||||
S_{\text{HIFFI-XP}} =
|
||||
w_Y N(S_{\text{YouTube}})
|
||||
+ w_I N(F_{\text{Instagram}})
|
||||
\]
|
||||
|
||||
Where:
|
||||
|
||||
- \(N(S_{\text{YouTube}})\) is the normalized YouTube score.
|
||||
- \(N(F_{\text{Instagram}})\) is the normalized Instagram follower count.
|
||||
- \(w_Y\) and \(w_I\) are the configured weights, which should sum to 1.
|
||||
|
||||
The actual configured weights should be documented from the scoring code rather than assumed from this formula.
|
||||
|
||||
Missing Instagram metrics should not automatically be treated as zero. Document the chosen missing-data policy and report the number of artists eligible for each ranking.
|
||||
|
||||
Spotify remains a planned additional signal until live metrics are available through an authorized source.
|
||||
|
||||
## Limitations
|
||||
|
||||
- Spotify API access is currently blocked for the configured request.
|
||||
- Instagram metadata may be unavailable, outdated, or affected by rate limits and website changes.
|
||||
- Candidate profile links may not always identify the correct artist.
|
||||
- Platform metrics measure different types of audience activity and require appropriate normalization.
|
||||
- Experimental scores should be validated before being used as production rankings.
|
||||
- Dataset coverage and ranking results may change between collection runs.
|
||||
|
||||
## Future Improvements
|
||||
|
||||
- Integrate an authorized source for Spotify performance metrics.
|
||||
- Improve cross-platform identity matching and confidence checks.
|
||||
- Add collection timestamps and source-level provenance to all metrics.
|
||||
- Strengthen retry handling, rate-limit management, and checkpoint recovery.
|
||||
- Expand automated tests for data quality and ranking stability.
|
||||
- Generate repeatable weekly ranking and coverage reports.
|
||||
|
||||
---
|
||||
|
||||
**Project principle:** Keep observed metrics, unresolved identities, and unavailable platform data clearly separated. This ensures that rankings remain traceable, reproducible, and honest about data limitations.
|
||||
Reference in new issue
Block a user