hiffi-artist-intelligence/README.md

# HIFFI Hip-Hop 500 — Artist Intelligence Engine

An end-to-end data pipeline and interactive analytics dashboard designed to collect, validate, analyze, and rank 500 hip-hop artists based on their digital performance.

The system combines artist identity verification, performance metrics, weighted ranking formulas, and breakout talent analysis to support reliable, data-driven artist discovery.

## 📌 Project Overview

Music industry datasets can contain incorrect artist links, unofficial fan accounts, missing metrics, and temporary viral spikes that distort rankings.

The HIFFI Hip-Hop 500 Intelligence Engine addresses these challenges through four key capabilities:

1. **Artist Identity Verification** — Cross-references artist information using sources such as MusicBrainz and Wikidata to help identify incorrect account matches.
2. **Channel Health Classification** — Categorizes artist channels as `ACTIVE_HEALTHY`, `CATALOG_EVERGREEN`, or `DORMANT_CATALOG` based on defined activity and performance criteria.
3. **Multi-Factor Ranking** — Calculates a composite HIFFI score using four weighted factors: recent momentum, view velocity, audience reach, and consistency.
4. **Breakout Artist Discovery** — Uses a Breakout Velocity Index (BVI) to highlight emerging artists with strong growth potential.
5. **Interactive Analytics Dashboard** — Provides rankings, performance visualizations, artist-level insights, filtering, and CSV/Excel exports through Streamlit.

## ✨ Key Features

- **Data Cleaning and Validation:** Identifies duplicate records, missing values, inconsistent data types, and potential outliers.
- **Artist Identity Matching:** Uses available artist metadata and external knowledge sources to support identity verification.
- **YouTube Metrics Collection:** Integrates YouTube Data API v3 to retrieve supported channel statistics and video information.
- **Historical Performance Tracking:** Uses SQLite to store metric snapshots and support growth and velocity calculations.
- **Weighted Ranking Model:** Combines four performance indicators into a composite score.
- **Channel Health Assessment:** Distinguishes active channels, evergreen catalogs, and potentially dormant channels.
- **Breakout Velocity Analysis:** Helps identify artists gaining attention relative to their existing audience size.
- **Interactive Dashboard:** Displays key performance indicators, leaderboards, charts, filters, and artist details.
- **Exportable Reports:** Generates enriched artist datasets and identity verification audit reports.

## 📊 HIFFI Ranking Methodology

The HIFFI composite score uses four weighted performance factors:

| Ranking Factor | Weight | Description |
|---|---:|---|
| Recent Momentum | 35% | Measures recent audience or performance growth. |
| View Velocity | 30% | Measures how quickly an artist's content gains views. |
| Audience Reach | 20% | Represents the size of the artist's audience. |
| Consistency | 15% | Measures performance stability over time. |
| **Total** | **100%** | |

The factors are normalized to a common scale before being combined into the raw score.

### Confidence-Adjusted Score

The system can adjust the raw score according to the confidence of the artist identity match:

`Final Score = Raw Score × (0.80 + 0.20 × Confidence Score)`

The confidence score ranges from `0.00` to `1.00`.

This adjustment reduces the influence of uncertain identity matches. Additional eligibility rules may be applied to prevent unverified accounts from entering the official leaderboard.

### Breakout Velocity Index (BVI)

The BVI provides a separate perspective on artist performance, focusing on growth and view velocity relative to the artist's existing audience size.

It complements the overall HIFFI ranking by helping surface emerging artists who may not yet have a large established audience.

*Note: The exact BVI calculation depends on the implementation in the scoring module.*

## 🏗️ System Architecture

The pipeline follows this workflow:

```text
Artist Source Data / Soundcharts Export
                  |
                  v
        Data Cleaning & Validation
                  |
                  v
       Artist Identity Verification
        (MusicBrainz / Wikidata)
                  |
                  v
       YouTube Metrics Collection
                  |
                  v
       Historical Snapshot Storage
                (SQLite)
                  |
                  v
        Scoring & Ranking Engine
          |                 |
          v                 v
   HIFFI Composite       Breakout Velocity
       Score                 Index
          |                 |
          +--------+--------+
                   |
                   v
          Enriched Artist Dataset
                   |
                   v
        Streamlit Analytics Dashboard
                   |
                   v
        CSV / Excel Report Exports
```

## 📁 Repository Structure

```text
hiffi-hip-hop-500/
├── collectors/                     # Data collection modules
├── verifiers/                      # Artist identity verification
├── core/                           # Data processing and scoring logic
├── data/                           # Input datasets and reference data
├── app.py                          # Streamlit analytics dashboard
├── main.py                         # Pipeline orchestration
├── enriched_artists.csv            # Enriched artist metrics and rankings
├── identity_verification_audit.xlsx # Identity audit and review flags
├── requirements.txt                # Python dependencies
└── README.md                       # Project documentation
```

*The structure above describes the expected project layout. File names and generated artifacts should match the actual repository.*

## 📦 Key Deliverables

| Deliverable | Purpose |
|---|---|
| `enriched_artists.csv` | Stores enriched artist records, performance metrics, scores, rankings, and health classifications. |
| `identity_verification_audit.xlsx` | Records identity matching results, confidence scores, verification flags, and review requirements. |
| `app.py` | Provides the interactive Streamlit dashboard for exploring artist performance. |
| SQLite database | Stores historical metric snapshots used for time-based performance analysis. |

## 🛠️ Technology Stack

- **Language:** Python
- **Data Processing:** pandas, NumPy
- **Data Sources:** Soundcharts exports and YouTube Data API v3
- **Identity Verification:** MusicBrainz and Wikidata, where available
- **Database:** SQLite
- **Dashboard:** Streamlit
- **Visualizations:** Plotly
- **Exports:** CSV and Excel

## 🚀 Getting Started

### 1. Clone the Repository

```bash
git clone <repository-url>
cd hiffi-hip-hop-500
```

Replace `<repository-url>` with the actual repository URL.

### 2. Create a Virtual Environment

```bash
python -m venv .venv
```

Activate it:

**Windows**
```bash
.venv\Scripts\activate
```

**macOS / Linux**
```bash
source .venv/bin/activate
```

### 3. Install Dependencies

```bash
pip install -r requirements.txt
```

### 4. Configure API Credentials

Set up the required API credentials, including the YouTube Data API v3 key, if the collection pipeline requires it.

Store credentials in environment variables or a local `.env` file. Do not commit API keys, passwords, or other secrets to the repository.

### 5. Run the Data Pipeline

```bash
python main.py
```

This runs the pipeline orchestrator, subject to the project's configured data sources, credentials, and implementation.

### 6. Launch the Dashboard

```bash
streamlit run app.py
```

Open the local URL displayed in the terminal to access the dashboard.

*Run commands should be verified against the actual repository before use.*

## 📈 Dashboard Capabilities

The dashboard is designed to provide:

- **KPI Overview:** Summary counts for monitored artists and channel health categories.
- **Top Artist Leaderboard:** Artists ranked by their HIFFI composite scores.
- **Breakout Leaderboard:** Artists highlighted by the BVI.
- **Identity Audit View:** Confidence scores, verification flags, and eligibility information.
- **Performance Charts:** Comparisons of view velocity and recent growth.
- **Category Filters:** Explore artist groups and subgenres.
- **Artist Details:** Review available channel information and verification results.
- **Data Exports:** Download filtered results in CSV or Excel format.

## 🔍 Data Quality and Reliability

The system is designed to improve the reliability of artist rankings by:

- Avoiding identity matches based only on similar names.
- Flagging uncertain or conflicting account information.
- Distinguishing unavailable metrics from genuine zero values.
- Considering both recent growth and established audience reach.
- Tracking historical snapshots to support time-based calculations.
- Maintaining audit records for review and transparency.

Automated identity matching supports verification but does not guarantee that every account is official. Uncertain matches should be reviewed before being treated as verified.

## 🔐 Security and Responsible Data Handling

- Keep API credentials outside source control.
- Use environment variables or approved secret-management tools.
- Respect API quotas, platform terms, and applicable data-use policies.
- Validate source data before including it in published rankings.

## 🎯 Project Objective

The goal of the HIFFI Hip-Hop 500 Intelligence Engine is to create a transparent, repeatable, and data-driven approach to hip-hop artist analysis.

By combining data quality checks, identity verification, historical performance tracking, weighted ranking, and interactive analytics, the platform supports more informed artist discovery and performance evaluation.