hiffi-artist-intelligence/README.md

# HIFFI Multi-Platform Artist Intelligence Pipeline

An auditable data pipeline for collecting, validating, and analyzing artist metrics across YouTube, Spotify, and Instagram for a cohort of 500 hip-hop artists.

The project combines platform data collection, artist verification, ranking calculations, data-quality checks, and audit reporting to support reliable artist analysis.

## 1. Project Objectives

- Process a cohort of 500 unique hip-hop artists.
- Collect available metrics from YouTube, Spotify, and Instagram.
- Validate artist profiles and track verification confidence.
- Preserve missing values and avoid misleading zero-imputation.
- Calculate HIFFI scores and Breakout Velocity Index (BVI).
- Compare artist rankings and maintain data provenance.
- Validate pipeline behavior through automated tests.

## 2. Key Features

- **Multi-platform data collection:** Collects available metrics from three platforms.
- **Spotify fallback strategy:** Uses a tiered collection approach involving Kworb, the Spotify Web API, and an embed fallback.
- **Instagram verification:** Uses candidate matching, confidence scoring, and manual overrides.
- **Ranking engine:** Calculates weighted HIFFI scores and a separate BVI.
- **Data governance:** Tracks missing values, verification status, and collection timestamps.
- **Automated QA:** Includes 14 documented Pytest checks.
- **Audit-ready output:** Exports consolidated artist metrics and ranking information.

## 3. System Workflow

```mermaid
flowchart TD
    A["500-Artist Cohort"] --> B["YouTube Data Collection"]
    A --> C["Spotify Data Collection"]
    A --> D["Instagram Profile Verification"]
    B --> E["Data Validation"]
    C --> E
    D --> E
    E --> F["HIFFI Score and BVI"]
    F --> G["Ranking and QA Checks"]
    G --> H["Consolidated Dataset and Report"]
```

## 4. Data Collection

### YouTube

Collects channel and video metrics used by the ranking engine, including:

- Subscriber count
- Lifetime views
- Daily view velocity
- Recent momentum
- Video count

### Spotify

Uses a three-tier collection strategy:

| Tier | Source | Purpose |
|---|---|---|
| 1 | Kworb | Collect available public streaming metrics. |
| 2 | Spotify Web API | Attempt authenticated metric retrieval. |
| 3 | Spotify embed | Attempt fallback retrieval for supported artist IDs. |

The collector includes handling for API errors and unavailable data. Actual coverage depends on source availability and access restrictions.

### Instagram

Evaluates candidate artist profiles using name matching, biography keywords, follower-count rules, and confidence checks. Manual overrides are supported through `data/handle_overrides.json`.

Automated matching is a verification aid, not a guarantee that an account is official.

## 5. Ranking Methodology

The ranking engine calculates two distinct metrics: the **HIFFI Score** and the **Breakout Velocity Index (BVI)**.

### 5.1 HIFFI Score

The HIFFI Score uses four weighted signals:

| Signal | Weight |
|---|---:|
| Recent Momentum | 35% |
| View Velocity | 30% |
| Audience Reach | 20% |
| Consistency | 15% |
| **Total** | **100%** |

#### Step 1: Normalize the metrics

Logarithmic normalization is used for daily view velocity, subscriber count, lifetime views, and video count.

\[
L(x)=
\frac{\log_{10}(1+x)-z_{\min}}
{z_{\max}-z_{\min}+10^{-9}}
\]

Here, \(z_{\min}\) and \(z_{\max}\) are the minimum and maximum log-transformed values.

Recent Momentum uses Min-Max normalization after clipping momentum to the range \([-100,300]\):

\[
M(x)=\frac{x-x_{\min}}{x_{\max}-x_{\min}+10^{-9}}
\]

The implementation returns zero scores when the valid values are constant or unavailable.

#### Step 2: Calculate Audience Reach

Audience Reach combines subscriber count and lifetime views:

\[
S_{\text{reach}}=
0.60L(\text{subscribers})
+0.40L(\text{lifetime views})
\]

#### Step 3: Calculate the raw score

\[
\begin{aligned}
H_{\text{raw}}=100\times(&0.35S_{\text{momentum}}\\
&+0.30S_{\text{velocity}}\\
&+0.20S_{\text{reach}}\\
&+0.15S_{\text{consistency}})
\end{aligned}
\]

#### Step 4: Apply confidence adjustment

\[
C_{\text{multiplier}}=0.8+0.2C
\]

\[
\boxed{H_{\text{final}}=H_{\text{raw}}\times C_{\text{multiplier}}}
\]

Here, \(C\) is the confidence score clipped to \([0,1]\). The final score is rounded to two decimal places.

For example, if the raw score is 68.50 and confidence is 0.90:

\[
68.50\times(0.8+0.2\times0.90)=67.13
\]

The final HIFFI Score is **67.13**.

### 5.2 HIFFI Ranking Eligibility

An artist qualifies for the primary ranking when:

- Subscriber count and lifetime views are available.
- Confidence score is at least 0.50 by default.
- Status is `COMPLETE` or `CATALOG_ACTIVE_NO_RECENT_UPLOAD`.

Eligible artists are ranked by final HIFFI Score in descending order. Tied scores receive the same minimum rank.

### 5.3 Breakout Velocity Index (BVI)

BVI evaluates relative view velocity and recent momentum among qualifying artists.

**Eligibility requirements:**

- At least 10,000 subscribers.
- Positive daily view velocity.
- Available subscriber count and daily view velocity.
- Status is neither `INACTIVE_OR_DORMANT` nor `FAILED_LOOKUP`.

The BVI calculation uses three stages.

**Audience velocity ratio**

\[
V_{\text{aud}}=
\frac{\text{daily view velocity}}
{\max(\text{subscribers},1000)}
\]

**Momentum signal**

\[
M_{\text{signal}}=
\frac{\text{momentum}_{30d}+100}{400}
\]

**Emergence dampener**

\[
W_{\text{emerge}}=
\frac{1}{1+\log_{10}(1+\text{subscribers})}
\]

The raw BVI is calculated as:

\[
BVI_{\text{raw}}=
\left(
0.65\log_{10}(1+V_{\text{aud}})
+0.35M_{\text{signal}}
\right)
W_{\text{emerge}}
\]

Finally, qualifying BVI values are min-max scaled to a 0–100 range. If all qualifying raw values are equal, the implementation assigns a BVI of 50.

The `breakout_rank` is calculated separately using BVI in descending order. Artists without a valid BVI are not included in this ranking.

**Note:** The 10,000-subscriber threshold applies to BVI eligibility, not primary HIFFI ranking eligibility.

## 6. Data Governance

The pipeline follows these principles:

- Missing platform metrics should not automatically be interpreted as genuine zero values.
- Artist profile verification status and confidence should be recorded where available.
- Collection timestamps and source information support traceability.
- Failed lookups and uncertain records must be distinguishable from successfully collected data.
- Primary HIFFI scores and BVI rankings use separate eligibility rules.

**Implementation note:** The current ranking engine fills several missing metric inputs with zero before scoring. This behavior should be distinguished from preserving missing values in the collected dataset.

## 7. Repository Structure

```text
hiffi-artist-intelligence/
├── collectors/
│   ├── instagram_collector.py
│   ├── instagram_verifier.py
│   ├── kworb_spotify_collector.py
│   ├── spotify_api_collector.py
│   ├── spotify_fallback_collector.py
│   └── youtube_collector.py
├── core/
│   ├── confidence_scorer.py
│   ├── deduplication.py
│   ├── health_detector.py
│   └── ranking_engine.py
├── data/
│   ├── handle_overrides.json
│   └── hiffi_multiplatform_500.csv
├── pipeline/
│   └── batch_500_runner.py
├── main.py
├── test_multiplatform_ranking.py
├── requirements.txt
└── README.md
```

## 8. Installation and Setup

### Prerequisites

- Python 3.12+
- Git
- Spotify API credentials, if required by the configured collector

### Clone the repository

```bash
git clone https://github.com/<your-username>/hiffi-artist-intelligence.git
cd hiffi-artist-intelligence
```

### Create a virtual environment

```powershell
python -m venv .venv
.\.venv\Scripts\Activate.ps1
```

### Install dependencies

```bash
pip install -r requirements.txt
```

### Configure Spotify API credentials

Create a `.env` file in the project root when using the official Spotify API collector:

```dotenv
SPOTIFY_CLIENT_ID=your_client_id_here
SPOTIFY_CLIENT_SECRET=your_client_secret_here
```

Never commit API credentials or secrets to GitHub.

## 9. Execution Commands

Run the following commands from the repository root.

| Operation | Command |
|---|---|
| Full workflow | `python main.py all` |
| Batch pipeline | `python main.py pipeline` |
| Health checks | `python main.py health` |
| QA tests | `pytest test_multiplatform_ranking.py -v` |
| Executive report | `python main.py report` |

Execution results depend on the configured inputs, dependencies, and availability of external data sources.

## 10. QA and Validation

The repository documents 14 tests covering:

1. Cohort size and uniqueness.
2. Missing-value and streaming-data governance.
3. Preservation of production rankings.
4. Score normalization bounds.
5. Ranking monotonicity.
6. Required provenance columns.
7. Instagram candidate matching.
8. Suspicious Instagram follower counts.
9. Spotify fallback null safety.
10. Mocked network-error handling.
11. Duplicate-record safeguards.
12. Unconfigured Spotify API behavior.
13. Empty or malformed Spotify IDs.
14. Additional validation of the pipeline's ranking and data-integrity rules.

Run the test suite with:

```bash
pytest test_multiplatform_ranking.py -v
```

The README documents the intended checks; test results should be verified against the current code before reporting them as passing.

## 11. Output Dataset

The consolidated dataset is exported to:

`data/hiffi_multiplatform_500.csv`

Important fields include:

| Column | Description |
|---|---|
| `name` | Artist name |
| `subscriber_count` | YouTube subscriber count |
| `lifetime_views` | Total YouTube views |
| `daily_view_velocity` | Daily view velocity |
| `momentum_30d` | Recent momentum |
| `video_count` | Video count |
| `confidence_score` | Confidence used for score adjustment and eligibility |
| `hiffi_score` | Final HIFFI Score |
| `hiffi_rank` | Primary ranking |
| `breakout_velocity_index` | BVI score |
| `breakout_rank` | BVI ranking |
| `is_rank_eligible` | Primary ranking eligibility flag |
| `status_flag` | Record or collection status |

The complete output schema may include additional Spotify, Instagram, provenance, and timestamp fields.

## 12. Limitations

- External APIs and public data sources may return incomplete data or restrict access.
- Artist names and unofficial profiles can create identity-matching errors.
- Missing metric handling can influence ranking outcomes.
- Min-Max normalization depends on the comparison cohort.
- Video count is a proxy for consistency, not a direct measure of upload regularity.
- BVI measures relative breakout signals under the implemented formula, not guaranteed future popularity.
- HIFFI scores reflect the configured metrics and weights, not artistic quality.

## 13. Future Improvements

- Improve artist identity verification and matching confidence.
- Strengthen API failure monitoring and source coverage reporting.
- Expand edge-case tests for normalization and missing metrics.
- Add historical snapshots and ranking-change reports.
- Improve documentation of data freshness and score eligibility.

## Conclusion

The HIFFI Multi-Platform Artist Intelligence Pipeline combines data collection, artist verification, ranking calculations, and automated quality checks to support structured analysis of a 500-artist hip-hop cohort.

Its key principles are **traceable data, explicit ranking rules, transparent confidence adjustments, and careful handling of missing information**.

The HIFFI Score and BVI represent the formulas implemented in this repository and should be distinguished from any separate official HIFFI website ranking methodology.