An auditable data pipeline for collecting, validating, and analyzing artist metrics across YouTube, Spotify, and Instagram for a cohort of 500 hip-hop artists.
The project combines platform data collection, artist verification, ranking calculations, data-quality checks, and audit reporting to support reliable artist analysis.
Collects channel and video metrics used by the ranking engine, including:
- Subscriber count
- Lifetime views
- Daily view velocity
- Recent momentum
- Video count
### Spotify
Uses a three-tier collection strategy:
| Tier | Source | Purpose |
|---|---|---|
| 1 | Kworb | Collect available public streaming metrics. |
| 2 | Spotify Web API | Attempt authenticated metric retrieval. |
| 3 | Spotify embed | Attempt fallback retrieval for supported artist IDs. |
The collector includes handling for API errors and unavailable data. Actual coverage depends on source availability and access restrictions.
### Instagram
Evaluates candidate artist profiles using name matching, biography keywords, follower-count rules, and confidence checks. Manual overrides are supported through `data/handle_overrides.json`.
Automated matching is a verification aid, not a guarantee that an account is official.
## 5. Ranking Methodology
The ranking engine calculates two distinct metrics: the **HIFFI Score** and the **Breakout Velocity Index (BVI)**.
### 5.1 HIFFI Score
The HIFFI Score uses four weighted signals:
| Signal | Weight |
|---|---:|
| Recent Momentum | 35% |
| View Velocity | 30% |
| Audience Reach | 20% |
| Consistency | 15% |
| **Total** | **100%** |
#### Step 1: Normalize the metrics
Logarithmic normalization is used for daily view velocity, subscriber count, lifetime views, and video count.
\[
L(x)=
\frac{\log_{10}(1+x)-z_{\min}}
{z_{\max}-z_{\min}+10^{-9}}
\]
Here, \(z_{\min}\) and \(z_{\max}\) are the minimum and maximum log-transformed values.
Recent Momentum uses Min-Max normalization after clipping momentum to the range \([-100,300]\):
\[
M(x)=\frac{x-x_{\min}}{x_{\max}-x_{\min}+10^{-9}}
\]
The implementation returns zero scores when the valid values are constant or unavailable.
#### Step 2: Calculate Audience Reach
Audience Reach combines subscriber count and lifetime views:
Here, \(C\) is the confidence score clipped to \([0,1]\). The final score is rounded to two decimal places.
For example, if the raw score is 68.50 and confidence is 0.90:
\[
68.50\times(0.8+0.2\times0.90)=67.13
\]
The final HIFFI Score is **67.13**.
### 5.2 HIFFI Ranking Eligibility
An artist qualifies for the primary ranking when:
- Subscriber count and lifetime views are available.
- Confidence score is at least 0.50 by default.
- Status is `COMPLETE` or `CATALOG_ACTIVE_NO_RECENT_UPLOAD`.
Eligible artists are ranked by final HIFFI Score in descending order. Tied scores receive the same minimum rank.
### 5.3 Breakout Velocity Index (BVI)
BVI evaluates relative view velocity and recent momentum among qualifying artists.
**Eligibility requirements:**
- At least 10,000 subscribers.
- Positive daily view velocity.
- Available subscriber count and daily view velocity.
- Status is neither `INACTIVE_OR_DORMANT` nor `FAILED_LOOKUP`.
The BVI calculation uses three stages.
**Audience velocity ratio**
\[
V_{\text{aud}}=
\frac{\text{daily view velocity}}
{\max(\text{subscribers},1000)}
\]
**Momentum signal**
\[
M_{\text{signal}}=
\frac{\text{momentum}_{30d}+100}{400}
\]
**Emergence dampener**
\[
W_{\text{emerge}}=
\frac{1}{1+\log_{10}(1+\text{subscribers})}
\]
The raw BVI is calculated as:
\[
BVI_{\text{raw}}=
\left(
0.65\log_{10}(1+V_{\text{aud}})
+0.35M_{\text{signal}}
\right)
W_{\text{emerge}}
\]
Finally, qualifying BVI values are min-max scaled to a 0–100 range. If all qualifying raw values are equal, the implementation assigns a BVI of 50.
The `breakout_rank` is calculated separately using BVI in descending order. Artists without a valid BVI are not included in this ranking.
**Note:** The 10,000-subscriber threshold applies to BVI eligibility, not primary HIFFI ranking eligibility.
## 6. Data Governance
The pipeline follows these principles:
- Missing platform metrics should not automatically be interpreted as genuine zero values.
- Artist profile verification status and confidence should be recorded where available.
- Collection timestamps and source information support traceability.
- Failed lookups and uncertain records must be distinguishable from successfully collected data.
- Primary HIFFI scores and BVI rankings use separate eligibility rules.
**Implementation note:** The current ranking engine fills several missing metric inputs with zero before scoring. This behavior should be distinguished from preserving missing values in the collected dataset.
The HIFFI Multi-Platform Artist Intelligence Pipeline combines data collection, artist verification, ranking calculations, and automated quality checks to support structured analysis of a 500-artist hip-hop cohort.
The HIFFI Score and BVI represent the formulas implemented in this repository and should be distinguished from any separate official HIFFI website ranking methodology.
