An auditable data pipeline for collecting, validating, and analyzing artist metrics across YouTube, Spotify, and Instagram for a cohort of 500 hip-hop artists.

The project combines platform data collection, artist verification, ranking calculations, data-quality checks, and audit reporting to support reliable artist analysis.

1. Project Objectives

Process a cohort of 500 unique hip-hop artists.
Collect available metrics from YouTube, Spotify, and Instagram.
Validate artist profiles and track verification confidence.
Preserve missing values and avoid misleading zero-imputation.
Calculate HIFFI scores and Breakout Velocity Index (BVI).
Compare artist rankings and maintain data provenance.
Validate pipeline behavior through automated tests.

2. Key Features

Multi-platform data collection: Collects available metrics from three platforms.
Spotify fallback strategy: Uses a tiered collection approach involving Kworb, the Spotify Web API, and an embed fallback.
Instagram verification: Uses candidate matching, confidence scoring, and manual overrides.
Ranking engine: Calculates weighted HIFFI scores and a separate BVI.
Data governance: Tracks missing values, verification status, and collection timestamps.
Automated QA: Includes 14 documented Pytest checks.
Audit-ready output: Exports consolidated artist metrics and ranking information.

3. System Workflow

flowchart TD
    A["500-Artist Cohort"] --> B["YouTube Data Collection"]
    A --> C["Spotify Data Collection"]
    A --> D["Instagram Profile Verification"]
    B --> E["Data Validation"]
    C --> E
    D --> E
    E --> F["HIFFI Score and BVI"]
    F --> G["Ranking and QA Checks"]
    G --> H["Consolidated Dataset and Report"]

4. Data Collection

YouTube

Collects channel and video metrics used by the ranking engine, including:

Subscriber count
Lifetime views
Daily view velocity
Recent momentum
Video count

Spotify

Uses a three-tier collection strategy:

Tier	Source	Purpose
1	Kworb	Collect available public streaming metrics.
2	Spotify Web API	Attempt authenticated metric retrieval.
3	Spotify embed	Attempt fallback retrieval for supported artist IDs.

The collector includes handling for API errors and unavailable data. Actual coverage depends on source availability and access restrictions.

Instagram

Evaluates candidate artist profiles using name matching, biography keywords, follower-count rules, and confidence checks. Manual overrides are supported through data/handle_overrides.json.

Automated matching is a verification aid, not a guarantee that an account is official.

5. Ranking Methodology

The ranking engine calculates two distinct metrics: the HIFFI Score and the Breakout Velocity Index (BVI).

5.1 HIFFI Score

The HIFFI Score uses four weighted signals:

Signal	Weight
Recent Momentum	35%
View Velocity	30%
Audience Reach	20%
Consistency	15%
Total	100%

Step 1: Normalize the metrics

Logarithmic normalization is used for daily view velocity, subscriber count, lifetime views, and video count.

[ L(x)= \frac{\log_{10}(1+x)-z_{\min}} {z_{\max}-z_{\min}+10^{-9}} ]

Here, (z_{\min}) and (z_{\max}) are the minimum and maximum log-transformed values.

Recent Momentum uses Min-Max normalization after clipping momentum to the range ([-100,300]):

[ M(x)=\frac{x-x_{\min}}{x_{\max}-x_{\min}+10^{-9}} ]

The implementation returns zero scores when the valid values are constant or unavailable.

Step 2: Calculate Audience Reach

Audience Reach combines subscriber count and lifetime views:

[ S_{\text{reach}}= 0.60L(\text{subscribers}) +0.40L(\text{lifetime views}) ]

Step 3: Calculate the raw score

[ \begin{aligned} H_{\text{raw}}=100\times(&0.35S_{\text{momentum}}\ &+0.30S_{\text{velocity}}\ &+0.20S_{\text{reach}}\ &+0.15S_{\text{consistency}}) \end{aligned} ]

Step 4: Apply confidence adjustment

[ C_{\text{multiplier}}=0.8+0.2C ]

[ \boxed{H_{\text{final}}=H_{\text{raw}}\times C_{\text{multiplier}}} ]

Here, (C) is the confidence score clipped to ([0,1]). The final score is rounded to two decimal places.

For example, if the raw score is 68.50 and confidence is 0.90:

[ 68.50\times(0.8+0.2\times0.90)=67.13 ]

The final HIFFI Score is 67.13.

5.2 HIFFI Ranking Eligibility

An artist qualifies for the primary ranking when:

Subscriber count and lifetime views are available.
Confidence score is at least 0.50 by default.
Status is COMPLETE or CATALOG_ACTIVE_NO_RECENT_UPLOAD.

Eligible artists are ranked by final HIFFI Score in descending order. Tied scores receive the same minimum rank.

5.3 Breakout Velocity Index (BVI)

BVI evaluates relative view velocity and recent momentum among qualifying artists.

Eligibility requirements:

At least 10,000 subscribers.
Positive daily view velocity.
Available subscriber count and daily view velocity.
Status is neither INACTIVE_OR_DORMANT nor FAILED_LOOKUP.

The BVI calculation uses three stages.

Audience velocity ratio

[ V_{\text{aud}}= \frac{\text{daily view velocity}} {\max(\text{subscribers},1000)} ]

Momentum signal

[ M_{\text{signal}}= \frac{\text{momentum}_{30d}+100}{400} ]

Emergence dampener

[ W_{\text{emerge}}= \frac{1}{1+\log_{10}(1+\text{subscribers})} ]

The raw BVI is calculated as:

[ BVI_{\text{raw}}= \left( 0.65\log_{10}(1+V_{\text{aud}}) +0.35M_{\text{signal}} \right) W_{\text{emerge}} ]

Finally, qualifying BVI values are min-max scaled to a 0–100 range. If all qualifying raw values are equal, the implementation assigns a BVI of 50.

The breakout_rank is calculated separately using BVI in descending order. Artists without a valid BVI are not included in this ranking.

Note: The 10,000-subscriber threshold applies to BVI eligibility, not primary HIFFI ranking eligibility.

6. Data Governance

The pipeline follows these principles:

Missing platform metrics should not automatically be interpreted as genuine zero values.
Artist profile verification status and confidence should be recorded where available.
Collection timestamps and source information support traceability.
Failed lookups and uncertain records must be distinguishable from successfully collected data.
Primary HIFFI scores and BVI rankings use separate eligibility rules.

Implementation note: The current ranking engine fills several missing metric inputs with zero before scoring. This behavior should be distinguished from preserving missing values in the collected dataset.

7. Repository Structure

hiffi-artist-intelligence/
├── collectors/
│   ├── instagram_collector.py
│   ├── instagram_verifier.py
│   ├── kworb_spotify_collector.py
│   ├── spotify_api_collector.py
│   ├── spotify_fallback_collector.py
│   └── youtube_collector.py
├── core/
│   ├── confidence_scorer.py
│   ├── deduplication.py
│   ├── health_detector.py
│   └── ranking_engine.py
├── data/
│   ├── handle_overrides.json
│   └── hiffi_multiplatform_500.csv
├── pipeline/
│   └── batch_500_runner.py
├── main.py
├── test_multiplatform_ranking.py
├── requirements.txt
└── README.md

8. Installation and Setup

Prerequisites

Python 3.12+
Git
Spotify API credentials, if required by the configured collector

Clone the repository

git clone https://github.com/<your-username>/hiffi-artist-intelligence.git
cd hiffi-artist-intelligence

Create a virtual environment

python -m venv .venv
.\.venv\Scripts\Activate.ps1

Install dependencies

pip install -r requirements.txt

Configure Spotify API credentials

Create a .env file in the project root when using the official Spotify API collector:

SPOTIFY_CLIENT_ID=your_client_id_here
SPOTIFY_CLIENT_SECRET=your_client_secret_here

Never commit API credentials or secrets to GitHub.

9. Execution Commands

Run the following commands from the repository root.

Operation	Command
Full workflow	`python main.py all`
Batch pipeline	`python main.py pipeline`
Health checks	`python main.py health`
QA tests	`pytest test_multiplatform_ranking.py -v`
Executive report	`python main.py report`

Execution results depend on the configured inputs, dependencies, and availability of external data sources.

10. QA and Validation

The repository documents 14 tests covering:

Cohort size and uniqueness.
Missing-value and streaming-data governance.
Preservation of production rankings.
Score normalization bounds.
Ranking monotonicity.
Required provenance columns.
Instagram candidate matching.
Suspicious Instagram follower counts.
Spotify fallback null safety.
Mocked network-error handling.
Duplicate-record safeguards.
Unconfigured Spotify API behavior.
Empty or malformed Spotify IDs.
Additional validation of the pipeline's ranking and data-integrity rules.

Run the test suite with:

pytest test_multiplatform_ranking.py -v

The README documents the intended checks; test results should be verified against the current code before reporting them as passing.

11. Output Dataset

The consolidated dataset is exported to:

data/hiffi_multiplatform_500.csv

Important fields include:

Column	Description
`name`	Artist name
`subscriber_count`	YouTube subscriber count
`lifetime_views`	Total YouTube views
`daily_view_velocity`	Daily view velocity
`momentum_30d`	Recent momentum
`video_count`	Video count
`confidence_score`	Confidence used for score adjustment and eligibility
`hiffi_score`	Final HIFFI Score
`hiffi_rank`	Primary ranking
`breakout_velocity_index`	BVI score
`breakout_rank`	BVI ranking
`is_rank_eligible`	Primary ranking eligibility flag
`status_flag`	Record or collection status

The complete output schema may include additional Spotify, Instagram, provenance, and timestamp fields.

12. Limitations

External APIs and public data sources may return incomplete data or restrict access.
Artist names and unofficial profiles can create identity-matching errors.
Missing metric handling can influence ranking outcomes.
Min-Max normalization depends on the comparison cohort.
Video count is a proxy for consistency, not a direct measure of upload regularity.
BVI measures relative breakout signals under the implemented formula, not guaranteed future popularity.
HIFFI scores reflect the configured metrics and weights, not artistic quality.

13. Future Improvements

Improve artist identity verification and matching confidence.
Strengthen API failure monitoring and source coverage reporting.
Expand edge-case tests for normalization and missing metrics.
Add historical snapshots and ranking-change reports.
Improve documentation of data freshness and score eligibility.

Conclusion

The HIFFI Multi-Platform Artist Intelligence Pipeline combines data collection, artist verification, ranking calculations, and automated quality checks to support structured analysis of a 500-artist hip-hop cohort.

Its key principles are traceable data, explicit ranking rules, transparent confidence adjustments, and careful handling of missing information.

The HIFFI Score and BVI represent the formulas implemented in this repository and should be distinguished from any separate official HIFFI website ranking methodology.

README.md Unescape Escape

HIFFI Multi-Platform Artist Intelligence Pipeline