An end-to-end Python data pipeline that collects, validates, and ranks a cohort of 500 hip-hop artists using data from YouTube, Spotify, and Instagram.

The project focuses on reliable data collection, transparent ranking calculations, artist identity verification, anomaly detection, and automated testing.

Key Features

Multi-platform data collection: Collects available artist metrics from YouTube, Spotify, and Instagram.
Artist verification: Uses identity matching, confidence checks, and manual review for uncertain profiles.
Weighted ranking: Calculates production scores, HIFFI-XP scores, and the Breakout Velocity Index (BVI).
Missing-data handling: Preserves missing source metrics as NaN and redistributes scoring weights across available signals.
Data freshness checks: Excludes stale or invalid signals from publication evidence.
Anomaly detection: Flags suspicious metrics for further review.
Ranking regression: Compares current rankings with historical snapshots.
Audit reporting: Generates a Markdown report with ranking and data-quality information.
Continuous Integration: Runs automated tests on Python 3.11 and 3.12 using GitHub Actions.

1. How the Pipeline Works

The pipeline follows five main stages:

Load and collect data — Load the artist cohort and retrieve available platform metrics.
Clean and validate — Check missing values, data quality, artist identities, and timestamps.
Calculate scores — Normalize available metrics and calculate weighted artist scores.
Apply governance checks — Evaluate freshness, platform evidence, eligibility, and suspicious patterns.
Publish and audit — Generate rankings, compare historical results, and create the audit report.

2. Ranking Methodology

The project calculates three main outputs: the YouTube production score, the multi-platform HIFFI-XP score, and the Breakout Velocity Index (BVI).

HIFFI-XP Composite Score

The default weights are:

Signal	Weight
YouTube production score	50%
Spotify monthly listeners	25%
Spotify daily velocity	5%
Instagram followers	20%
Total	100%

Each metric is normalized before combining it into the composite score.

Dynamic Weight Redistribution

When some platform metrics are unavailable, the remaining available signals share the weight proportionally.

Examples:

YouTube only: YouTube receives 100%.
YouTube + Spotify monthly listeners: YouTube receives 66.7%, and Spotify receives 33.3%.
All signals available: The original weights are retained.

The effective_weights column records the actual weights used for each artist.

Missing metrics are preserved as NaN in the source dataset rather than being treated as genuine zero values. Missing inputs may be handled separately during individual scoring calculations, so score interpretation should account for the implemented normalization and missing-data rules.

YouTube Production Score

The production score combines four signals:

Signal	Weight
Recent momentum	35%
Daily view velocity	30%
Audience reach	20%
Consistency proxy	15%

Audience reach combines subscriber count and lifetime views. Video count is used as a consistency proxy.

Breakout Velocity Index (BVI)

BVI measures relative breakout signals using view velocity, recent momentum, and an audience-size adjustment.

Eligibility requires at least 10,000 subscribers, positive daily view velocity, and an eligible collection status. BVI is calculated and ranked separately from the primary production score.

Note: The production score, HIFFI-XP composite score, and BVI are separate calculations. They should not be confused with one another or assumed to be identical to any separate official HIFFI website formula.

3. Data Governance

Freshness Rules

Platform	Maximum signal age
YouTube	7 days
Spotify	7 days
Instagram	14 days

Missing, malformed, future-dated, or stale timestamps fail the freshness check. Excluded signals are recorded in evidence_exclusions.

Artist Identity Verification

Instagram identity status is checked before the platform contributes to publication evidence. The pipeline supports automated verification, manual verification, and a review process for uncertain accounts.

Automated matching helps identify likely official profiles, but it cannot guarantee that every matched account belongs to the correct artist.

Publication Eligibility

Top 10: Requires Tier 3 platform evidence.
Top 50: Requires at least Tier 2 platform evidence.
Published ranks: Assigned consecutively.
Shortfall handling: When strict evidence requirements cannot fill the target positions, fallback selections are recorded in the shortfall audit metadata.

The audit report should be reviewed for any shortfalls before the rankings are treated as fully compliant with the intended eligibility rules.

Anomaly Detection

The anomaly_flags field records suspicious patterns, including:

SUSPICIOUS_IG_TO_YT_RATIO — Instagram followers exceed 10 times YouTube lifetime views.
EXTREME_SPOTIFY_VELOCITY_RATIO — Spotify daily change exceeds 25% of monthly listeners.

These rules flag records for investigation. An anomaly is not proof of fraud or incorrect data.

4. Project Structure

hiffi-artist-intelligence/
├── .github/
│   └── workflows/
│       └── ci.yml
├── collectors/
│   ├── youtube_collector.py
│   ├── spotify_api_collector.py
│   ├── spotify_collector.py
│   ├── kworb_spotify_collector.py
│   ├── spotify_fallback_collector.py
│   ├── instagram_collector.py
│   ├── instagram_verifier.py
│   └── wikidata_resolver.py
├── config/
├── core/
│   ├── ranking_engine.py
│   ├── confidence_scorer.py
│   ├── deduplication.py
│   ├── health_detector.py
│   └── regression_auditor.py
├── data/
│   ├── snapshots/
│   ├── checkpoints/
│   ├── handle_overrides.json
│   ├── review_queue.csv
│   └── hiffi_multiplatform_500.csv
├── pipeline/
├── reports/
│   ├── audit_report_latest.md
│   └── provenance_summary.py
├── run_pipeline.py
├── test_multiplatform_ranking.py
├── requirements.txt
└── README.md

5. Installation and Setup

Prerequisites

Python 3.11 or 3.12
Git
API credentials when required by the configured collectors

Clone the repository

git clone https://github.com/<your-username>/hiffi-artist-intelligence.git
cd hiffi-artist-intelligence

Create a virtual environment

Windows PowerShell

python -m venv .venv
.\.venv\Scripts\Activate.ps1
pip install -r requirements.txt

Configure API credentials

If the Spotify API collector requires credentials, create a local .env file:

SPOTIFY_CLIENT_ID=your_client_id
SPOTIFY_CLIENT_SECRET=your_client_secret

Never commit real credentials, API keys, or tokens to GitHub. Keep .env excluded through .gitignore and use GitHub Secrets for CI credentials when needed.

6. Run the Project

Run commands from the repository root.

Execute the full pipeline

python run_pipeline.py

This runs the configured stages for data loading, scoring, validation, governance, and audit generation.

Run the test suite

python -m pytest test_multiplatform_ranking.py -v

The current governance suite contains 27 tests, based on the latest reported test run.

Compare historical rankings

python -m core.regression_auditor

The regression auditor compares current rankings with saved snapshots and flags movements greater than 100 positions for review.

Review the audit report

reports/audit_report_latest.md

Review the report for missing data, evidence exclusions, anomalies, ranking movements, and publication shortfalls.

7. Continuous Integration

GitHub Actions runs automated tests on Ubuntu across Python 3.11 and Python 3.12.

The CI workflow is configured in:

.github/workflows/ci.yml

The latest reported run passed on both Python versions, with all 27 governance tests passing.

CI results confirm that the tested code and scenarios pass in the configured environments. They do not independently guarantee the accuracy of external platform data or the reliability of every live collection run.

8. Output Dataset

The consolidated dataset is stored at:

data/hiffi_multiplatform_500.csv

Important columns include:

Column	Purpose
`name`	Artist name
`production_hiffi_score`	YouTube production score
`hiffi_xp_score`	Multi-platform composite score
`hiffi_xp_rank`	Raw composite ranking
`hiffi_xp_published_rank`	Publication ranking after eligibility rules
`breakout_velocity_index`	BVI score
`breakout_rank`	Separate BVI ranking
`effective_weights`	Actual scoring weights used
`platform_evidence_tier`	Count of qualifying platform signals
`evidence_exclusions`	Reasons signals were excluded from evidence
`anomaly_flags`	Suspicious patterns identified
`rank_delta`	Difference between production and composite ranks

Additional columns may contain platform metrics, verification statuses, collection timestamps, confidence values, and source provenance.

9. Limitations

External APIs and public sources may provide incomplete data or restrict access.
Artist names and similar handles can cause identity-matching errors.
Missing metrics and normalization choices can affect ranking outcomes.
Large ranking movements require investigation, not automatic correction.
Anomaly flags identify patterns for review rather than confirmed wrongdoing.
Video count is a proxy for consistency, not a direct measurement of upload regularity.
BVI measures relative breakout signals; it does not predict future success with certainty.
Rankings represent the configured metrics and weights, not artistic quality.

10. Future Improvements

Expand source coverage and identity verification.
Improve API failure monitoring and recovery.
Automate scheduled data collection and weekly audit generation.
Strengthen historical regression reporting.
Improve observability, release checks, and production deployment safeguards.

Conclusion

The HIFFI Hip-Hop 500 pipeline brings together multi-platform data collection, artist verification, ranking calculations, data governance, anomaly detection, regression analysis, and automated testing.

Its main goal is to make artist rankings more transparent, traceable, and repeatable, while clearly identifying missing data and uncertain evidence.