hiffi-artist-intelligence/README.md

HIFFI Hip-Hop 500 — Multi-Platform Artist Intelligence Pipeline

A Python-based pipeline for collecting, validating, scoring, and ranking hip-hop artists using YouTube, Spotify, and Instagram signals. It includes data freshness checks, identity verification, anomaly detection, publication eligibility rules, and regression auditing.

Key Features

• Multi-platform artist metrics and scoring.
• Instagram identity matching with review-queue support.
• Missing-data handling and dynamic weight redistribution.
• Platform freshness and publication eligibility gates.
• Cross-platform anomaly detection.
• Regression auditing and rank-shift reports.
• Automated tests and GitHub Actions CI for Python 3.11 and 3.12.

Pipeline Architecture: Telemetry Ingestion vs. Ranking Core

The project separates data collection from ranking so that platform data can be refreshed independently of score calculation.

1. Telemetry Ingestion Layer — pipeline/batch_500_runner.py

• Collects multi-platform telemetry through configured API calls and web scraping.
• Uses checkpoints in data/checkpoints/ to support recovery after interrupted runs and reduce repeated data requests.
• Saves the refreshed master cohort dataset to data/hiffi_multiplatform_500.csv.

2. Ranking and Governance Core — run_pipeline.py

• Loads data/hiffi_multiplatform_500.csv and executes the ranking workflow independently of data collection.
• Handles missing metrics, redistributes weights across available signals, checks data freshness, and flags suspicious records.
• Applies publication eligibility rules and assigns published ranks.
• Generates an audit report at reports/audit_report_latest.md.

Ranking Methodology

The project uses separate scores for artist production performance, multi-platform ranking, and breakout potential.

HIFFI-XP Multi-Platform Score

Signal	Base Weight
YouTube production score	50%
Spotify monthly listeners	25%
Spotify daily stream velocity	5%
Instagram followers	20%

When some signals are unavailable, the scoring logic redistributes their weights proportionally across the available signals. The effective weights are recorded in effective_weights for auditability.

YouTube Production Score

The production score uses the following components:

Component	Weight
Recent momentum	35%
View velocity	30%
Audience reach	20%
Consistency proxy	15%

Audience reach combines subscriber and lifetime-view signals. The score also uses confidence and eligibility checks where supported by the ranking implementation.

Breakout Velocity Index (BVI)

The Breakout Velocity Index identifies artists showing growth potential based on available velocity signals. A 10,000-subscriber floor is used by the BVI logic to reduce the impact of very small accounts.

Note: The project's internal scoring methodology is not necessarily identical to the public HIFFI website's ranking formula.

Identity Verification Gates

Only Instagram profiles that pass the configured verification rules qualify for multi-platform credit.

• AUTOMATED_VERIFIED: Confidence ≥ 0.55, or confidence ≥ 0.40 with follower corroboration ≥ 10,000.
• UNVERIFIED: Accounts that do not meet verification requirements.
• UNCERTAIN_NEEDS_REVIEW: Ambiguous account matches requiring human review.

Unverified accounts, low-follower outliers below 10,000, and ambiguous handles are routed to data/review_queue.csv for review and excluded from multi-platform tier credit.

Follower count is a supporting signal, not proof of identity. The verification status and publication eligibility rules determine whether an account can contribute evidence.

Data Freshness and Publication Eligibility

The ranking engine uses freshness windows to determine whether platform data qualifies as current evidence.

Platform	Maximum Data Age
YouTube	7 days
Spotify	7 days
Instagram	14 days

Missing, unparseable, or stale timestamps do not qualify for platform evidence credit. The engine records exclusion reasons in evidence_exclusions, such as STALE_OR_MISSING_SPOTIFY_TIMESTAMP.

A platform's evidence may be excluded from publication eligibility while its available metric can still contribute to a continuous score. Therefore, score calculation and publication eligibility are separate decisions.

Missing-Data Handling

• Preserve missing values as NaN where supported by the data and scoring logic.
• Do not treat failed API requests as genuine zero-valued metrics.
• Redistribute weights across available signals according to the scoring implementation.
• Record effective weights so ranking results can be inspected and explained.

Some individual scoring functions may apply documented defaults. Review the relevant implementation before assuming every missing value remains NaN throughout all calculations.

Anomaly Detection

Before publication, the pipeline flags suspicious cross-platform metric relationships for audit review.

Flag	Condition
SUSPICIOUS_IG_TO_YT_RATIO	Instagram followers exceed YouTube lifetime views by more than 10×.
EXTREME_SPOTIFY_VELOCITY_RATIO	Daily stream growth exceeds 25% of monthly listeners.

Flags are recorded in anomaly_flags so suspicious records can be investigated rather than silently ignored. A flag should trigger review; it does not, by itself, prove that the data is fraudulent.

Regression Auditing

The regression auditor compares current ranking output against saved baselines.

• Baselines are stored in data/snapshots/.
• Rank movements greater than 100 positions (|rank shift| > 100) are identified for review.
• The latest report is written to reports/audit_report_latest.md.

Large ranking changes may reflect genuine changes in platform signals, data availability, scoring weights, or identity verification. They should be reviewed before publication.

Project Structure

hiffi-artist-intelligence/
├── collectors/
│   ├── instagram_verifier.py
│   ├── spotify_api_collector.py
│   └── spotify_fallback_collector.py
├── core/
│   ├── ranking_engine.py
│   └── regression_auditor.py
├── data/
│   ├── checkpoints/
│   ├── snapshots/
│   ├── review_queue.csv
│   └── hiffi_multiplatform_500.csv
├── pipeline/
│   └── batch_500_runner.py
├── reports/
│   └── audit_report_latest.md
├── tests/
├── run_pipeline.py
├── test_multiplatform_ranking.py
├── requirements.txt
├── .env.example
└── README.md

The tree shows the intended layout. Keep only files and folders that actually exist in the repository.

Requirements

• Python 3.11 or 3.12
• Git
• API credentials for any integrations that require authentication

Setup

1. Clone the repository

git clone <YOUR_REPOSITORY_URL>
cd hiffi-artist-intelligence

Replace <YOUR_REPOSITORY_URL> with your repository's clone URL.

2. Create a virtual environment

Windows PowerShell

python -m venv .venv
.\.venv\Scripts\Activate.ps1

macOS/Linux

python3 -m venv .venv
source .venv/bin/activate

3. Install dependencies

python -m pip install --upgrade pip
pip install -r requirements.txt

4. Configure environment variables

Create a local .env file using .env.example as a guide. Add the credentials required by your enabled integrations, for example:

SPOTIFY_CLIENT_ID=your_spotify_client_id
SPOTIFY_CLIENT_SECRET=your_spotify_client_secret

Use the actual variable names expected by your collectors. Never commit .env files or live API credentials to Git.

Running the Pipeline

Refresh platform data

If pipeline/batch_500_runner.py is implemented as the ingestion entry point, run:

python -m pipeline.batch_500_runner

This stage should refresh the master dataset and use checkpoints where supported.

Calculate rankings and generate the audit report

python run_pipeline.py

This stage consumes the available master dataset, applies scoring and governance rules, and produces ranking outputs and the audit report.

Run tests

python -m pytest test_multiplatform_ranking.py -v

Run the regression auditor

python -m core.regression_auditor

Use the exact commands and test paths defined by the current repository.

Testing and Continuous Integration

The project reports 27 passing unit and integration tests, with GitHub Actions checks passing on Python 3.11 and 3.12 on Ubuntu.

The test suite is intended to cover ranking behavior, missing-data handling, freshness and eligibility gates, anomaly detection, and regression safeguards.

Test results should be confirmed against the latest commit before a release is declared complete.

Important Output Fields

Field	Purpose
production_hiffi_score	YouTube production-performance score.
hiffi_xp_score	Combined multi-platform score.
hiffi_xp_rank	Raw multi-platform ranking position.
hiffi_xp_published_rank	Published ranking position after eligibility rules.
breakout_velocity_index	Breakout potential score.
breakout_rank	Breakout ranking position.
effective_weights	Weights applied to the available signals.
platform_evidence_tier	Platform evidence tier used for eligibility.
evidence_exclusions	Reasons platform evidence was excluded.
anomaly_flags	Potentially suspicious metric relationships.
rank_delta	Change in ranking position relative to the comparison baseline.

Actual output columns depend on the current implementation and input dataset.

Security and Data Governance

• Keep API keys in environment variables, not source code.
• Keep .env and local credentials out of version control.
• Review uncertain artist identities before granting platform evidence credit.
• Keep data timestamps and exclusion reasons available for auditing.
• Investigate large rank shifts and anomaly flags before publication.
• Do not interpret a missing metric as a confirmed zero unless the relevant scoring rule explicitly defines that behavior.

Limitations

• API access, quotas, and authentication can affect data availability.
• Platform metrics may have different refresh schedules and definitions.
• Similar artist names and unofficial social accounts can produce ambiguous identity matches.
• Anomaly rules identify unusual relationships, not necessarily fraudulent activity.
• Rankings depend on data quality, available signals, and the configured weighting methodology.
• Automated tests cannot guarantee that every live platform account or metric is correct.

Future Improvements

• Expand identity verification using reliable official account evidence.
• Improve source-level data validation and stale-data monitoring.
• Add operational alerts for failed or incomplete ingestion runs.
• Strengthen checkpoint recovery and API retry behavior.
• Track ranking changes across releases and review significant shifts.
• Automate recurring regression reports and release checks.

Production Deployment & Operations Runbook

Dashboard Service (systemd)

The multi-platform Streamlit dashboard runs in the background as a Linux systemd service.

Check service status

sudo systemctl status hiffi-dashboard --no-pager

Restart after approved code or dataset updates

sudo systemctl restart hiffi-dashboard

View live application logs

sudo journalctl -u hiffi-dashboard -f

View the latest 100 log entries

sudo journalctl -u hiffi-dashboard -n 100 --no-pager

Enable automatic startup after server reboot

sudo systemctl enable hiffi-dashboard

Server Health Checks

Check available disk space:

df -h

Check memory usage:

free -h

List running application sessions:

tmux ls

Operational Notes

• Confirm the service is active before restarting or stopping it.
• Check application logs if the dashboard becomes unavailable.
• Verify dashboard access after any restart or deployment.
• Back up important files before making production changes.
• Do not expose API keys, passwords, or other secrets in the repository.
• Confirm the configured service name and deployment paths before running commands.

Summary

HIFFI Hip-Hop 500 separates platform data ingestion from ranking and governance. Its scoring, identity checks, freshness controls, anomaly flags, and regression reports are designed to make multi-platform rankings more consistent, explainable, and auditable.

For production use, verify that the documented entry points, thresholds, checkpoint behavior, and output paths match the implementation in the released repository.