HIFFI Artist Intelligence Engine

Multi-Platform Artist Intelligence, Commercial Analytics & Simulation

HIFFI Artist Intelligence Engine is a Python-based analytics platform designed to evaluate the top 500 hip-hop artists using multi-platform performance signals and commercial sales intelligence.

The system combines YouTube, Spotify, and Instagram telemetry with commercial sales data to provide artist rankings, commercial performance analysis, isolated ranking simulations, data governance, and executive audit reporting.

It includes an automated data pipeline, a FastAPI backend, an interactive Streamlit dashboard, executive audit exporters, and a PM2-managed deployment environment.

---

1. System Architecture

The platform separates data ingestion, data governance, ranking, commercial simulation, and application services into independent components.

                    DATA INGESTION SOURCES
                              |
          +-------------------+-------------------+
          |                   |                   |
       YouTube              Spotify            Instagram
          |                   |                   |
          +-------------------+-------------------+
                              |
                    COMMERCIAL DATA SOURCES
                              |
          +-------------------+-------------------+
          |                   |                   |
      ChartMasters            RIAA              Luminate
          |                   |                   |
          +-------------------+-------------------+
                              |
                              v
                    DATA GOVERNANCE & HYGIENE
                       core/health_detector.py
                              |
                    - Missing-data validation
                    - Artist identity verification
                    - Alias and handle overrides
                    - Dynamic weight redistribution
                              |
                              v
                      MASTER COHORT DATASET
                  data/hiffi_multiplatform_500.csv
                              |
                 +------------+------------+
                 |                         |
                 v                         v
          RANKING ENGINE          COMMERCIAL SIMULATION
          Multi-platform scoring       core/cpi_engine.py
                 |                         |
                 |                    - CPI calculation
                 |                    - Dynamic re-ranking
                 |                    - Sensitivity analysis
                 |                         |
                 +------------+------------+
                              |
                   +----------+----------+
                   |                     |
                   v                     v
              FASTAPI SERVICE       AUDIT EXPORTERS
                  api.py              exporters/
                   |                     |
                   |                Excel / Markdown
                   |                     |
                   +----------+----------+
                              |
                       STREAMLIT UI
                          app.py
                              |
                      PM2 PROCESS MANAGER
                              |
                 +------------+------------+
                 |                         |
              Port 8000                 Port 8501
              FastAPI                   Streamlit

Core Components

Component	Responsibility
Data Collectors	Collect platform and commercial telemetry.
Data Governance	Validate metrics, identities, freshness, and missing values.
Ranking Engine	Calculate multi-platform artist scores and rankings.
CPI Engine	Normalize commercial sales performance.
Simulation Engine	Evaluate hypothetical ranking changes without modifying baseline rankings.
FastAPI	Expose ranking, telemetry, and simulation endpoints.
Streamlit	Provide an interactive analytics dashboard.
Exporters	Generate executive Excel and Markdown audit reports.
PM2	Manage API and dashboard processes.

---

2. Ranking Methodology

The HIFFI ranking system evaluates artists through three analytical dimensions:

1. YouTube Production Performance.
2. Multi-Platform Artist Performance (HIFFI-XP).
3. Breakout Velocity Index (BVI).

Each dimension serves a different purpose and is calculated independently.

A. YouTube Production Score

The YouTube Production Score evaluates artist performance using four components.

Component	Weight
Recent Momentum	35%
View Velocity	30%
Audience Reach	20%
Consistency Proxy	15%
Total	100%

Scoring Formula

Production Score =
    (Recent Momentum × 0.35)
  + (View Velocity × 0.30)
  + (Audience Reach × 0.20)
  + (Consistency Proxy × 0.15)

Where:

• Recent Momentum: Measures recent performance and growth.
• View Velocity: Measures the rate at which video views are generated.
• Audience Reach: Combines subscriber and lifetime-view signals.
• Consistency Proxy: Represents consistency in available performance signals.

Each component contributes according to its assigned weight.

Example Calculation

Consider an artist with the following normalized component scores:

Component	Score	Weight
Recent Momentum	80	35%
View Velocity	70	30%
Audience Reach	90	20%
Consistency Proxy	60	15%

Production Score =
    (80 × 0.35)
  + (70 × 0.30)
  + (90 × 0.20)
  + (60 × 0.15)

Production Score =
    28 + 21 + 18 + 9

Production Score = 76

The resulting production score is 76.

The normalized component values are illustrative; the actual normalization and preprocessing rules are determined by the ranking implementation.

B. HIFFI-XP Multi-Platform Score

The HIFFI-XP scoring engine evaluates artist performance across YouTube, Spotify, and Instagram.

Unlike the YouTube Production Score, this calculation combines multiple platform signals into one multi-platform score.

Base Signal Weights

Platform Signal	Base Weight
YouTube Production Score	50%
Spotify Monthly Listeners	25%
Spotify Daily Stream Velocity	5%
Instagram Followers	20%
Total	100%

Scoring Formula

HIFFI-XP Score =
    (YouTube Production Score × 0.50)
  + (Spotify Monthly Listeners Score × 0.25)
  + (Spotify Daily Stream Velocity Score × 0.05)
  + (Instagram Followers Score × 0.20)

The platform inputs in this formula represent the corresponding normalized scoring values, rather than raw listener or follower counts.

Example Calculation

Assume the following normalized platform scores:

Platform Signal	Score	Weight
YouTube Production	80	50%
Spotify Monthly Listeners	70	25%
Spotify Daily Stream Velocity	90	5%
Instagram Followers	60	20%

HIFFI-XP Score =
    (80 × 0.50)
  + (70 × 0.25)
  + (90 × 0.05)
  + (60 × 0.20)

HIFFI-XP Score =
    40 + 17.5 + 4.5 + 12

HIFFI-XP Score = 74

The resulting HIFFI-XP score is 74.

C. Dynamic Weight Redistribution

Not every artist has complete data across all platforms.

The scoring engine handles missing platform signals by redistributing the weights of the available signals.

This prevents unavailable metrics from automatically reducing an artist's score.

Redistribution Formula

Effective Weight =
    (Original Weight / Sum of Available Original Weights) × 100

The total effective weight is normalized to 100%.

Example

Suppose an artist has YouTube, Spotify Monthly Listeners, and Instagram data, but Spotify Daily Stream Velocity is unavailable.

The available original weights are:

YouTube              = 50%
Spotify Listeners    = 25%
Instagram            = 20%

Available Weight     = 95%

The redistributed weights become:

YouTube =
    (50 / 95) × 100
    = 52.63%

Spotify Listeners =
    (25 / 95) × 100
    = 26.32%

Instagram =
    (20 / 95) × 100
    = 21.05%

Resulting Effective Weights

Available Signal	Effective Weight
YouTube Production Score	52.63%
Spotify Monthly Listeners	26.32%
Instagram Followers	21.05%
Spotify Daily Stream Velocity	Excluded
Total	100%

The resulting score is calculated using the available signals and their redistributed weights.

HIFFI-XP Score =
    (YouTube Score × 0.5263)
  + (Spotify Listeners Score × 0.2632)
  + (Instagram Score × 0.2105)

The effective weights are recorded in effective_weights for auditability.

D. Breakout Velocity Index (BVI)

The Breakout Velocity Index identifies artists demonstrating growth potential based on available velocity signals.

The BVI is a separate analytical measure and is not interchangeable with the HIFFI-XP multi-platform score.

Key considerations:

• Evaluates available artist growth and velocity signals.
• Uses a 10,000-subscriber eligibility floor.
• Reduces the influence of very small accounts.
• Generates a separate breakout score and ranking.

Important: The exact BVI component weights and calculation should follow the implemented BVI scoring function. No additional formula is assumed where the implementation does not define one.

E. Ranking Output

The ranking engine generates separate outputs for artist performance and publication eligibility.

Output Field	Description
production_hiffi_score	YouTube production-performance score.
hiffi_xp_score	Combined multi-platform score.
hiffi_xp_rank	Raw multi-platform ranking position.
hiffi_xp_published_rank	Published ranking position after eligibility checks.
breakout_velocity_index	Artist breakout potential score.
breakout_rank	Breakout ranking position.

Note: Internal HIFFI scoring methodology is separate from the public HIFFI website's ranking methodology.

---

3. Commercial Power Index (CPI)

The Commercial Power Index introduces commercial sales intelligence into the artist analytics platform.

It evaluates the commercial performance of artists using Equivalent Album Sales (EAS) obtained from available sales and certification sources.

The objective is to normalize commercial performance across artists with significantly different sales volumes.

CPI Calculation

The engine applies base-10 logarithmic normalization to convert EAS values into a standardized score between 0 and 100.

Formula

CPI = clip(
    ((log10(max(EAS, 1.0)) - MIN_LOG)
    / (MAX_LOG - MIN_LOG)) × 100,
    0,
    100
)

Reference Values

Parameter	Value	Description
MIN_LOG	5.000000	Logarithm of 100,000 EAS.
MAX_LOG	8.397940	Logarithm of 250,000,000 EAS.
Minimum Reference EAS	100,000	Lower normalization boundary.
Maximum Reference EAS	250,000,000	Upper normalization boundary.

The clip operation ensures that the resulting CPI remains within the range of 0 to 100.

Commercial Sales Data Sources

Source	Purpose
ChartMasters CSPC	Global commercial sales and equivalent album sales estimates.
RIAA	US domestic album and digital certification milestones.
Luminate	Commercial music consumption and sales intelligence.
Kworb	Spotify streaming telemetry and listener activity.

Sales Provenance Tiers

Commercial data is classified according to its source and evidence level.

Tier	Classification	Artists
Tier 1	VERIFIED_CSPC	62
Tier 2	VERIFIED_RIAA	259
Tier 3	ESTIMATED_SEA	179
	Total	500

Provenance Definitions

• VERIFIED_CSPC: Global comprehensive sales data sourced from ChartMasters.
• VERIFIED_RIAA: US domestic certified album and digital milestones.
• ESTIMATED_SEA: Estimated streaming-equivalent album sales using configured conversion assumptions.

Streaming conversion assumptions:

• 1,500 audio streams = 1 EAS.
• 3,750 video streams = 1 EAS.

These are configured estimation assumptions, not universal industry accounting standards.

Sales provenance tiers distinguish the strength and coverage of available commercial evidence.

---

4. Isolated Dynamic Simulation Engine

The simulation engine evaluates how commercial performance could influence artist rankings under different commercial weighting scenarios.

It operates independently of the baseline published ranking system.

Simulation Methodology

The commercial weight (w_sales) is configurable between 0% and 50%.

A. Base Weight Calculation

The base weight represents the remaining contribution of the original multi-platform score.

Base Weight = 1 - Commercial Weight

For example:

Commercial Weight = 0.20
Base Weight       = 1 - 0.20
                  = 0.80

B. Simulated Score Calculation

When CPI data is available:

Simulated Score =
    (Base Weight × Base Score)
    + (Commercial Weight × CPI)

When CPI data is missing:

Simulated Score = Base Score

This ensures that artists without commercial sales data retain their original base score rather than receiving an artificial zero CPI contribution.

C. Ranking Movement Calculation

Rank Delta = Published Rank - Simulated Rank

A positive rank delta indicates an improvement in ranking position, while a negative value indicates a decline.

Simulation Example

Consider an artist with the following values:

Parameter	Value
Base Score	80
CPI	60
Commercial Weight	20% (0.20)
Base Weight	80% (0.80)

Calculation:

Simulated Score =
    (0.80 × 80) + (0.20 × 60)

Simulated Score =
    64 + 12

Simulated Score = 76

The resulting simulated score is 76.

The simulation compares this result against the original ranking to identify potential rank movements.

Simulation Features

• Adjustable commercial weighting.
• Dynamic weight re-ranking.
• Global redistribution mode.
• Commercial sensitivity analysis.
• Rank movement comparison.
• Baseline ranking preservation.

Baseline Invariance

Simulations do not overwrite or modify the original published ranking field:

hiffi_xp_published_rank

This ensures that commercial simulations remain isolated from the production ranking methodology.

Simulation results represent hypothetical ranking scenarios and do not automatically change official HIFFI rankings.

---

5. Data Governance & Validation

Data governance ensures that ranking and commercial calculations use traceable and validated inputs.

Zero-Imputation Policy

Missing streaming telemetry is preserved as NaN wherever supported by the scoring implementation.

Failed API requests or unavailable platform metrics must not automatically be interpreted as genuine zero values.

Artist Identity Verification

Instagram profiles must pass configured identity verification rules before receiving multi-platform evidence credit.

Supported verification statuses include:

• AUTOMATED_VERIFIED
• UNVERIFIED
• UNCERTAIN_NEEDS_REVIEW

Ambiguous artist matches and uncertain handles are routed to:

data/review_queue.csv

Artist follower counts are supporting evidence, not independent proof of identity.

Freshness Validation

Platform	Maximum Data Age
YouTube	7 days
Spotify	7 days
Instagram	14 days

Missing, invalid, or stale timestamps are recorded as evidence exclusions.

Example:

STALE_OR_MISSING_SPOTIFY_TIMESTAMP

Scoring eligibility and publication eligibility are treated as separate decisions.

Dynamic Weight Governance

The system records the effective weights applied to each artist.

This makes it possible to inspect how missing platform signals influence individual ranking calculations.

Ranking Continuity

The baseline ranking system maintains unique, contiguous ranking positions from 1 to 500, without duplicate ranks or gaps.

---

6. Anomaly Detection & Regression Auditing

Cross-Platform Anomaly Detection

The system identifies unusual relationships between platform metrics.

Anomaly Flag	Detection Condition
SUSPICIOUS_IG_TO_YT_RATIO	Instagram followers exceed YouTube lifetime views by more than 10x.
EXTREME_SPOTIFY_VELOCITY_RATIO	Daily stream growth exceeds 25% of monthly listeners.

Anomaly flags are recorded for investigation and do not independently establish fraudulent activity.

Regression Auditing

The regression auditor compares current ranking outputs against previously saved ranking snapshots.

Key checks include:

• Significant ranking movements.
• Rank changes greater than 100 positions.
• Changes in platform evidence availability.
• Potential scoring inconsistencies.
• Ranking continuity and data integrity.

Baselines are maintained in:

data/snapshots/

The latest audit report is generated at:

reports/audit_report_latest.md

Large ranking changes should be reviewed before publication.

---

7. REST API Reference

The FastAPI service exposes endpoints for system health, telemetry coverage, artist rankings, and commercial simulations.

Service: FastAPI
Default Port: 8000
Application Entry Point: api:app

Available Endpoints

Method	Endpoint	Description
GET	/health	Service health and total artist cohort count.
GET	/telemetry/coverage	Data provenance tier counts and coverage percentages.
GET	/artists	Ranked artist catalog.
POST	/simulate	Executes an isolated commercial ranking simulation.

Artist Endpoint Parameters

GET /artists

Parameter	Description
scope	Artist selection: all or verified.
limit	Maximum number of artists, between 1 and 500.
vertical	Optional vertical filter.

Simulation Request

POST /simulate

Example request:

{
  "commercial_weight": 0.15,
  "mode": "global_redistribution"
}

The simulation accepts a commercial weighting value and execution mode to evaluate alternative ranking scenarios.

Example API Requests

Check service health

Invoke-RestMethod -Uri "http://localhost:8000/health"

Check telemetry coverage

Invoke-RestMethod -Uri "http://localhost:8000/telemetry/coverage"

View the artist catalog

Invoke-RestMethod -Uri "http://localhost:8000/artists?scope=all&limit=10"

---

8. Executive Audit & Reporting

The platform generates structured reports for commercial analysis and data governance.

Excel Executive Sales Audit

File:

reports/hiffi_executive_sales_audit.xlsx

Provides structured commercial sales and artist-level analysis.

Sales Provenance Audit

File:

reports/sales_provenance_audit.md

Documents commercial source classifications and data provenance.

Ranking Audit Report

File:

reports/audit_report_latest.md

Contains ranking regression findings, rank movements, and data quality observations.

These reports support traceability and review of ranking and commercial simulation outputs.

---

9. Project Structure

hiffi-artist-intelligence/
│
├── collectors/
│   ├── instagram_verifier.py
│   ├── spotify_api_collector.py
│   └── spotify_fallback_collector.py
│
├── core/
│   ├── ranking_engine.py
│   ├── regression_auditor.py
│   ├── health_detector.py
│   └── cpi_engine.py
│
├── exporters/
│   └── excel_workbook.py
│
├── pipeline/
│   └── batch_500_runner.py
│
├── data/
│   ├── checkpoints/
│   ├── snapshots/
│   ├── review_queue.csv
│   ├── handle_overrides.json
│   └── hiffi_multiplatform_500.csv
│
├── reports/
│   ├── audit_report_latest.md
│   ├── hiffi_executive_sales_audit.xlsx
│   └── sales_provenance_audit.md
│
├── tests/
│   └── test_platform_invariants.py
│
├── app.py
├── api.py
├── run_pipeline.py
├── audit_system_integrity.py
├── ecosystem.config.js
├── requirements.txt
├── .env.example
└── README.md

The project structure represents the documented architecture. Confirm filenames and folders against the current repository before treating this tree as exhaustive.

---

10. Requirements & Setup

Prerequisites

• Python 3.11 or 3.12
• Git
• Node.js and npm
• PM2
• Required API credentials for enabled integrations

Step 1: Clone the Repository

git clone <YOUR_REPOSITORY_URL>
cd hiffi-artist-intelligence

Replace the placeholder with the repository's actual clone URL.

Step 2: Create a Virtual Environment

Windows PowerShell

python -m venv .venv
.\.venv\Scripts\Activate.ps1

macOS/Linux

python3 -m venv .venv
source .venv/bin/activate

Step 3: Install Python Dependencies

python -m pip install --upgrade pip
pip install -r requirements.txt

Step 4: Configure Environment Variables

Create a local .env file using .env.example as a reference.

Add the required credentials for integrations that need authentication.

Example:

SPOTIFY_CLIENT_ID=your_spotify_client_id
SPOTIFY_CLIENT_SECRET=your_spotify_client_secret

Use the exact environment variable names expected by the configured collectors.

Never commit API keys, credentials, or .env files to Git.

Step 5: Install PM2

npm install -g pm2

---

11. Running the Application

Refresh Platform Data

Run the ingestion pipeline:

python -m pipeline.batch_500_runner

This stage collects available platform telemetry and refreshes the master artist dataset.

Execute the Ranking Pipeline

python run_pipeline.py

This stage processes the available dataset, calculates ranking scores, applies governance rules, and generates audit outputs.

Launch the FastAPI Service

uvicorn api:app --host 0.0.0.0 --port 8000

Launch the Streamlit Dashboard

streamlit run app.py --server.port 8501 --server.address 0.0.0.0

The two services can also be managed together through PM2.

---

12. Testing & Continuous Integration

The project includes automated tests for mathematical invariants, ranking behavior, and data governance.

Run the Test Suite

pytest tests/test_platform_invariants.py -v

Run the System Integrity Audit

python audit_system_integrity.py

Continuous Integration

GitHub Actions is configured to validate the project across:

• Python 3.11
• Python 3.12
• Ubuntu environments

The previously reported validation baseline includes 27 passing tests and successful CI checks on both Python versions.

Always verify the latest test results against the current repository commit before deployment.

---

13. Production Deployment & Operations

The application uses PM2 to supervise the FastAPI backend and Streamlit dashboard.

PM2 Process Configuration

The ecosystem.config.js file defines the application processes.

Service	Port	Framework
API Service	8000	FastAPI
Dashboard	8501	Streamlit

Start Both Services

pm2 start ecosystem.config.js

Check Process Status

pm2 status

Restart Services

pm2 restart ecosystem.config.js

View Logs

pm2 logs

Save Process Configuration

pm2 save

Verify API Availability

Invoke-RestMethod -Uri "http://localhost:8000/health"

Verify Telemetry Coverage

Invoke-RestMethod -Uri "http://localhost:8000/telemetry/coverage"

Operational Checklist

• Confirm both processes are running.
• Verify API health and telemetry endpoints.
• Check dashboard accessibility.
• Review application logs after deployment.
• Validate the dataset before restarting services.
• Preserve baseline ranking data before running simulations.
• Keep API credentials outside the repository.

---

14. Important Output Fields

Field	Purpose
production_hiffi_score	YouTube production-performance score.
hiffi_xp_score	Combined multi-platform artist score.
hiffi_xp_rank	Raw multi-platform ranking position.
hiffi_xp_published_rank	Published rank after eligibility checks.
breakout_velocity_index	Artist breakout potential score.
breakout_rank	Breakout ranking position.
effective_weights	Weights applied to available platform signals.
platform_evidence_tier	Platform evidence classification.
evidence_exclusions	Reasons for excluding platform evidence.
anomaly_flags	Potentially suspicious metric relationships.
rank_delta	Difference between comparison ranking positions.
CPI	Normalized Commercial Power Index.
sales_provenance_tier	Commercial sales evidence classification.

Actual output fields depend on the current implementation and input dataset.

---

15. Security & Data Governance

The system follows basic operational and data governance practices.

• Store API credentials in environment variables.
• Never commit secrets or .env files.
• Preserve missing values instead of introducing synthetic zeros.
• Validate artist identity before assigning platform evidence credit.
• Maintain source timestamps and sales provenance.
• Investigate anomaly flags and significant ranking movements.
• Preserve published baseline rankings during simulations.
• Review commercial estimates before interpreting them as verified sales.

---

16. Limitations

• Platform API access, authentication, and request quotas can affect data availability.
• Platform metrics may have different refresh schedules and definitions.
• Artist aliases and unofficial social accounts can create identity-matching challenges.
• Commercial sales estimates may vary across data providers.
• RIAA certifications represent US domestic certification evidence, not complete global sales.
• Estimated streaming-equivalent sales depend on configured conversion assumptions.
• Anomaly detection identifies unusual relationships but does not prove fraudulent activity.
• Simulation results are hypothetical and do not automatically change official published rankings.
• Automated tests cannot guarantee the accuracy of every live platform metric or commercial estimate.

---

17. Future Improvements

• Expand reliable multi-platform identity verification.
• Improve commercial sales source reconciliation and provenance validation.
• Introduce additional artist growth and engagement signals.
• Strengthen ingestion retry mechanisms and checkpoint recovery.
• Improve automated anomaly detection and data freshness monitoring.
• Expand commercial sensitivity and simulation reporting.
• Introduce operational alerts for failed or incomplete pipeline runs.
• Explore LLM-based artist skill extraction and music-industry demand/supply intelligence.
• Track changes in artist skills, genre demand, and market opportunities over time.

---

Summary

HIFFI Artist Intelligence Engine combines multi-platform artist performance analytics with commercial sales intelligence in a modular and auditable architecture.

Its ranking engine evaluates streaming and audience signals through weighted scoring, while the Commercial Power Index introduces normalized commercial performance into an isolated simulation environment.

With automated data governance, provenance tracking, regression auditing, REST API services, executive reporting, and an interactive dashboard, the platform provides a structured foundation for artist intelligence and commercial analytics.

Built on measurable signals, validated evidence, and explainable analytics.