28 KiB
HIFFI Artist Intelligence Engine
Multi-Platform Artist Intelligence, Commercial Analytics & Simulation
HIFFI Artist Intelligence Engine is a Python-based analytics platform designed to evaluate the top 500 hip-hop artists using multi-platform performance signals and commercial sales intelligence.
The system combines YouTube, Spotify, and Instagram telemetry with commercial sales data to provide artist rankings, commercial performance analysis, isolated ranking simulations, data governance, and executive audit reporting.
It includes an automated data pipeline, a FastAPI backend, an interactive Streamlit dashboard, executive audit exporters, and a PM2-managed deployment environment.
1. System Architecture
The platform separates data ingestion, data governance, ranking, commercial simulation, and application services into independent components.
DATA INGESTION SOURCES
|
+-------------------+-------------------+
| | |
YouTube Spotify Instagram
| | |
+-------------------+-------------------+
|
COMMERCIAL DATA SOURCES
|
+-------------------+-------------------+
| | |
ChartMasters RIAA Luminate
| | |
+-------------------+-------------------+
|
v
DATA GOVERNANCE & HYGIENE
core/health_detector.py
|
- Missing-data validation
- Artist identity verification
- Alias and handle overrides
- Dynamic weight redistribution
|
v
MASTER COHORT DATASET
data/hiffi_multiplatform_500.csv
|
+------------+------------+
| |
v v
RANKING ENGINE COMMERCIAL SIMULATION
Multi-platform scoring core/cpi_engine.py
| |
| - CPI calculation
| - Dynamic re-ranking
| - Sensitivity analysis
| |
+------------+------------+
|
+----------+----------+
| |
v v
FASTAPI SERVICE AUDIT EXPORTERS
api.py exporters/
| |
| Excel / Markdown
| |
+----------+----------+
|
STREAMLIT UI
app.py
|
PM2 PROCESS MANAGER
|
+------------+------------+
| |
Port 8000 Port 8501
FastAPI Streamlit
Core Components
| Component | Responsibility |
|---|---|
| Data Collectors | Collect platform and commercial telemetry. |
| Data Governance | Validate metrics, identities, freshness, and missing values. |
| Ranking Engine | Calculate multi-platform artist scores and rankings. |
| CPI Engine | Normalize commercial sales performance. |
| Simulation Engine | Evaluate hypothetical ranking changes without modifying baseline rankings. |
| FastAPI | Expose ranking, telemetry, and simulation endpoints. |
| Streamlit | Provide an interactive analytics dashboard. |
| Exporters | Generate executive Excel and Markdown audit reports. |
| PM2 | Manage API and dashboard processes. |
2. Ranking Methodology
The HIFFI ranking system evaluates artists through three analytical dimensions:
- YouTube Production Performance.
- Multi-Platform Artist Performance (HIFFI-XP).
- Breakout Velocity Index (BVI).
Each dimension serves a different purpose and is calculated independently.
A. YouTube Production Score
The YouTube Production Score evaluates artist performance using four components.
| Component | Weight |
|---|---|
| Recent Momentum | 35% |
| View Velocity | 30% |
| Audience Reach | 20% |
| Consistency Proxy | 15% |
| Total | 100% |
Scoring Formula
Production Score =
(Recent Momentum × 0.35)
+ (View Velocity × 0.30)
+ (Audience Reach × 0.20)
+ (Consistency Proxy × 0.15)
Where:
- Recent Momentum: Measures recent performance and growth.
- View Velocity: Measures the rate at which video views are generated.
- Audience Reach: Combines subscriber and lifetime-view signals.
- Consistency Proxy: Represents consistency in available performance signals.
Each component contributes according to its assigned weight.
Example Calculation
Consider an artist with the following normalized component scores:
| Component | Score | Weight |
|---|---|---|
| Recent Momentum | 80 | 35% |
| View Velocity | 70 | 30% |
| Audience Reach | 90 | 20% |
| Consistency Proxy | 60 | 15% |
Production Score =
(80 × 0.35)
+ (70 × 0.30)
+ (90 × 0.20)
+ (60 × 0.15)
Production Score =
28 + 21 + 18 + 9
Production Score = 76
The resulting production score is 76.
The normalized component values are illustrative; the actual normalization and preprocessing rules are determined by the ranking implementation.
B. HIFFI-XP Multi-Platform Score
The HIFFI-XP scoring engine evaluates artist performance across YouTube, Spotify, and Instagram.
Unlike the YouTube Production Score, this calculation combines multiple platform signals into one multi-platform score.
Base Signal Weights
| Platform Signal | Base Weight |
|---|---|
| YouTube Production Score | 50% |
| Spotify Monthly Listeners | 25% |
| Spotify Daily Stream Velocity | 5% |
| Instagram Followers | 20% |
| Total | 100% |
Scoring Formula
HIFFI-XP Score =
(YouTube Production Score × 0.50)
+ (Spotify Monthly Listeners Score × 0.25)
+ (Spotify Daily Stream Velocity Score × 0.05)
+ (Instagram Followers Score × 0.20)
The platform inputs in this formula represent the corresponding normalized scoring values, rather than raw listener or follower counts.
Example Calculation
Assume the following normalized platform scores:
| Platform Signal | Score | Weight |
|---|---|---|
| YouTube Production | 80 | 50% |
| Spotify Monthly Listeners | 70 | 25% |
| Spotify Daily Stream Velocity | 90 | 5% |
| Instagram Followers | 60 | 20% |
HIFFI-XP Score =
(80 × 0.50)
+ (70 × 0.25)
+ (90 × 0.05)
+ (60 × 0.20)
HIFFI-XP Score =
40 + 17.5 + 4.5 + 12
HIFFI-XP Score = 74
The resulting HIFFI-XP score is 74.
C. Dynamic Weight Redistribution
Not every artist has complete data across all platforms.
The scoring engine handles missing platform signals by redistributing the weights of the available signals.
This prevents unavailable metrics from automatically reducing an artist's score.
Redistribution Formula
Effective Weight =
(Original Weight / Sum of Available Original Weights) × 100
The total effective weight is normalized to 100%.
Example
Suppose an artist has YouTube, Spotify Monthly Listeners, and Instagram data, but Spotify Daily Stream Velocity is unavailable.
The available original weights are:
YouTube = 50%
Spotify Listeners = 25%
Instagram = 20%
Available Weight = 95%
The redistributed weights become:
YouTube =
(50 / 95) × 100
= 52.63%
Spotify Listeners =
(25 / 95) × 100
= 26.32%
Instagram =
(20 / 95) × 100
= 21.05%
Resulting Effective Weights
| Available Signal | Effective Weight |
|---|---|
| YouTube Production Score | 52.63% |
| Spotify Monthly Listeners | 26.32% |
| Instagram Followers | 21.05% |
| Spotify Daily Stream Velocity | Excluded |
| Total | 100% |
The resulting score is calculated using the available signals and their redistributed weights.
HIFFI-XP Score =
(YouTube Score × 0.5263)
+ (Spotify Listeners Score × 0.2632)
+ (Instagram Score × 0.2105)
The effective weights are recorded in effective_weights for auditability.
D. Breakout Velocity Index (BVI)
The Breakout Velocity Index identifies artists demonstrating growth potential based on available velocity signals.
The BVI is a separate analytical measure and is not interchangeable with the HIFFI-XP multi-platform score.
Key considerations:
- Evaluates available artist growth and velocity signals.
- Uses a 10,000-subscriber eligibility floor.
- Reduces the influence of very small accounts.
- Generates a separate breakout score and ranking.
Important: The exact BVI component weights and calculation should follow the implemented BVI scoring function. No additional formula is assumed where the implementation does not define one.
E. Ranking Output
The ranking engine generates separate outputs for artist performance and publication eligibility.
| Output Field | Description |
|---|---|
production_hiffi_score |
YouTube production-performance score. |
hiffi_xp_score |
Combined multi-platform score. |
hiffi_xp_rank |
Raw multi-platform ranking position. |
hiffi_xp_published_rank |
Published ranking position after eligibility checks. |
breakout_velocity_index |
Artist breakout potential score. |
breakout_rank |
Breakout ranking position. |
Note: Internal HIFFI scoring methodology is separate from the public HIFFI website's ranking methodology.
3. Commercial Power Index (CPI)
The Commercial Power Index introduces commercial sales intelligence into the artist analytics platform.
It evaluates the commercial performance of artists using Equivalent Album Sales (EAS) obtained from available sales and certification sources.
The objective is to normalize commercial performance across artists with significantly different sales volumes.
CPI Calculation
The engine applies base-10 logarithmic normalization to convert EAS values into a standardized score between 0 and 100.
Formula
CPI = clip(
((log10(max(EAS, 1.0)) - MIN_LOG)
/ (MAX_LOG - MIN_LOG)) × 100,
0,
100
)
Reference Values
| Parameter | Value | Description |
|---|---|---|
| MIN_LOG | 5.000000 | Logarithm of 100,000 EAS. |
| MAX_LOG | 8.397940 | Logarithm of 250,000,000 EAS. |
| Minimum Reference EAS | 100,000 | Lower normalization boundary. |
| Maximum Reference EAS | 250,000,000 | Upper normalization boundary. |
The clip operation ensures that the resulting CPI remains within the range of 0 to 100.
Commercial Sales Data Sources
| Source | Purpose |
|---|---|
| ChartMasters CSPC | Global commercial sales and equivalent album sales estimates. |
| RIAA | US domestic album and digital certification milestones. |
| Luminate | Commercial music consumption and sales intelligence. |
| Kworb | Spotify streaming telemetry and listener activity. |
Sales Provenance Tiers
Commercial data is classified according to its source and evidence level.
| Tier | Classification | Artists |
|---|---|---|
| Tier 1 | VERIFIED_CSPC |
62 |
| Tier 2 | VERIFIED_RIAA |
259 |
| Tier 3 | ESTIMATED_SEA |
179 |
| Total | 500 |
Provenance Definitions
VERIFIED_CSPC: Global comprehensive sales data sourced from ChartMasters.VERIFIED_RIAA: US domestic certified album and digital milestones.ESTIMATED_SEA: Estimated streaming-equivalent album sales using configured conversion assumptions.
Streaming conversion assumptions:
- 1,500 audio streams = 1 EAS.
- 3,750 video streams = 1 EAS.
These are configured estimation assumptions, not universal industry accounting standards.
Sales provenance tiers distinguish the strength and coverage of available commercial evidence.
4. Isolated Dynamic Simulation Engine
The simulation engine evaluates how commercial performance could influence artist rankings under different commercial weighting scenarios.
It operates independently of the baseline published ranking system.
Simulation Methodology
The commercial weight (w_sales) is configurable between 0% and 50%.
A. Base Weight Calculation
The base weight represents the remaining contribution of the original multi-platform score.
Base Weight = 1 - Commercial Weight
For example:
Commercial Weight = 0.20
Base Weight = 1 - 0.20
= 0.80
B. Simulated Score Calculation
When CPI data is available:
Simulated Score =
(Base Weight × Base Score)
+ (Commercial Weight × CPI)
When CPI data is missing:
Simulated Score = Base Score
This ensures that artists without commercial sales data retain their original base score rather than receiving an artificial zero CPI contribution.
C. Ranking Movement Calculation
Rank Delta = Published Rank - Simulated Rank
A positive rank delta indicates an improvement in ranking position, while a negative value indicates a decline.
Simulation Example
Consider an artist with the following values:
| Parameter | Value |
|---|---|
| Base Score | 80 |
| CPI | 60 |
| Commercial Weight | 20% (0.20) |
| Base Weight | 80% (0.80) |
Calculation:
Simulated Score =
(0.80 × 80) + (0.20 × 60)
Simulated Score =
64 + 12
Simulated Score = 76
The resulting simulated score is 76.
The simulation compares this result against the original ranking to identify potential rank movements.
Simulation Features
- Adjustable commercial weighting.
- Dynamic weight re-ranking.
- Global redistribution mode.
- Commercial sensitivity analysis.
- Rank movement comparison.
- Baseline ranking preservation.
Baseline Invariance
Simulations do not overwrite or modify the original published ranking field:
hiffi_xp_published_rank
This ensures that commercial simulations remain isolated from the production ranking methodology.
Simulation results represent hypothetical ranking scenarios and do not automatically change official HIFFI rankings.
5. Data Governance & Validation
Data governance ensures that ranking and commercial calculations use traceable and validated inputs.
Zero-Imputation Policy
Missing streaming telemetry is preserved as NaN wherever supported by the scoring implementation.
Failed API requests or unavailable platform metrics must not automatically be interpreted as genuine zero values.
Artist Identity Verification
Instagram profiles must pass configured identity verification rules before receiving multi-platform evidence credit.
Supported verification statuses include:
AUTOMATED_VERIFIEDUNVERIFIEDUNCERTAIN_NEEDS_REVIEW
Ambiguous artist matches and uncertain handles are routed to:
data/review_queue.csv
Artist follower counts are supporting evidence, not independent proof of identity.
Freshness Validation
| Platform | Maximum Data Age |
|---|---|
| YouTube | 7 days |
| Spotify | 7 days |
| 14 days |
Missing, invalid, or stale timestamps are recorded as evidence exclusions.
Example:
STALE_OR_MISSING_SPOTIFY_TIMESTAMP
Scoring eligibility and publication eligibility are treated as separate decisions.
Dynamic Weight Governance
The system records the effective weights applied to each artist.
This makes it possible to inspect how missing platform signals influence individual ranking calculations.
Ranking Continuity
The baseline ranking system maintains unique, contiguous ranking positions from 1 to 500, without duplicate ranks or gaps.
6. Anomaly Detection & Regression Auditing
Cross-Platform Anomaly Detection
The system identifies unusual relationships between platform metrics.
| Anomaly Flag | Detection Condition |
|---|---|
SUSPICIOUS_IG_TO_YT_RATIO |
Instagram followers exceed YouTube lifetime views by more than 10x. |
EXTREME_SPOTIFY_VELOCITY_RATIO |
Daily stream growth exceeds 25% of monthly listeners. |
Anomaly flags are recorded for investigation and do not independently establish fraudulent activity.
Regression Auditing
The regression auditor compares current ranking outputs against previously saved ranking snapshots.
Key checks include:
- Significant ranking movements.
- Rank changes greater than 100 positions.
- Changes in platform evidence availability.
- Potential scoring inconsistencies.
- Ranking continuity and data integrity.
Baselines are maintained in:
data/snapshots/
The latest audit report is generated at:
reports/audit_report_latest.md
Large ranking changes should be reviewed before publication.
7. REST API Reference
The FastAPI service exposes endpoints for system health, telemetry coverage, artist rankings, and commercial simulations.
Service: FastAPI
Default Port: 8000
Application Entry Point: api:app
Available Endpoints
| Method | Endpoint | Description |
|---|---|---|
| GET | /health |
Service health and total artist cohort count. |
| GET | /telemetry/coverage |
Data provenance tier counts and coverage percentages. |
| GET | /artists |
Ranked artist catalog. |
| POST | /simulate |
Executes an isolated commercial ranking simulation. |
Artist Endpoint Parameters
GET /artists
| Parameter | Description |
|---|---|
scope |
Artist selection: all or verified. |
limit |
Maximum number of artists, between 1 and 500. |
vertical |
Optional vertical filter. |
Simulation Request
POST /simulate
Example request:
{
"commercial_weight": 0.15,
"mode": "global_redistribution"
}
The simulation accepts a commercial weighting value and execution mode to evaluate alternative ranking scenarios.
Example API Requests
Check service health
Invoke-RestMethod -Uri "http://localhost:8000/health"
Check telemetry coverage
Invoke-RestMethod -Uri "http://localhost:8000/telemetry/coverage"
View the artist catalog
Invoke-RestMethod -Uri "http://localhost:8000/artists?scope=all&limit=10"
8. Executive Audit & Reporting
The platform generates structured reports for commercial analysis and data governance.
Excel Executive Sales Audit
File:
reports/hiffi_executive_sales_audit.xlsx
Provides structured commercial sales and artist-level analysis.
Sales Provenance Audit
File:
reports/sales_provenance_audit.md
Documents commercial source classifications and data provenance.
Ranking Audit Report
File:
reports/audit_report_latest.md
Contains ranking regression findings, rank movements, and data quality observations.
These reports support traceability and review of ranking and commercial simulation outputs.
9. Project Structure
hiffi-artist-intelligence/
│
├── collectors/
│ ├── instagram_verifier.py
│ ├── spotify_api_collector.py
│ └── spotify_fallback_collector.py
│
├── core/
│ ├── ranking_engine.py
│ ├── regression_auditor.py
│ ├── health_detector.py
│ └── cpi_engine.py
│
├── exporters/
│ └── excel_workbook.py
│
├── pipeline/
│ └── batch_500_runner.py
│
├── data/
│ ├── checkpoints/
│ ├── snapshots/
│ ├── review_queue.csv
│ ├── handle_overrides.json
│ └── hiffi_multiplatform_500.csv
│
├── reports/
│ ├── audit_report_latest.md
│ ├── hiffi_executive_sales_audit.xlsx
│ └── sales_provenance_audit.md
│
├── tests/
│ └── test_platform_invariants.py
│
├── app.py
├── api.py
├── run_pipeline.py
├── audit_system_integrity.py
├── ecosystem.config.js
├── requirements.txt
├── .env.example
└── README.md
The project structure represents the documented architecture. Confirm filenames and folders against the current repository before treating this tree as exhaustive.
10. Requirements & Setup
Prerequisites
- Python 3.11 or 3.12
- Git
- Node.js and npm
- PM2
- Required API credentials for enabled integrations
Step 1: Clone the Repository
git clone <YOUR_REPOSITORY_URL>
cd hiffi-artist-intelligence
Replace the placeholder with the repository's actual clone URL.
Step 2: Create a Virtual Environment
Windows PowerShell
python -m venv .venv
.\.venv\Scripts\Activate.ps1
macOS/Linux
python3 -m venv .venv
source .venv/bin/activate
Step 3: Install Python Dependencies
python -m pip install --upgrade pip
pip install -r requirements.txt
Step 4: Configure Environment Variables
Create a local .env file using .env.example as a reference.
Add the required credentials for integrations that need authentication.
Example:
SPOTIFY_CLIENT_ID=your_spotify_client_id
SPOTIFY_CLIENT_SECRET=your_spotify_client_secret
Use the exact environment variable names expected by the configured collectors.
Never commit API keys, credentials, or .env files to Git.
Step 5: Install PM2
npm install -g pm2
11. Running the Application
Refresh Platform Data
Run the ingestion pipeline:
python -m pipeline.batch_500_runner
This stage collects available platform telemetry and refreshes the master artist dataset.
Execute the Ranking Pipeline
python run_pipeline.py
This stage processes the available dataset, calculates ranking scores, applies governance rules, and generates audit outputs.
Launch the FastAPI Service
uvicorn api:app --host 0.0.0.0 --port 8000
Launch the Streamlit Dashboard
streamlit run app.py --server.port 8501 --server.address 0.0.0.0
The two services can also be managed together through PM2.
12. Testing & Continuous Integration
The project includes automated tests for mathematical invariants, ranking behavior, and data governance.
Run the Test Suite
pytest tests/test_platform_invariants.py -v
Run the System Integrity Audit
python audit_system_integrity.py
Continuous Integration
GitHub Actions is configured to validate the project across:
- Python 3.11
- Python 3.12
- Ubuntu environments
The previously reported validation baseline includes 27 passing tests and successful CI checks on both Python versions.
Always verify the latest test results against the current repository commit before deployment.
13. Production Deployment & Operations
The application uses PM2 to supervise the FastAPI backend and Streamlit dashboard.
PM2 Process Configuration
The ecosystem.config.js file defines the application processes.
| Service | Port | Framework |
|---|---|---|
| API Service | 8000 | FastAPI |
| Dashboard | 8501 | Streamlit |
Start Both Services
pm2 start ecosystem.config.js
Check Process Status
pm2 status
Restart Services
pm2 restart ecosystem.config.js
View Logs
pm2 logs
Save Process Configuration
pm2 save
Verify API Availability
Invoke-RestMethod -Uri "http://localhost:8000/health"
Verify Telemetry Coverage
Invoke-RestMethod -Uri "http://localhost:8000/telemetry/coverage"
Operational Checklist
- Confirm both processes are running.
- Verify API health and telemetry endpoints.
- Check dashboard accessibility.
- Review application logs after deployment.
- Validate the dataset before restarting services.
- Preserve baseline ranking data before running simulations.
- Keep API credentials outside the repository.
14. Important Output Fields
| Field | Purpose |
|---|---|
production_hiffi_score |
YouTube production-performance score. |
hiffi_xp_score |
Combined multi-platform artist score. |
hiffi_xp_rank |
Raw multi-platform ranking position. |
hiffi_xp_published_rank |
Published rank after eligibility checks. |
breakout_velocity_index |
Artist breakout potential score. |
breakout_rank |
Breakout ranking position. |
effective_weights |
Weights applied to available platform signals. |
platform_evidence_tier |
Platform evidence classification. |
evidence_exclusions |
Reasons for excluding platform evidence. |
anomaly_flags |
Potentially suspicious metric relationships. |
rank_delta |
Difference between comparison ranking positions. |
CPI |
Normalized Commercial Power Index. |
sales_provenance_tier |
Commercial sales evidence classification. |
Actual output fields depend on the current implementation and input dataset.
15. Security & Data Governance
The system follows basic operational and data governance practices.
- Store API credentials in environment variables.
- Never commit secrets or
.envfiles. - Preserve missing values instead of introducing synthetic zeros.
- Validate artist identity before assigning platform evidence credit.
- Maintain source timestamps and sales provenance.
- Investigate anomaly flags and significant ranking movements.
- Preserve published baseline rankings during simulations.
- Review commercial estimates before interpreting them as verified sales.
16. Limitations
- Platform API access, authentication, and request quotas can affect data availability.
- Platform metrics may have different refresh schedules and definitions.
- Artist aliases and unofficial social accounts can create identity-matching challenges.
- Commercial sales estimates may vary across data providers.
- RIAA certifications represent US domestic certification evidence, not complete global sales.
- Estimated streaming-equivalent sales depend on configured conversion assumptions.
- Anomaly detection identifies unusual relationships but does not prove fraudulent activity.
- Simulation results are hypothetical and do not automatically change official published rankings.
- Automated tests cannot guarantee the accuracy of every live platform metric or commercial estimate.
17. Future Improvements
- Expand reliable multi-platform identity verification.
- Improve commercial sales source reconciliation and provenance validation.
- Introduce additional artist growth and engagement signals.
- Strengthen ingestion retry mechanisms and checkpoint recovery.
- Improve automated anomaly detection and data freshness monitoring.
- Expand commercial sensitivity and simulation reporting.
- Introduce operational alerts for failed or incomplete pipeline runs.
- Explore LLM-based artist skill extraction and music-industry demand/supply intelligence.
- Track changes in artist skills, genre demand, and market opportunities over time.
Summary
HIFFI Artist Intelligence Engine combines multi-platform artist performance analytics with commercial sales intelligence in a modular and auditable architecture.
Its ranking engine evaluates streaming and audience signals through weighted scoring, while the Commercial Power Index introduces normalized commercial performance into an isolated simulation environment.
With automated data governance, provenance tracking, regression auditing, REST API services, executive reporting, and an interactive dashboard, the platform provides a structured foundation for artist intelligence and commercial analytics.
Built on measurable signals, validated evidence, and explainable analytics.