1000 lines
28 KiB
Markdown
1000 lines
28 KiB
Markdown
# HIFFI Artist Intelligence Engine
|
||
### Multi-Platform Artist Intelligence, Commercial Analytics & Simulation
|
||
|
||
HIFFI Artist Intelligence Engine is a Python-based analytics platform designed to evaluate the top 500 hip-hop artists using multi-platform performance signals and commercial sales intelligence.
|
||
|
||
The system combines YouTube, Spotify, and Instagram telemetry with commercial sales data to provide artist rankings, commercial performance analysis, isolated ranking simulations, data governance, and executive audit reporting.
|
||
|
||
It includes an automated data pipeline, a FastAPI backend, an interactive Streamlit dashboard, executive audit exporters, and a PM2-managed deployment environment.
|
||
|
||
---
|
||
|
||
## 1. System Architecture
|
||
|
||
The platform separates data ingestion, data governance, ranking, commercial simulation, and application services into independent components.
|
||
|
||
```text
|
||
DATA INGESTION SOURCES
|
||
|
|
||
+-------------------+-------------------+
|
||
| | |
|
||
YouTube Spotify Instagram
|
||
| | |
|
||
+-------------------+-------------------+
|
||
|
|
||
COMMERCIAL DATA SOURCES
|
||
|
|
||
+-------------------+-------------------+
|
||
| | |
|
||
ChartMasters RIAA Luminate
|
||
| | |
|
||
+-------------------+-------------------+
|
||
|
|
||
v
|
||
DATA GOVERNANCE & HYGIENE
|
||
core/health_detector.py
|
||
|
|
||
- Missing-data validation
|
||
- Artist identity verification
|
||
- Alias and handle overrides
|
||
- Dynamic weight redistribution
|
||
|
|
||
v
|
||
MASTER COHORT DATASET
|
||
data/hiffi_multiplatform_500.csv
|
||
|
|
||
+------------+------------+
|
||
| |
|
||
v v
|
||
RANKING ENGINE COMMERCIAL SIMULATION
|
||
Multi-platform scoring core/cpi_engine.py
|
||
| |
|
||
| - CPI calculation
|
||
| - Dynamic re-ranking
|
||
| - Sensitivity analysis
|
||
| |
|
||
+------------+------------+
|
||
|
|
||
+----------+----------+
|
||
| |
|
||
v v
|
||
FASTAPI SERVICE AUDIT EXPORTERS
|
||
api.py exporters/
|
||
| |
|
||
| Excel / Markdown
|
||
| |
|
||
+----------+----------+
|
||
|
|
||
STREAMLIT UI
|
||
app.py
|
||
|
|
||
PM2 PROCESS MANAGER
|
||
|
|
||
+------------+------------+
|
||
| |
|
||
Port 8000 Port 8501
|
||
FastAPI Streamlit
|
||
```
|
||
|
||
### Core Components
|
||
|
||
| Component | Responsibility |
|
||
|---|---|
|
||
| Data Collectors | Collect platform and commercial telemetry. |
|
||
| Data Governance | Validate metrics, identities, freshness, and missing values. |
|
||
| Ranking Engine | Calculate multi-platform artist scores and rankings. |
|
||
| CPI Engine | Normalize commercial sales performance. |
|
||
| Simulation Engine | Evaluate hypothetical ranking changes without modifying baseline rankings. |
|
||
| FastAPI | Expose ranking, telemetry, and simulation endpoints. |
|
||
| Streamlit | Provide an interactive analytics dashboard. |
|
||
| Exporters | Generate executive Excel and Markdown audit reports. |
|
||
| PM2 | Manage API and dashboard processes. |
|
||
|
||
---
|
||
|
||
## 2. Ranking Methodology
|
||
|
||
The HIFFI ranking system evaluates artists through three analytical dimensions:
|
||
|
||
1. YouTube Production Performance.
|
||
2. Multi-Platform Artist Performance (HIFFI-XP).
|
||
3. Breakout Velocity Index (BVI).
|
||
|
||
Each dimension serves a different purpose and is calculated independently.
|
||
|
||
### A. YouTube Production Score
|
||
|
||
The YouTube Production Score evaluates artist performance using four components.
|
||
|
||
| Component | Weight |
|
||
|---|---:|
|
||
| Recent Momentum | 35% |
|
||
| View Velocity | 30% |
|
||
| Audience Reach | 20% |
|
||
| Consistency Proxy | 15% |
|
||
| **Total** | **100%** |
|
||
|
||
**Scoring Formula**
|
||
|
||
```text
|
||
Production Score =
|
||
(Recent Momentum × 0.35)
|
||
+ (View Velocity × 0.30)
|
||
+ (Audience Reach × 0.20)
|
||
+ (Consistency Proxy × 0.15)
|
||
```
|
||
|
||
Where:
|
||
|
||
- **Recent Momentum:** Measures recent performance and growth.
|
||
- **View Velocity:** Measures the rate at which video views are generated.
|
||
- **Audience Reach:** Combines subscriber and lifetime-view signals.
|
||
- **Consistency Proxy:** Represents consistency in available performance signals.
|
||
|
||
Each component contributes according to its assigned weight.
|
||
|
||
**Example Calculation**
|
||
|
||
Consider an artist with the following normalized component scores:
|
||
|
||
| Component | Score | Weight |
|
||
|---|---:|---:|
|
||
| Recent Momentum | 80 | 35% |
|
||
| View Velocity | 70 | 30% |
|
||
| Audience Reach | 90 | 20% |
|
||
| Consistency Proxy | 60 | 15% |
|
||
|
||
```text
|
||
Production Score =
|
||
(80 × 0.35)
|
||
+ (70 × 0.30)
|
||
+ (90 × 0.20)
|
||
+ (60 × 0.15)
|
||
|
||
Production Score =
|
||
28 + 21 + 18 + 9
|
||
|
||
Production Score = 76
|
||
```
|
||
|
||
The resulting production score is 76.
|
||
|
||
The normalized component values are illustrative; the actual normalization and preprocessing rules are determined by the ranking implementation.
|
||
|
||
### B. HIFFI-XP Multi-Platform Score
|
||
|
||
The HIFFI-XP scoring engine evaluates artist performance across YouTube, Spotify, and Instagram.
|
||
|
||
Unlike the YouTube Production Score, this calculation combines multiple platform signals into one multi-platform score.
|
||
|
||
**Base Signal Weights**
|
||
|
||
| Platform Signal | Base Weight |
|
||
|---|---:|
|
||
| YouTube Production Score | 50% |
|
||
| Spotify Monthly Listeners | 25% |
|
||
| Spotify Daily Stream Velocity | 5% |
|
||
| Instagram Followers | 20% |
|
||
| **Total** | **100%** |
|
||
|
||
**Scoring Formula**
|
||
|
||
```text
|
||
HIFFI-XP Score =
|
||
(YouTube Production Score × 0.50)
|
||
+ (Spotify Monthly Listeners Score × 0.25)
|
||
+ (Spotify Daily Stream Velocity Score × 0.05)
|
||
+ (Instagram Followers Score × 0.20)
|
||
```
|
||
|
||
The platform inputs in this formula represent the corresponding normalized scoring values, rather than raw listener or follower counts.
|
||
|
||
**Example Calculation**
|
||
|
||
Assume the following normalized platform scores:
|
||
|
||
| Platform Signal | Score | Weight |
|
||
|---|---:|---:|
|
||
| YouTube Production | 80 | 50% |
|
||
| Spotify Monthly Listeners | 70 | 25% |
|
||
| Spotify Daily Stream Velocity | 90 | 5% |
|
||
| Instagram Followers | 60 | 20% |
|
||
|
||
```text
|
||
HIFFI-XP Score =
|
||
(80 × 0.50)
|
||
+ (70 × 0.25)
|
||
+ (90 × 0.05)
|
||
+ (60 × 0.20)
|
||
|
||
HIFFI-XP Score =
|
||
40 + 17.5 + 4.5 + 12
|
||
|
||
HIFFI-XP Score = 74
|
||
```
|
||
|
||
The resulting HIFFI-XP score is 74.
|
||
|
||
### C. Dynamic Weight Redistribution
|
||
|
||
Not every artist has complete data across all platforms.
|
||
|
||
The scoring engine handles missing platform signals by redistributing the weights of the available signals.
|
||
|
||
This prevents unavailable metrics from automatically reducing an artist's score.
|
||
|
||
**Redistribution Formula**
|
||
|
||
```text
|
||
Effective Weight =
|
||
(Original Weight / Sum of Available Original Weights) × 100
|
||
```
|
||
|
||
The total effective weight is normalized to 100%.
|
||
|
||
**Example**
|
||
|
||
Suppose an artist has YouTube, Spotify Monthly Listeners, and Instagram data, but Spotify Daily Stream Velocity is unavailable.
|
||
|
||
The available original weights are:
|
||
|
||
```text
|
||
YouTube = 50%
|
||
Spotify Listeners = 25%
|
||
Instagram = 20%
|
||
|
||
Available Weight = 95%
|
||
```
|
||
|
||
The redistributed weights become:
|
||
|
||
```text
|
||
YouTube =
|
||
(50 / 95) × 100
|
||
= 52.63%
|
||
|
||
Spotify Listeners =
|
||
(25 / 95) × 100
|
||
= 26.32%
|
||
|
||
Instagram =
|
||
(20 / 95) × 100
|
||
= 21.05%
|
||
```
|
||
|
||
**Resulting Effective Weights**
|
||
|
||
| Available Signal | Effective Weight |
|
||
|---|---:|
|
||
| YouTube Production Score | 52.63% |
|
||
| Spotify Monthly Listeners | 26.32% |
|
||
| Instagram Followers | 21.05% |
|
||
| Spotify Daily Stream Velocity | Excluded |
|
||
| **Total** | **100%** |
|
||
|
||
The resulting score is calculated using the available signals and their redistributed weights.
|
||
|
||
```text
|
||
HIFFI-XP Score =
|
||
(YouTube Score × 0.5263)
|
||
+ (Spotify Listeners Score × 0.2632)
|
||
+ (Instagram Score × 0.2105)
|
||
```
|
||
|
||
The effective weights are recorded in `effective_weights` for auditability.
|
||
|
||
### D. Breakout Velocity Index (BVI)
|
||
|
||
The Breakout Velocity Index identifies artists demonstrating growth potential based on available velocity signals.
|
||
|
||
The BVI is a separate analytical measure and is not interchangeable with the HIFFI-XP multi-platform score.
|
||
|
||
Key considerations:
|
||
|
||
- Evaluates available artist growth and velocity signals.
|
||
- Uses a 10,000-subscriber eligibility floor.
|
||
- Reduces the influence of very small accounts.
|
||
- Generates a separate breakout score and ranking.
|
||
|
||
**Important:** The exact BVI component weights and calculation should follow the implemented BVI scoring function. No additional formula is assumed where the implementation does not define one.
|
||
|
||
### E. Ranking Output
|
||
|
||
The ranking engine generates separate outputs for artist performance and publication eligibility.
|
||
|
||
| Output Field | Description |
|
||
|---|---|
|
||
| `production_hiffi_score` | YouTube production-performance score. |
|
||
| `hiffi_xp_score` | Combined multi-platform score. |
|
||
| `hiffi_xp_rank` | Raw multi-platform ranking position. |
|
||
| `hiffi_xp_published_rank` | Published ranking position after eligibility checks. |
|
||
| `breakout_velocity_index` | Artist breakout potential score. |
|
||
| `breakout_rank` | Breakout ranking position. |
|
||
|
||
**Note:** Internal HIFFI scoring methodology is separate from the public HIFFI website's ranking methodology.
|
||
|
||
---
|
||
|
||
## 3. Commercial Power Index (CPI)
|
||
|
||
The Commercial Power Index introduces commercial sales intelligence into the artist analytics platform.
|
||
|
||
It evaluates the commercial performance of artists using Equivalent Album Sales (EAS) obtained from available sales and certification sources.
|
||
|
||
The objective is to normalize commercial performance across artists with significantly different sales volumes.
|
||
|
||
### CPI Calculation
|
||
|
||
The engine applies base-10 logarithmic normalization to convert EAS values into a standardized score between 0 and 100.
|
||
|
||
**Formula**
|
||
|
||
```text
|
||
CPI = clip(
|
||
((log10(max(EAS, 1.0)) - MIN_LOG)
|
||
/ (MAX_LOG - MIN_LOG)) × 100,
|
||
0,
|
||
100
|
||
)
|
||
```
|
||
|
||
**Reference Values**
|
||
|
||
| Parameter | Value | Description |
|
||
|---|---:|---|
|
||
| MIN_LOG | 5.000000 | Logarithm of 100,000 EAS. |
|
||
| MAX_LOG | 8.397940 | Logarithm of 250,000,000 EAS. |
|
||
| Minimum Reference EAS | 100,000 | Lower normalization boundary. |
|
||
| Maximum Reference EAS | 250,000,000 | Upper normalization boundary. |
|
||
|
||
The `clip` operation ensures that the resulting CPI remains within the range of 0 to 100.
|
||
|
||
### Commercial Sales Data Sources
|
||
|
||
| Source | Purpose |
|
||
|---|---|
|
||
| ChartMasters CSPC | Global commercial sales and equivalent album sales estimates. |
|
||
| RIAA | US domestic album and digital certification milestones. |
|
||
| Luminate | Commercial music consumption and sales intelligence. |
|
||
| Kworb | Spotify streaming telemetry and listener activity. |
|
||
|
||
### Sales Provenance Tiers
|
||
|
||
Commercial data is classified according to its source and evidence level.
|
||
|
||
| Tier | Classification | Artists |
|
||
|---|---|---:|
|
||
| Tier 1 | `VERIFIED_CSPC` | 62 |
|
||
| Tier 2 | `VERIFIED_RIAA` | 259 |
|
||
| Tier 3 | `ESTIMATED_SEA` | 179 |
|
||
| | **Total** | **500** |
|
||
|
||
**Provenance Definitions**
|
||
|
||
- `VERIFIED_CSPC`: Global comprehensive sales data sourced from ChartMasters.
|
||
- `VERIFIED_RIAA`: US domestic certified album and digital milestones.
|
||
- `ESTIMATED_SEA`: Estimated streaming-equivalent album sales using configured conversion assumptions.
|
||
|
||
Streaming conversion assumptions:
|
||
|
||
- 1,500 audio streams = 1 EAS.
|
||
- 3,750 video streams = 1 EAS.
|
||
|
||
These are configured estimation assumptions, not universal industry accounting standards.
|
||
|
||
Sales provenance tiers distinguish the strength and coverage of available commercial evidence.
|
||
|
||
---
|
||
|
||
## 4. Isolated Dynamic Simulation Engine
|
||
|
||
The simulation engine evaluates how commercial performance could influence artist rankings under different commercial weighting scenarios.
|
||
|
||
It operates independently of the baseline published ranking system.
|
||
|
||
### Simulation Methodology
|
||
|
||
The commercial weight (`w_sales`) is configurable between 0% and 50%.
|
||
|
||
**A. Base Weight Calculation**
|
||
|
||
The base weight represents the remaining contribution of the original multi-platform score.
|
||
|
||
```text
|
||
Base Weight = 1 - Commercial Weight
|
||
```
|
||
|
||
For example:
|
||
|
||
```text
|
||
Commercial Weight = 0.20
|
||
Base Weight = 1 - 0.20
|
||
= 0.80
|
||
```
|
||
|
||
**B. Simulated Score Calculation**
|
||
|
||
When CPI data is available:
|
||
|
||
```text
|
||
Simulated Score =
|
||
(Base Weight × Base Score)
|
||
+ (Commercial Weight × CPI)
|
||
```
|
||
|
||
When CPI data is missing:
|
||
|
||
```text
|
||
Simulated Score = Base Score
|
||
```
|
||
|
||
This ensures that artists without commercial sales data retain their original base score rather than receiving an artificial zero CPI contribution.
|
||
|
||
**C. Ranking Movement Calculation**
|
||
|
||
```text
|
||
Rank Delta = Published Rank - Simulated Rank
|
||
```
|
||
|
||
A positive rank delta indicates an improvement in ranking position, while a negative value indicates a decline.
|
||
|
||
### Simulation Example
|
||
|
||
Consider an artist with the following values:
|
||
|
||
| Parameter | Value |
|
||
|---|---:|
|
||
| Base Score | 80 |
|
||
| CPI | 60 |
|
||
| Commercial Weight | 20% (0.20) |
|
||
| Base Weight | 80% (0.80) |
|
||
|
||
Calculation:
|
||
|
||
```text
|
||
Simulated Score =
|
||
(0.80 × 80) + (0.20 × 60)
|
||
|
||
Simulated Score =
|
||
64 + 12
|
||
|
||
Simulated Score = 76
|
||
```
|
||
|
||
The resulting simulated score is 76.
|
||
|
||
The simulation compares this result against the original ranking to identify potential rank movements.
|
||
|
||
### Simulation Features
|
||
|
||
- Adjustable commercial weighting.
|
||
- Dynamic weight re-ranking.
|
||
- Global redistribution mode.
|
||
- Commercial sensitivity analysis.
|
||
- Rank movement comparison.
|
||
- Baseline ranking preservation.
|
||
|
||
### Baseline Invariance
|
||
|
||
Simulations do not overwrite or modify the original published ranking field:
|
||
|
||
`hiffi_xp_published_rank`
|
||
|
||
This ensures that commercial simulations remain isolated from the production ranking methodology.
|
||
|
||
Simulation results represent hypothetical ranking scenarios and do not automatically change official HIFFI rankings.
|
||
|
||
---
|
||
|
||
## 5. Data Governance & Validation
|
||
|
||
Data governance ensures that ranking and commercial calculations use traceable and validated inputs.
|
||
|
||
### Zero-Imputation Policy
|
||
|
||
Missing streaming telemetry is preserved as `NaN` wherever supported by the scoring implementation.
|
||
|
||
Failed API requests or unavailable platform metrics must not automatically be interpreted as genuine zero values.
|
||
|
||
### Artist Identity Verification
|
||
|
||
Instagram profiles must pass configured identity verification rules before receiving multi-platform evidence credit.
|
||
|
||
Supported verification statuses include:
|
||
|
||
- `AUTOMATED_VERIFIED`
|
||
- `UNVERIFIED`
|
||
- `UNCERTAIN_NEEDS_REVIEW`
|
||
|
||
Ambiguous artist matches and uncertain handles are routed to:
|
||
|
||
`data/review_queue.csv`
|
||
|
||
Artist follower counts are supporting evidence, not independent proof of identity.
|
||
|
||
### Freshness Validation
|
||
|
||
| Platform | Maximum Data Age |
|
||
|---|---:|
|
||
| YouTube | 7 days |
|
||
| Spotify | 7 days |
|
||
| Instagram | 14 days |
|
||
|
||
Missing, invalid, or stale timestamps are recorded as evidence exclusions.
|
||
|
||
Example:
|
||
|
||
`STALE_OR_MISSING_SPOTIFY_TIMESTAMP`
|
||
|
||
Scoring eligibility and publication eligibility are treated as separate decisions.
|
||
|
||
### Dynamic Weight Governance
|
||
|
||
The system records the effective weights applied to each artist.
|
||
|
||
This makes it possible to inspect how missing platform signals influence individual ranking calculations.
|
||
|
||
### Ranking Continuity
|
||
|
||
The baseline ranking system maintains unique, contiguous ranking positions from 1 to 500, without duplicate ranks or gaps.
|
||
|
||
---
|
||
|
||
## 6. Anomaly Detection & Regression Auditing
|
||
|
||
### Cross-Platform Anomaly Detection
|
||
|
||
The system identifies unusual relationships between platform metrics.
|
||
|
||
| Anomaly Flag | Detection Condition |
|
||
|---|---|
|
||
| `SUSPICIOUS_IG_TO_YT_RATIO` | Instagram followers exceed YouTube lifetime views by more than 10x. |
|
||
| `EXTREME_SPOTIFY_VELOCITY_RATIO` | Daily stream growth exceeds 25% of monthly listeners. |
|
||
|
||
Anomaly flags are recorded for investigation and do not independently establish fraudulent activity.
|
||
|
||
### Regression Auditing
|
||
|
||
The regression auditor compares current ranking outputs against previously saved ranking snapshots.
|
||
|
||
Key checks include:
|
||
|
||
- Significant ranking movements.
|
||
- Rank changes greater than 100 positions.
|
||
- Changes in platform evidence availability.
|
||
- Potential scoring inconsistencies.
|
||
- Ranking continuity and data integrity.
|
||
|
||
Baselines are maintained in:
|
||
|
||
`data/snapshots/`
|
||
|
||
The latest audit report is generated at:
|
||
|
||
`reports/audit_report_latest.md`
|
||
|
||
Large ranking changes should be reviewed before publication.
|
||
|
||
---
|
||
|
||
## 7. REST API Reference
|
||
|
||
The FastAPI service exposes endpoints for system health, telemetry coverage, artist rankings, and commercial simulations.
|
||
|
||
**Service:** FastAPI
|
||
**Default Port:** 8000
|
||
**Application Entry Point:** `api:app`
|
||
|
||
### Available Endpoints
|
||
|
||
| Method | Endpoint | Description |
|
||
|---|---|---|
|
||
| GET | `/health` | Service health and total artist cohort count. |
|
||
| GET | `/telemetry/coverage` | Data provenance tier counts and coverage percentages. |
|
||
| GET | `/artists` | Ranked artist catalog. |
|
||
| POST | `/simulate` | Executes an isolated commercial ranking simulation. |
|
||
|
||
### Artist Endpoint Parameters
|
||
|
||
`GET /artists`
|
||
|
||
| Parameter | Description |
|
||
|---|---|
|
||
| `scope` | Artist selection: `all` or `verified`. |
|
||
| `limit` | Maximum number of artists, between 1 and 500. |
|
||
| `vertical` | Optional vertical filter. |
|
||
|
||
### Simulation Request
|
||
|
||
`POST /simulate`
|
||
|
||
Example request:
|
||
|
||
```json
|
||
{
|
||
"commercial_weight": 0.15,
|
||
"mode": "global_redistribution"
|
||
}
|
||
```
|
||
|
||
The simulation accepts a commercial weighting value and execution mode to evaluate alternative ranking scenarios.
|
||
|
||
### Example API Requests
|
||
|
||
**Check service health**
|
||
|
||
```powershell
|
||
Invoke-RestMethod -Uri "http://localhost:8000/health"
|
||
```
|
||
|
||
**Check telemetry coverage**
|
||
|
||
```powershell
|
||
Invoke-RestMethod -Uri "http://localhost:8000/telemetry/coverage"
|
||
```
|
||
|
||
**View the artist catalog**
|
||
|
||
```powershell
|
||
Invoke-RestMethod -Uri "http://localhost:8000/artists?scope=all&limit=10"
|
||
```
|
||
|
||
---
|
||
|
||
## 8. Executive Audit & Reporting
|
||
|
||
The platform generates structured reports for commercial analysis and data governance.
|
||
|
||
### Excel Executive Sales Audit
|
||
|
||
File:
|
||
|
||
`reports/hiffi_executive_sales_audit.xlsx`
|
||
|
||
Provides structured commercial sales and artist-level analysis.
|
||
|
||
### Sales Provenance Audit
|
||
|
||
File:
|
||
|
||
`reports/sales_provenance_audit.md`
|
||
|
||
Documents commercial source classifications and data provenance.
|
||
|
||
### Ranking Audit Report
|
||
|
||
File:
|
||
|
||
`reports/audit_report_latest.md`
|
||
|
||
Contains ranking regression findings, rank movements, and data quality observations.
|
||
|
||
These reports support traceability and review of ranking and commercial simulation outputs.
|
||
|
||
---
|
||
|
||
## 9. Project Structure
|
||
|
||
```text
|
||
hiffi-artist-intelligence/
|
||
│
|
||
├── collectors/
|
||
│ ├── instagram_verifier.py
|
||
│ ├── spotify_api_collector.py
|
||
│ └── spotify_fallback_collector.py
|
||
│
|
||
├── core/
|
||
│ ├── ranking_engine.py
|
||
│ ├── regression_auditor.py
|
||
│ ├── health_detector.py
|
||
│ └── cpi_engine.py
|
||
│
|
||
├── exporters/
|
||
│ └── excel_workbook.py
|
||
│
|
||
├── pipeline/
|
||
│ └── batch_500_runner.py
|
||
│
|
||
├── data/
|
||
│ ├── checkpoints/
|
||
│ ├── snapshots/
|
||
│ ├── review_queue.csv
|
||
│ ├── handle_overrides.json
|
||
│ └── hiffi_multiplatform_500.csv
|
||
│
|
||
├── reports/
|
||
│ ├── audit_report_latest.md
|
||
│ ├── hiffi_executive_sales_audit.xlsx
|
||
│ └── sales_provenance_audit.md
|
||
│
|
||
├── tests/
|
||
│ └── test_platform_invariants.py
|
||
│
|
||
├── app.py
|
||
├── api.py
|
||
├── run_pipeline.py
|
||
├── audit_system_integrity.py
|
||
├── ecosystem.config.js
|
||
├── requirements.txt
|
||
├── .env.example
|
||
└── README.md
|
||
```
|
||
|
||
*The project structure represents the documented architecture. Confirm filenames and folders against the current repository before treating this tree as exhaustive.*
|
||
|
||
---
|
||
|
||
## 10. Requirements & Setup
|
||
|
||
### Prerequisites
|
||
|
||
- Python 3.11 or 3.12
|
||
- Git
|
||
- Node.js and npm
|
||
- PM2
|
||
- Required API credentials for enabled integrations
|
||
|
||
### Step 1: Clone the Repository
|
||
|
||
```bash
|
||
git clone <YOUR_REPOSITORY_URL>
|
||
cd hiffi-artist-intelligence
|
||
```
|
||
|
||
Replace the placeholder with the repository's actual clone URL.
|
||
|
||
### Step 2: Create a Virtual Environment
|
||
|
||
**Windows PowerShell**
|
||
|
||
```powershell
|
||
python -m venv .venv
|
||
.\.venv\Scripts\Activate.ps1
|
||
```
|
||
|
||
**macOS/Linux**
|
||
|
||
```bash
|
||
python3 -m venv .venv
|
||
source .venv/bin/activate
|
||
```
|
||
|
||
### Step 3: Install Python Dependencies
|
||
|
||
```bash
|
||
python -m pip install --upgrade pip
|
||
pip install -r requirements.txt
|
||
```
|
||
|
||
### Step 4: Configure Environment Variables
|
||
|
||
Create a local `.env` file using `.env.example` as a reference.
|
||
|
||
Add the required credentials for integrations that need authentication.
|
||
|
||
Example:
|
||
|
||
```env
|
||
SPOTIFY_CLIENT_ID=your_spotify_client_id
|
||
SPOTIFY_CLIENT_SECRET=your_spotify_client_secret
|
||
```
|
||
|
||
Use the exact environment variable names expected by the configured collectors.
|
||
|
||
Never commit API keys, credentials, or `.env` files to Git.
|
||
|
||
### Step 5: Install PM2
|
||
|
||
```powershell
|
||
npm install -g pm2
|
||
```
|
||
|
||
---
|
||
|
||
## 11. Running the Application
|
||
|
||
### Refresh Platform Data
|
||
|
||
Run the ingestion pipeline:
|
||
|
||
```bash
|
||
python -m pipeline.batch_500_runner
|
||
```
|
||
|
||
This stage collects available platform telemetry and refreshes the master artist dataset.
|
||
|
||
### Execute the Ranking Pipeline
|
||
|
||
```bash
|
||
python run_pipeline.py
|
||
```
|
||
|
||
This stage processes the available dataset, calculates ranking scores, applies governance rules, and generates audit outputs.
|
||
|
||
### Launch the FastAPI Service
|
||
|
||
```bash
|
||
uvicorn api:app --host 0.0.0.0 --port 8000
|
||
```
|
||
|
||
### Launch the Streamlit Dashboard
|
||
|
||
```bash
|
||
streamlit run app.py --server.port 8501 --server.address 0.0.0.0
|
||
```
|
||
|
||
The two services can also be managed together through PM2.
|
||
|
||
---
|
||
|
||
## 12. Testing & Continuous Integration
|
||
|
||
The project includes automated tests for mathematical invariants, ranking behavior, and data governance.
|
||
|
||
### Run the Test Suite
|
||
|
||
```powershell
|
||
pytest tests/test_platform_invariants.py -v
|
||
```
|
||
|
||
### Run the System Integrity Audit
|
||
|
||
```powershell
|
||
python audit_system_integrity.py
|
||
```
|
||
|
||
### Continuous Integration
|
||
|
||
GitHub Actions is configured to validate the project across:
|
||
|
||
- Python 3.11
|
||
- Python 3.12
|
||
- Ubuntu environments
|
||
|
||
The previously reported validation baseline includes 27 passing tests and successful CI checks on both Python versions.
|
||
|
||
Always verify the latest test results against the current repository commit before deployment.
|
||
|
||
---
|
||
|
||
## 13. Production Deployment & Operations
|
||
|
||
The application uses PM2 to supervise the FastAPI backend and Streamlit dashboard.
|
||
|
||
### PM2 Process Configuration
|
||
|
||
The `ecosystem.config.js` file defines the application processes.
|
||
|
||
| Service | Port | Framework |
|
||
|---|---:|---|
|
||
| API Service | 8000 | FastAPI |
|
||
| Dashboard | 8501 | Streamlit |
|
||
|
||
### Start Both Services
|
||
|
||
```powershell
|
||
pm2 start ecosystem.config.js
|
||
```
|
||
|
||
### Check Process Status
|
||
|
||
```powershell
|
||
pm2 status
|
||
```
|
||
|
||
### Restart Services
|
||
|
||
```powershell
|
||
pm2 restart ecosystem.config.js
|
||
```
|
||
|
||
### View Logs
|
||
|
||
```powershell
|
||
pm2 logs
|
||
```
|
||
|
||
### Save Process Configuration
|
||
|
||
```powershell
|
||
pm2 save
|
||
```
|
||
|
||
### Verify API Availability
|
||
|
||
```powershell
|
||
Invoke-RestMethod -Uri "http://localhost:8000/health"
|
||
```
|
||
|
||
### Verify Telemetry Coverage
|
||
|
||
```powershell
|
||
Invoke-RestMethod -Uri "http://localhost:8000/telemetry/coverage"
|
||
```
|
||
|
||
### Operational Checklist
|
||
|
||
- Confirm both processes are running.
|
||
- Verify API health and telemetry endpoints.
|
||
- Check dashboard accessibility.
|
||
- Review application logs after deployment.
|
||
- Validate the dataset before restarting services.
|
||
- Preserve baseline ranking data before running simulations.
|
||
- Keep API credentials outside the repository.
|
||
|
||
---
|
||
|
||
## 14. Important Output Fields
|
||
|
||
| Field | Purpose |
|
||
|---|---|
|
||
| `production_hiffi_score` | YouTube production-performance score. |
|
||
| `hiffi_xp_score` | Combined multi-platform artist score. |
|
||
| `hiffi_xp_rank` | Raw multi-platform ranking position. |
|
||
| `hiffi_xp_published_rank` | Published rank after eligibility checks. |
|
||
| `breakout_velocity_index` | Artist breakout potential score. |
|
||
| `breakout_rank` | Breakout ranking position. |
|
||
| `effective_weights` | Weights applied to available platform signals. |
|
||
| `platform_evidence_tier` | Platform evidence classification. |
|
||
| `evidence_exclusions` | Reasons for excluding platform evidence. |
|
||
| `anomaly_flags` | Potentially suspicious metric relationships. |
|
||
| `rank_delta` | Difference between comparison ranking positions. |
|
||
| `CPI` | Normalized Commercial Power Index. |
|
||
| `sales_provenance_tier` | Commercial sales evidence classification. |
|
||
|
||
Actual output fields depend on the current implementation and input dataset.
|
||
|
||
---
|
||
|
||
## 15. Security & Data Governance
|
||
|
||
The system follows basic operational and data governance practices.
|
||
|
||
- Store API credentials in environment variables.
|
||
- Never commit secrets or `.env` files.
|
||
- Preserve missing values instead of introducing synthetic zeros.
|
||
- Validate artist identity before assigning platform evidence credit.
|
||
- Maintain source timestamps and sales provenance.
|
||
- Investigate anomaly flags and significant ranking movements.
|
||
- Preserve published baseline rankings during simulations.
|
||
- Review commercial estimates before interpreting them as verified sales.
|
||
|
||
---
|
||
|
||
## 16. Limitations
|
||
|
||
- Platform API access, authentication, and request quotas can affect data availability.
|
||
- Platform metrics may have different refresh schedules and definitions.
|
||
- Artist aliases and unofficial social accounts can create identity-matching challenges.
|
||
- Commercial sales estimates may vary across data providers.
|
||
- RIAA certifications represent US domestic certification evidence, not complete global sales.
|
||
- Estimated streaming-equivalent sales depend on configured conversion assumptions.
|
||
- Anomaly detection identifies unusual relationships but does not prove fraudulent activity.
|
||
- Simulation results are hypothetical and do not automatically change official published rankings.
|
||
- Automated tests cannot guarantee the accuracy of every live platform metric or commercial estimate.
|
||
|
||
---
|
||
|
||
## 17. Future Improvements
|
||
|
||
- Expand reliable multi-platform identity verification.
|
||
- Improve commercial sales source reconciliation and provenance validation.
|
||
- Introduce additional artist growth and engagement signals.
|
||
- Strengthen ingestion retry mechanisms and checkpoint recovery.
|
||
- Improve automated anomaly detection and data freshness monitoring.
|
||
- Expand commercial sensitivity and simulation reporting.
|
||
- Introduce operational alerts for failed or incomplete pipeline runs.
|
||
- Explore LLM-based artist skill extraction and music-industry demand/supply intelligence.
|
||
- Track changes in artist skills, genre demand, and market opportunities over time.
|
||
|
||
---
|
||
|
||
## Summary
|
||
|
||
HIFFI Artist Intelligence Engine combines multi-platform artist performance analytics with commercial sales intelligence in a modular and auditable architecture.
|
||
|
||
Its ranking engine evaluates streaming and audience signals through weighted scoring, while the Commercial Power Index introduces normalized commercial performance into an isolated simulation environment.
|
||
|
||
With automated data governance, provenance tracking, regression auditing, REST API services, executive reporting, and an interactive dashboard, the platform provides a structured foundation for artist intelligence and commercial analytics.
|
||
|
||
**Built on measurable signals, validated evidence, and explainable analytics.** |