diff --git a/README.md b/README.md index 3627d0a..8e0a6d7 100644 --- a/README.md +++ b/README.md @@ -1,83 +1,518 @@ -# HIFFI Hip-Hop 500 — Multi-Platform Artist Intelligence Pipeline +# HIFFI Artist Intelligence Engine +### Multi-Platform Artist Intelligence, Commercial Analytics & Simulation -A Python-based pipeline for collecting, validating, scoring, and ranking hip-hop artists using YouTube, Spotify, and Instagram signals. It includes data freshness checks, identity verification, anomaly detection, publication eligibility rules, and regression auditing. +HIFFI Artist Intelligence Engine is a Python-based analytics platform designed to evaluate the top 500 hip-hop artists using multi-platform performance signals and commercial sales intelligence. -## Key Features +The system combines YouTube, Spotify, and Instagram telemetry with commercial sales data to provide artist rankings, commercial performance analysis, isolated ranking simulations, data governance, and executive audit reporting. -- Multi-platform artist metrics and scoring. -- Instagram identity matching with review-queue support. -- Missing-data handling and dynamic weight redistribution. -- Platform freshness and publication eligibility gates. -- Cross-platform anomaly detection. -- Regression auditing and rank-shift reports. -- Automated tests and GitHub Actions CI for Python 3.11 and 3.12. +It includes an automated data pipeline, a FastAPI backend, an interactive Streamlit dashboard, executive audit exporters, and a PM2-managed deployment environment. -## Pipeline Architecture: Telemetry Ingestion vs. Ranking Core +--- -The project separates data collection from ranking so that platform data can be refreshed independently of score calculation. +## 1. System Architecture -### 1. Telemetry Ingestion Layer — `pipeline/batch_500_runner.py` +The platform separates data ingestion, data governance, ranking, commercial simulation, and application services into independent components. -- Collects multi-platform telemetry through configured API calls and web scraping. -- Uses checkpoints in `data/checkpoints/` to support recovery after interrupted runs and reduce repeated data requests. -- Saves the refreshed master cohort dataset to `data/hiffi_multiplatform_500.csv`. +```text + DATA INGESTION SOURCES + | + +-------------------+-------------------+ + | | | + YouTube Spotify Instagram + | | | + +-------------------+-------------------+ + | + COMMERCIAL DATA SOURCES + | + +-------------------+-------------------+ + | | | + ChartMasters RIAA Luminate + | | | + +-------------------+-------------------+ + | + v + DATA GOVERNANCE & HYGIENE + core/health_detector.py + | + - Missing-data validation + - Artist identity verification + - Alias and handle overrides + - Dynamic weight redistribution + | + v + MASTER COHORT DATASET + data/hiffi_multiplatform_500.csv + | + +------------+------------+ + | | + v v + RANKING ENGINE COMMERCIAL SIMULATION + Multi-platform scoring core/cpi_engine.py + | | + | - CPI calculation + | - Dynamic re-ranking + | - Sensitivity analysis + | | + +------------+------------+ + | + +----------+----------+ + | | + v v + FASTAPI SERVICE AUDIT EXPORTERS + api.py exporters/ + | | + | Excel / Markdown + | | + +----------+----------+ + | + STREAMLIT UI + app.py + | + PM2 PROCESS MANAGER + | + +------------+------------+ + | | + Port 8000 Port 8501 + FastAPI Streamlit +``` -### 2. Ranking and Governance Core — `run_pipeline.py` +### Core Components -- Loads `data/hiffi_multiplatform_500.csv` and executes the ranking workflow independently of data collection. -- Handles missing metrics, redistributes weights across available signals, checks data freshness, and flags suspicious records. -- Applies publication eligibility rules and assigns published ranks. -- Generates an audit report at `reports/audit_report_latest.md`. +| Component | Responsibility | +|---|---| +| Data Collectors | Collect platform and commercial telemetry. | +| Data Governance | Validate metrics, identities, freshness, and missing values. | +| Ranking Engine | Calculate multi-platform artist scores and rankings. | +| CPI Engine | Normalize commercial sales performance. | +| Simulation Engine | Evaluate hypothetical ranking changes without modifying baseline rankings. | +| FastAPI | Expose ranking, telemetry, and simulation endpoints. | +| Streamlit | Provide an interactive analytics dashboard. | +| Exporters | Generate executive Excel and Markdown audit reports. | +| PM2 | Manage API and dashboard processes. | -## Ranking Methodology +--- -The project uses separate scores for artist production performance, multi-platform ranking, and breakout potential. +## 2. Ranking Methodology -### HIFFI-XP Multi-Platform Score +The HIFFI ranking system evaluates artists through three analytical dimensions: -| Signal | Base Weight | -|---|---:| -| YouTube production score | 50% | -| Spotify monthly listeners | 25% | -| Spotify daily stream velocity | 5% | -| Instagram followers | 20% | +1. YouTube Production Performance. +2. Multi-Platform Artist Performance (HIFFI-XP). +3. Breakout Velocity Index (BVI). -When some signals are unavailable, the scoring logic redistributes their weights proportionally across the available signals. The effective weights are recorded in `effective_weights` for auditability. +Each dimension serves a different purpose and is calculated independently. -### YouTube Production Score +### A. YouTube Production Score -The production score uses the following components: +The YouTube Production Score evaluates artist performance using four components. | Component | Weight | |---|---:| -| Recent momentum | 35% | -| View velocity | 30% | -| Audience reach | 20% | -| Consistency proxy | 15% | +| Recent Momentum | 35% | +| View Velocity | 30% | +| Audience Reach | 20% | +| Consistency Proxy | 15% | +| **Total** | **100%** | -Audience reach combines subscriber and lifetime-view signals. The score also uses confidence and eligibility checks where supported by the ranking implementation. +**Scoring Formula** -### Breakout Velocity Index (BVI) +```text +Production Score = + (Recent Momentum × 0.35) + + (View Velocity × 0.30) + + (Audience Reach × 0.20) + + (Consistency Proxy × 0.15) +``` -The Breakout Velocity Index identifies artists showing growth potential based on available velocity signals. A 10,000-subscriber floor is used by the BVI logic to reduce the impact of very small accounts. +Where: -**Note:** The project's internal scoring methodology is not necessarily identical to the public HIFFI website's ranking formula. +- **Recent Momentum:** Measures recent performance and growth. +- **View Velocity:** Measures the rate at which video views are generated. +- **Audience Reach:** Combines subscriber and lifetime-view signals. +- **Consistency Proxy:** Represents consistency in available performance signals. -## Identity Verification Gates +Each component contributes according to its assigned weight. -Only Instagram profiles that pass the configured verification rules qualify for multi-platform credit. +**Example Calculation** -- **`AUTOMATED_VERIFIED`:** Confidence ≥ `0.55`, or confidence ≥ `0.40` with follower corroboration ≥ `10,000`. -- **`UNVERIFIED`:** Accounts that do not meet verification requirements. -- **`UNCERTAIN_NEEDS_REVIEW`:** Ambiguous account matches requiring human review. +Consider an artist with the following normalized component scores: -Unverified accounts, low-follower outliers below `10,000`, and ambiguous handles are routed to `data/review_queue.csv` for review and excluded from multi-platform tier credit. +| Component | Score | Weight | +|---|---:|---:| +| Recent Momentum | 80 | 35% | +| View Velocity | 70 | 30% | +| Audience Reach | 90 | 20% | +| Consistency Proxy | 60 | 15% | -Follower count is a supporting signal, not proof of identity. The verification status and publication eligibility rules determine whether an account can contribute evidence. +```text +Production Score = + (80 × 0.35) + + (70 × 0.30) + + (90 × 0.20) + + (60 × 0.15) -## Data Freshness and Publication Eligibility +Production Score = + 28 + 21 + 18 + 9 -The ranking engine uses freshness windows to determine whether platform data qualifies as current evidence. +Production Score = 76 +``` + +The resulting production score is 76. + +The normalized component values are illustrative; the actual normalization and preprocessing rules are determined by the ranking implementation. + +### B. HIFFI-XP Multi-Platform Score + +The HIFFI-XP scoring engine evaluates artist performance across YouTube, Spotify, and Instagram. + +Unlike the YouTube Production Score, this calculation combines multiple platform signals into one multi-platform score. + +**Base Signal Weights** + +| Platform Signal | Base Weight | +|---|---:| +| YouTube Production Score | 50% | +| Spotify Monthly Listeners | 25% | +| Spotify Daily Stream Velocity | 5% | +| Instagram Followers | 20% | +| **Total** | **100%** | + +**Scoring Formula** + +```text +HIFFI-XP Score = + (YouTube Production Score × 0.50) + + (Spotify Monthly Listeners Score × 0.25) + + (Spotify Daily Stream Velocity Score × 0.05) + + (Instagram Followers Score × 0.20) +``` + +The platform inputs in this formula represent the corresponding normalized scoring values, rather than raw listener or follower counts. + +**Example Calculation** + +Assume the following normalized platform scores: + +| Platform Signal | Score | Weight | +|---|---:|---:| +| YouTube Production | 80 | 50% | +| Spotify Monthly Listeners | 70 | 25% | +| Spotify Daily Stream Velocity | 90 | 5% | +| Instagram Followers | 60 | 20% | + +```text +HIFFI-XP Score = + (80 × 0.50) + + (70 × 0.25) + + (90 × 0.05) + + (60 × 0.20) + +HIFFI-XP Score = + 40 + 17.5 + 4.5 + 12 + +HIFFI-XP Score = 74 +``` + +The resulting HIFFI-XP score is 74. + +### C. Dynamic Weight Redistribution + +Not every artist has complete data across all platforms. + +The scoring engine handles missing platform signals by redistributing the weights of the available signals. + +This prevents unavailable metrics from automatically reducing an artist's score. + +**Redistribution Formula** + +```text +Effective Weight = + (Original Weight / Sum of Available Original Weights) × 100 +``` + +The total effective weight is normalized to 100%. + +**Example** + +Suppose an artist has YouTube, Spotify Monthly Listeners, and Instagram data, but Spotify Daily Stream Velocity is unavailable. + +The available original weights are: + +```text +YouTube = 50% +Spotify Listeners = 25% +Instagram = 20% + +Available Weight = 95% +``` + +The redistributed weights become: + +```text +YouTube = + (50 / 95) × 100 + = 52.63% + +Spotify Listeners = + (25 / 95) × 100 + = 26.32% + +Instagram = + (20 / 95) × 100 + = 21.05% +``` + +**Resulting Effective Weights** + +| Available Signal | Effective Weight | +|---|---:| +| YouTube Production Score | 52.63% | +| Spotify Monthly Listeners | 26.32% | +| Instagram Followers | 21.05% | +| Spotify Daily Stream Velocity | Excluded | +| **Total** | **100%** | + +The resulting score is calculated using the available signals and their redistributed weights. + +```text +HIFFI-XP Score = + (YouTube Score × 0.5263) + + (Spotify Listeners Score × 0.2632) + + (Instagram Score × 0.2105) +``` + +The effective weights are recorded in `effective_weights` for auditability. + +### D. Breakout Velocity Index (BVI) + +The Breakout Velocity Index identifies artists demonstrating growth potential based on available velocity signals. + +The BVI is a separate analytical measure and is not interchangeable with the HIFFI-XP multi-platform score. + +Key considerations: + +- Evaluates available artist growth and velocity signals. +- Uses a 10,000-subscriber eligibility floor. +- Reduces the influence of very small accounts. +- Generates a separate breakout score and ranking. + +**Important:** The exact BVI component weights and calculation should follow the implemented BVI scoring function. No additional formula is assumed where the implementation does not define one. + +### E. Ranking Output + +The ranking engine generates separate outputs for artist performance and publication eligibility. + +| Output Field | Description | +|---|---| +| `production_hiffi_score` | YouTube production-performance score. | +| `hiffi_xp_score` | Combined multi-platform score. | +| `hiffi_xp_rank` | Raw multi-platform ranking position. | +| `hiffi_xp_published_rank` | Published ranking position after eligibility checks. | +| `breakout_velocity_index` | Artist breakout potential score. | +| `breakout_rank` | Breakout ranking position. | + +**Note:** Internal HIFFI scoring methodology is separate from the public HIFFI website's ranking methodology. + +--- + +## 3. Commercial Power Index (CPI) + +The Commercial Power Index introduces commercial sales intelligence into the artist analytics platform. + +It evaluates the commercial performance of artists using Equivalent Album Sales (EAS) obtained from available sales and certification sources. + +The objective is to normalize commercial performance across artists with significantly different sales volumes. + +### CPI Calculation + +The engine applies base-10 logarithmic normalization to convert EAS values into a standardized score between 0 and 100. + +**Formula** + +```text +CPI = clip( + ((log10(max(EAS, 1.0)) - MIN_LOG) + / (MAX_LOG - MIN_LOG)) × 100, + 0, + 100 +) +``` + +**Reference Values** + +| Parameter | Value | Description | +|---|---:|---| +| MIN_LOG | 5.000000 | Logarithm of 100,000 EAS. | +| MAX_LOG | 8.397940 | Logarithm of 250,000,000 EAS. | +| Minimum Reference EAS | 100,000 | Lower normalization boundary. | +| Maximum Reference EAS | 250,000,000 | Upper normalization boundary. | + +The `clip` operation ensures that the resulting CPI remains within the range of 0 to 100. + +### Commercial Sales Data Sources + +| Source | Purpose | +|---|---| +| ChartMasters CSPC | Global commercial sales and equivalent album sales estimates. | +| RIAA | US domestic album and digital certification milestones. | +| Luminate | Commercial music consumption and sales intelligence. | +| Kworb | Spotify streaming telemetry and listener activity. | + +### Sales Provenance Tiers + +Commercial data is classified according to its source and evidence level. + +| Tier | Classification | Artists | +|---|---|---:| +| Tier 1 | `VERIFIED_CSPC` | 62 | +| Tier 2 | `VERIFIED_RIAA` | 259 | +| Tier 3 | `ESTIMATED_SEA` | 179 | +| | **Total** | **500** | + +**Provenance Definitions** + +- `VERIFIED_CSPC`: Global comprehensive sales data sourced from ChartMasters. +- `VERIFIED_RIAA`: US domestic certified album and digital milestones. +- `ESTIMATED_SEA`: Estimated streaming-equivalent album sales using configured conversion assumptions. + +Streaming conversion assumptions: + +- 1,500 audio streams = 1 EAS. +- 3,750 video streams = 1 EAS. + +These are configured estimation assumptions, not universal industry accounting standards. + +Sales provenance tiers distinguish the strength and coverage of available commercial evidence. + +--- + +## 4. Isolated Dynamic Simulation Engine + +The simulation engine evaluates how commercial performance could influence artist rankings under different commercial weighting scenarios. + +It operates independently of the baseline published ranking system. + +### Simulation Methodology + +The commercial weight (`w_sales`) is configurable between 0% and 50%. + +**A. Base Weight Calculation** + +The base weight represents the remaining contribution of the original multi-platform score. + +```text +Base Weight = 1 - Commercial Weight +``` + +For example: + +```text +Commercial Weight = 0.20 +Base Weight = 1 - 0.20 + = 0.80 +``` + +**B. Simulated Score Calculation** + +When CPI data is available: + +```text +Simulated Score = + (Base Weight × Base Score) + + (Commercial Weight × CPI) +``` + +When CPI data is missing: + +```text +Simulated Score = Base Score +``` + +This ensures that artists without commercial sales data retain their original base score rather than receiving an artificial zero CPI contribution. + +**C. Ranking Movement Calculation** + +```text +Rank Delta = Published Rank - Simulated Rank +``` + +A positive rank delta indicates an improvement in ranking position, while a negative value indicates a decline. + +### Simulation Example + +Consider an artist with the following values: + +| Parameter | Value | +|---|---:| +| Base Score | 80 | +| CPI | 60 | +| Commercial Weight | 20% (0.20) | +| Base Weight | 80% (0.80) | + +Calculation: + +```text +Simulated Score = + (0.80 × 80) + (0.20 × 60) + +Simulated Score = + 64 + 12 + +Simulated Score = 76 +``` + +The resulting simulated score is 76. + +The simulation compares this result against the original ranking to identify potential rank movements. + +### Simulation Features + +- Adjustable commercial weighting. +- Dynamic weight re-ranking. +- Global redistribution mode. +- Commercial sensitivity analysis. +- Rank movement comparison. +- Baseline ranking preservation. + +### Baseline Invariance + +Simulations do not overwrite or modify the original published ranking field: + +`hiffi_xp_published_rank` + +This ensures that commercial simulations remain isolated from the production ranking methodology. + +Simulation results represent hypothetical ranking scenarios and do not automatically change official HIFFI rankings. + +--- + +## 5. Data Governance & Validation + +Data governance ensures that ranking and commercial calculations use traceable and validated inputs. + +### Zero-Imputation Policy + +Missing streaming telemetry is preserved as `NaN` wherever supported by the scoring implementation. + +Failed API requests or unavailable platform metrics must not automatically be interpreted as genuine zero values. + +### Artist Identity Verification + +Instagram profiles must pass configured identity verification rules before receiving multi-platform evidence credit. + +Supported verification statuses include: + +- `AUTOMATED_VERIFIED` +- `UNVERIFIED` +- `UNCERTAIN_NEEDS_REVIEW` + +Ambiguous artist matches and uncertain handles are routed to: + +`data/review_queue.csv` + +Artist follower counts are supporting evidence, not independent proof of identity. + +### Freshness Validation | Platform | Maximum Data Age | |---|---:| @@ -85,88 +520,230 @@ The ranking engine uses freshness windows to determine whether platform data qua | Spotify | 7 days | | Instagram | 14 days | -Missing, unparseable, or stale timestamps do not qualify for platform evidence credit. The engine records exclusion reasons in `evidence_exclusions`, such as `STALE_OR_MISSING_SPOTIFY_TIMESTAMP`. +Missing, invalid, or stale timestamps are recorded as evidence exclusions. -A platform's evidence may be excluded from publication eligibility while its available metric can still contribute to a continuous score. Therefore, score calculation and publication eligibility are separate decisions. +Example: -## Missing-Data Handling +`STALE_OR_MISSING_SPOTIFY_TIMESTAMP` -- Preserve missing values as `NaN` where supported by the data and scoring logic. -- Do not treat failed API requests as genuine zero-valued metrics. -- Redistribute weights across available signals according to the scoring implementation. -- Record effective weights so ranking results can be inspected and explained. +Scoring eligibility and publication eligibility are treated as separate decisions. -Some individual scoring functions may apply documented defaults. Review the relevant implementation before assuming every missing value remains `NaN` throughout all calculations. +### Dynamic Weight Governance -## Anomaly Detection +The system records the effective weights applied to each artist. -Before publication, the pipeline flags suspicious cross-platform metric relationships for audit review. +This makes it possible to inspect how missing platform signals influence individual ranking calculations. -| Flag | Condition | +### Ranking Continuity + +The baseline ranking system maintains unique, contiguous ranking positions from 1 to 500, without duplicate ranks or gaps. + +--- + +## 6. Anomaly Detection & Regression Auditing + +### Cross-Platform Anomaly Detection + +The system identifies unusual relationships between platform metrics. + +| Anomaly Flag | Detection Condition | |---|---| -| `SUSPICIOUS_IG_TO_YT_RATIO` | Instagram followers exceed YouTube lifetime views by more than 10×. | +| `SUSPICIOUS_IG_TO_YT_RATIO` | Instagram followers exceed YouTube lifetime views by more than 10x. | | `EXTREME_SPOTIFY_VELOCITY_RATIO` | Daily stream growth exceeds 25% of monthly listeners. | -Flags are recorded in `anomaly_flags` so suspicious records can be investigated rather than silently ignored. A flag should trigger review; it does not, by itself, prove that the data is fraudulent. +Anomaly flags are recorded for investigation and do not independently establish fraudulent activity. -## Regression Auditing +### Regression Auditing -The regression auditor compares current ranking output against saved baselines. +The regression auditor compares current ranking outputs against previously saved ranking snapshots. -- Baselines are stored in `data/snapshots/`. -- Rank movements greater than 100 positions (`|rank shift| > 100`) are identified for review. -- The latest report is written to `reports/audit_report_latest.md`. +Key checks include: -Large ranking changes may reflect genuine changes in platform signals, data availability, scoring weights, or identity verification. They should be reviewed before publication. +- Significant ranking movements. +- Rank changes greater than 100 positions. +- Changes in platform evidence availability. +- Potential scoring inconsistencies. +- Ranking continuity and data integrity. -## Project Structure +Baselines are maintained in: + +`data/snapshots/` + +The latest audit report is generated at: + +`reports/audit_report_latest.md` + +Large ranking changes should be reviewed before publication. + +--- + +## 7. REST API Reference + +The FastAPI service exposes endpoints for system health, telemetry coverage, artist rankings, and commercial simulations. + +**Service:** FastAPI +**Default Port:** 8000 +**Application Entry Point:** `api:app` + +### Available Endpoints + +| Method | Endpoint | Description | +|---|---|---| +| GET | `/health` | Service health and total artist cohort count. | +| GET | `/telemetry/coverage` | Data provenance tier counts and coverage percentages. | +| GET | `/artists` | Ranked artist catalog. | +| POST | `/simulate` | Executes an isolated commercial ranking simulation. | + +### Artist Endpoint Parameters + +`GET /artists` + +| Parameter | Description | +|---|---| +| `scope` | Artist selection: `all` or `verified`. | +| `limit` | Maximum number of artists, between 1 and 500. | +| `vertical` | Optional vertical filter. | + +### Simulation Request + +`POST /simulate` + +Example request: + +```json +{ + "commercial_weight": 0.15, + "mode": "global_redistribution" +} +``` + +The simulation accepts a commercial weighting value and execution mode to evaluate alternative ranking scenarios. + +### Example API Requests + +**Check service health** + +```powershell +Invoke-RestMethod -Uri "http://localhost:8000/health" +``` + +**Check telemetry coverage** + +```powershell +Invoke-RestMethod -Uri "http://localhost:8000/telemetry/coverage" +``` + +**View the artist catalog** + +```powershell +Invoke-RestMethod -Uri "http://localhost:8000/artists?scope=all&limit=10" +``` + +--- + +## 8. Executive Audit & Reporting + +The platform generates structured reports for commercial analysis and data governance. + +### Excel Executive Sales Audit + +File: + +`reports/hiffi_executive_sales_audit.xlsx` + +Provides structured commercial sales and artist-level analysis. + +### Sales Provenance Audit + +File: + +`reports/sales_provenance_audit.md` + +Documents commercial source classifications and data provenance. + +### Ranking Audit Report + +File: + +`reports/audit_report_latest.md` + +Contains ranking regression findings, rank movements, and data quality observations. + +These reports support traceability and review of ranking and commercial simulation outputs. + +--- + +## 9. Project Structure ```text hiffi-artist-intelligence/ +│ ├── collectors/ │ ├── instagram_verifier.py │ ├── spotify_api_collector.py │ └── spotify_fallback_collector.py +│ ├── core/ │ ├── ranking_engine.py -│ └── regression_auditor.py +│ ├── regression_auditor.py +│ ├── health_detector.py +│ └── cpi_engine.py +│ +├── exporters/ +│ └── excel_workbook.py +│ +├── pipeline/ +│ └── batch_500_runner.py +│ ├── data/ │ ├── checkpoints/ │ ├── snapshots/ │ ├── review_queue.csv +│ ├── handle_overrides.json │ └── hiffi_multiplatform_500.csv -├── pipeline/ -│ └── batch_500_runner.py +│ ├── reports/ -│ └── audit_report_latest.md +│ ├── audit_report_latest.md +│ ├── hiffi_executive_sales_audit.xlsx +│ └── sales_provenance_audit.md +│ ├── tests/ +│ └── test_platform_invariants.py +│ +├── app.py +├── api.py ├── run_pipeline.py -├── test_multiplatform_ranking.py +├── audit_system_integrity.py +├── ecosystem.config.js ├── requirements.txt ├── .env.example └── README.md ``` -The tree shows the intended layout. Keep only files and folders that actually exist in the repository. +*The project structure represents the documented architecture. Confirm filenames and folders against the current repository before treating this tree as exhaustive.* -## Requirements +--- + +## 10. Requirements & Setup + +### Prerequisites - Python 3.11 or 3.12 - Git -- API credentials for any integrations that require authentication +- Node.js and npm +- PM2 +- Required API credentials for enabled integrations -## Setup - -### 1. Clone the repository +### Step 1: Clone the Repository ```bash git clone cd hiffi-artist-intelligence ``` -Replace `` with your repository's clone URL. +Replace the placeholder with the repository's actual clone URL. -### 2. Create a virtual environment +### Step 2: Create a Virtual Environment **Windows PowerShell** @@ -182,169 +759,242 @@ python3 -m venv .venv source .venv/bin/activate ``` -### 3. Install dependencies +### Step 3: Install Python Dependencies ```bash python -m pip install --upgrade pip pip install -r requirements.txt ``` -### 4. Configure environment variables +### Step 4: Configure Environment Variables -Create a local `.env` file using `.env.example` as a guide. Add the credentials required by your enabled integrations, for example: +Create a local `.env` file using `.env.example` as a reference. -```dotenv +Add the required credentials for integrations that need authentication. + +Example: + +```env SPOTIFY_CLIENT_ID=your_spotify_client_id SPOTIFY_CLIENT_SECRET=your_spotify_client_secret ``` -Use the actual variable names expected by your collectors. Never commit `.env` files or live API credentials to Git. +Use the exact environment variable names expected by the configured collectors. -## Running the Pipeline +Never commit API keys, credentials, or `.env` files to Git. -### Refresh platform data +### Step 5: Install PM2 -If `pipeline/batch_500_runner.py` is implemented as the ingestion entry point, run: +```powershell +npm install -g pm2 +``` + +--- + +## 11. Running the Application + +### Refresh Platform Data + +Run the ingestion pipeline: ```bash python -m pipeline.batch_500_runner ``` -This stage should refresh the master dataset and use checkpoints where supported. +This stage collects available platform telemetry and refreshes the master artist dataset. -### Calculate rankings and generate the audit report +### Execute the Ranking Pipeline ```bash python run_pipeline.py ``` -This stage consumes the available master dataset, applies scoring and governance rules, and produces ranking outputs and the audit report. +This stage processes the available dataset, calculates ranking scores, applies governance rules, and generates audit outputs. -### Run tests +### Launch the FastAPI Service ```bash -python -m pytest test_multiplatform_ranking.py -v +uvicorn api:app --host 0.0.0.0 --port 8000 ``` -### Run the regression auditor +### Launch the Streamlit Dashboard ```bash -python -m core.regression_auditor +streamlit run app.py --server.port 8501 --server.address 0.0.0.0 ``` -Use the exact commands and test paths defined by the current repository. +The two services can also be managed together through PM2. -## Testing and Continuous Integration +--- -The project reports 27 passing unit and integration tests, with GitHub Actions checks passing on Python 3.11 and 3.12 on Ubuntu. +## 12. Testing & Continuous Integration -The test suite is intended to cover ranking behavior, missing-data handling, freshness and eligibility gates, anomaly detection, and regression safeguards. +The project includes automated tests for mathematical invariants, ranking behavior, and data governance. -Test results should be confirmed against the latest commit before a release is declared complete. +### Run the Test Suite -## Important Output Fields +```powershell +pytest tests/test_platform_invariants.py -v +``` + +### Run the System Integrity Audit + +```powershell +python audit_system_integrity.py +``` + +### Continuous Integration + +GitHub Actions is configured to validate the project across: + +- Python 3.11 +- Python 3.12 +- Ubuntu environments + +The previously reported validation baseline includes 27 passing tests and successful CI checks on both Python versions. + +Always verify the latest test results against the current repository commit before deployment. + +--- + +## 13. Production Deployment & Operations + +The application uses PM2 to supervise the FastAPI backend and Streamlit dashboard. + +### PM2 Process Configuration + +The `ecosystem.config.js` file defines the application processes. + +| Service | Port | Framework | +|---|---:|---| +| API Service | 8000 | FastAPI | +| Dashboard | 8501 | Streamlit | + +### Start Both Services + +```powershell +pm2 start ecosystem.config.js +``` + +### Check Process Status + +```powershell +pm2 status +``` + +### Restart Services + +```powershell +pm2 restart ecosystem.config.js +``` + +### View Logs + +```powershell +pm2 logs +``` + +### Save Process Configuration + +```powershell +pm2 save +``` + +### Verify API Availability + +```powershell +Invoke-RestMethod -Uri "http://localhost:8000/health" +``` + +### Verify Telemetry Coverage + +```powershell +Invoke-RestMethod -Uri "http://localhost:8000/telemetry/coverage" +``` + +### Operational Checklist + +- Confirm both processes are running. +- Verify API health and telemetry endpoints. +- Check dashboard accessibility. +- Review application logs after deployment. +- Validate the dataset before restarting services. +- Preserve baseline ranking data before running simulations. +- Keep API credentials outside the repository. + +--- + +## 14. Important Output Fields | Field | Purpose | |---|---| | `production_hiffi_score` | YouTube production-performance score. | -| `hiffi_xp_score` | Combined multi-platform score. | +| `hiffi_xp_score` | Combined multi-platform artist score. | | `hiffi_xp_rank` | Raw multi-platform ranking position. | -| `hiffi_xp_published_rank` | Published ranking position after eligibility rules. | -| `breakout_velocity_index` | Breakout potential score. | +| `hiffi_xp_published_rank` | Published rank after eligibility checks. | +| `breakout_velocity_index` | Artist breakout potential score. | | `breakout_rank` | Breakout ranking position. | -| `effective_weights` | Weights applied to the available signals. | -| `platform_evidence_tier` | Platform evidence tier used for eligibility. | -| `evidence_exclusions` | Reasons platform evidence was excluded. | +| `effective_weights` | Weights applied to available platform signals. | +| `platform_evidence_tier` | Platform evidence classification. | +| `evidence_exclusions` | Reasons for excluding platform evidence. | | `anomaly_flags` | Potentially suspicious metric relationships. | -| `rank_delta` | Change in ranking position relative to the comparison baseline. | +| `rank_delta` | Difference between comparison ranking positions. | +| `CPI` | Normalized Commercial Power Index. | +| `sales_provenance_tier` | Commercial sales evidence classification. | -Actual output columns depend on the current implementation and input dataset. +Actual output fields depend on the current implementation and input dataset. -## Security and Data Governance +--- -- Keep API keys in environment variables, not source code. -- Keep `.env` and local credentials out of version control. -- Review uncertain artist identities before granting platform evidence credit. -- Keep data timestamps and exclusion reasons available for auditing. -- Investigate large rank shifts and anomaly flags before publication. -- Do not interpret a missing metric as a confirmed zero unless the relevant scoring rule explicitly defines that behavior. +## 15. Security & Data Governance -## Limitations +The system follows basic operational and data governance practices. -- API access, quotas, and authentication can affect data availability. +- Store API credentials in environment variables. +- Never commit secrets or `.env` files. +- Preserve missing values instead of introducing synthetic zeros. +- Validate artist identity before assigning platform evidence credit. +- Maintain source timestamps and sales provenance. +- Investigate anomaly flags and significant ranking movements. +- Preserve published baseline rankings during simulations. +- Review commercial estimates before interpreting them as verified sales. + +--- + +## 16. Limitations + +- Platform API access, authentication, and request quotas can affect data availability. - Platform metrics may have different refresh schedules and definitions. -- Similar artist names and unofficial social accounts can produce ambiguous identity matches. -- Anomaly rules identify unusual relationships, not necessarily fraudulent activity. -- Rankings depend on data quality, available signals, and the configured weighting methodology. -- Automated tests cannot guarantee that every live platform account or metric is correct. +- Artist aliases and unofficial social accounts can create identity-matching challenges. +- Commercial sales estimates may vary across data providers. +- RIAA certifications represent US domestic certification evidence, not complete global sales. +- Estimated streaming-equivalent sales depend on configured conversion assumptions. +- Anomaly detection identifies unusual relationships but does not prove fraudulent activity. +- Simulation results are hypothetical and do not automatically change official published rankings. +- Automated tests cannot guarantee the accuracy of every live platform metric or commercial estimate. -## Future Improvements +--- -- Expand identity verification using reliable official account evidence. -- Improve source-level data validation and stale-data monitoring. -- Add operational alerts for failed or incomplete ingestion runs. -- Strengthen checkpoint recovery and API retry behavior. -- Track ranking changes across releases and review significant shifts. -- Automate recurring regression reports and release checks. -## Production Deployment & Operations Runbook +## 17. Future Improvements -### Dashboard Service (`systemd`) +- Expand reliable multi-platform identity verification. +- Improve commercial sales source reconciliation and provenance validation. +- Introduce additional artist growth and engagement signals. +- Strengthen ingestion retry mechanisms and checkpoint recovery. +- Improve automated anomaly detection and data freshness monitoring. +- Expand commercial sensitivity and simulation reporting. +- Introduce operational alerts for failed or incomplete pipeline runs. +- Explore LLM-based artist skill extraction and music-industry demand/supply intelligence. +- Track changes in artist skills, genre demand, and market opportunities over time. -The multi-platform Streamlit dashboard runs in the background as a Linux systemd service. - -**Check service status** -```bash -sudo systemctl status hiffi-dashboard --no-pager -``` - -**Restart after approved code or dataset updates** -```bash -sudo systemctl restart hiffi-dashboard -``` - -**View live application logs** -```bash -sudo journalctl -u hiffi-dashboard -f -``` - -**View the latest 100 log entries** -```bash -sudo journalctl -u hiffi-dashboard -n 100 --no-pager -``` - -**Enable automatic startup after server reboot** -```bash -sudo systemctl enable hiffi-dashboard -``` - -### Server Health Checks - -Check available disk space: -```bash -df -h -``` - -Check memory usage: -```bash -free -h -``` - -List running application sessions: -```bash -tmux ls -``` - -### Operational Notes - -- Confirm the service is active before restarting or stopping it. -- Check application logs if the dashboard becomes unavailable. -- Verify dashboard access after any restart or deployment. -- Back up important files before making production changes. -- Do not expose API keys, passwords, or other secrets in the repository. -- Confirm the configured service name and deployment paths before running commands. +--- ## Summary -HIFFI Hip-Hop 500 separates platform data ingestion from ranking and governance. Its scoring, identity checks, freshness controls, anomaly flags, and regression reports are designed to make multi-platform rankings more consistent, explainable, and auditable. +HIFFI Artist Intelligence Engine combines multi-platform artist performance analytics with commercial sales intelligence in a modular and auditable architecture. -For production use, verify that the documented entry points, thresholds, checkpoint behavior, and output paths match the implementation in the released repository. +Its ranking engine evaluates streaming and audience signals through weighted scoring, while the Commercial Power Index introduces normalized commercial performance into an isolated simulation environment. + +With automated data governance, provenance tracking, regression auditing, REST API services, executive reporting, and an interactive dashboard, the platform provides a structured foundation for artist intelligence and commercial analytics. + +**Built on measurable signals, validated evidence, and explainable analytics.** \ No newline at end of file