srilekha/hiffi-artist-intelligence
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(pipeline): complete end-to-end multiplatform ingestion, QA governance, zero anomalies, and docs

Browse Source
This commit is contained in:
Srilekha committed 2026-10-24 15:57:20 +05:30
1 parent 62aea7a85b
commit 05da5050df
13 files changed
+1874 -1103
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

README.md
+308 -211
View File
@@ -1,283 +1,380 @@
# HIFFI Artist Intelligence Pipeline
# HIFFI Multi-Platform Artist Intelligence Pipeline
An automated data pipeline for analyzing and ranking 500 hip-hop artists using YouTube performance, Instagram audience reach, and cross-platform artist identity resolution.
An auditable data pipeline for collecting, validating, and analyzing artist metrics across YouTube, Spotify, and Instagram for a cohort of 500 hip-hop artists.
The project maintains an existing YouTube-based production ranking alongside an experimental multi-platform ranking, with explicit handling of missing data and platform access limitations.
The project combines platform data collection, artist verification, ranking calculations, data-quality checks, and audit reporting to support reliable artist analysis.
## Table of Contents
## 1. Project Objectives
- [Overview](#overview)
- [Architecture](#architecture)
- [Project Structure](#project-structure)
- [Data Rules and Governance](#data-rules-and-governance)
- [Setup](#setup)
- [How to Run](#how-to-run)
- [Output Dataset](#output-dataset)
- [Limitations](#limitations)
- Process a cohort of 500 unique hip-hop artists.
- Collect available metrics from YouTube, Spotify, and Instagram.
- Validate artist profiles and track verification confidence.
- Preserve missing values and avoid misleading zero-imputation.
- Calculate HIFFI scores and Breakout Velocity Index (BVI).
- Compare artist rankings and maintain data provenance.
- Validate pipeline behavior through automated tests.
## Overview
## 2. Key Features
The pipeline is designed to:
- **Multi-platform data collection:** Collects available metrics from three platforms.
- **Spotify fallback strategy:** Uses a tiered collection approach involving Kworb, the Spotify Web API, and an embed fallback.
- **Instagram verification:** Uses candidate matching, confidence scoring, and manual overrides.
- **Ranking engine:** Calculates weighted HIFFI scores and a separate BVI.
- **Data governance:** Tracks missing values, verification status, and collection timestamps.
- **Automated QA:** Includes 14 documented Pytest checks.
- **Audit-ready output:** Exports consolidated artist metrics and ranking information.
1. **Collect YouTube metrics** — Process video views, recent view velocity, and available channel statistics.
2. **Resolve artist identities** — Use Wikidata and identifier matching to find candidate Spotify IDs and Instagram profiles.
3. **Collect Instagram metrics** — Extract publicly accessible follower information where available.
4. **Validate data quality** — Detect missing values, duplicate records, suspicious handles, and potential identity mismatches.
5. **Calculate artist rankings** — Maintain two separate ranking outputs:
- **Production Rank:** The existing baseline based on validated YouTube performance data.
- **HIFFI-XP Rank:** An experimental ranking combining normalized YouTube performance and Instagram audience reach.
## Architecture
## 3. System Workflow
```mermaid
flowchart TD
A[Seed Artist Dataset - 500 Artists] --> B[Artist Identity Resolution]
B --> C[Candidate Platform Profiles]
A --> D[YouTube Metrics]
C --> E[Instagram Metrics]
C --> F[Spotify API Diagnostics]
D --> G[Data Quality Validation]
E --> G
F --> G
G --> H[Production YouTube Ranking]
G --> I[Experimental HIFFI-XP Ranking]
H --> J[Rankings and Reports]
I --> J
J --> K[CSV Output and Checkpoints]
A["500-Artist Cohort"] --> B["YouTube Data Collection"]
A --> C["Spotify Data Collection"]
A --> D["Instagram Profile Verification"]
B --> E["Data Validation"]
C --> E
D --> E
E --> F["HIFFI Score and BVI"]
F --> G["Ranking and QA Checks"]
G --> H["Consolidated Dataset and Report"]
```
## Project Structure
## 4. Data Collection
### YouTube
Collects channel and video metrics used by the ranking engine, including:
- Subscriber count
- Lifetime views
- Daily view velocity
- Recent momentum
- Video count
### Spotify
Uses a three-tier collection strategy:
| Tier | Source | Purpose |
|---|---|---|
| 1 | Kworb | Collect available public streaming metrics. |
| 2 | Spotify Web API | Attempt authenticated metric retrieval. |
| 3 | Spotify embed | Attempt fallback retrieval for supported artist IDs. |
The collector includes handling for API errors and unavailable data. Actual coverage depends on source availability and access restrictions.
### Instagram
Evaluates candidate artist profiles using name matching, biography keywords, follower-count rules, and confidence checks. Manual overrides are supported through `data/handle_overrides.json`.
Automated matching is a verification aid, not a guarantee that an account is official.
## 5. Ranking Methodology
The ranking engine calculates two distinct metrics: the **HIFFI Score** and the **Breakout Velocity Index (BVI)**.
### 5.1 HIFFI Score
The HIFFI Score uses four weighted signals:
| Signal | Weight |
|---|---:|
| Recent Momentum | 35% |
| View Velocity | 30% |
| Audience Reach | 20% |
| Consistency | 15% |
| **Total** | **100%** |
#### Step 1: Normalize the metrics
Logarithmic normalization is used for daily view velocity, subscriber count, lifetime views, and video count.
\[
L(x)=
\frac{\log_{10}(1+x)-z_{\min}}
{z_{\max}-z_{\min}+10^{-9}}
\]
Here, \(z_{\min}\) and \(z_{\max}\) are the minimum and maximum log-transformed values.
Recent Momentum uses Min-Max normalization after clipping momentum to the range \([-100,300]\):
\[
M(x)=\frac{x-x_{\min}}{x_{\max}-x_{\min}+10^{-9}}
\]
The implementation returns zero scores when the valid values are constant or unavailable.
#### Step 2: Calculate Audience Reach
Audience Reach combines subscriber count and lifetime views:
\[
S_{\text{reach}}=
0.60L(\text{subscribers})
+0.40L(\text{lifetime views})
\]
#### Step 3: Calculate the raw score
\[
\begin{aligned}
H_{\text{raw}}=100\times(&0.35S_{\text{momentum}}\\
&+0.30S_{\text{velocity}}\\
&+0.20S_{\text{reach}}\\
&+0.15S_{\text{consistency}})
\end{aligned}
\]
#### Step 4: Apply confidence adjustment
\[
C_{\text{multiplier}}=0.8+0.2C
\]
\[
\boxed{H_{\text{final}}=H_{\text{raw}}\times C_{\text{multiplier}}}
\]
Here, \(C\) is the confidence score clipped to \([0,1]\). The final score is rounded to two decimal places.
For example, if the raw score is 68.50 and confidence is 0.90:
\[
68.50\times(0.8+0.2\times0.90)=67.13
\]
The final HIFFI Score is **67.13**.
### 5.2 HIFFI Ranking Eligibility
An artist qualifies for the primary ranking when:
- Subscriber count and lifetime views are available.
- Confidence score is at least 0.50 by default.
- Status is `COMPLETE` or `CATALOG_ACTIVE_NO_RECENT_UPLOAD`.
Eligible artists are ranked by final HIFFI Score in descending order. Tied scores receive the same minimum rank.
### 5.3 Breakout Velocity Index (BVI)
BVI evaluates relative view velocity and recent momentum among qualifying artists.
**Eligibility requirements:**
- At least 10,000 subscribers.
- Positive daily view velocity.
- Available subscriber count and daily view velocity.
- Status is neither `INACTIVE_OR_DORMANT` nor `FAILED_LOOKUP`.
The BVI calculation uses three stages.
**Audience velocity ratio**
\[
V_{\text{aud}}=
\frac{\text{daily view velocity}}
{\max(\text{subscribers},1000)}
\]
**Momentum signal**
\[
M_{\text{signal}}=
\frac{\text{momentum}_{30d}+100}{400}
\]
**Emergence dampener**
\[
W_{\text{emerge}}=
\frac{1}{1+\log_{10}(1+\text{subscribers})}
\]
The raw BVI is calculated as:
\[
BVI_{\text{raw}}=
\left(
0.65\log_{10}(1+V_{\text{aud}})
+0.35M_{\text{signal}}
\right)
W_{\text{emerge}}
\]
Finally, qualifying BVI values are min-max scaled to a 0–100 range. If all qualifying raw values are equal, the implementation assigns a BVI of 50.
The `breakout_rank` is calculated separately using BVI in descending order. Artists without a valid BVI are not included in this ranking.
**Note:** The 10,000-subscriber threshold applies to BVI eligibility, not primary HIFFI ranking eligibility.
## 6. Data Governance
The pipeline follows these principles:
- Missing platform metrics should not automatically be interpreted as genuine zero values.
- Artist profile verification status and confidence should be recorded where available.
- Collection timestamps and source information support traceability.
- Failed lookups and uncertain records must be distinguishable from successfully collected data.
- Primary HIFFI scores and BVI rankings use separate eligibility rules.
**Implementation note:** The current ranking engine fills several missing metric inputs with zero before scoring. This behavior should be distinguished from preserving missing values in the collected dataset.
## 7. Repository Structure
```text
hiffi-artist-intelligence/
├── collectors/
│ ├── instagram_collector.py
│ ├── instagram_verifier.py
│ ├── kworb_spotify_collector.py
│ ├── spotify_api_collector.py
│ ├── spotify_fallback_collector.py
│ └── youtube_collector.py
├── config/
│ └── Pipeline and API configuration
├── core/
│ ├── confidence_scorer.py
│ ├── deduplication.py
│ ├── health_detector.py
│ └── ranking_engine.py
├── data/
│ ├── checkpoints/
│ ├── generate_seed_500.py
│ ├── handle_overrides.json
│ ├── hiffi_multiplatform_500.csv
│ ├── roster_loader.py
│ ├── seed_artists_50.py
│ └── seed_artists_500.csv
│ └── hiffi_multiplatform_500.csv
├── pipeline/
│ ├── audit_instagram_handles.py
│ ├── batch_500_runner.py
│ ├── generate_report.py
│ ├── multiplatform_scoring.py
│ └── patch_anomalies.py
├── storage/
├── verifiers/
├── app.py
│ └── batch_500_runner.py
├── main.py
├── enriched_artists.csv
├── requirements.txt
├── test_multiplatform_ranking.py
├── .env
└── .gitignore
├── requirements.txt
└── README.md
```
*Directory descriptions are illustrative. Keep only files and directories that exist in the actual repository.*
## Data Rules and Governance
### 1. No Fabricated Metrics
- Missing metrics remain `NaN` or `NULL`, according to the storage format.
- Estimated values must not be presented as live platform measurements.
- Missing data must be handled consistently during normalization and scoring.
### 2. Spotify API Status
Spotify API access currently returns HTTP 403 for the configured request.
- Spotify IDs and Spotify performance metrics are treated separately.
- Unavailable Spotify metrics remain missing.
- The pipeline records the relevant error status for traceability.
- Spotify metrics must not be described as collected until a working, authorized data source supplies them.
Example status:
```text
spotify_status = BLOCKED_HTTP_403
```
Use `BLOCKED_HTTP_403_PREMIUM_REQUIRED` only if the actual diagnostic confirms that specific reason. An HTTP 403 alone does not establish its cause.
### 3. Artist Identity Validation
- Treat external identifiers as candidate matches until validated.
- Use official cross-links and other reliable evidence where available.
- Flag ambiguous accounts and suspicious follower counts for review.
- Record manual handle corrections in the appropriate override file.
### 4. Checkpoint Protection
The batch runner is designed to save intermediate progress so processing can resume after interruption. Checkpoint recovery depends on the runner's implementation and the availability of compatible saved checkpoint files.
## Setup
## 8. Installation and Setup
### Prerequisites
- Python 3.10 or later
- Python 3.12+
- Git
- A virtual environment
- Valid credentials for any external APIs used
- Spotify API credentials, if required by the configured collector
### 1. Clone the Repository
### Clone the repository
```bash
git clone <repository-url>
git clone https://github.com/<your-username>/hiffi-artist-intelligence.git
cd hiffi-artist-intelligence
```
Replace `<repository-url>` with the actual repository clone URL.
### 2. Create a Virtual Environment
**Windows PowerShell**
### Create a virtual environment
```powershell
python -m venv .venv
.venv\Scripts\Activate.ps1
.\.venv\Scripts\Activate.ps1
```
**macOS / Linux**
### Install dependencies
```bash
python3 -m venv .venv
source .venv/bin/activate
```
### 3. Install Dependencies
```bash
python -m pip install --upgrade pip
pip install -r requirements.txt
playwright install chromium
```
## How to Run
### Configure Spotify API credentials
Run the commands from the repository root with the virtual environment activated.
Create a `.env` file in the project root when using the official Spotify API collector:
### 1. Run the 500-Artist Pipeline
```dotenv
SPOTIFY_CLIENT_ID=your_client_id_here
SPOTIFY_CLIENT_SECRET=your_client_secret_here
```
Never commit API credentials or secrets to GitHub.
## 9. Execution Commands
Run the following commands from the repository root.
| Operation | Command |
|---|---|
| Full workflow | `python main.py all` |
| Batch pipeline | `python main.py pipeline` |
| Health checks | `python main.py health` |
| QA tests | `pytest test_multiplatform_ranking.py -v` |
| Executive report | `python main.py report` |
Execution results depend on the configured inputs, dependencies, and availability of external data sources.
## 10. QA and Validation
The repository documents 14 tests covering:
1. Cohort size and uniqueness.
2. Missing-value and streaming-data governance.
3. Preservation of production rankings.
4. Score normalization bounds.
5. Ranking monotonicity.
6. Required provenance columns.
7. Instagram candidate matching.
8. Suspicious Instagram follower counts.
9. Spotify fallback null safety.
10. Mocked network-error handling.
11. Duplicate-record safeguards.
12. Unconfigured Spotify API behavior.
13. Empty or malformed Spotify IDs.
14. Additional validation of the pipeline's ranking and data-integrity rules.
Run the test suite with:
```bash
python -m pipeline.batch_500_runner
pytest test_multiplatform_ranking.py -v
```
This is the main batch-processing entry point. Depending on the implementation, it processes the configured artist dataset, collects available signals, validates records, and calculates ranking scores.
The README documents the intended checks; test results should be verified against the current code before reporting them as passing.
### 2. Audit Instagram Handles
## 11. Output Dataset
```bash
python -m pipeline.audit_instagram_handles
```
Use the audit output to identify potential incorrect profiles, fan pages, or suspicious metric values.
### 3. Apply Targeted Corrections
```bash
python -m pipeline.patch_anomalies
```
Review proposed corrections before applying them. Preserve the original mappings and document changes.
### 4. Generate the Executive Report
```bash
python -m pipeline.generate_report
```
The report is intended to summarize ranking changes, leading gainers and fallers, and dataset-level metrics.
### 5. Run Ranking Tests
```bash
pytest -q test_multiplatform_ranking.py
```
Run the tests after modifying normalization, weighting, missing-value handling, or ranking logic.
## Output Dataset
The intended final dataset is:
The consolidated dataset is exported to:
`data/hiffi_multiplatform_500.csv`
Key columns may include the following, depending on the latest pipeline output.
Important fields include:
| Column | Description |
|---|---|
| `name` | Artist stage name |
| `production_hiffi_rank` | Rank from the existing YouTube-based production baseline |
| `hiffi_xp_rank` | Rank from the experimental multi-platform model |
| `hiffi_xp_score` | Experimental composite score |
| `instagram_followers` | Collected Instagram follower count, when available |
| `spotify_id` | Resolved Spotify artist identifier, when available |
| `spotify_status` | Spotify collection or API status |
| `name` | Artist name |
| `subscriber_count` | YouTube subscriber count |
| `lifetime_views` | Total YouTube views |
| `daily_view_velocity` | Daily view velocity |
| `momentum_30d` | Recent momentum |
| `video_count` | Video count |
| `confidence_score` | Confidence used for score adjustment and eligibility |
| `hiffi_score` | Final HIFFI Score |
| `hiffi_rank` | Primary ranking |
| `breakout_velocity_index` | BVI score |
| `breakout_rank` | BVI ranking |
| `is_rank_eligible` | Primary ranking eligibility flag |
| `status_flag` | Record or collection status |
Check the generated CSV to confirm the exact column names and available values.
The complete output schema may include additional Spotify, Instagram, provenance, and timestamp fields.
## Ranking Methodology
## 12. Limitations
### Production Ranking
- External APIs and public data sources may return incomplete data or restrict access.
- Artist names and unofficial profiles can create identity-matching errors.
- Missing metric handling can influence ranking outcomes.
- Min-Max normalization depends on the comparison cohort.
- Video count is a proxy for consistency, not a direct measure of upload regularity.
- BVI measures relative breakout signals under the implemented formula, not guaranteed future popularity.
- HIFFI scores reflect the configured metrics and weights, not artistic quality.
The production ranking uses the existing YouTube scoring implementation:
## 13. Future Improvements
\[
S_{\text{production}} = S_{\text{YouTube}}
\]
- Improve artist identity verification and matching confidence.
- Strengthen API failure monitoring and source coverage reporting.
- Expand edge-case tests for normalization and missing metrics.
- Add historical snapshots and ranking-change reports.
- Improve documentation of data freshness and score eligibility.
### Experimental Ranking — HIFFI-XP
## Conclusion
The experimental model combines normalized YouTube performance and Instagram audience reach.
The HIFFI Multi-Platform Artist Intelligence Pipeline combines data collection, artist verification, ranking calculations, and automated quality checks to support structured analysis of a 500-artist hip-hop cohort.
\[
S_{\text{HIFFI-XP}} =
w_Y N(S_{\text{YouTube}})
+ w_I N(F_{\text{Instagram}})
\]
Its key principles are **traceable data, explicit ranking rules, transparent confidence adjustments, and careful handling of missing information**.
Where:
- \(N(S_{\text{YouTube}})\) is the normalized YouTube score.
- \(N(F_{\text{Instagram}})\) is the normalized Instagram follower count.
- \(w_Y\) and \(w_I\) are the configured weights, which should sum to 1.
The actual configured weights should be documented from the scoring code rather than assumed from this formula.
Missing Instagram metrics should not automatically be treated as zero. Document the chosen missing-data policy and report the number of artists eligible for each ranking.
Spotify remains a planned additional signal until live metrics are available through an authorized source.
## Limitations
- Spotify API access is currently blocked for the configured request.
- Instagram metadata may be unavailable, outdated, or affected by rate limits and website changes.
- Candidate profile links may not always identify the correct artist.
- Platform metrics measure different types of audience activity and require appropriate normalization.
- Experimental scores should be validated before being used as production rankings.
- Dataset coverage and ranking results may change between collection runs.
## Future Improvements
- Integrate an authorized source for Spotify performance metrics.
- Improve cross-platform identity matching and confidence checks.
- Add collection timestamps and source-level provenance to all metrics.
- Strengthen retry handling, rate-limit management, and checkpoint recovery.
- Expand automated tests for data quality and ranking stability.
- Generate repeatable weekly ranking and coverage reports.
---
**Project principle:** Keep observed metrics, unresolved identities, and unavailable platform data clearly separated. This ensures that rankings remain traceable, reproducible, and honest about data limitations.
The HIFFI Score and BVI represent the formulas implemented in this repository and should be distinguished from any separate official HIFFI website ranking methodology.