srilekha/hiffi-artist-intelligence
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: complete production README with formulas, governance invariants, and CI setup
HIFFI Multi-Platform CI / test (3.11) (push) Successful in 30s
Details
HIFFI Multi-Platform CI / test (3.12) (push) Successful in 31s
Details

Browse Source
This commit is contained in:
Srilekha committed 2026-10-26 15:18:14 +05:30
1 parent d8af95cd90
commit 183af17263
1 file changed
+181 -268
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+181 -268
View File
@@ -1,259 +1,166 @@
# HIFFI Multi-Platform Artist Intelligence Pipeline
# HIFFI Hip-Hop 500 — Multi-Platform Artist Intelligence Pipeline
An auditable data pipeline for collecting, validating, and analyzing artist metrics across YouTube, Spotify, and Instagram for a cohort of 500 hip-hop artists.
An end-to-end Python data pipeline that collects, validates, and ranks a cohort of 500 hip-hop artists using data from **YouTube, Spotify, and Instagram**.
The project combines platform data collection, artist verification, ranking calculations, data-quality checks, and audit reporting to support reliable artist analysis.
The project focuses on reliable data collection, transparent ranking calculations, artist identity verification, anomaly detection, and automated testing.
## 1. Project Objectives
## Key Features
- Process a cohort of 500 unique hip-hop artists.
- Collect available metrics from YouTube, Spotify, and Instagram.
- Validate artist profiles and track verification confidence.
- Preserve missing values and avoid misleading zero-imputation.
- Calculate HIFFI scores and Breakout Velocity Index (BVI).
- Compare artist rankings and maintain data provenance.
- Validate pipeline behavior through automated tests.
- **Multi-platform data collection:** Collects available artist metrics from YouTube, Spotify, and Instagram.
- **Artist verification:** Uses identity matching, confidence checks, and manual review for uncertain profiles.
- **Weighted ranking:** Calculates production scores, HIFFI-XP scores, and the Breakout Velocity Index (BVI).
- **Missing-data handling:** Preserves missing source metrics as `NaN` and redistributes scoring weights across available signals.
- **Data freshness checks:** Excludes stale or invalid signals from publication evidence.
- **Anomaly detection:** Flags suspicious metrics for further review.
- **Ranking regression:** Compares current rankings with historical snapshots.
- **Audit reporting:** Generates a Markdown report with ranking and data-quality information.
- **Continuous Integration:** Runs automated tests on Python 3.11 and 3.12 using GitHub Actions.
## 2. Key Features
## 1. How the Pipeline Works
- **Multi-platform data collection:** Collects available metrics from three platforms.
- **Spotify fallback strategy:** Uses a tiered collection approach involving Kworb, the Spotify Web API, and an embed fallback.
- **Instagram verification:** Uses candidate matching, confidence scoring, and manual overrides.
- **Ranking engine:** Calculates weighted HIFFI scores and a separate BVI.
- **Data governance:** Tracks missing values, verification status, and collection timestamps.
- **Automated QA:** Includes 14 documented Pytest checks.
- **Audit-ready output:** Exports consolidated artist metrics and ranking information.
The pipeline follows five main stages:
## 3. System Workflow
1. **Load and collect data** — Load the artist cohort and retrieve available platform metrics.
2. **Clean and validate** — Check missing values, data quality, artist identities, and timestamps.
3. **Calculate scores** — Normalize available metrics and calculate weighted artist scores.
4. **Apply governance checks** — Evaluate freshness, platform evidence, eligibility, and suspicious patterns.
5. **Publish and audit** — Generate rankings, compare historical results, and create the audit report.
```mermaid
flowchart TD
A["500-Artist Cohort"] --> B["YouTube Data Collection"]
A --> C["Spotify Data Collection"]
A --> D["Instagram Profile Verification"]
B --> E["Data Validation"]
C --> E
D --> E
E --> F["HIFFI Score and BVI"]
F --> G["Ranking and QA Checks"]
G --> H["Consolidated Dataset and Report"]
```
## 2. Ranking Methodology
## 4. Data Collection
The project calculates three main outputs: the YouTube production score, the multi-platform HIFFI-XP score, and the Breakout Velocity Index (BVI).
### YouTube
### HIFFI-XP Composite Score
Collects channel and video metrics used by the ranking engine, including:
- Subscriber count
- Lifetime views
- Daily view velocity
- Recent momentum
- Video count
### Spotify
Uses a three-tier collection strategy:
| Tier | Source | Purpose |
|---|---|---|
| 1 | Kworb | Collect available public streaming metrics. |
| 2 | Spotify Web API | Attempt authenticated metric retrieval. |
| 3 | Spotify embed | Attempt fallback retrieval for supported artist IDs. |
The collector includes handling for API errors and unavailable data. Actual coverage depends on source availability and access restrictions.
### Instagram
Evaluates candidate artist profiles using name matching, biography keywords, follower-count rules, and confidence checks. Manual overrides are supported through `data/handle_overrides.json`.
Automated matching is a verification aid, not a guarantee that an account is official.
## 5. Ranking Methodology
The ranking engine calculates two distinct metrics: the **HIFFI Score** and the **Breakout Velocity Index (BVI)**.
### 5.1 HIFFI Score
The HIFFI Score uses four weighted signals:
The default weights are:
| Signal | Weight |
|---|---:|
| Recent Momentum | 35% |
| View Velocity | 30% |
| Audience Reach | 20% |
| Consistency | 15% |
| YouTube production score | 50% |
| Spotify monthly listeners | 25% |
| Spotify daily velocity | 5% |
| Instagram followers | 20% |
| **Total** | **100%** |
#### Step 1: Normalize the metrics
Each metric is normalized before combining it into the composite score.
Logarithmic normalization is used for daily view velocity, subscriber count, lifetime views, and video count.
### Dynamic Weight Redistribution
\[
L(x)=
\frac{\log_{10}(1+x)-z_{\min}}
{z_{\max}-z_{\min}+10^{-9}}
\]
When some platform metrics are unavailable, the remaining available signals share the weight proportionally.
Here, \(z_{\min}\) and \(z_{\max}\) are the minimum and maximum log-transformed values.
Examples:
Recent Momentum uses Min-Max normalization after clipping momentum to the range \([-100,300]\):
- **YouTube only:** YouTube receives 100%.
- **YouTube + Spotify monthly listeners:** YouTube receives 66.7%, and Spotify receives 33.3%.
- **All signals available:** The original weights are retained.
\[
M(x)=\frac{x-x_{\min}}{x_{\max}-x_{\min}+10^{-9}}
\]
The `effective_weights` column records the actual weights used for each artist.
The implementation returns zero scores when the valid values are constant or unavailable.
Missing metrics are preserved as `NaN` in the source dataset rather than being treated as genuine zero values. Missing inputs may be handled separately during individual scoring calculations, so score interpretation should account for the implemented normalization and missing-data rules.
#### Step 2: Calculate Audience Reach
### YouTube Production Score
Audience Reach combines subscriber count and lifetime views:
The production score combines four signals:
\[
S_{\text{reach}}=
0.60L(\text{subscribers})
+0.40L(\text{lifetime views})
\]
| Signal | Weight |
|---|---:|
| Recent momentum | 35% |
| Daily view velocity | 30% |
| Audience reach | 20% |
| Consistency proxy | 15% |
#### Step 3: Calculate the raw score
Audience reach combines subscriber count and lifetime views. Video count is used as a consistency proxy.
\[
\begin{aligned}
H_{\text{raw}}=100\times(&0.35S_{\text{momentum}}\\
&+0.30S_{\text{velocity}}\\
&+0.20S_{\text{reach}}\\
&+0.15S_{\text{consistency}})
\end{aligned}
\]
### Breakout Velocity Index (BVI)
#### Step 4: Apply confidence adjustment
BVI measures relative breakout signals using view velocity, recent momentum, and an audience-size adjustment.
\[
C_{\text{multiplier}}=0.8+0.2C
\]
Eligibility requires at least 10,000 subscribers, positive daily view velocity, and an eligible collection status. BVI is calculated and ranked separately from the primary production score.
\[
\boxed{H_{\text{final}}=H_{\text{raw}}\times C_{\text{multiplier}}}
\]
> **Note:** The production score, HIFFI-XP composite score, and BVI are separate calculations. They should not be confused with one another or assumed to be identical to any separate official HIFFI website formula.
Here, \(C\) is the confidence score clipped to \([0,1]\). The final score is rounded to two decimal places.
## 3. Data Governance
For example, if the raw score is 68.50 and confidence is 0.90:
### Freshness Rules
\[
68.50\times(0.8+0.2\times0.90)=67.13
\]
| Platform | Maximum signal age |
|---|---:|
| YouTube | 7 days |
| Spotify | 7 days |
| Instagram | 14 days |
The final HIFFI Score is **67.13**.
Missing, malformed, future-dated, or stale timestamps fail the freshness check. Excluded signals are recorded in `evidence_exclusions`.
### 5.2 HIFFI Ranking Eligibility
### Artist Identity Verification
An artist qualifies for the primary ranking when:
Instagram identity status is checked before the platform contributes to publication evidence. The pipeline supports automated verification, manual verification, and a review process for uncertain accounts.
- Subscriber count and lifetime views are available.
- Confidence score is at least 0.50 by default.
- Status is `COMPLETE` or `CATALOG_ACTIVE_NO_RECENT_UPLOAD`.
Automated matching helps identify likely official profiles, but it cannot guarantee that every matched account belongs to the correct artist.
Eligible artists are ranked by final HIFFI Score in descending order. Tied scores receive the same minimum rank.
### Publication Eligibility
### 5.3 Breakout Velocity Index (BVI)
- **Top 10:** Requires Tier 3 platform evidence.
- **Top 50:** Requires at least Tier 2 platform evidence.
- **Published ranks:** Assigned consecutively.
- **Shortfall handling:** When strict evidence requirements cannot fill the target positions, fallback selections are recorded in the shortfall audit metadata.
BVI evaluates relative view velocity and recent momentum among qualifying artists.
The audit report should be reviewed for any shortfalls before the rankings are treated as fully compliant with the intended eligibility rules.
**Eligibility requirements:**
### Anomaly Detection
- At least 10,000 subscribers.
- Positive daily view velocity.
- Available subscriber count and daily view velocity.
- Status is neither `INACTIVE_OR_DORMANT` nor `FAILED_LOOKUP`.
The `anomaly_flags` field records suspicious patterns, including:
The BVI calculation uses three stages.
- `SUSPICIOUS_IG_TO_YT_RATIO` — Instagram followers exceed 10 times YouTube lifetime views.
- `EXTREME_SPOTIFY_VELOCITY_RATIO` — Spotify daily change exceeds 25% of monthly listeners.
**Audience velocity ratio**
These rules flag records for investigation. An anomaly is not proof of fraud or incorrect data.
\[
V_{\text{aud}}=
\frac{\text{daily view velocity}}
{\max(\text{subscribers},1000)}
\]
**Momentum signal**
\[
M_{\text{signal}}=
\frac{\text{momentum}_{30d}+100}{400}
\]
**Emergence dampener**
\[
W_{\text{emerge}}=
\frac{1}{1+\log_{10}(1+\text{subscribers})}
\]
The raw BVI is calculated as:
\[
BVI_{\text{raw}}=
\left(
0.65\log_{10}(1+V_{\text{aud}})
+0.35M_{\text{signal}}
\right)
W_{\text{emerge}}
\]
Finally, qualifying BVI values are min-max scaled to a 0–100 range. If all qualifying raw values are equal, the implementation assigns a BVI of 50.
The `breakout_rank` is calculated separately using BVI in descending order. Artists without a valid BVI are not included in this ranking.
**Note:** The 10,000-subscriber threshold applies to BVI eligibility, not primary HIFFI ranking eligibility.
## 6. Data Governance
The pipeline follows these principles:
- Missing platform metrics should not automatically be interpreted as genuine zero values.
- Artist profile verification status and confidence should be recorded where available.
- Collection timestamps and source information support traceability.
- Failed lookups and uncertain records must be distinguishable from successfully collected data.
- Primary HIFFI scores and BVI rankings use separate eligibility rules.
**Implementation note:** The current ranking engine fills several missing metric inputs with zero before scoring. This behavior should be distinguished from preserving missing values in the collected dataset.
## 7. Repository Structure
## 4. Project Structure
```text
hiffi-artist-intelligence/
├── .github/
│ └── workflows/
│ └── ci.yml
├── collectors/
│ ├── youtube_collector.py
│ ├── spotify_api_collector.py
│ ├── spotify_collector.py
│ ├── kworb_spotify_collector.py
│ ├── spotify_fallback_collector.py
│ ├── instagram_collector.py
│ ├── instagram_verifier.py
│ ├── kworb_spotify_collector.py
│ ├── spotify_api_collector.py
│ ├── spotify_fallback_collector.py
│ └── youtube_collector.py
│ └── wikidata_resolver.py
├── config/
├── core/
│ ├── ranking_engine.py
│ ├── confidence_scorer.py
│ ├── deduplication.py
│ ├── health_detector.py
│ └── ranking_engine.py
│ └── regression_auditor.py
├── data/
│ ├── snapshots/
│ ├── checkpoints/
│ ├── handle_overrides.json
│ ├── review_queue.csv
│ └── hiffi_multiplatform_500.csv
├── pipeline/
│ └── batch_500_runner.py
├── main.py
├── reports/
│ ├── audit_report_latest.md
│ └── provenance_summary.py
├── run_pipeline.py
├── test_multiplatform_ranking.py
├── requirements.txt
└── README.md
```
## 8. Installation and Setup
## 5. Installation and Setup
### Prerequisites
- Python 3.12+
- Python 3.11 or 3.12
- Git
- Spotify API credentials, if required by the configured collector
- API credentials when required by the configured collectors
### Clone the repository
@@ -264,117 +171,123 @@ cd hiffi-artist-intelligence
### Create a virtual environment
**Windows PowerShell**
```powershell
python -m venv .venv
.\.venv\Scripts\Activate.ps1
```
### Install dependencies
```bash
pip install -r requirements.txt
```
### Configure Spotify API credentials
### Configure API credentials
Create a `.env` file in the project root when using the official Spotify API collector:
If the Spotify API collector requires credentials, create a local `.env` file:
```dotenv
SPOTIFY_CLIENT_ID=your_client_id_here
SPOTIFY_CLIENT_SECRET=your_client_secret_here
```text
SPOTIFY_CLIENT_ID=your_client_id
SPOTIFY_CLIENT_SECRET=your_client_secret
```
Never commit API credentials or secrets to GitHub.
Never commit real credentials, API keys, or tokens to GitHub. Keep `.env` excluded through `.gitignore` and use GitHub Secrets for CI credentials when needed.
## 9. Execution Commands
## 6. Run the Project
Run the following commands from the repository root.
Run commands from the repository root.
| Operation | Command |
|---|---|
| Full workflow | `python main.py all` |
| Batch pipeline | `python main.py pipeline` |
| Health checks | `python main.py health` |
| QA tests | `pytest test_multiplatform_ranking.py -v` |
| Executive report | `python main.py report` |
Execution results depend on the configured inputs, dependencies, and availability of external data sources.
## 10. QA and Validation
The repository documents 14 tests covering:
1. Cohort size and uniqueness.
2. Missing-value and streaming-data governance.
3. Preservation of production rankings.
4. Score normalization bounds.
5. Ranking monotonicity.
6. Required provenance columns.
7. Instagram candidate matching.
8. Suspicious Instagram follower counts.
9. Spotify fallback null safety.
10. Mocked network-error handling.
11. Duplicate-record safeguards.
12. Unconfigured Spotify API behavior.
13. Empty or malformed Spotify IDs.
14. Additional validation of the pipeline's ranking and data-integrity rules.
Run the test suite with:
### Execute the full pipeline
```bash
pytest test_multiplatform_ranking.py -v
python run_pipeline.py
```
The README documents the intended checks; test results should be verified against the current code before reporting them as passing.
This runs the configured stages for data loading, scoring, validation, governance, and audit generation.
## 11. Output Dataset
### Run the test suite
The consolidated dataset is exported to:
```bash
python -m pytest test_multiplatform_ranking.py -v
```
`data/hiffi_multiplatform_500.csv`
The current governance suite contains 27 tests, based on the latest reported test run.
Important fields include:
### Compare historical rankings
| Column | Description |
```bash
python -m core.regression_auditor
```
The regression auditor compares current rankings with saved snapshots and flags movements greater than 100 positions for review.
### Review the audit report
```text
reports/audit_report_latest.md
```
Review the report for missing data, evidence exclusions, anomalies, ranking movements, and publication shortfalls.
## 7. Continuous Integration
GitHub Actions runs automated tests on Ubuntu across Python 3.11 and Python 3.12.
The CI workflow is configured in:
```text
.github/workflows/ci.yml
```
The latest reported run passed on both Python versions, with all 27 governance tests passing.
CI results confirm that the tested code and scenarios pass in the configured environments. They do not independently guarantee the accuracy of external platform data or the reliability of every live collection run.
## 8. Output Dataset
The consolidated dataset is stored at:
```text
data/hiffi_multiplatform_500.csv
```
Important columns include:
| Column | Purpose |
|---|---|
| `name` | Artist name |
| `subscriber_count` | YouTube subscriber count |
| `lifetime_views` | Total YouTube views |
| `daily_view_velocity` | Daily view velocity |
| `momentum_30d` | Recent momentum |
| `video_count` | Video count |
| `confidence_score` | Confidence used for score adjustment and eligibility |
| `hiffi_score` | Final HIFFI Score |
| `hiffi_rank` | Primary ranking |
| `production_hiffi_score` | YouTube production score |
| `hiffi_xp_score` | Multi-platform composite score |
| `hiffi_xp_rank` | Raw composite ranking |
| `hiffi_xp_published_rank` | Publication ranking after eligibility rules |
| `breakout_velocity_index` | BVI score |
| `breakout_rank` | BVI ranking |
| `is_rank_eligible` | Primary ranking eligibility flag |
| `status_flag` | Record or collection status |
| `breakout_rank` | Separate BVI ranking |
| `effective_weights` | Actual scoring weights used |
| `platform_evidence_tier` | Count of qualifying platform signals |
| `evidence_exclusions` | Reasons signals were excluded from evidence |
| `anomaly_flags` | Suspicious patterns identified |
| `rank_delta` | Difference between production and composite ranks |
The complete output schema may include additional Spotify, Instagram, provenance, and timestamp fields.
Additional columns may contain platform metrics, verification statuses, collection timestamps, confidence values, and source provenance.
## 12. Limitations
## 9. Limitations
- External APIs and public data sources may return incomplete data or restrict access.
- Artist names and unofficial profiles can create identity-matching errors.
- Missing metric handling can influence ranking outcomes.
- Min-Max normalization depends on the comparison cohort.
- Video count is a proxy for consistency, not a direct measure of upload regularity.
- BVI measures relative breakout signals under the implemented formula, not guaranteed future popularity.
- HIFFI scores reflect the configured metrics and weights, not artistic quality.
- External APIs and public sources may provide incomplete data or restrict access.
- Artist names and similar handles can cause identity-matching errors.
- Missing metrics and normalization choices can affect ranking outcomes.
- Large ranking movements require investigation, not automatic correction.
- Anomaly flags identify patterns for review rather than confirmed wrongdoing.
- Video count is a proxy for consistency, not a direct measurement of upload regularity.
- BVI measures relative breakout signals; it does not predict future success with certainty.
- Rankings represent the configured metrics and weights, not artistic quality.
## 13. Future Improvements
## 10. Future Improvements
- Improve artist identity verification and matching confidence.
- Strengthen API failure monitoring and source coverage reporting.
- Expand edge-case tests for normalization and missing metrics.
- Add historical snapshots and ranking-change reports.
- Improve documentation of data freshness and score eligibility.
- Expand source coverage and identity verification.
- Improve API failure monitoring and recovery.
- Automate scheduled data collection and weekly audit generation.
- Strengthen historical regression reporting.
- Improve observability, release checks, and production deployment safeguards.
## Conclusion
The HIFFI Multi-Platform Artist Intelligence Pipeline combines data collection, artist verification, ranking calculations, and automated quality checks to support structured analysis of a 500-artist hip-hop cohort.
The HIFFI Hip-Hop 500 pipeline brings together multi-platform data collection, artist verification, ranking calculations, data governance, anomaly detection, regression analysis, and automated testing.
Its key principles are **traceable data, explicit ranking rules, transparent confidence adjustments, and careful handling of missing information**.
The HIFFI Score and BVI represent the formulas implemented in this repository and should be distinguished from any separate official HIFFI website ranking methodology.
Its main goal is to make artist rankings more **transparent, traceable, and repeatable**, while clearly identifying missing data and uncertain evidence.