srilekha/hiffi-artist-intelligence
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: align Instagram verification thresholds and document decoupled ingestion architecture

Browse Source
This commit is contained in:
Srilekha committed 2026-10-26 15:48:26 +05:30
1 parent 443ee92654
commit c63387e5b0
1 file changed
+166 -165
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+166 -165
View File
@@ -1,293 +1,294 @@
# HIFFI Hip-Hop 500 — Multi-Platform Artist Intelligence Pipeline # HIFFI Hip-Hop 500 — Multi-Platform Artist Intelligence Pipeline
An end-to-end Python data pipeline that collects, validates, and ranks a cohort of 500 hip-hop artists using data from **YouTube, Spotify, and Instagram**. A Python-based pipeline for collecting, validating, scoring, and ranking hip-hop artists using YouTube, Spotify, and Instagram signals. It includes data freshness checks, identity verification, anomaly detection, publication eligibility rules, and regression auditing.
The project focuses on reliable data collection, transparent ranking calculations, artist identity verification, anomaly detection, and automated testing.
## Key Features ## Key Features
- **Multi-platform data collection:** Collects available artist metrics from YouTube, Spotify, and Instagram. - Multi-platform artist metrics and scoring.
- **Artist verification:** Uses identity matching, confidence checks, and manual review for uncertain profiles. - Instagram identity matching with review-queue support.
- **Weighted ranking:** Calculates production scores, HIFFI-XP scores, and the Breakout Velocity Index (BVI). - Missing-data handling and dynamic weight redistribution.
- **Missing-data handling:** Preserves missing source metrics as `NaN` and redistributes scoring weights across available signals. - Platform freshness and publication eligibility gates.
- **Data freshness checks:** Excludes stale or invalid signals from publication evidence. - Cross-platform anomaly detection.
- **Anomaly detection:** Flags suspicious metrics for further review. - Regression auditing and rank-shift reports.
- **Ranking regression:** Compares current rankings with historical snapshots. - Automated tests and GitHub Actions CI for Python 3.11 and 3.12.
- **Audit reporting:** Generates a Markdown report with ranking and data-quality information.
- **Continuous Integration:** Runs automated tests on Python 3.11 and 3.12 using GitHub Actions.
## 1. How the Pipeline Works ## Pipeline Architecture: Telemetry Ingestion vs. Ranking Core
The pipeline follows five main stages: The project separates data collection from ranking so that platform data can be refreshed independently of score calculation.
1. **Load and collect data** — Load the artist cohort and retrieve available platform metrics. ### 1. Telemetry Ingestion Layer — `pipeline/batch_500_runner.py`
2. **Clean and validate** — Check missing values, data quality, artist identities, and timestamps.
3. **Calculate scores** — Normalize available metrics and calculate weighted artist scores.
4. **Apply governance checks** — Evaluate freshness, platform evidence, eligibility, and suspicious patterns.
5. **Publish and audit** — Generate rankings, compare historical results, and create the audit report.
## 2. Ranking Methodology - Collects multi-platform telemetry through configured API calls and web scraping.
- Uses checkpoints in `data/checkpoints/` to support recovery after interrupted runs and reduce repeated data requests.
- Saves the refreshed master cohort dataset to `data/hiffi_multiplatform_500.csv`.
The project calculates three main outputs: the YouTube production score, the multi-platform HIFFI-XP score, and the Breakout Velocity Index (BVI). ### 2. Ranking and Governance Core — `run_pipeline.py`
### HIFFI-XP Composite Score - Loads `data/hiffi_multiplatform_500.csv` and executes the ranking workflow independently of data collection.
- Handles missing metrics, redistributes weights across available signals, checks data freshness, and flags suspicious records.
- Applies publication eligibility rules and assigns published ranks.
- Generates an audit report at `reports/audit_report_latest.md`.
The default weights are: ## Ranking Methodology
| Signal | Weight | The project uses separate scores for artist production performance, multi-platform ranking, and breakout potential.
### HIFFI-XP Multi-Platform Score
| Signal | Base Weight |
|---|---:| |---|---:|
| YouTube production score | 50% | | YouTube production score | 50% |
| Spotify monthly listeners | 25% | | Spotify monthly listeners | 25% |
| Spotify daily velocity | 5% | | Spotify daily stream velocity | 5% |
| Instagram followers | 20% | | Instagram followers | 20% |
| **Total** | **100%** |
Each metric is normalized before combining it into the composite score. When some signals are unavailable, the scoring logic redistributes their weights proportionally across the available signals. The effective weights are recorded in `effective_weights` for auditability.
### Dynamic Weight Redistribution
When some platform metrics are unavailable, the remaining available signals share the weight proportionally.
Examples:
- **YouTube only:** YouTube receives 100%.
- **YouTube + Spotify monthly listeners:** YouTube receives 66.7%, and Spotify receives 33.3%.
- **All signals available:** The original weights are retained.
The `effective_weights` column records the actual weights used for each artist.
Missing metrics are preserved as `NaN` in the source dataset rather than being treated as genuine zero values. Missing inputs may be handled separately during individual scoring calculations, so score interpretation should account for the implemented normalization and missing-data rules.
### YouTube Production Score ### YouTube Production Score
The production score combines four signals: The production score uses the following components:
| Signal | Weight | | Component | Weight |
|---|---:| |---|---:|
| Recent momentum | 35% | | Recent momentum | 35% |
| Daily view velocity | 30% | | View velocity | 30% |
| Audience reach | 20% | | Audience reach | 20% |
| Consistency proxy | 15% | | Consistency proxy | 15% |
Audience reach combines subscriber count and lifetime views. Video count is used as a consistency proxy. Audience reach combines subscriber and lifetime-view signals. The score also uses confidence and eligibility checks where supported by the ranking implementation.
### Breakout Velocity Index (BVI) ### Breakout Velocity Index (BVI)
BVI measures relative breakout signals using view velocity, recent momentum, and an audience-size adjustment. The Breakout Velocity Index identifies artists showing growth potential based on available velocity signals. A 10,000-subscriber floor is used by the BVI logic to reduce the impact of very small accounts.
Eligibility requires at least 10,000 subscribers, positive daily view velocity, and an eligible collection status. BVI is calculated and ranked separately from the primary production score. **Note:** The project's internal scoring methodology is not necessarily identical to the public HIFFI website's ranking formula.
> **Note:** The production score, HIFFI-XP composite score, and BVI are separate calculations. They should not be confused with one another or assumed to be identical to any separate official HIFFI website formula. ## Identity Verification Gates
## 3. Data Governance Only Instagram profiles that pass the configured verification rules qualify for multi-platform credit.
### Freshness Rules - **`AUTOMATED_VERIFIED`:** Confidence ≥ `0.55`, or confidence ≥ `0.40` with follower corroboration ≥ `10,000`.
- **`UNVERIFIED`:** Accounts that do not meet verification requirements.
- **`UNCERTAIN_NEEDS_REVIEW`:** Ambiguous account matches requiring human review.
| Platform | Maximum signal age | Unverified accounts, low-follower outliers below `10,000`, and ambiguous handles are routed to `data/review_queue.csv` for review and excluded from multi-platform tier credit.
Follower count is a supporting signal, not proof of identity. The verification status and publication eligibility rules determine whether an account can contribute evidence.
## Data Freshness and Publication Eligibility
The ranking engine uses freshness windows to determine whether platform data qualifies as current evidence.
| Platform | Maximum Data Age |
|---|---:| |---|---:|
| YouTube | 7 days | | YouTube | 7 days |
| Spotify | 7 days | | Spotify | 7 days |
| Instagram | 14 days | | Instagram | 14 days |
Missing, malformed, future-dated, or stale timestamps fail the freshness check. Excluded signals are recorded in `evidence_exclusions`. Missing, unparseable, or stale timestamps do not qualify for platform evidence credit. The engine records exclusion reasons in `evidence_exclusions`, such as `STALE_OR_MISSING_SPOTIFY_TIMESTAMP`.
### Artist Identity Verification A platform's evidence may be excluded from publication eligibility while its available metric can still contribute to a continuous score. Therefore, score calculation and publication eligibility are separate decisions.
Instagram identity status is checked before the platform contributes to publication evidence. The pipeline supports automated verification, manual verification, and a review process for uncertain accounts. ## Missing-Data Handling
Automated matching helps identify likely official profiles, but it cannot guarantee that every matched account belongs to the correct artist. - Preserve missing values as `NaN` where supported by the data and scoring logic.
- Do not treat failed API requests as genuine zero-valued metrics.
- Redistribute weights across available signals according to the scoring implementation.
- Record effective weights so ranking results can be inspected and explained.
### Publication Eligibility Some individual scoring functions may apply documented defaults. Review the relevant implementation before assuming every missing value remains `NaN` throughout all calculations.
- **Top 10:** Requires Tier 3 platform evidence. ## Anomaly Detection
- **Top 50:** Requires at least Tier 2 platform evidence.
- **Published ranks:** Assigned consecutively.
- **Shortfall handling:** When strict evidence requirements cannot fill the target positions, fallback selections are recorded in the shortfall audit metadata.
The audit report should be reviewed for any shortfalls before the rankings are treated as fully compliant with the intended eligibility rules. Before publication, the pipeline flags suspicious cross-platform metric relationships for audit review.
### Anomaly Detection | Flag | Condition |
|---|---|
| `SUSPICIOUS_IG_TO_YT_RATIO` | Instagram followers exceed YouTube lifetime views by more than 10×. |
| `EXTREME_SPOTIFY_VELOCITY_RATIO` | Daily stream growth exceeds 25% of monthly listeners. |
The `anomaly_flags` field records suspicious patterns, including: Flags are recorded in `anomaly_flags` so suspicious records can be investigated rather than silently ignored. A flag should trigger review; it does not, by itself, prove that the data is fraudulent.
- `SUSPICIOUS_IG_TO_YT_RATIO` — Instagram followers exceed 10 times YouTube lifetime views. ## Regression Auditing
- `EXTREME_SPOTIFY_VELOCITY_RATIO` — Spotify daily change exceeds 25% of monthly listeners.
These rules flag records for investigation. An anomaly is not proof of fraud or incorrect data. The regression auditor compares current ranking output against saved baselines.
## 4. Project Structure - Baselines are stored in `data/snapshots/`.
- Rank movements greater than 100 positions (`|rank shift| > 100`) are identified for review.
- The latest report is written to `reports/audit_report_latest.md`.
Large ranking changes may reflect genuine changes in platform signals, data availability, scoring weights, or identity verification. They should be reviewed before publication.
## Project Structure
```text ```text
hiffi-artist-intelligence/ hiffi-artist-intelligence/
├── .github/
│ └── workflows/
│ └── ci.yml
├── collectors/ ├── collectors/
│ ├── youtube_collector.py
│ ├── spotify_api_collector.py
│ ├── spotify_collector.py
│ ├── kworb_spotify_collector.py
│ ├── spotify_fallback_collector.py
│ ├── instagram_collector.py
│ ├── instagram_verifier.py │ ├── instagram_verifier.py
│ └── wikidata_resolver.py │ ├── spotify_api_collector.py
├── config/ │ └── spotify_fallback_collector.py
├── core/ ├── core/
│ ├── ranking_engine.py │ ├── ranking_engine.py
│ ├── confidence_scorer.py
│ ├── deduplication.py
│ ├── health_detector.py
│ └── regression_auditor.py │ └── regression_auditor.py
├── data/ ├── data/
│ ├── snapshots/
│ ├── checkpoints/ │ ├── checkpoints/
│ ├── handle_overrides.json │ ├── snapshots/
│ ├── review_queue.csv │ ├── review_queue.csv
│ └── hiffi_multiplatform_500.csv │ └── hiffi_multiplatform_500.csv
├── pipeline/ ├── pipeline/
│ └── batch_500_runner.py
├── reports/ ├── reports/
│ ├── audit_report_latest.md │ └── audit_report_latest.md
│ └── provenance_summary.py ├── tests/
├── run_pipeline.py ├── run_pipeline.py
├── test_multiplatform_ranking.py ├── test_multiplatform_ranking.py
├── requirements.txt ├── requirements.txt
├── .env.example
└── README.md └── README.md
``` ```
## 5. Installation and Setup The tree shows the intended layout. Keep only files and folders that actually exist in the repository.
### Prerequisites ## Requirements
- Python 3.11 or 3.12 - Python 3.11 or 3.12
- Git - Git
- API credentials when required by the configured collectors - API credentials for any integrations that require authentication
### Clone the repository ## Setup
### 1. Clone the repository
```bash ```bash
git clone https://github.com/<your-username>/hiffi-artist-intelligence.git git clone <YOUR_REPOSITORY_URL>
cd hiffi-artist-intelligence cd hiffi-artist-intelligence
``` ```
### Create a virtual environment Replace `<YOUR_REPOSITORY_URL>` with your repository's clone URL.
### 2. Create a virtual environment
**Windows PowerShell** **Windows PowerShell**
```powershell ```powershell
python -m venv .venv python -m venv .venv
.\.venv\Scripts\Activate.ps1 .\.venv\Scripts\Activate.ps1
```
**macOS/Linux**
```bash
python3 -m venv .venv
source .venv/bin/activate
```
### 3. Install dependencies
```bash
python -m pip install --upgrade pip
pip install -r requirements.txt pip install -r requirements.txt
``` ```
### Configure API credentials ### 4. Configure environment variables
If the Spotify API collector requires credentials, create a local `.env` file: Create a local `.env` file using `.env.example` as a guide. Add the credentials required by your enabled integrations, for example:
```text ```dotenv
SPOTIFY_CLIENT_ID=your_client_id SPOTIFY_CLIENT_ID=your_spotify_client_id
SPOTIFY_CLIENT_SECRET=your_client_secret SPOTIFY_CLIENT_SECRET=your_spotify_client_secret
``` ```
Never commit real credentials, API keys, or tokens to GitHub. Keep `.env` excluded through `.gitignore` and use GitHub Secrets for CI credentials when needed. Use the actual variable names expected by your collectors. Never commit `.env` files or live API credentials to Git.
## 6. Run the Project ## Running the Pipeline
Run commands from the repository root. ### Refresh platform data
### Execute the full pipeline If `pipeline/batch_500_runner.py` is implemented as the ingestion entry point, run:
```bash
python -m pipeline.batch_500_runner
```
This stage should refresh the master dataset and use checkpoints where supported.
### Calculate rankings and generate the audit report
```bash ```bash
python run_pipeline.py python run_pipeline.py
``` ```
This runs the configured stages for data loading, scoring, validation, governance, and audit generation. This stage consumes the available master dataset, applies scoring and governance rules, and produces ranking outputs and the audit report.
### Run the test suite ### Run tests
```bash ```bash
python -m pytest test_multiplatform_ranking.py -v python -m pytest test_multiplatform_ranking.py -v
``` ```
The current governance suite contains 27 tests, based on the latest reported test run. ### Run the regression auditor
### Compare historical rankings
```bash ```bash
python -m core.regression_auditor python -m core.regression_auditor
``` ```
The regression auditor compares current rankings with saved snapshots and flags movements greater than 100 positions for review. Use the exact commands and test paths defined by the current repository.
### Review the audit report ## Testing and Continuous Integration
```text The project reports 27 passing unit and integration tests, with GitHub Actions checks passing on Python 3.11 and 3.12 on Ubuntu.
reports/audit_report_latest.md
```
Review the report for missing data, evidence exclusions, anomalies, ranking movements, and publication shortfalls. The test suite is intended to cover ranking behavior, missing-data handling, freshness and eligibility gates, anomaly detection, and regression safeguards.
## 7. Continuous Integration Test results should be confirmed against the latest commit before a release is declared complete.
GitHub Actions runs automated tests on Ubuntu across Python 3.11 and Python 3.12. ## Important Output Fields
The CI workflow is configured in: | Field | Purpose |
```text
.github/workflows/ci.yml
```
The latest reported run passed on both Python versions, with all 27 governance tests passing.
CI results confirm that the tested code and scenarios pass in the configured environments. They do not independently guarantee the accuracy of external platform data or the reliability of every live collection run.
## 8. Output Dataset
The consolidated dataset is stored at:
```text
data/hiffi_multiplatform_500.csv
```
Important columns include:
| Column | Purpose |
|---|---| |---|---|
| `name` | Artist name | | `production_hiffi_score` | YouTube production-performance score. |
| `production_hiffi_score` | YouTube production score | | `hiffi_xp_score` | Combined multi-platform score. |
| `hiffi_xp_score` | Multi-platform composite score | | `hiffi_xp_rank` | Raw multi-platform ranking position. |
| `hiffi_xp_rank` | Raw composite ranking | | `hiffi_xp_published_rank` | Published ranking position after eligibility rules. |
| `hiffi_xp_published_rank` | Publication ranking after eligibility rules | | `breakout_velocity_index` | Breakout potential score. |
| `breakout_velocity_index` | BVI score | | `breakout_rank` | Breakout ranking position. |
| `breakout_rank` | Separate BVI ranking | | `effective_weights` | Weights applied to the available signals. |
| `effective_weights` | Actual scoring weights used | | `platform_evidence_tier` | Platform evidence tier used for eligibility. |
| `platform_evidence_tier` | Count of qualifying platform signals | | `evidence_exclusions` | Reasons platform evidence was excluded. |
| `evidence_exclusions` | Reasons signals were excluded from evidence | | `anomaly_flags` | Potentially suspicious metric relationships. |
| `anomaly_flags` | Suspicious patterns identified | | `rank_delta` | Change in ranking position relative to the comparison baseline. |
| `rank_delta` | Difference between production and composite ranks |
Additional columns may contain platform metrics, verification statuses, collection timestamps, confidence values, and source provenance. Actual output columns depend on the current implementation and input dataset.
## 9. Limitations ## Security and Data Governance
- External APIs and public sources may provide incomplete data or restrict access. - Keep API keys in environment variables, not source code.
- Artist names and similar handles can cause identity-matching errors. - Keep `.env` and local credentials out of version control.
- Missing metrics and normalization choices can affect ranking outcomes. - Review uncertain artist identities before granting platform evidence credit.
- Large ranking movements require investigation, not automatic correction. - Keep data timestamps and exclusion reasons available for auditing.
- Anomaly flags identify patterns for review rather than confirmed wrongdoing. - Investigate large rank shifts and anomaly flags before publication.
- Video count is a proxy for consistency, not a direct measurement of upload regularity. - Do not interpret a missing metric as a confirmed zero unless the relevant scoring rule explicitly defines that behavior.
- BVI measures relative breakout signals; it does not predict future success with certainty.
- Rankings represent the configured metrics and weights, not artistic quality.
## 10. Future Improvements ## Limitations
- Expand source coverage and identity verification. - API access, quotas, and authentication can affect data availability.
- Improve API failure monitoring and recovery. - Platform metrics may have different refresh schedules and definitions.
- Automate scheduled data collection and weekly audit generation. - Similar artist names and unofficial social accounts can produce ambiguous identity matches.
- Strengthen historical regression reporting. - Anomaly rules identify unusual relationships, not necessarily fraudulent activity.
- Improve observability, release checks, and production deployment safeguards. - Rankings depend on data quality, available signals, and the configured weighting methodology.
- Automated tests cannot guarantee that every live platform account or metric is correct.
## Conclusion ## Future Improvements
The HIFFI Hip-Hop 500 pipeline brings together multi-platform data collection, artist verification, ranking calculations, data governance, anomaly detection, regression analysis, and automated testing. - Expand identity verification using reliable official account evidence.
- Improve source-level data validation and stale-data monitoring.
- Add operational alerts for failed or incomplete ingestion runs.
- Strengthen checkpoint recovery and API retry behavior.
- Track ranking changes across releases and review significant shifts.
- Automate recurring regression reports and release checks.
Its main goal is to make artist rankings more **transparent, traceable, and repeatable**, while clearly identifying missing data and uncertain evidence. ## Summary
HIFFI Hip-Hop 500 separates platform data ingestion from ranking and governance. Its scoring, identity checks, freshness controls, anomaly flags, and regression reports are designed to make multi-platform rankings more consistent, explainable, and auditable.
For production use, verify that the documented entry points, thresholds, checkpoint behavior, and output paths match the implementation in the released repository.