docs: add clear end-to-end README, update app.py and requirements, remove temp lock file
This commit is contained in:
Srilekha committed
1 parent
9710cf9948
commit
ef0c0ca258
4 files changed
+347
-22
No files matched your search
@@ -1,3 +1,237 @@
|
||||
# HIFFI Top 500 Hip-Hop Artist Intelligence
|
||||
# HIFFI Hip-Hop 500 — Artist Intelligence Engine
|
||||
|
||||
Automated data pipeline for multi-source verification and composite metric ranking.
|
||||
An end-to-end data pipeline and interactive analytics dashboard designed to collect, validate, analyze, and rank 500 hip-hop artists based on their digital performance.
|
||||
|
||||
The system combines artist identity verification, performance metrics, weighted ranking formulas, and breakout talent analysis to support reliable, data-driven artist discovery.
|
||||
|
||||
## 📌 Project Overview
|
||||
|
||||
Music industry datasets can contain incorrect artist links, unofficial fan accounts, missing metrics, and temporary viral spikes that distort rankings.
|
||||
|
||||
The HIFFI Hip-Hop 500 Intelligence Engine addresses these challenges through four key capabilities:
|
||||
|
||||
1. **Artist Identity Verification** — Cross-references artist information using sources such as MusicBrainz and Wikidata to help identify incorrect account matches.
|
||||
2. **Channel Health Classification** — Categorizes artist channels as `ACTIVE_HEALTHY`, `CATALOG_EVERGREEN`, or `DORMANT_CATALOG` based on defined activity and performance criteria.
|
||||
3. **Multi-Factor Ranking** — Calculates a composite HIFFI score using four weighted factors: recent momentum, view velocity, audience reach, and consistency.
|
||||
4. **Breakout Artist Discovery** — Uses a Breakout Velocity Index (BVI) to highlight emerging artists with strong growth potential.
|
||||
5. **Interactive Analytics Dashboard** — Provides rankings, performance visualizations, artist-level insights, filtering, and CSV/Excel exports through Streamlit.
|
||||
|
||||
## ✨ Key Features
|
||||
|
||||
- **Data Cleaning and Validation:** Identifies duplicate records, missing values, inconsistent data types, and potential outliers.
|
||||
- **Artist Identity Matching:** Uses available artist metadata and external knowledge sources to support identity verification.
|
||||
- **YouTube Metrics Collection:** Integrates YouTube Data API v3 to retrieve supported channel statistics and video information.
|
||||
- **Historical Performance Tracking:** Uses SQLite to store metric snapshots and support growth and velocity calculations.
|
||||
- **Weighted Ranking Model:** Combines four performance indicators into a composite score.
|
||||
- **Channel Health Assessment:** Distinguishes active channels, evergreen catalogs, and potentially dormant channels.
|
||||
- **Breakout Velocity Analysis:** Helps identify artists gaining attention relative to their existing audience size.
|
||||
- **Interactive Dashboard:** Displays key performance indicators, leaderboards, charts, filters, and artist details.
|
||||
- **Exportable Reports:** Generates enriched artist datasets and identity verification audit reports.
|
||||
|
||||
## 📊 HIFFI Ranking Methodology
|
||||
|
||||
The HIFFI composite score uses four weighted performance factors:
|
||||
|
||||
| Ranking Factor | Weight | Description |
|
||||
|---|---:|---|
|
||||
| Recent Momentum | 35% | Measures recent audience or performance growth. |
|
||||
| View Velocity | 30% | Measures how quickly an artist's content gains views. |
|
||||
| Audience Reach | 20% | Represents the size of the artist's audience. |
|
||||
| Consistency | 15% | Measures performance stability over time. |
|
||||
| **Total** | **100%** | |
|
||||
|
||||
The factors are normalized to a common scale before being combined into the raw score.
|
||||
|
||||
### Confidence-Adjusted Score
|
||||
|
||||
The system can adjust the raw score according to the confidence of the artist identity match:
|
||||
|
||||
`Final Score = Raw Score × (0.80 + 0.20 × Confidence Score)`
|
||||
|
||||
The confidence score ranges from `0.00` to `1.00`.
|
||||
|
||||
This adjustment reduces the influence of uncertain identity matches. Additional eligibility rules may be applied to prevent unverified accounts from entering the official leaderboard.
|
||||
|
||||
### Breakout Velocity Index (BVI)
|
||||
|
||||
The BVI provides a separate perspective on artist performance, focusing on growth and view velocity relative to the artist's existing audience size.
|
||||
|
||||
It complements the overall HIFFI ranking by helping surface emerging artists who may not yet have a large established audience.
|
||||
|
||||
*Note: The exact BVI calculation depends on the implementation in the scoring module.*
|
||||
|
||||
## 🏗️ System Architecture
|
||||
|
||||
The pipeline follows this workflow:
|
||||
|
||||
```text
|
||||
Artist Source Data / Soundcharts Export
|
||||
|
|
||||
v
|
||||
Data Cleaning & Validation
|
||||
|
|
||||
v
|
||||
Artist Identity Verification
|
||||
(MusicBrainz / Wikidata)
|
||||
|
|
||||
v
|
||||
YouTube Metrics Collection
|
||||
|
|
||||
v
|
||||
Historical Snapshot Storage
|
||||
(SQLite)
|
||||
|
|
||||
v
|
||||
Scoring & Ranking Engine
|
||||
| |
|
||||
v v
|
||||
HIFFI Composite Breakout Velocity
|
||||
Score Index
|
||||
| |
|
||||
+--------+--------+
|
||||
|
|
||||
v
|
||||
Enriched Artist Dataset
|
||||
|
|
||||
v
|
||||
Streamlit Analytics Dashboard
|
||||
|
|
||||
v
|
||||
CSV / Excel Report Exports
|
||||
```
|
||||
|
||||
## 📁 Repository Structure
|
||||
|
||||
```text
|
||||
hiffi-hip-hop-500/
|
||||
├── collectors/ # Data collection modules
|
||||
├── verifiers/ # Artist identity verification
|
||||
├── core/ # Data processing and scoring logic
|
||||
├── data/ # Input datasets and reference data
|
||||
├── app.py # Streamlit analytics dashboard
|
||||
├── main.py # Pipeline orchestration
|
||||
├── enriched_artists.csv # Enriched artist metrics and rankings
|
||||
├── identity_verification_audit.xlsx # Identity audit and review flags
|
||||
├── requirements.txt # Python dependencies
|
||||
└── README.md # Project documentation
|
||||
```
|
||||
|
||||
*The structure above describes the expected project layout. File names and generated artifacts should match the actual repository.*
|
||||
|
||||
## 📦 Key Deliverables
|
||||
|
||||
| Deliverable | Purpose |
|
||||
|---|---|
|
||||
| `enriched_artists.csv` | Stores enriched artist records, performance metrics, scores, rankings, and health classifications. |
|
||||
| `identity_verification_audit.xlsx` | Records identity matching results, confidence scores, verification flags, and review requirements. |
|
||||
| `app.py` | Provides the interactive Streamlit dashboard for exploring artist performance. |
|
||||
| SQLite database | Stores historical metric snapshots used for time-based performance analysis. |
|
||||
|
||||
## 🛠️ Technology Stack
|
||||
|
||||
- **Language:** Python
|
||||
- **Data Processing:** pandas, NumPy
|
||||
- **Data Sources:** Soundcharts exports and YouTube Data API v3
|
||||
- **Identity Verification:** MusicBrainz and Wikidata, where available
|
||||
- **Database:** SQLite
|
||||
- **Dashboard:** Streamlit
|
||||
- **Visualizations:** Plotly
|
||||
- **Exports:** CSV and Excel
|
||||
|
||||
## 🚀 Getting Started
|
||||
|
||||
### 1. Clone the Repository
|
||||
|
||||
```bash
|
||||
git clone <repository-url>
|
||||
cd hiffi-hip-hop-500
|
||||
```
|
||||
|
||||
Replace `<repository-url>` with the actual repository URL.
|
||||
|
||||
### 2. Create a Virtual Environment
|
||||
|
||||
```bash
|
||||
python -m venv .venv
|
||||
```
|
||||
|
||||
Activate it:
|
||||
|
||||
**Windows**
|
||||
```bash
|
||||
.venv\Scripts\activate
|
||||
```
|
||||
|
||||
**macOS / Linux**
|
||||
```bash
|
||||
source .venv/bin/activate
|
||||
```
|
||||
|
||||
### 3. Install Dependencies
|
||||
|
||||
```bash
|
||||
pip install -r requirements.txt
|
||||
```
|
||||
|
||||
### 4. Configure API Credentials
|
||||
|
||||
Set up the required API credentials, including the YouTube Data API v3 key, if the collection pipeline requires it.
|
||||
|
||||
Store credentials in environment variables or a local `.env` file. Do not commit API keys, passwords, or other secrets to the repository.
|
||||
|
||||
### 5. Run the Data Pipeline
|
||||
|
||||
```bash
|
||||
python main.py
|
||||
```
|
||||
|
||||
This runs the pipeline orchestrator, subject to the project's configured data sources, credentials, and implementation.
|
||||
|
||||
### 6. Launch the Dashboard
|
||||
|
||||
```bash
|
||||
streamlit run app.py
|
||||
```
|
||||
|
||||
Open the local URL displayed in the terminal to access the dashboard.
|
||||
|
||||
*Run commands should be verified against the actual repository before use.*
|
||||
|
||||
## 📈 Dashboard Capabilities
|
||||
|
||||
The dashboard is designed to provide:
|
||||
|
||||
- **KPI Overview:** Summary counts for monitored artists and channel health categories.
|
||||
- **Top Artist Leaderboard:** Artists ranked by their HIFFI composite scores.
|
||||
- **Breakout Leaderboard:** Artists highlighted by the BVI.
|
||||
- **Identity Audit View:** Confidence scores, verification flags, and eligibility information.
|
||||
- **Performance Charts:** Comparisons of view velocity and recent growth.
|
||||
- **Category Filters:** Explore artist groups and subgenres.
|
||||
- **Artist Details:** Review available channel information and verification results.
|
||||
- **Data Exports:** Download filtered results in CSV or Excel format.
|
||||
|
||||
## 🔍 Data Quality and Reliability
|
||||
|
||||
The system is designed to improve the reliability of artist rankings by:
|
||||
|
||||
- Avoiding identity matches based only on similar names.
|
||||
- Flagging uncertain or conflicting account information.
|
||||
- Distinguishing unavailable metrics from genuine zero values.
|
||||
- Considering both recent growth and established audience reach.
|
||||
- Tracking historical snapshots to support time-based calculations.
|
||||
- Maintaining audit records for review and transparency.
|
||||
|
||||
Automated identity matching supports verification but does not guarantee that every account is official. Uncertain matches should be reviewed before being treated as verified.
|
||||
|
||||
## 🔐 Security and Responsible Data Handling
|
||||
|
||||
- Keep API credentials outside source control.
|
||||
- Use environment variables or approved secret-management tools.
|
||||
- Respect API quotas, platform terms, and applicable data-use policies.
|
||||
- Validate source data before including it in published rankings.
|
||||
|
||||
## 🎯 Project Objective
|
||||
|
||||
The goal of the HIFFI Hip-Hop 500 Intelligence Engine is to create a transparent, repeatable, and data-driven approach to hip-hop artist analysis.
|
||||
|
||||
By combining data quality checks, identity verification, historical performance tracking, weighted ranking, and interactive analytics, the platform supports more informed artist discovery and performance evaluation.
|
||||
Reference in new issue
Block a user