docs: restructure roadmap and deliverables into phases

README.md

@@ -1,17 +1,27 @@
-# AugmentX – Job Market Data Platform
+# AugmentX – Job Market Intelligence Platform
 
-A data platform designed to collect verified job postings directly from official company career pages, transform them into clean and structured data, and provide transparent insights into job-market demand and skills trends.
+A Python-based job market intelligence platform designed to collect verified job postings directly from official company career pages and ATS platforms, transform raw postings into structured datasets, and generate actionable insights into hiring demand, role distribution, and skill trends.
 
 ## Project Principles & Architecture
 
-- **Primary Data Source:** Official company career pages and Applicant Tracking Systems (ATS).
-- **Secondary Sources:** Job discovery platforms are used only to identify and locate official company career pages—not as sources for scraping job postings.
-- **Core Technology Stack:** Python, HTTP clients, HTML/JSON parsing, SQLite/PostgreSQL, and Pandas.
+- **Primary Data Sources:** Official company career pages and Applicant Tracking Systems (ATS).
+- **Secondary Discovery Sources:** Job discovery platforms are used exclusively to identify and locate official company career pages, not to scrape job postings.
+- **Core Technology Stack:** Python, Requests/HTTP clients, HTML/JSON parsing, SQLite/PostgreSQL, and Pandas.
+- **Data Integrity:** Source verification, structured ingestion, validation, and deduplication throughout the pipeline.
 
-## Roadmap
+## Project Roadmap
 
-- **Week 1 – Job Source Identification & Prioritization:** Identify, evaluate, and prioritize official company career pages.
-- **Week 2 – Job Data Collection & Automated Crawlers:** Build automated crawlers to collect job postings from prioritized sources.
-- **Week 3 – Data Cleaning & Deduplication:** Clean, standardize, and remove duplicate job records.
-- **Week 4 – Search Experience & Exploratory Analytics:** Develop a simple job search experience and explore the collected market data.
-- **Week 5 – Market Trends & Skills Analytics:** Extract skills from job postings and analyze demand trends across roles and skills.
+### Phase 1: Job Source Intelligence & Crawl Prioritization
+Build a structured source registry by identifying target companies, discovering official career pages and ATS platforms, validating accessible endpoints, and ranking sources based on job availability, data accessibility, and crawling feasibility.
+
+### Phase 2: Job Data Collection & Pipeline Automation
+Develop modular crawlers, automate job-posting extraction, capture raw records, and establish a centralized database ingestion workflow.
+
+### Phase 3: Job Data Cleaning & Deduplication
+Standardize job titles, company names, locations, and descriptions while implementing schema validation, entity normalization, and duplicate detection.
+
+### Phase 4: Search Interface & Market Exploration
+Develop a searchable job discovery interface with role-based filtering, location-based exploration, and initial market analytics.
+
+### Phase 5: Labor Market Demand & Skills Intelligence
+Extract and categorize skills using NLP/LLM techniques, analyze hiring demand across roles and technologies, and identify emerging skill trends and market patterns.

docs/sources_priority.md

@@ -0,0 +1,105 @@
+# AugmentX – Phase 1: Job Source Intelligence & Crawl Prioritization
+
+## 1. Executive Summary
+
+Established the foundation for the AugmentX job-data ingestion pipeline by identifying target companies, discovering official career portals, evaluating ATS platforms, validating accessible endpoints, and assessing crawling feasibility.
+
+Sources were prioritized based on job availability, data accessibility, source reliability, and implementation complexity.
+
+The outcome is a structured source registry and technical assessment that will guide automated job collection in Phase 2.
+
+## 2. Key Deliverables & Current Status
+
+- **Source Registry:** Created a structured catalog of target companies, official career URLs, ATS platforms, and verified endpoints in `config/sources.json`.
+- **ATS Feasibility Assessment:** Evaluated Greenhouse, SmartRecruiters, and custom career portals, identifying direct JSON API access and HTML-based extraction strategies.
+- **Crawl Priority Matrix:** Classified target companies into priority tiers based on expected data coverage, accessibility, and crawler implementation complexity.
+- **Data Schema Analysis:** Identified key job attributes and source-specific field structures required for standardized ingestion.
+
+---
+
+## 3. Crawl Priority Matrix
+
+| Priority | Company | ATS / Platform | Access Endpoint | Status | Extraction Strategy |
+|---|---|---|---|---|---|
+| **Tier 1 – High** | **Razorpay** | Greenhouse | `https://boards-api.greenhouse.io/v1/boards/razorpaysoftwareprivatelimited/jobs` | Verified – HTTP 200, ~30 active roles | Direct JSON API; structured payload with minimal extraction complexity. |
+| **Tier 1 – High** | **Freshworks** | SmartRecruiters | `https://api.smartrecruiters.com/v1/companies/Freshworks/postings` | Verified – HTTP 200, ~125 active roles | Direct JSON API with pagination and structured job metadata. |
+| **Tier 2 – Medium** | **Zoho** | Zoho Recruit / Custom | `https://careers.zohocorp.com/` | Pending | Investigate network requests, internal endpoints, and server-rendered HTML. |
+| **Tier 2 – Medium** | **Swiggy** | Custom / Workday | `https://careers.swiggy.com/` | Pending | Inspect dynamic page rendering and network XHR/API payloads. |
+
+## 4. ATS Schema & Data Access Analysis
+
+### 4.1 Greenhouse – Razorpay
+
+**Access Method:** Public job board API.
+
+**Endpoint:**
+`https://boards-api.greenhouse.io/v1/boards/razorpaysoftwareprivatelimited/jobs`
+
+**Observed Data Fields:**
+
+| Field | Description |
+|---|---|
+| `title` | Job position title |
+| `location.name` | Job location |
+| `absolute_url` | Official job application URL |
+| `updated_at` | Last updated timestamp |
+| `departments` | Department classification |
+
+**Technical Assessment:** The endpoint returns structured JSON data, making it suitable for direct API-based extraction with minimal parsing overhead.
+
+### 4.2 SmartRecruiters – Freshworks
+
+**Access Method:** Public job postings API.
+
+**Endpoint:**
+`https://api.smartrecruiters.com/v1/companies/Freshworks/postings`
+
+**Observed Data Fields:**
+
+| Field | Description |
+|---|---|
+| `name` | Job position title |
+| `location.city` | City |
+| `location.region` | State / Region |
+| `location.country` | Country |
+| `refNumber` | Unique job reference identifier |
+| `releasedDate` | Job posting release date |
+| `department` | Business unit classification |
+
+**Technical Assessment:** The API provides structured job records with pagination metadata, supporting scalable extraction across multiple result pages.
+
+---
+
+## 5. Crawl Prioritization Strategy
+
+The initial crawling sequence is based on implementation feasibility and expected data availability.
+
+1. **Tier 1 – API-Based Sources:** Begin with Razorpay and Freshworks due to their accessible JSON endpoints and structured responses.
+2. **Tier 2 – Custom Sources:** Investigate Zoho and Swiggy to identify their underlying data-access mechanisms before implementing dedicated extraction logic.
+
+This approach prioritizes reliable data collection while reducing unnecessary crawler development complexity.
+
+---
+
+## 6. Phase 2 – Implementation Roadmap
+
+### Job Data Collection & Pipeline Automation
+
+**Milestone 1 – Modular Crawler Architecture**
+- Design a reusable `BaseCrawler` interface.
+- Implement `GreenhouseCrawler` and `SmartRecruitersCrawler`.
+- Standardize source-specific extraction outputs.
+
+**Milestone 2 – Raw Data Ingestion**
+- Capture original API responses and raw job payloads.
+- Organize raw data under `data/raw/YYYY-MM-DD/`.
+- Implement caching and idempotent ingestion to avoid unnecessary duplicate processing.
+
+**Milestone 3 – Crawl Orchestration & Monitoring**
+- Build a centralized crawl execution runner.
+- Track execution status, records collected, response errors, and source availability.
+- Generate source-wise crawl metrics and execution logs.
+
+## 7. Expected Phase 2 Outcome
+
+A modular and automated job-data collection pipeline capable of retrieving job postings from prioritized official company sources, preserving raw responses, and storing collected records for subsequent cleaning, normalization, and deduplication in Phase 3.