augmentx-job-market/docs/sources_priority.md

# AugmentX – Phase 1: Job Source Intelligence & Crawl Prioritization

## 1. Executive Summary

Established the foundation for the AugmentX job-data ingestion pipeline by identifying target companies, discovering official career portals, evaluating ATS platforms, validating accessible endpoints, and assessing crawling feasibility.

Sources were prioritized based on job availability, data accessibility, source reliability, and implementation complexity.

The outcome is a structured source registry and technical assessment that will guide automated job collection in Phase 2.

## 2. Key Deliverables & Current Status

- **Source Registry:** Created a structured catalog of target companies, official career URLs, ATS platforms, and verified endpoints in `config/sources.json`.
- **ATS Feasibility Assessment:** Evaluated Greenhouse, SmartRecruiters, and custom career portals, identifying direct JSON API access and HTML-based extraction strategies.
- **Crawl Priority Matrix:** Classified target companies into priority tiers based on expected data coverage, accessibility, and crawler implementation complexity.
- **Data Schema Analysis:** Identified key job attributes and source-specific field structures required for standardized ingestion.

---

## 3. Crawl Priority Matrix

| Priority | Company | ATS / Platform | Access Endpoint | Status | Extraction Strategy |
|---|---|---|---|---|---|
| **Tier 1 – High** | **Razorpay** | Greenhouse | `https://boards-api.greenhouse.io/v1/boards/razorpaysoftwareprivatelimited/jobs` | Verified – HTTP 200, ~30 active roles | Direct JSON API; structured payload with minimal extraction complexity. |
| **Tier 1 – High** | **Freshworks** | SmartRecruiters | `https://api.smartrecruiters.com/v1/companies/Freshworks/postings` | Verified – HTTP 200, ~125 active roles | Direct JSON API with pagination and structured job metadata. |
| **Tier 2 – Medium** | **Zoho** | Zoho Recruit / Custom | `https://careers.zohocorp.com/` | Pending | Investigate network requests, internal endpoints, and server-rendered HTML. |
| **Tier 2 – Medium** | **Swiggy** | Custom / Workday | `https://careers.swiggy.com/` | Pending | Inspect dynamic page rendering and network XHR/API payloads. |

## 4. ATS Schema & Data Access Analysis

### 4.1 Greenhouse – Razorpay

**Access Method:** Public job board API.

**Endpoint:**
`https://boards-api.greenhouse.io/v1/boards/razorpaysoftwareprivatelimited/jobs`

**Observed Data Fields:**

| Field | Description |
|---|---|
| `title` | Job position title |
| `location.name` | Job location |
| `absolute_url` | Official job application URL |
| `updated_at` | Last updated timestamp |
| `departments` | Department classification |

**Technical Assessment:** The endpoint returns structured JSON data, making it suitable for direct API-based extraction with minimal parsing overhead.

### 4.2 SmartRecruiters – Freshworks

**Access Method:** Public job postings API.

**Endpoint:**
`https://api.smartrecruiters.com/v1/companies/Freshworks/postings`

**Observed Data Fields:**

| Field | Description |
|---|---|
| `name` | Job position title |
| `location.city` | City |
| `location.region` | State / Region |
| `location.country` | Country |
| `refNumber` | Unique job reference identifier |
| `releasedDate` | Job posting release date |
| `department` | Business unit classification |

**Technical Assessment:** The API provides structured job records with pagination metadata, supporting scalable extraction across multiple result pages.

---

## 5. Crawl Prioritization Strategy

The initial crawling sequence is based on implementation feasibility and expected data availability.

1. **Tier 1 – API-Based Sources:** Begin with Razorpay and Freshworks due to their accessible JSON endpoints and structured responses.
2. **Tier 2 – Custom Sources:** Investigate Zoho and Swiggy to identify their underlying data-access mechanisms before implementing dedicated extraction logic.

This approach prioritizes reliable data collection while reducing unnecessary crawler development complexity.

---

## 6. Phase 2 – Implementation Roadmap

### Job Data Collection & Pipeline Automation

**Milestone 1 – Modular Crawler Architecture**
- Design a reusable `BaseCrawler` interface.
- Implement `GreenhouseCrawler` and `SmartRecruitersCrawler`.
- Standardize source-specific extraction outputs.

**Milestone 2 – Raw Data Ingestion**
- Capture original API responses and raw job payloads.
- Organize raw data under `data/raw/YYYY-MM-DD/`.
- Implement caching and idempotent ingestion to avoid unnecessary duplicate processing.

**Milestone 3 – Crawl Orchestration & Monitoring**
- Build a centralized crawl execution runner.
- Track execution status, records collected, response errors, and source availability.
- Generate source-wise crawl metrics and execution logs.

## 7. Expected Phase 2 Outcome

A modular and automated job-data collection pipeline capable of retrieving job postings from prioritized official company sources, preserving raw responses, and storing collected records for subsequent cleaning, normalization, and deduplication in Phase 3.