feat: initial project scaffold and repository setup
This commit is contained in:
commit
ff31dc55a5
5 files changed
+46
No files matched your search
+24
@@ -0,0 +1,24 @@
|
|||||||
|
# Python bytecode & cache
|
||||||
|
__pycache__/
|
||||||
|
*.py[cod]
|
||||||
|
*$py.class
|
||||||
|
|
||||||
|
# Virtual environments
|
||||||
|
venv/
|
||||||
|
env/
|
||||||
|
.venv/
|
||||||
|
|
||||||
|
# Environment files
|
||||||
|
.env
|
||||||
|
|
||||||
|
# Data files (track folders via .gitkeep, ignore data dumps)
|
||||||
|
data/raw/*
|
||||||
|
!data/raw/.gitkeep
|
||||||
|
data/processed/*
|
||||||
|
!data/processed/.gitkeep
|
||||||
|
|
||||||
|
# IDE & system artifacts
|
||||||
|
.vscode/
|
||||||
|
.idea/
|
||||||
|
.DS_Store
|
||||||
|
Thumbs.db
|
||||||
@@ -0,0 +1,17 @@
|
|||||||
|
# AugmentX – Job Market Data Platform
|
||||||
|
|
||||||
|
A data platform designed to collect verified job postings directly from official company career pages, transform them into clean and structured data, and provide transparent insights into job-market demand and skills trends.
|
||||||
|
|
||||||
|
## Project Principles & Architecture
|
||||||
|
|
||||||
|
- **Primary Data Source:** Official company career pages and Applicant Tracking Systems (ATS).
|
||||||
|
- **Secondary Sources:** Job discovery platforms are used only to identify and locate official company career pages—not as sources for scraping job postings.
|
||||||
|
- **Core Technology Stack:** Python, HTTP clients, HTML/JSON parsing, SQLite/PostgreSQL, and Pandas.
|
||||||
|
|
||||||
|
## Roadmap
|
||||||
|
|
||||||
|
- **Week 1 – Job Source Identification & Prioritization:** Identify, evaluate, and prioritize official company career pages.
|
||||||
|
- **Week 2 – Job Data Collection & Automated Crawlers:** Build automated crawlers to collect job postings from prioritized sources.
|
||||||
|
- **Week 3 – Data Cleaning & Deduplication:** Clean, standardize, and remove duplicate job records.
|
||||||
|
- **Week 4 – Search Experience & Exploratory Analytics:** Develop a simple job search experience and explore the collected market data.
|
||||||
|
- **Week 5 – Market Trends & Skills Analytics:** Extract skills from job postings and analyze demand trends across roles and skills.
|
||||||
Whitespace-only changes.
Whitespace-only changes.
@@ -0,0 +1,5 @@
|
|||||||
|
requests>=2.31.0
|
||||||
|
beautifulsoup4>=4.12.0
|
||||||
|
pandas>=2.1.0
|
||||||
|
pydantic>=2.0.0
|
||||||
|
python-dotenv>=1.0.0
|
||||||
Reference in new issue
Block a user