Method
Where the data comes from
Four job APIs, all used as their operators intend — official endpoints with published terms, not scraped pages. Nothing here comes from a site whose robots.txt disallows it.
| Channel | Access | Covers |
|---|---|---|
| Adzuna (India) | Official API, free tier | The broad Indian job market |
| Greenhouse | Public job board API | Per-company boards |
| Lever | Public postings API | Per-company boards |
| Ashby | Public job board API | Per-company boards, structured pay |
The original plan for this project was to scrape Internshala. That was
abandoned after reading its robots.txt, which disallows
/internship/search/, /internship/details/ and every query-string URL —
exactly the pages a scraper would need. Beyond the compliance problem, a
scraper fighting anti-bot defences stops working within weeks, which would
defeat the entire premise of a repository that stays current.
How a posting becomes a row
- Fetch
Each channel is queried daily. A channel that fails is reported and skipped; it never takes the run down with it.
- Normalize
Four different response shapes collapse into one schema. Ashby nests compensation, Lever uses epoch milliseconds, Greenhouse needs a flag to return descriptions at all, Adzuna paginates.
- Filter
A posting is kept only if its title names a data or AI role. Descriptions are deliberately not used: nearly every engineering job mentions machine learning somewhere, and matching on that would inflate every count on this site.
- Store
Full records are written once, on the day a posting is first seen, plus a small daily log of which postings were still open. That is what makes
days_openandis_activepossible without re-committing every description every day. - Transform
dbt builds staging → intermediate → marts on DuckDB.
- Gate
The build fails, and nothing is published, if a channel disappears, if volume collapses against its trailing average, if a required field stops being populated, or if a pay figure is incoherent.
What these numbers do not say
These limits are the reason to trust the rest. They are stated here rather than buried because a measurement without its error bars is decoration.
Inferred from the job title. Most postings do not state a level, so a large share sits in unknown — and that bucket is excluded from charts rather than quietly folded into another level.
A lower bound. It counts from the first day this pipeline saw a posting, not from when the employer published it.
Sparse and self-selected: only 304 of 4,508 roles advertise pay, and employers who publish a salary are not a random sample. Predicted salaries are discarded.
Bounded by the company registry. Roles at companies not in it, and not surfaced by Adzuna, are invisible here.
One row per requisition. A role advertised in three cities appears three times, because each is a separate opening.
Verifiable scale
Every figure below is derived from files committed to the repository. Clone it, run dbt build, and you will get these same numbers.