For site operators

IndiaMasala-bot

India Masala helps people find where an Indian film is streaming, in their language and their country. This page describes the crawler that keeps that information current.

User-Agent

Every request we make carries this string, unchanged:

IndiaMasala-bot/0.1 (+https://indiamasala.com/bot; mohan@mohaniyer.com)

We never impersonate a browser and never rotate user agents. If a request does not carry that string, it is not us.

What it collects

Only what a site publishes for public indexing: the sitemap, and from each film’s page the title, release year, audio languages, and the og:image and og:title tags a platform provides specifically so third parties can display them.

We do not collect subscriber data, pricing, video, or subtitles. We do not attempt to access anything behind a login, and we do not read pages your robots.txt disallows — see the one stated exception below, which concerns sitemap files only.

How it behaves

  • Obeys robots.txt, including Crawl-delay.
  • One deliberate exception, stated plainly: if your robots.txt publishes a Sitemap: line, we read that sitemap and the child sitemaps it names, even where a blanket rule such as Disallow: /*.xml would also cover them. We read the specific invitation as speaking to crawlers and the blanket rule as keeping XML out of search results. This applies to sitemap files only — never to a page. If you would rather we did not, say so and we will stop.
  • Requests at most two pages per second, single-threaded, from one host at a time.
  • Runs on a schedule — a full pass monthly, and a weekly incremental pass that fetches only URLs whose sitemap <lastmod> has changed. Most weeks that is a small fraction of a catalogue.
  • Backs off on 429 and 5xx rather than retrying immediately.

What you get from it

Every listing links back to the film’s page on your service, and we name the platform as the place to watch it. We do not host or stream video, and we do not reproduce review or synopsis text.

How to control or block it

Add this to your robots.txt and we will stop within a day:

User-agent: IndiaMasala-bot
Disallow: /

To slow us down instead, set Crawl-delay and we will honour it. To have specific pages removed from the site, or to ask anything at all, write to mohan@mohaniyer.com — a real address, read by a person.