Crawling a site
Give Minaya a starting URL and it follows links from that page, staying on the same origin.
Defaults
| Setting | Default | Meaning |
|---|---|---|
| Depth | 2 | The start page, pages it links to, and pages those link to. |
| Max pages | 100 | Hard stop, so a large site cannot crawl indefinitely. |
| Timeout | 10s | Per page. A slow page is skipped rather than blocking the crawl. |
Depth 1 only reaches pages linked directly from the start URL. Hub pages like /portfolio list their children one level deeper, which is why the default is 2.
What is skipped
- Other origins. Only same-origin links are followed.
- Assets. Images, CSS, JS, fonts, archives, media, and JSON.
- Framework internals. Paths like
/_next/,/cdn-cgi/, and/api/. - Auth and commerce pages. Login, register, cart, checkout, and search.
- Non-HTTP schemes.
mailto:,tel:,javascript:.
URL fragments are stripped before queuing, so /about#team and /about are treated as one page and fetched once.
What is extracted
From each page, Minaya keeps the title, headings, meta description, and body prose. It removes navigation, footers, scripts, and styling.
Two details worth knowing:
- Contact details are captured before footers are stripped, so your email and phone number survive even though the footer itself is discarded.
- Cloudflare-obfuscated emails are decoded. Sites behind Cloudflare often replace addresses with a placeholder that only JavaScript can decode; Minaya recovers the real address.
JavaScript-rendered sites
The crawler reads server-rendered HTML and does not execute JavaScript. Most frameworks — Next.js, Nuxt, Astro — server-render their content, so this is usually fine. A fully client-rendered SPA that ships an empty shell will yield little or nothing.
To check, view source on a page and search for a sentence you can see in the browser. If it is absent from the HTML, paste that content as a text source instead.
Verifying a crawl
After a crawl completes, the source shows how many chunks were produced. A one-page result for a large site usually means the content is client-rendered, or the pages you expected were not linked from the start URL.