# Web Crawl through toolfare

> Start from one URL and crawl a public website, getting each page back as Markdown — with a page limit, a link depth and paths to include or skip. Ask from Claude, ChatGPT, Cursor, Codex or your own code; the crawl runs as a task the agent collects with `get_result`. No crawler to host and no vendor account, quoted before each call.

Page: https://toolfare.ai/web-crawl

## Use it from your agent

Paste this into Claude Code, Codex, Cursor or any agent with a shell:

> Set up https://toolfare.ai/skill.md, then use toolfare to crawl the /docs section of https://example.com, up to 30 pages, and build me a table of contents from what you find

Apps without a terminal connect toolfare as an MCP server: https://toolfare.ai/docs#connect

## Use it from a script

```sh
mkdir -p ~/.local/bin
curl -fsSL 'https://toolfare.ai/toolfare/toolfare.mjs' -o ~/.local/bin/toolfare
chmod +x ~/.local/bin/toolfare
export PATH="$HOME/.local/bin:$PATH"
export TOOLFARE_MCP_URL='https://api.toolfare.ai/mcp'
toolfare login
```

Sign in once per machine; the CLI renews the sign-in itself. On a CI runner or another server,
create an API key in the toolfare console and set it as `TOOLFARE_API_KEY` instead of signing in.
Paid calls on one sign-in or key run one at a time. `--max-cost` refuses a call quoted above
that many credits (1,000 credits = $3).

## Tools

### A site, page by page, as Markdown

Tool: `web_crawl_crawls`

Crawl a public website and return each page as Markdown

Required parameters: url

Optional parameters: exclude_paths, include_paths, limit, max_depth

Cost: From $0.06

Ask:
- Crawl this documentation site and give me every page as Markdown.
- Crawl only the /blog pages of this site, up to 50, and list their titles.
- Pull this competitor’s help centre so we can compare it with ours.
## Price

From $0.06 Quoted before each call; the final charge follows the returned usage. Pay per call from one balance.

Balances and top-ups are in US dollars. Quotes and `--max-cost` count credits: 1,000 credits = $3 ($0.003 each).

## Use it when

- You want a documentation site or help centre as Markdown for a retrieval index or a knowledge base.
- You need the text of many pages under one site, not just one.
- You want to limit a crawl to certain sections with include or exclude paths.

## Look elsewhere when

- You need just one page.
- You want pages on a topic from across the web, not from one site.
- You want a quick answer to a question, not the source pages.

## Limits

- Asynchronous: the crawl returns a task, and the pages are collected with `get_result` once it finishes.
- The page count stops at `limit`, ten by default — set it higher for larger sites.
- Public pages only — nothing behind a login or paywall.
- Pay as you go from your toolfare balance, in US dollars; there is no subscription. A call runs when your balance covers its quote.