Point your AI at a docs site and get its pages back as Markdown.
Start from one URL and crawl a public website, getting each page back as Markdown — with a page limit, a link depth and paths to include or skip. Ask from Claude, ChatGPT, Cursor, Codex or your own code; the crawl runs as a task the agent collects with `get_result`. No crawler to host and no vendor account, quoted before each call.
1Connect your AI app
Pick the app you use. You do this once, and every API on toolfare is then available in it.
Add toolfare as a connector in Claude
Add to Claudehttps://api.toolfare.ai/mcp
- 1Open Customize → Connectors and choose "Add custom connector".
- 2Paste this URL as the remote MCP server URL.
- 3Leave the OAuth client id and secret under Advanced settings empty.
- 4Complete the browser authorization, then return to your conversation.
Custom connectors are available on Free, Pro, Max, Team and Enterprise. A Free account may hold one custom connector in total.
Claude’s own instructions2What to ask
Copy a prompt into your AI app once you’re connected. It picks the right one of the 1 tools and tells you the price first.
A site, page by page, as Markdown
From $0.06Crawl a public website and return each page as Markdown
- “Crawl this documentation site and give me every page as Markdown.”
- “Crawl only the /blog pages of this site, up to 50, and list their titles.”
- “Pull this competitor’s help centre so we can compare it with ours.”
1Create an API key
One key and one endpoint for every API on toolfare.
export TOOLFARE_API_KEY='tf_live_…'
2Every tool and its input
Send the tool name and its arguments as JSON; the reply is JSON too.
web_crawl_crawlsCrawl a public website and return each page as Markdown
From $0.06
Required parameters:
urlOptional parameters:
exclude_paths, include_paths, limit, max_depth
Use it when
- You want a documentation site or help centre as Markdown for a retrieval index or a knowledge base.
- You need the text of many pages under one site, not just one.
- You want to limit a crawl to certain sections with include or exclude paths.
Look elsewhere when
- You need just one page. Use this instead
- You want pages on a topic from across the web, not from one site. Use this instead
- You want a quick answer to a question, not the source pages. Use this instead
Web Crawl directly, or through toolfare
Directly
- A crawling API account and key, or your own crawler and queue
- Code to follow links, respect depth and path rules, and convert HTML to Markdown
- Polling and storage for long-running jobs
Through toolfare
- One call starts the crawl, quoted before it runs; `get_result` collects it
- No crawler to host and no vendor key in an agent’s config
- Single-page reading and search on the same connection
toolfare does not schedule repeat crawls, store the pages for you or keep an index; each crawl is a one-off job you collect.
Limits
- Asynchronous: the crawl returns a task, and the pages are collected with `get_result` once it finishes.
- The page count stops at `limit`, ten by default — set it higher for larger sites.
- Public pages only — nothing behind a login or paywall.
- Pay as you go from your toolfare balance, in US dollars; there is no subscription. A call runs when your balance covers its quote.
Questions
For public sites, yes: it returns each crawled page as Markdown, ready for a model or a retrieval index. It does not log in to sites or reach private pages.
Up to `limit` pages, ten by default. You can also cap the link depth from the start URL with `max_depth`.
Yes. Pass path patterns in `include_paths` to keep, or in `exclude_paths` to skip.
Crawls take time, so the first call returns a task and your agent collects the pages with `get_result` once it has finished.
Each tool’s current price is shown on this page and quoted before the call runs.