ProConfig Tutorial
  • Overview & Setup
  • Tutorial Structure
  • Tutorial
    • Hello World with Pro Config
    • Building Workflow
    • Transitions
    • Expressions and Variables
    • An Advanced Example
    • Function Calling Example
    • Random Routing Example
  • API Reference
    • Widgets
      • Bark TTS
      • Champ
      • CoinGecko
      • ControlNet with Civitai
      • Crawler
      • Crypto News
      • Data Visualizer
      • Email Sender
      • Google Flight Search
      • Google Hotel Search
      • Google Image Search
      • Google Map Search
      • Google News Search
      • Google Scholar Search
      • Google Search
      • GroundedSAM
      • Image Text Fuser
      • Information Extractor - OpenAI Schema Generator
      • Information Extractor
      • Instagram Search
      • JSON to Table
      • LinkedIn
      • MS Word to Markdown
      • Markdown to MS Word
      • Markdown to PDF
      • Mindmap Generator
      • Notion Database
      • OCR
      • Pdf to Markdown
      • RMBG
      • Stabel-Video-Diffusion
      • Stable Diffusion Inpaint
      • Stable Diffusion Recommend
      • Stable Diffusion Transform
      • Stable Diffusion Upscale
      • Stable Diffusion with 6 fixed category
      • Stable Diffusion with Civitai
      • Storydiffusion
      • Suno Lyrics Generator
      • Suno Music Generator
      • Table to Markdown
      • TripAdvisor
      • Twitter Search
      • UDOP: Document Question Answering
      • Weather forecasting
      • Whisper large-v3
      • Wikipedia
      • Wolfram Alpha Search
      • Yelp Search
      • YouTube Downloader
      • YouTube Transcriber
      • Youtube Search
  • Tools
    • AutoConfig Bot
    • Cache Mode
Powered by GitBook
On this page
  • Try it in the Widget Center
  • Usage
  1. API Reference
  2. Widgets

Crawler

A naive crawler for website. Return raw content in markdwon format.

PreviousControlNet with CivitaiNextCrypto News

Last updated 1 year ago

Try it in the Widget Center

Click this to try this widget and copy the Pro Config template.

Usage

Given the target website url, return the markdown string of the content

Input Parameters

Name
Type
Description
Default
Required

url

string

The target website url

Output Parameters

Name
Type
Description
File Type

markdown_string

string

The raw content of given url

Output Example

{
  "markdown_string": "Title: MyShell - An Ultimate Place for AI Apps Creation, Discover, and Earn\n\nURL Source: https://app.myshell.ai/explore\n\nMarkdown Content:\nExplore\n-------\n"
}

url