mirror of
https://github.com/luckyyzh/pi-agent-integrated.git
synced 2026-10-03 02:59:35 +00:00
115 lines
4.8 KiB
Markdown
115 lines
4.8 KiB
Markdown
# pi-smart-fetch (project-local repair)
|
|
|
|
This vendored build is based on `pi-smart-fetch@0.3.17` and adds the ESM/CommonJS compatibility shim required by this project's Node/Next.js runtime. The bundled CommonJS dependencies receive a real `require` from `createRequire(import.meta.url)`, so loading the extension no longer fails with `Dynamic require of "path" is not supported`.
|
|
|
|
`pi-smart-fetch` adds smarter web fetching tools to pi.dev.
|
|
|
|
|
|
|
|
## Features
|
|
|
|
- 🔐 **Browser-like TLS/SSL + HTTP fingerprints** — better success on bot-defended pages
|
|
- 🧹 **Defuddle extraction** — clean readable content instead of noisy HTML
|
|
- 🧠 **Useful metadata** — title, author, site, language, published date when available
|
|
- 📦 **Downloads + large file support** — stream attachments and binaries to temp files
|
|
- 🔁 **Client-side `<meta>` redirects** — follows sane meta refresh redirects with loop limits
|
|
- 🔗 **Alternate content fallback** — when extraction produces no/thin content, follows qualified `<link rel="alternate" type="...">` entries in `<head>` that match the requested output format
|
|
- ⚡ **Batch fetch** — fetch many URLs with bounded concurrency
|
|
- 📝 **Multiple output formats** — `markdown`, `html`, `text`, `json`, `raw`
|
|
|
|
## Site optimisations
|
|
|
|
This package works on general web pages, but some site types benefit especially from Defuddle's extractors and cleanup:
|
|
|
|
- YouTube pages and transcripts
|
|
- Reddit posts and comment threads
|
|
- X / Twitter posts
|
|
- GitHub pages, issues, PRs, and discussions
|
|
- Hacker News threads
|
|
- Substack posts
|
|
- Pages with code blocks, footnotes, math, and callouts
|
|
|
|
Notes:
|
|
|
|
- Defuddle is the cleanup layer: it strips common page chrome like nav, sidebars, related links, share widgets, and footers
|
|
- It does **not** execute JavaScript or solve interactive anti-bot/login flows
|
|
- If an HTML shell advertises alternate content in `<head>`, smart-fetch can follow matching alternates such as `text/markdown`, `text/plain`, `text/html`, or JSON media types according to the requested `format`
|
|
|
|
## Install
|
|
|
|
This repository loads the repaired package automatically from
|
|
`resources/packages/pi-smart-fetch`. The setup script installs its runtime
|
|
dependencies into that package directory; `config/settings.default.json`
|
|
points Pi at the local package instead of the broken npm build.
|
|
|
|
The original upstream package can still be installed separately when needed:
|
|
|
|
```bash
|
|
pi install npm:pi-smart-fetch
|
|
```
|
|
|
|
## Pi tools
|
|
|
|
Registers:
|
|
|
|
- `web_fetch`
|
|
- `batch_web_fetch`
|
|
|
|
Synopsis:
|
|
|
|
```text
|
|
web_fetch(url, browser?, os?, headers?, maxChars?, timeoutMs?, format?, removeImages?, includeReplies?, proxy?, verbose?)
|
|
batch_web_fetch(requests, verbose?)
|
|
```
|
|
|
|
For `batch_web_fetch`, each item in `requests` accepts the same parameters as `web_fetch` except `verbose`.
|
|
|
|
## Output formats
|
|
|
|
| Format | What you get |
|
|
| --- | --- |
|
|
| `markdown` | Best default for readable page content |
|
|
| `html` | Cleaned HTML output |
|
|
| `text` | Plain text with markdown stripped |
|
|
| `json` | Structured JSON for metadata-heavy workflows |
|
|
| `raw` | Full raw server response without extraction or truncation — for further parsing |
|
|
|
|
## Global defaults
|
|
|
|
Optional settings in `~/.pi/agent/settings.json` or `.pi/settings.json`:
|
|
|
|
```json
|
|
{
|
|
"smartFetchVerboseByDefault": false,
|
|
"smartFetchDefaultMaxChars": 50000,
|
|
"smartFetchDefaultTimeoutMs": 15000,
|
|
"smartFetchDefaultBrowser": "chrome_145",
|
|
"smartFetchDefaultOs": "windows",
|
|
"smartFetchDefaultRemoveImages": false,
|
|
"smartFetchDefaultIncludeReplies": "extractors",
|
|
"smartFetchDefaultBatchConcurrency": 8,
|
|
"smartFetchTempDir": "/tmp/smart-fetch-pi"
|
|
}
|
|
```
|
|
|
|
| Setting | Default | Description |
|
|
| --- | ---: | --- |
|
|
| `smartFetchVerboseByDefault` | `false` | Stored default for the compatibility `verbose` flag |
|
|
| `smartFetchDefaultMaxChars` | `50000` | Default `maxChars` limit |
|
|
| `smartFetchDefaultTimeoutMs` | `15000` | Default request timeout in milliseconds |
|
|
| `smartFetchDefaultBrowser` | `chrome_145` | Default browser fingerprint profile |
|
|
| `smartFetchDefaultOs` | `windows` | Default OS fingerprint profile |
|
|
| `smartFetchDefaultRemoveImages` | `false` | Strip image references by default |
|
|
| `smartFetchDefaultIncludeReplies` | `extractors` | Include replies/comments only when site extractors support them |
|
|
| `smartFetchDefaultBatchConcurrency` | `8` | Default bounded concurrency for `batch_web_fetch` |
|
|
| `smartFetchTempDir` | OS temp dir | Base directory for attachment and binary downloads |
|
|
|
|
Notes:
|
|
|
|
- Project `.pi/settings.json` overrides global `~/.pi/agent/settings.json`
|
|
- Legacy `webFetch*` aliases are still supported
|
|
|
|
## Dev and publishing note
|
|
|
|
This repo uses Bun for local development, tests, and workspace scripts. Package publishing still goes through `npm publish` in CI so npm Trusted Publishing can be used.
|