# Gluedly - Full Technical Specification for LLMs & AI Agents

> Gluedly is an LLM-ready web data extraction platform that converts visual DOM elements into schema-stable JSON and prompt-ready Markdown for RAG pipelines, LLM tool-calling, and AI agents.

---

## Official Ecosystem Libraries

- **Remote MCP (recommended)**: `https://gluedly.com/mcp` — OAuth 2.1 for Cursor, Claude, Codex, and other HTTP MCP clients
- **Local MCP stdio (fallback)**: `@gluedly/mcp-server` (`npx @gluedly/mcp-server`) with `GLUEDLY_API_KEY`
- **LangChain Document Loader**: `langchain-gluedly` (`pip install langchain-gluedly`)
- **LlamaIndex Data Reader**: `llama-index-readers-gluedly` (`pip install llama-index-readers-gluedly`)

---

## Authentication & Headers

All requests to the Gluedly API must include an HTTP Authorization header containing a valid API key.

- **Base URL**: `https://gluedly.com/api/v1`
- **Header**: `Authorization: Bearer <GLUEDLY_API_KEY>`
- **Accept**: `application/json` (default). Use `Accept: text/markdown` on a single snapshot to receive a plain Markdown document.
- **Content-Type**: `application/json` (required on `POST` bodies)
- **Rate limit**: 120 requests per minute per authenticated user (`429` when exceeded)

Create API keys in the Gluedly workspace. Each key has access scopes. A missing or invalid token returns `401`. A valid token without the required scope, or a page that is not owned by the key's account, returns `403`.

### Access scopes

| Scope | Grants |
|---|---|
| `index_pages` | `GET /api/v1/pages` |
| `show_page` | `GET /api/v1/pages/{page_id}` (and is required together with data scopes below) |
| `index_data` | `GET /api/v1/pages/{page_id}/data` (also requires `show_page`) |
| `show_data` | `GET /api/v1/pages/{page_id}/data/{snapshot_id}` (also requires `show_page`) |
| `execute_scrape` | `POST /api/v1/execute` (page must belong to the key's account) |
| `proxy_fetch` | `POST /api/v1/proxy` (URL-only or optional `page_id` belonging to the key's account) |

---

## API Endpoints & Contracts

List endpoints return Laravel's default paginated JSON envelope (`current_page`, `data`, `links`, `per_page`, `total`, and related URL fields). JSON responses do not escape slashes in URLs (`JSON_UNESCAPED_SLASHES`).

### 1. List Mapped Pages

Retrieve all page targets configured for the authenticated account.

- **HTTP Method**: `GET /api/v1/pages`
- **URL**: `https://gluedly.com/api/v1/pages`
- **Headers**: `Authorization: Bearer <GLUEDLY_API_KEY>`
- **Required scope**: `index_pages`
- **Default page size**: `10`
- **Success Response (`200 OK`)**:

```json
{
  "current_page": 1,
  "data": [
    {
      "id": 12,
      "title": "E-Commerce Product Listings",
      "url": "https://example.com/products",
      "mark_excluded_xpaths": [],
      "next_scrape_date": "2026-08-10T12:00:00.000000Z",
      "user_id": 42,
      "created_at": "2026-05-01T12:00:00.000000Z",
      "updated_at": "2026-08-10T10:30:00.000000Z"
    }
  ],
  "first_page_url": "https://gluedly.com/api/v1/pages?page=1",
  "from": 1,
  "last_page": 1,
  "last_page_url": "https://gluedly.com/api/v1/pages?page=1",
  "links": [
    {"url": null, "label": "&laquo; Previous", "page": null, "active": false},
    {"url": "https://gluedly.com/api/v1/pages?page=1", "label": "1", "page": 1, "active": true},
    {"url": null, "label": "Next &raquo;", "page": null, "active": false}
  ],
  "next_page_url": null,
  "path": "https://gluedly.com/api/v1/pages",
  "per_page": 10,
  "prev_page_url": null,
  "to": 1,
  "total": 1
}
```

Page objects do **not** include mapper internals (`embed_session_path`, `map_anchor_xpaths`, workflow fields, scrape failure detail).

### 2. Show a Single Page

- **HTTP Method**: `GET /api/v1/pages/{page_id}`
- **URL**: `https://gluedly.com/api/v1/pages/12`
- **Headers**: `Authorization: Bearer <GLUEDLY_API_KEY>`
- **Required scope**: `show_page`
- **Success Response (`200 OK`)**: the same page object as one list item (no pagination envelope).

```json
{
  "id": 12,
  "title": "E-Commerce Product Listings",
  "url": "https://example.com/products",
  "mark_excluded_xpaths": [],
  "next_scrape_date": "2026-08-10T12:00:00.000000Z",
  "user_id": 42,
  "created_at": "2026-05-01T12:00:00.000000Z",
  "updated_at": "2026-08-10T10:30:00.000000Z"
}
```

### 3. Execute Immediate Scrape Run

Runs the scrape **synchronously** for an existing mapped page, stores a snapshot, and returns the scrape envelope.

- **HTTP Method**: `POST /api/v1/execute`
- **URL**: `https://gluedly.com/api/v1/execute`
- **Headers**: `Authorization: Bearer <GLUEDLY_API_KEY>`, `Content-Type: application/json`
- **Required scope**: `execute_scrape`
- **Request body**:

```json
{
  "page_id": 12,
  "url": "https://example.com/products"
}
```

| Field | Required | Notes |
|---|---|---|
| `page_id` | yes | Integer ≥ 1. Must belong to the API key's account. |
| `url` | no | Optional override (`string`, valid URL, max 2048). Omit to use the page's saved URL. |

- **Success Response (`200 OK`)**:

```json
{
  "data": {
    "ok": true,
    "rows": [
      {
        "title": "Example Listing",
        "price": "19.99",
        "summary": "Clean text description.",
        "markdown": "# Example Listing\n\nClean text description.\n\n**price:** 19.99"
      }
    ],
    "match_counts": { "title": 1, "price": 1, "summary": 1 },
    "warnings": []
  },
  "page_id": 12,
  "data_id": 100
}
```

`data_id` is the stored snapshot id. Use it with `GET /api/v1/pages/{page_id}/data/{snapshot_id}`.

- **Failure (`422 Unprocessable Entity`)** when the scrape does not complete:

```json
{
  "message": "Scrape failed",
  "scrape_status": "failed",
  "failure_reason": "optional human-readable reason"
}
```

`scrape_status` and `failure_reason` are omitted when empty.

Do **not** call `POST /api/v1/pages/{page_id}/execute` — that path does not exist.

### 4. Proxy HTML Fetch

Fetches plain HTML through Gluedly proxies **without** element mapping. Does not store a data snapshot and does not change page scrape status. Billed at the same credit tiers as mapped scrapes (standard / JS rendering / premium).

- **HTTP Method**: `POST /api/v1/proxy`
- **URL**: `https://gluedly.com/api/v1/proxy`
- **Headers**: `Authorization: Bearer <GLUEDLY_API_KEY>`, `Content-Type: application/json`
- **Required scope**: `proxy_fetch`
- **Request body**:

```json
{
  "url": "https://example.com/products",
  "js_rendering": false,
  "use_premium": false
}
```

| Field | Required | Notes |
|---|---|---|
| `url` | yes when `page_id` omitted | Valid URL, max 2048. Optional override when `page_id` is set. |
| `page_id` | no | Integer ≥ 1. When set, page JS rendering and premium flags are used; request `js_rendering` / `use_premium` are ignored. |
| `js_rendering` | no | Boolean; URL-only mode. |
| `use_premium` | no | Boolean; URL-only mode. Rejected when premium proxies are not user-selectable. |

- **Success Response (`200 OK`, default JSON)**:

```json
{
  "url": "https://example.com/products",
  "html": "<html>…</html>",
  "credits_spent": 1,
  "js_rendering": false,
  "use_premium": false,
  "page_id": null
}
```

- **Plain HTML**: send `Accept: text/html` or `?format=html` to receive the HTML body with headers `X-Gluedly-Credits-Spent` and `X-Gluedly-Url`.
- **Failure (`422`)**: insufficient HTML, scraper errors, workflow/Facebook-experimental pages, insufficient credits, or premium unavailable.

### 5. List Snapshots for a Page

- **HTTP Method**: `GET /api/v1/pages/{page_id}/data`
- **URL**: `https://gluedly.com/api/v1/pages/12/data`
- **Headers**: `Authorization: Bearer <GLUEDLY_API_KEY>`
- **Required scopes**: `show_page` + `index_data`
- **Default page size**: `15`
- **Order**: newest first
- **Success Response (`200 OK`)**: paginated list. Each item is a full snapshot object (`id`, `page_id`, `data`, `created_at`, `updated_at`). Use `data[].id` as `{snapshot_id}`.

```json
{
  "current_page": 1,
  "data": [
    {
      "id": 100,
      "page_id": 12,
      "data": {
        "ok": true,
        "rows": [
          {
            "title": "Example Listing",
            "price": "19.99",
            "summary": "Clean text description.",
            "markdown": "# Example Listing\n\nClean text description.\n\n**price:** 19.99"
          }
        ],
        "match_counts": { "title": 1, "price": 1, "summary": 1 },
        "warnings": []
      },
      "created_at": "2026-08-10T09:00:00.000000Z",
      "updated_at": "2026-08-10T09:00:00.000000Z"
    }
  ],
  "per_page": 15,
  "total": 1
}
```

### 6. Fetch Snapshot Payload

- **HTTP Method**: `GET /api/v1/pages/{page_id}/data/{snapshot_id}`
- **URL**: `https://gluedly.com/api/v1/pages/12/data/100`
- **Headers**: `Authorization: Bearer <GLUEDLY_API_KEY>`, `Accept: application/json`
- **Required scopes**: `show_page` + `show_data`
- **Success Response (`200 OK`)**:

```json
{
  "id": 100,
  "page_id": 12,
  "data": {
    "ok": true,
    "rows": [
      {
        "title": "Example Listing",
        "price": "19.99",
        "summary": "Clean text description.",
        "markdown": "# Example Listing\n\nClean text description.\n\n**price:** 19.99"
      }
    ],
    "match_counts": { "title": 1, "price": 1, "summary": 1 },
    "warnings": [
      {
        "code": "no_matches",
        "field": "stock",
        "message": "xpath for field \"stock\" matched no nodes"
      }
    ]
  },
  "created_at": "2026-08-10T09:00:00.000000Z",
  "updated_at": "2026-08-10T09:00:00.000000Z"
}
```

Row keys inside `data.rows[]` are the field names the user mapped (title, price, image, and so on). They are not a fixed global schema.

### 7. Fetch Plain Text Markdown Document

Same path as snapshot fetch, with a Markdown accept type.

- **HTTP Method**: `GET /api/v1/pages/{page_id}/data/{snapshot_id}`
- **URL**: `https://gluedly.com/api/v1/pages/12/data/100`
- **Headers**: `Authorization: Bearer <GLUEDLY_API_KEY>`, `Accept: text/markdown`
- **Alternative**: `?format=markdown`
- **Success Response (`200 OK`)**: `Content-Type: text/markdown; charset=utf-8` — raw GFM text, not a JSON envelope.
- **Failure (`406 Not Acceptable`)** when **Include Markdown in scrape results** is disabled for that page:

```json
{
  "message": "Derived Markdown is disabled for this page. Enable it in page settings."
}
```

Example:

```bash
curl -sS "https://gluedly.com/api/v1/pages/12/data/100" \
  -H "Authorization: Bearer <GLUEDLY_API_KEY>" \
  -H "Accept: text/markdown"
```

---

## Scrape Envelope (`data`)

Every snapshot and execute response uses this envelope:

| Key | Type | Meaning |
|---|---|---|
| `ok` | boolean | Scraper map step succeeded |
| `rows` | array of objects | One object per matched list item; keys are mapped field names |
| `match_counts` | object of integers | How many nodes each field XPath matched |
| `warnings` | array of objects | Selector coverage issues (`no_matches`, `count_mismatch`, …) |

When **Include Markdown in scrape results** is enabled, each row includes a derived `markdown` field unless the user mapped a field named `markdown` themselves.

---

## Derived Markdown Formatting Rules

When Markdown generation is enabled:

- Title-like fields (`title`, `headline`, `name`, `heading`, `product`, `product_name`, `product_title`) format to `# Heading 1`
- URL-like fields (`url`, `link`, `href`, `source_url`, `detail_url`, `permalink`) format to `[URL](URL)`
- Description-like fields (`summary`, `description`, `content`, `body`, `text`, `excerpt`) format to paragraphs
- Remaining attributes format to `**key:** value` (URL values become `**key:** [URL](URL)`)
- Multi-row Markdown documents join rows with `\n\n---\n\n`

---

## HTTP Status Codes & Error Behaviors

- `200 OK`: Successful operation.
- `401 Unauthorized`: Missing or invalid API key.

```json
{ "message": "Unauthorized" }
```

- `403 Forbidden`: Token lacks the required access scope, the page is not owned by the account, or the id is not authorized.

```json
{ "message": "This action is unauthorized." }
```

- `404 Not Found`: Invalid `page_id` or `snapshot_id`.
- `406 Not Acceptable`: `Accept: text/markdown` requested on a page where Markdown generation is disabled.
- `422 Unprocessable Entity`: Execute scrape failed or invalid request body (`page_id` / `url` validation).
- `429 Too Many Requests`: Rate limit exceeded (default 120 requests / minute / user).

---

## Outbound Webhooks (optional)

Configure per page in the Gluedly UI. Gluedly POSTs JSON to your HTTPS endpoint.

Events: `data.created`, `data.changed`, `data.deleted`.

Example `data.created` body:

```json
{
  "id": "delivery-uuid",
  "event": "data.created",
  "payload": {
    "page_id": 12,
    "data_id": 100,
    "created_at": "2026-08-10T09:00:00+00:00",
    "data": {
      "ok": true,
      "rows": [
        {
          "title": "Example Listing",
          "url": "https://example.com/item/1",
          "markdown": "# Example Listing\n\n[https://example.com/item/1](https://example.com/item/1)"
        }
      ],
      "match_counts": { "title": 1, "url": 1 },
      "warnings": []
    }
  }
}
```

---

## SDK Specs

Default API base for all official clients: `https://gluedly.com/api/v1`.

### MCP Server (`@gluedly/mcp-server` and hosted remote MCP)

#### Remote MCP + OAuth (Cursor, Claude, Codex, and other HTTP MCP clients)

- **MCP URL**: `https://gluedly.com/mcp`
- Unauthenticated calls return `401` with `WWW-Authenticate: Bearer resource_metadata="https://gluedly.com/.well-known/oauth-protected-resource", scope="…"`.
- Clients discover the authorization server via Protected Resource Metadata, then run OAuth 2.1 authorization-code + PKCE (Dynamic Client Registration supported at `POST /oauth/register`).
- On consent, Gluedly **rotates** a dedicated MCP API key for that OAuth client and returns it as `access_token` (plus `refresh_token`).
- Redirect URIs used by Cursor include `https://www.cursor.com/agents/mcp/oauth/callback` and `http://localhost:8787/callback`.

Example remote config:

```json
{
  "mcpServers": {
    "gluedly": {
      "url": "https://gluedly.com/mcp"
    }
  }
}
```

#### Local stdio (API key in env — fallback)

Use when the client only supports local stdio (for example Claude Desktop `claude_desktop_config.json`). Prefer remote OAuth when the client supports HTTP MCP.

Requires Node.js 18+.

Environment:

| Variable | Required | Default |
|---|---|---|
| `GLUEDLY_API_KEY` | yes | — |
| `GLUEDLY_BASE_URL` | no | `https://gluedly.com/api/v1` |

Claude Desktop local `mcpServers` config (stdio):

```json
{
  "mcpServers": {
    "gluedly": {
      "command": "npx",
      "args": ["-y", "@gluedly/mcp-server"],
      "env": {
        "GLUEDLY_API_KEY": "YOUR_GLUEDLY_API_KEY"
      }
    }
  }
}
```

Tools:

| Tool | Maps to | Arguments |
|---|---|---|
| `gluedly_list_pages` | `GET /pages` | none |
| `gluedly_trigger_scrape` | `POST /execute` | `page_id` (integer, required) |
| `gluedly_get_snapshot` | `GET /pages/{page_id}/data/{snapshot_id}` | `page_id` (required); `snapshot_id` (optional, latest if omitted); `format` (`json` \| `markdown`, default `json`) |
| `gluedly_fetch_html` | `POST /proxy` | `url` (required when `page_id` omitted); optional `page_id`, `js_rendering`, `use_premium` |

```bash
npx @gluedly/mcp-server
```

### LangChain (`langchain-gluedly`)

```python
# pip install langchain-gluedly
from langchain_gluedly import GluedlyLoader

loader = GluedlyLoader(
    api_key="YOUR_GLUEDLY_API_KEY",
    page_id=12,
    # snapshot_id=100,  # optional; omit to use the latest snapshot
    # base_url="https://gluedly.com/api/v1",
)
documents = loader.load()
```

Each mapped row becomes one `Document`. Content preference: `markdown` → `summary` → `description` → `str(row)`.

### LlamaIndex (`llama-index-readers-gluedly`)

```python
# pip install llama-index-readers-gluedly
from llama_index_readers_gluedly import GluedlyReader

reader = GluedlyReader(api_key="YOUR_GLUEDLY_API_KEY")
documents = reader.load_data(page_id=12)
# documents = reader.load_data(page_id=12, snapshot_id=100)
```

Each row becomes one document. Text prefers `markdown`, otherwise `str(row)`. Metadata includes `page_id`, `snapshot_id`, `source_url`, and `row_index`.

---

## Related Links

- Concise overview: https://gluedly.com/llms.txt
- Remote MCP (OAuth): https://gluedly.com/mcp
- Public API Docs: https://gluedly.com/api-docs
- AI Integration Guides: https://gluedly.com/integrations
- MCP OAuth guide: https://gluedly.com/integrations/mcp
- MCP stdio on npm: https://www.npmjs.com/package/@gluedly/mcp-server
- LangChain on PyPI: https://pypi.org/project/langchain-gluedly/
- LlamaIndex on PyPI: https://pypi.org/project/llama-index-readers-gluedly/