curl-paging

A curl-compatible CLI wrapper that optionally fetches all pages from a paginated API and outputs aggregated JSON.

Modes

Default: curl wrapper mode

Without --cp, all arguments are passed through to curl as-is.

curl-paging https://api.example.com/items
# → equivalent to: curl https://api.example.com/items

Pagination mode

With --cp, fetches all pages and aggregates the results.

curl-paging --cp https://api.example.com/items
# → fetches all pages and outputs aggregated JSON

Options

curl-paging options (only effective with --cp)

Intercepted curl options (in pagination mode)

In curl wrapper mode, -o is intercepted and -D is passed through to curl.

curl options

All other options are passed through to curl.

Examples

# Simple curl wrapper
curl-paging https://api.example.com/items

# Pagination mode
curl-paging --cp https://api.example.com/items

# With headers
curl-paging --cp -H "Authorization: Bearer token" https://api.example.com/items

# Custom keys
curl-paging --cp --cp-data-key items --cp-pagination-key meta https://api.example.com/items

# Limit to 10 pages
curl-paging --cp --cp-max-pages 10 https://api.example.com/items

# Output to file
curl-paging --cp -o result.json https://api.example.com/items

Output

stdout

On success, outputs aggregated JSON:

{"data":[{"id":1},{"id":2},...]}

Artifacts (./paging/)

A directory is created for each page:

./paging/
  0001/
    cmd          - curl command executed
    req.header   - Request headers sent
    req.body     - Request body (if any)
    res.header   - Response headers received
    res.body     - Raw response body
    res.json     - JSON with pagination metadata stripped
  0002/
    ...

Debugging on error

On failure, the .wip suffixed directory is left for debugging:

./paging/
  0001/           # Successful page
  0002.wip/       # Failed page (for debugging)
    req.header
    res.header
    res.body

Safety

All safety checks are designed to prevent client-side mistakes from causing excessive or repeated requests to the server.

Build & Test

Written in Crystal. Builds via Docker.

make build   # Build binary
make test    # Run E2E tests
make         # Show available tasks

Known Limitations

• Only supports page/total_pages pagination (no cursor/next_url/Link header support)
• JSON keys are top-level only (no nested path support)
• No retry mechanism
• Sequential fetching only (no parallel requests)