ScrapingBytes API - Documentation

Welcome to the official ScrapingBytes documentation. Our goal is to make this process as simple as possible. Web scraping shouldn't be hard and neither should our implementation. To get started you'll need to know a few things:

  • Your API Key
  • The URL you want to scrape
  • Do you need to render Javascript?
  • Do you want to use a premium/residential proxy?

Finding Your API Key

You have to generate an API key. It will only be shown once. Please store this in a secure place and keep it private. To generate an API key first head to the dashboard.

generating an API key on the ScrapingBytes dashboard

An API key will be shown like the following. example API key

Copy this and integrate it into your application. Keep your API key private and secure. How to do this is beyond the scope of this documentation. A brief example from Twilio by using environmental variables can be watched here.

Sending a Request

Now that you have your API key you can send a POST request to our API:

https://scrapingbytes.com/api/v1/scrape
Authorization: Bearer YOUR-API-KEY

You'll need to build a request body, and send it as JSON. At a minimum you'll need to provide the target URL.

{
    "url": "https://books.toscrape.com/",
    "browser": true,
    "premium_proxy": true
}

If you do not want to use the headless browser or premium proxies set the values to false. Don't know how to do this? Try using the Request Builder on the dashboard.

The API will return the raw HTML from the target URL:

<html>
<head>
...
</head>
<body>
...
</body>
</html>

To know if a request was successful plus the credit costs, check the response headers.

SB-Credit-Cost This header will return a number with the total credits used. SB-Success This header will return a boolean on whether the request was successful or not.

In this case the response headers would be:

Key Value
SB-Credit-Cost 25
SB-Success true

Credit Costs

Every ScrapingBytes plan includes a certain amount of credits to use per month.

The credits used per request depends on the parameters provided with your API calls. It will cost 1 to 25 credits to make a request with the ScrapingBytes API.

A breakdown of credit costs:

Feature Credit Cost
Rotating Proxy without JavaScript rendering 1
Rotating Proxy with JavaScript rendering 5
Premium Proxy without JavaScript rendering 10
Premium Proxy with JavaScript rendering 25

Response Status Codes

Every response from the ScrapingBytes API returns a HTTP code with a specific/general reason. You can find all status codes and their reasons below:

Code Charged Meaning Solution
200 Yes Successful request
400 No Bad request Invalid parameters provided. Check the response message for further details.
402 No Not enough credits Please upgrade your plan, or contact support/sales for assistance.
429 No Too many concurrent connections Please upgrade your plan, or contact support/sales for assistance.
404 Yes Invalid URL You provided an invalid URL or we couldn't find the requested URL.
500 No Error Please try again, an unknown error has occurred.