Documentation

API Access Key & Authentication

After signing up, every user is assigned a personal API access key, a unique combination of letters and digits provided to access to our API endpoint. To authenticate with the Codery API, simply include your bearer token in the Authorization header.

Headers

Header Description

Authorization  [Required] Should be Bearer access_key. See "Your API Access Key" above.

API Features

Object Description

URL address    [Required] URL to look up using Codery.

Example API Request:

curl 'https://app.mycodery.com/api/scrape?url=' -X GET  -H 'Accept: application/json' -H 'Authorization: YOUR_API_KEY

Overview

Here is the list of the different parameters you can use with Codery's HTML API. You can also discover this API using our Postman collection covering every Codery's features.

name description [type] (default)
URL The URL of the page you want to scrape [boolean] (required)
API key Your API key [boolean] (required)
block_ads Block ads on the page you want to scrape [boolean] (false)
block_resources Block images and CSS on the page you want to scrape [boolean] (true)
country_code Premium proxy geolocation [Countries Available] [string] ("")
render_js Render the JavaScript on the page with a headless browser [boolean] (true)
premium_proxy Use premium proxies to bypass difficult to scrape websites [boolean] (false)
screenshot Return a screenshot of the page you want to scrape [boolean] (false)
screenshot_full_page Return a screenshot of the full page you want to scrape [boolean] (false)

EXAMPLES

Ej: YOUR_API_KEY

PHP CURL

            $curl = curl_init();

            curl_setopt_array($curl, array(
              CURLOPT_URL => "https://app.mycodery.com/api/scrape/?access_token=YOUR_API_KEY&url=YOUR-URL",
              CURLOPT_RETURNTRANSFER => true,
              CURLOPT_ENCODING => "",
              CURLOPT_MAXREDIRS => 10,
              CURLOPT_TIMEOUT => 30,
              CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_1,
              CURLOPT_HTTPHEADER => array(
                "Accept: application/json",
                "Authorization:  YOUR_API_KEY",
                "Content-Type: application/json",
                "cache-control: no-cache"
              ),
            ));

            $response = curl_exec($curl);
            $err = curl_error($curl);

            curl_close($curl);

            if ($err) {
              echo "cURL Error #:" . $err;
            } else {
              echo $response;
            }

Python

            import requests

            url = "https://app.mycodery.com/api/scrape/?access_token=YOUR_API_KEY&url=YOUR-URL"

            headers = {
                'Accept': "application/json",
                'Content-Type': "application/json",
                'Authorization': "YOUR_API_KEY",
                'cache-control': "no-cache"
                }

            response = requests.request("GET", url, headers=headers)

            print(response.text)

JavaScript Jquery AJAX

            var settings = {
            "async": true,
            "crossDomain": true,
            "url": "https://app.mycodery.com/api/scrape/?access_token=YOUR_API_KEY &url=YOUR-URL",
            "method": "GET",
            "headers": {
              "Accept": "application/json",
              "Content-Type": "application/json",
              "Authorization": "YOUR_API_KEY"
              "cache-control": "no-cache"
            },
            "processData": false
          }

          $.ajax(settings).done(function (response) {
            console.log(response);
          });


Country List

            Countries_list = {
                            "AF" => "Afghanistan"
                            "AL" => "Albania"
                            "DZ" => "Algeria"
                            "AS" => "American Samoa"
                            "AD" => "Andorra"
                            "AO" => "Angola"
                            "AI" => "Anguilla"
                            "AQ" => "Antarctica"
                            "AG" => "Antigua and Barbuda"
                            "AR" => "Argentina"
                            "AM" => "Armenia"
                            "AW" => "Aruba"
                            "AU" => "Australia"
                            "AT" => "Austria"
                            "AZ" => "Azerbaijan"
                            "BS" => "Bahamas"
                            "BH" => "Bahrain"
                            "BD" => "Bangladesh"
                            "BB" => "Barbados"
                            "BY" => "Belarus"
                            "BE" => "Belgium"
                            "BZ" => "Belize"
                            "BJ" => "Benin"
                            "BM" => "Bermuda"
                            "BT" => "Bhutan"
                            "BO" => "Bolivia"
                            "BA" => "Bosnia and Herzegovina"
                            "BW" => "Botswana"
                            "BV" => "Bouvet Island"
                            "BR" => "Brazil"
                            "IO" => "British Indian Ocean Territory"
                            "BN" => "Brunei Darussalam"
                            "BG" => "Bulgaria"
                            "BF" => "Burkina Faso"
                            "BI" => "Burundi"
                            "KH" => "Cambodia"
                            "CM" => "Cameroon"
                            "CA" => "Canada"
                            "CV" => "Cape Verde"
                            "KY" => "Cayman Islands"
                            "CF" => "Central African Republic"
                            "TD" => "Chad"
                            "CL" => "Chile"
                            "CN" => "China"
                            "CX" => "Christmas Island"
                            "CC" => "Cocos (Keeling) Islands"
                            "CO" => "Colombia"
                            "KM" => "Comoros"
                            "CG" => "Congo"
                            "CD" => "Congo, the Democratic Republic of the"
                            "CK" => "Cook Islands"
                            "CR" => "Costa Rica"
                            "CI" => "Cote D'Ivoire"
                            "HR" => "Croatia"
                            "CU" => "Cuba"
                            "CY" => "Cyprus"
                            "CZ" => "Czech Republic"
                            "DK" => "Denmark"
                            "DJ" => "Djibouti"
                            "DM" => "Dominica"
                            "DO" => "Dominican Republic"
                            "EC" => "Ecuador"
                            "EG" => "Egypt"
                            "SV" => "El Salvador"
                            "GQ" => "Equatorial Guinea"
                            "ER" => "Eritrea"
                            "EE" => "Estonia"
                            "ET" => "Ethiopia"
                            "FK" => "Falkland Islands (Malvinas)"
                            "FO" => "Faroe Islands"
                            "FJ" => "Fiji"
                            "FI" => "Finland"
                            "FR" => "France"
                            "GF" => "French Guiana"
                            "PF" => "French Polynesia"
                            "TF" => "French Southern Territories"
                            "GA" => "Gabon"
                            "GM" => "Gambia"
                            "GE" => "Georgia"
                            "DE" => "Germany"
                            "GH" => "Ghana"
                            "GI" => "Gibraltar"
                            "GR" => "Greece"
                            "GL" => "Greenland"
                            "GD" => "Grenada"
                            "GP" => "Guadeloupe"
                            "GU" => "Guam"
                            "GT" => "Guatemala"
                            "GN" => "Guinea"
                            "GW" => "Guinea-Bissau"
                            "GY" => "Guyana"
                            "HT" => "Haiti"
                            "HM" => "Heard Island and Mcdonald Islands"
                            "VA" => "Holy See (Vatican City State)"
                            "HN" => "Honduras"
                            "HK" => "Hong Kong"
                            "HU" => "Hungary"
                            "IS" => "Iceland"
                            "IN" => "India"
                            "ID" => "Indonesia"
                            "IR" => "Iran, Islamic Republic of"
                            "IQ" => "Iraq"
                            "IE" => "Ireland"
                            "IL" => "Israel"
                            "IT" => "Italy"
                            "JM" => "Jamaica"
                            "JP" => "Japan"
                            "JO" => "Jordan"
                            "KZ" => "Kazakhstan"
                            "KE" => "Kenya"
                            "KI" => "Kiribati"
                            "KP" => "Korea, Democratic People's Republic of"
                            "KR" => "Korea, Republic of"
                            "KW" => "Kuwait"
                            "KG" => "Kyrgyzstan"
                            "LA" => "Lao People's Democratic Republic"
                            "LV" => "Latvia"
                            "LB" => "Lebanon"
                            "LS" => "Lesotho"
                            "LR" => "Liberia"
                            "LY" => "Libyan Arab Jamahiriya"
                            "LI" => "Liechtenstein"
                            "LT" => "Lithuania"
                            "LU" => "Luxembourg"
                            "MO" => "Macao"
                            "MK" => "Macedonia, the Former Yugoslav Republic of"
                            "MG" => "Madagascar"
                            "MW" => "Malawi"
                            "MY" => "Malaysia"
                            "MV" => "Maldives"
                            "ML" => "Mali"
                            "MT" => "Malta"
                            "MH" => "Marshall Islands"
                            "MQ" => "Martinique"
                            "MR" => "Mauritania"
                            "MU" => "Mauritius"
                            "YT" => "Mayotte"
                            "MX" => "Mexico"
                            "FM" => "Micronesia, Federated States of"
                            "MD" => "Moldova, Republic of"
                            "MC" => "Monaco"
                            "MN" => "Mongolia"
                            "MS" => "Montserrat"
                            "MA" => "Morocco"
                            "MZ" => "Mozambique"
                            "MM" => "Myanmar"
                            "NA" => "Namibia"
                            "NR" => "Nauru"
                            "NP" => "Nepal"
                            "NL" => "Netherlands"
                            "AN" => "Netherlands Antilles"
                            "NC" => "New Caledonia"
                            "NZ" => "New Zealand"
                            "NI" => "Nicaragua"
                            "NE" => "Niger"
                            "NG" => "Nigeria"
                            "NU" => "Niue"
                            "NF" => "Norfolk Island"
                            "MP" => "Northern Mariana Islands"
                            "NO" => "Norway"
                            "OM" => "Oman"
                            "PK" => "Pakistan"
                            "PW" => "Palau"
                            "PS" => "Palestinian Territory, Occupied"
                            "PA" => "Panama"
                            "PG" => "Papua New Guinea"
                            "PY" => "Paraguay"
                            "PE" => "Peru"
                            "PH" => "Philippines"
                            "PN" => "Pitcairn"
                            "PL" => "Poland"
                            "PT" => "Portugal"
                            "PR" => "Puerto Rico"
                            "QA" => "Qatar"
                            "RE" => "Reunion"
                            "RO" => "Romania"
                            "RU" => "Russian Federation"
                            "RW" => "Rwanda"
                            "SH" => "Saint Helena"
                            "KN" => "Saint Kitts and Nevis"
                            "LC" => "Saint Lucia"
                            "PM" => "Saint Pierre and Miquelon"
                            "VC" => "Saint Vincent and the Grenadines"
                            "WS" => "Samoa"
                            "SM" => "San Marino"
                            "ST" => "Sao Tome and Principe"
                            "SA" => "Saudi Arabia"
                            "SN" => "Senegal"
                            "CS" => "Serbia and Montenegro"
                            "SC" => "Seychelles"
                            "SL" => "Sierra Leone"
                            "SG" => "Singapore"
                            "SK" => "Slovakia"
                            "SI" => "Slovenia"
                            "SB" => "Solomon Islands"
                            "SO" => "Somalia"
                            "ZA" => "South Africa"
                            "GS" => "South Georgia and the South Sandwich Islands"
                            "ES" => "Spain"
                            "LK" => "Sri Lanka"
                            "SD" => "Sudan"
                            "SR" => "Suriname"
                            "SJ" => "Svalbard and Jan Mayen"
                            "SZ" => "Swaziland"
                            "SE" => "Sweden"
                            "CH" => "Switzerland"
                            "SY" => "Syrian Arab Republic"
                            "TW" => "Taiwan, Province of China"
                            "TJ" => "Tajikistan"
                            "TZ" => "Tanzania, United Republic of"
                            "TH" => "Thailand"
                            "TL" => "Timor-Leste"
                            "TG" => "Togo"
                            "TK" => "Tokelau"
                            "TO" => "Tonga"
                            "TT" => "Trinidad and Tobago"
                            "TN" => "Tunisia"
                            "TR" => "Turkey"
                            "TM" => "Turkmenistan"
                            "TC" => "Turks and Caicos Islands"
                            "TV" => "Tuvalu"
                            "UG" => "Uganda"
                            "UA" => "Ukraine"
                            "AE" => "United Arab Emirates"
                            "GB" => "United Kingdom"
                            "US" => "United States"
                            "UM" => "United States Minor Outlying Islands"
                            "UY" => "Uruguay"
                            "UZ" => "Uzbekistan"
                            "VU" => "Vanuatu"
                            "VE" => "Venezuela"
                            "VN" => "Viet Nam"
                            "VG" => "Virgin Islands, British"
                            "VI" => "Virgin Islands, U.s."
                            "WF" => "Wallis and Futuna"
                            "EH" => "Western Sahara"
                            "YE" => "Yemen"
                            "ZM" => "Zambia"
                            "ZW" => "Zimbabwe"
                          }

Extraction Rules

Usage Instructions

If you wish to get data from web pages but don't want to analyze the HTML on your side, it's possible to add extraction rules to your API call. The only thing you have to do is to add an extraction rule to your API call employing the following format:

{"key_name" : "css_selector"}

If you need to extract a webpage's title and subtitle, for example, you'll need to apply these parameters.

{
"title" : "h1",
"subtitle" : "#subtitle",
}

And this will be the JSON response

{
"title" : "The Codery API",
"subtitle" : "Extract and collect data from any website in only seconds.",
}

It's important to remind that extraction rules are JSON-formatted, and you must stringify them before passing them to a GET request.

Here's how to get the information above in your preferred language.

*Install the Python Codery library:*
*pip install Codery*
from Codery import CoderyClient

client = CoderyClient(api_key='YOUR-API-KEY')
response = client.get(
'https://www.mycodery.com/',
params={
    'extract_rules':{"title": "h1", "subtitle": "#subtitle"},
     },
)
print('Response HTTP Status Code: ', response.status_code)
print('Response HTTP Response Body: ', response.content)

Please note that using:

{
"title" : "h1",
}

Is the same as using:

{
"title" : {
    "selector": "h1",
    "output": "text",
    "type": "item"
   }
}

Below are more details about all those different options.

Output Format output [ text | html | @...](default= text)

By using the output option, you can extract various types of data for a specified selector:

text: the selector's text content (default)
@...: attribute of selector (prefixed by @)
HTML: HTML content of selector.

A variety of output options utilizing the same selector are shown below.

{
"title_text" : {
    "selector": "h1",
    "output": "text"
},
"title_html" : {
    "selector": "h1",
    "output": "html"
},
"title_id" : {
    "selector": "h1",
    "output": "@id"
},
}

The information extracted by the above rules on Codery's page will be:

{
"title_text": "The Codery API",
"title_html": "<h1 id=\"the-Codery-API\"<The <a href=\"https://www.Codery.com/\"<Codery</a< api</h1<",
"title_id": "the-Codery-api"
}

A single item or list type [ item | list ] (default= item)

We'll return the first HTML element that matches the selector by default. The type option should be used if you wish to receive all components that match the selector. The following are examples of type:

  • item return the first element that matches the selector (default)
  • list gives you a list of all the elements that match the selector.

Here's an example of how to get the titles of posts from our webpage:

{
"first_post_title" : {
    "selector": ".post-title",
    "type": "item"
},
"all_post_title" : {
    "selector": ".post-title",
    "type": "list"
    },
}

Clean Text clean [ true | false ] (default= true)

By default, we will clean the material before returning it to you. We will delete trailing spaces and empty characters ('n', 't', etc...) from them. If you don't want to enable this behavior, use the clean: false option instead.
Here's an example of using "clean": true to retrieve post descriptions from our webpage.

{
"first_post_description" : {
    "selector": ".card > div",
    "clean": true #default
    }
}

The information extracted by the above rules on Codery's webpage would be

{
"first_post_description": "Extract and collect data from any website Turns websites into accurate data in only seconds",
}

If you use "clean": false.

{
"first_post_description" : {
    "selector": ".card > div",
    "clean": false
    }
}

You would get this result instead:

{
"first_post_description": "\n                Extract and collect data from any website\n        \n            \n            \n            Turns websites into accurate data in only seconds.\n        read more\n        ",

Extract linked items

In order to develop powerful extractors, you may additionally add extraction rules to the output option.
The rules for extracting general information and all blog post specifics from Codery's homepage are listed below.

{
"title" : "h1",
"subtitle" : "#subtitle",
"articles": {
    "selector": ".card",
    "type": "list",
    "output": {
        "title": ".post-title",
        "link": {
            "selector": ".post-title",
            "output": "@href"
        },
        "description": ".post-description"
        }
     }
}

The information extracted by the above rules on Codery's homepage would be

{
 "title": "The Codery Blog",
 "subtitle": " The Codery API crawls a webpage and gets all 
  structured data from it",
 "articles": [
{
  "title": "  Extract and collect data from any website",
  "link": "https://www.mycodery.com/extract-and-collect-data-from-any-website/",
  "description": "Select the required web elements and generate a comprehensive data structure in the most flexible way possible."
},
...
{
  "title": "The Codery API crawls a webpage and gets all structured data from it",
  "link": "https://www.mycodery.com/the-codery-api-crawls-webpage-and-gets-structured-data-from-it",
  "description": "Select the required web elements and generate a comprehensive data structure in the most flexible way possible."
      },
   ]
}

Popular Use Cases

Our users frequently use the extraction rules listed below.

Extract all links from a page

Quickly extracting all connections from a single page can be important for SEO, lead creation, or simply data collecting.
With just one API call, you may accomplish this with the extract rules listed below:

{
"all_links" : {
    "selector": "a",
    "type": "list",
    "output": "@href"
   }
}

The JSON response will be as follow:

{
"all_links": [
    "https://www.mycodery.com/",
    ...,
    "https://www.mycodery.com//"
   ]
}

Instead, you may use these rules to retrieve both the href and the anchors of links:

{
"all_links" : {
    "selector": "a",
    "type": "list",
    "output": {
        "anchor": "a",
        "href": {
            "selector": "a",
            "output": "@href"
          }
      }
   }
}

The JSON response will be as follow:

{
"all_links":[
  {
     "link":"Documentation API",
     "anchor":"https://www.mycodery.com/documentationapi"
  },
  ...

Extract all text from a page

You can use these rules to get all of a web page's text, and only the text, meaning no HTML tags or attributes:

{
"text": "body"
}

Using those criteria with this Codery landing page, for example, yields the following:

{
"text": "Login Sign Up Pricing FAQ Blog Other Features Screenshots Google search API Data extraction JavaScript scenario No code scraping with Integromat Documentation Tired of getting blocked while scraping the web? Codery API handles headless browsers and rotates proxies for you. Try Codery for Free based on 25+ reviews. Render your web page as if it were a real browser. We manage thousands of headless instances using the latest Chrome version. Focus on extracting the data you need, and not dealing with concurrent headless browsers that will eat up all your RAM and CPU. Latest Chrome version Fast, no matter what! Codery simplified our day-to-day marketing and engineering operations a lot . We no longer have to worry about managing our own fleet of headless browsers, and we no longer have to spend days sourcing the right proxy provider Mike Ritchie CEO @ SeekWell Javascript Rendering We render Javascript with a simple parameter so you can scrape every website, even Single Page Applications using React, AngularJS, Vue.js or any other libraries. Execute custom JS snippet Custom wait for all JS to be executed Codery is helping us scrape many job boards and company websites without having to deal with proxies or chrome browsers. It drastically simplified our data pipeline Russel Taylor CEO @ HelloOutbound Rotating Proxies Thanks to our large proxy pool, you can bypass rate limiting website, lower the chance to get blocked and hide your bots! Large proxy pool Geotargeting Automatic proxy rotation Codery clear documentation, easy-to-use API, and great success rate made it a no-brainer. Dominic Phillips Co-Founder @ CodeSubmit Three specific ways to use Codery How our customers use our API: 1. ..."
}

Extract all email addresses from a page

You may use these rules to get all of the email addresses on a web page:

{
"email_addresses": {
    "selector": "a[href^='mailto']",
    "output": "@href",
    "type": "list"
}

Using those rules with this Codery landing page returns this result:

{
"email_addresses": [
    "mailto:hello@codery.com"
  ]
}