Tips and tricks

Collection of handy tips and tricks you can use for crawler setup

Lukáš Křivka avatar Jaroslav Hejlek avatar Václav Růt avatar +5
11 articles in this collection
Written by Lukáš Křivka, Jaroslav Hejlek, and Václav Růt and 5 others

How to pass data from crawler to actor

Post-processing crawler data in an Apify actor
Lukáš Křivka avatar
Written by Lukáš Křivka
Updated over a week ago

Several tips how to bypass website anti-scraping protections

Learn how to scrape data from websites that employ IP rate limiting, browser fingerprinting and other modern bot detection techniques
Jaroslav Hejlek avatar
Written by Jaroslav Hejlek
Updated over a week ago

Request labels and how to pass data to request in Puppeteer

How to handle request labels in Apify actors with Puppeteer
Václav Růt avatar
Written by Václav Růt
Updated over a week ago

How to upload larger payloads to key-value store or dataset

Learn how to upload payloads that are larger than 9 MB into a dataset or key-value store via Apify API.
Juan S. Gaitán V. avatar
Written by Juan S. Gaitán V.
Updated over a week ago

Crawl multiple pages with the same URL and different POST data

Crawlers skip pages that have the same URL but only differ in POST data. Learn how to make the crawler visit all such pages.
Jan Čurn avatar
Written by Jan Čurn
Updated over a week ago

Processing the same page multiple times with different setups

Solving a common problem with crawler features such as customData, enqueuePage, interceptRequest function and interceptRequestData.
Petr Čermák avatar
Written by Petr Čermák
Updated over a week ago

Waiting for dynamic content

Code to wait for dynamic content to load
Jakub Balada avatar
Written by Jakub Balada
Updated over a week ago

Using key-value store from crawlers

Sharing or persisting data for future crawler runs
Petr Čermák avatar
Written by Petr Čermák
Updated over a week ago

Scraping data from websites using schema.org Microdata

JavaScript code to automatically extract data using schema.org tags
Jan Čurn avatar
Written by Jan Čurn
Updated over a week ago

Submitting forms on .ASPX pages

How to handle pages created with ASP.NET - with bonus community crawler template
Jakub Drobník avatar
Written by Jakub Drobník
Updated over a week ago

Debugging your Page Function in browser's console

Test your Page Function's code directly in your browser's console
Jakub Balada avatar
Written by Jakub Balada
Updated over a week ago