How to Scroll With Puppeteer

Let's talk about a basic skill you need to know when you are scraping pages. Scrolling, without scrolling, you will only get a fraction of the content you are looking for. Scrolling with Puppeteer can be a simple task or quite complicated depending on the website. In this article, we will walk you through the basics and show you a few advanced options to try out.

Getting Started

Use the command npm install puppeteer. Once you have installed Puppeteer, you are ready to go. You do not need to download Chrome or any extra drivers like you would with Selenium. If you would like to use Firefox instead of Chrome, install Puppeteer with npm install puppeteer-firefox. Please note this package has been deprecated.

Let's start with a basic script to scroll to the bottom of the page.

const puppeteer = require('puppeteer');

(async () => {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();
  await page.goto('https://example.com');

  await page.evaluate(() => window.scrollTo(0, document.body.scrollHeight));

  await browser.close();
})();

The page.evaluate() function tells the script a set of instructions to run. The function position.window.scrollTo() is the location we want to scroll to on the page. In our case, we are going to the bottom of the page using x coordinate 0, and y coordinate document.body.scrollHeight.

window.scrollTo is useful when you need to scroll to a specific location every time. window.scrollTo lets us choose the coordinates of the page by changing the x and y values.

window.scrollTo(250, 1000);

This will move 250 pixels to the right and 1000 pixels down.

On the other hand, if you need to scroll the page and are not looking for a specific location, you can use window.scrollBy. This will scroll the page based on the x and y coordinates relative to your current position.

window.scrollBy(0, 250);

The difference between window.scrollTo and window.scrollBy is that we are not going directly to the pixel at 250, 1000. We are moving down 250 pixels from our starting point. If we were to put window.scrollBy(0, 250); in a loop, the script would continue to scroll down by 250 pixels. If we were to loop window.scrollTo it would go back to the exact location every time.

Helpful tip:

If you are going to be using a loop on your scroll action, be sure to add a counter to limit the number of loops, so it does not run endlessly.

let maxScrolls = 20; // Set a maximum scroll limit
let scrollCount = 0;
while (scrollCount < maxScrolls) { 
  await page.evaluate(() => window.scrollBy(0, window.innerHeight)); 
  await page.waitForTimeout(1000); // Wait for content to load
  scrollCount++; // Increment the scroll counter
}

Advanced Scrolling Methods

Now that you know the basics of scrolling with page.evaluate, window.scrollTo, and window.scrollBy lets's explore a few different options. There are many different types of load methods for websites, but some of the most popular ones are:

• Infinite Scrolling: Content loads as you scroll down. Normally, this page will have a "Load More" button at the bottom of the page.
• Pagination: Content is separated into multiple pages, requiring you to click on a button like “next page.”
• Lazy Loading: Images or other elements only load once they are needed and in your browser's view.

Each website will do things slightly differently, which is why we need a toolbox full of different scrolling methods. Let's take a look at a few more options we can use to scroll with Puppeteer.

Method	Description
window.scrollBy	Scroll by a specific amount.
window.scrollTo	Go to a specific position.
scrollIntoView	Targets a specific element
page.keyboard.press	imitates scrolling with the arrow keys
page.mouse.wheel	scrolls like a mouse would

page.keyboard.press and page.mouse.wheel are great methods for avoiding bot detection and appearing more human-like.

page.keyboard.press works by using the up and down arrow keys on your keyboard to scroll the website smoothly. Using a method like window.scrollTo may cause problems on websites that dynamically load content, as the script is trying to scroll to content that has not loaded yet. Keyboard scrolling offers a smoother way of scrolling, causing fewer layout shifts on the page.

Another option is page.mouse.wheel. As the name implies, it will imitate using your mouse wheel to scroll. With page.mouse.wheel, you have more control over how you scroll by changing the speed and direction. Mouse wheel scrolling is also useful for a few specific use cases where an event will only trigger on mouse wheel movement. For example, zooming in and out of Google Maps.

Now let’s talk about a powerful function called puppeteer scroll to element. This function will make Puppeteer scroll down the page until the element you are targeting is in view. This is particularly helpful when you need to find a button or a specific area to interact with. Using element.scrollIntoView() will solve the problem of clicking a button before the button is in view.

await page.evaluate(() => {
    const element = document.querySelector('#my-element'); // Selects the element
    if (element) {
	  element.scrollIntoView();
	}
});

Websites are constantly updating and changing their element names. Be sure to check back periodically to make sure the element you are looking for has not changed its name.

Infinite Scrolling

A common problem when trying to scrape pages with infinite scrolling is that you only load content as you keep scrolling down, not allowing you to capture all the content. We can solve this problem by scrolling, adding a delay, then checking if we have reached the bottom of the page.

await page.evaluate(() => { window.scrollTo(0, document.body.scrollHeight); });

Then we want to make sure the new content has loaded, so we add in a delay:

await page.waitForTimeout(2000);

After that, we will need to check the new height of the content we just loaded to begin scrolling again:

const newHeight = await page.evaluate(() =>document.body.scrollHeight);

Let's also make sure to add a way to stop the scrolling if no more content has loaded:

if (newHeight === previousHeight) {
	break; // Break from the loop if present
}
previousHeight = newHeight;

Now that you know the basics of the Puppeteer scroll, get out there and put it to good use. If you want to practice, feel free to use https://playground.scrapingbytes.com. We have designed a page for users to practice their scraping techniques. Try using each method of scrolling and see which one works best for you.