How to wait for all downloads to complete with Puppeteer?

Question

I have a small web scraping application that downloads multiple files from a web application where the URLs require visting the page.

It works fine if I keep the browser instance alive in between runs, but I want to close the instance in between runs. When I call browser.close() my downloads are stopped because the chrome instance is closed before the downloads have finished.

Does puppeteer provide a way to check if downloads are still active, and wait for them to complete? I've tried page.waitForNavigation({ waitUntil: "networkidle0" }) and "networkidle2", but those seem to wait indefinitely.

---

• node.js 8.10
• puppeteer 1.10.0

Answer 1

My solution is to use chrome's own chrome://downloads/ page to managing download files. This solution can be very easily to auto restart a failed download using chrome's own feature

This example is 'single thread' currently, because it's only monitoring the first item appear in the download manager page. But you can easily adapt it to 'infinite threads' by iterating through all download items (#frb0~#frbn) in that page, well, take care of your network:)

dmPage = await browser.newPage()
await dmPage.goto('chrome://downloads/')

await your_download_button.click() // start download

await dmPage.bringToFront() // this is necessary
await dmPage.waitForFunction(
    () => {
        // monitoring the state of the first download item
        // if finish than return true; if fail click
        const dm = document.querySelector('downloads-manager').shadowRoot
        const firstItem = dm.querySelector('#frb0')
        if (firstItem) {
            const thatArea = firstItem.shadowRoot.querySelector('.controls')
            const atag = thatArea.querySelector('a')
            if (atag && atag.textContent === '在文件夹中显示') { // may be 'show in file explorer...'? you can try some ids, classess and do a better job than me lol
                return true
            }
            const btn = thatArea.querySelector('cr-button')
            if (btn && btn.textContent === '重试') { // may be 'try again'
                btn.click()
            }
        }
    },
    { polling: 'raf', timeout: 0 }, // polling? yes. there is a 'polling: "mutation"' which kind of async
)
console.log('finish')

Answer 2

An alternative if you have the file name or a suggestion for other ways to check.

async function waitFile (filename) {

    return new Promise(async (resolve, reject) => {
        if (!fs.existsSync(filename)) {
            await delay(3000);    
            await waitFile(filename);
            resolve();
        }else{
          resolve();
        }

    })   
}

function delay(time) {
    return new Promise(function(resolve) { 
        setTimeout(resolve, time)
    });
}

Implementation:

var filename = `${yyyy}${mm}_TAC.csv`;
var pathWithFilename = `${config.path}\\${filename}`;
await waitFile(pathWithFilename);

Answer 3

Using puppeteer and chrome I have one more solution which might help you.

If you are downloading the file from chrome it will always have ".crdownload" extension. And when file is completely downloaded that extension will vanish.

So, I am using recurring function and maximum number of times it can iterate, If it doesn't download the file in that time.. I am deleting it. And I am constantly checking a folder for that extention.

async checkFileDownloaded(path, timer) {
    return new Promise(async (resolve, reject) => {
        let noOfFile;
        try {
            noOfFile = await fs.readdirSync(path);
        } catch (err) {
            return resolve("null");
        }
        for (let i in noOfFile) {
            if (noOfFile[i].includes('.crdownload')) {
                await this.delay(20000);
                if (timer == 0) {
                    fs.unlink(path + '/' + noOfFile[i], (err) => {
                    });
                    return resolve("Success");
                } else {
                    timer = timer - 1;
                    await this.checkFileDownloaded(path, timer);
                }
            }
        }
        return resolve("Success");
    });
}

Answer 4

You need check request response.

await page.on('response', (response)=>{ console.log(response, response._url)}

You should check what is coming from response then find status, it comes with status 200

Answer 5

Tried doing an await page.waitFor(50000); with a time as long as the download should take.

Or look at watching for file changes on complete file transfer

Answer 6

Created simple await function that will check for file rapidly or timeout in 10 seconds

import fs from "fs";

awaitFileDownloaded: async (filePath) => {
    let timeout = 10000
    const delay = 200

    return new Promise(async (resolve, reject) => {
        while (timeout > 0) {
            if (fs.existsSync(filePath)) {
                resolve(true);
                return
            } else {
                await HelperUI.delay(delay)
                timeout -= delay
            }
        }
        reject("awaitFileDownloaded timed out")
    });
},

Answer 7

Here is another function, its just wait for the pause button to disappear:

async function waitForDownload(browser: Browser) {
  const dmPage = await browser.newPage();
  await dmPage.goto("chrome://downloads/");

  await dmPage.bringToFront();
  await dmPage.waitForFunction(() => {
    try {
      const donePath = document.querySelector("downloads-manager")!.shadowRoot!
        .querySelector(
          "#frb0",
        )!.shadowRoot!.querySelector("#pauseOrResume")!;
      if ((donePath as HTMLButtonElement).innerText != "Pause") {
        return true;
      }
    } catch {
      //
    }
  }, { timeout: 0 });
  console.log("Download finished");
}

Answer 8

You can use node-watch to report the updates to the target directory. When the file upload is complete you will receive an update event with the name of the new file that has been downloaded.

Run npm to install node-watch:

npm install node-watch

Sample code:

const puppeteer = require('puppeteer');
const watch = require('node-watch');
const path = require('path');

// Add code to initiate the download ...
const watchDir = '/Users/home/Downloads'
const filepath = path.join(watchDir, "download_file");
(async() => {
    watch(watchDir, function(event, name) {
    if (event == "update") {
        if (name === filepath)) {
            browser.close(); // use case specific
            process.exit();  // use case specific
        }
    }
})