How can I skip a promise if it is not completed

Question

I have an array of 100,000 URLs. I want to loop through the array and fetch the URL contents. I am splitting the array dynamically so each time it will take 20 URLs from array (In order to prevent memory overflow due to size of content fetched from every URL) and form new array with the 20 URLs. The problem I am facing is since I am using Promise.all if any one of the request is in pending state it will pause the further execution. I want to skip the pending promise after certain time and the loop should continue to work

(async () => {
    while(allUrls.length) {
        const splittedUrlsArray = allUrls.splice(0, 20); // splitting large array of urls to small arrays of 20 urls
        const urlContents = await getUrlContents(splittedUrlsArray);
        urlContents.map(async (item) => {
        })
    }
})()

const getUrlContents = (urls) => {
    const promises = urls.map( async (url) => {
        try {
            const res = await axios.get(url);
            return res;
        } catch(err) {
            console.log(err);
        }
    })
    return Promise.all(promises);
}

Could you try using Promise.allSettled? It should be resolved when all promises have been resolved or rejected. — Gustavo Cesário, Mar 21 '22 at 17:40
This is called a "timeout". It's typically applied to the network request, not the promise. — user229044, Mar 21 '22 at 17:50

score -1 · Answer 1 · answered Mar 21 '22 at 17:46

-1

Standard operation for web robots is to use a queue. Since you're using promises, try looking at something like this: https://www.npmjs.com/package/async-await-queue

answered Mar 21 '22 at 17:46

Richard Wheeldon

576
9
17

How can I skip a promise if it is not completed

1 Answers1