How to automate a web scraping script with Python?

Question

I completed this web scraping tutorial : https://www.youtube.com/watch?v=XQgXKtPSzUI

(tl;dw: A dude teaches how to scrape an online store with Python and Beautiful Soup to create a database in csv).

My question is, since online stores constantly update their content, how do I set the script to run every, for example, ten minutes in order to update my database?

score 0 · Accepted Answer · answered Mar 27 '17 at 12:19

0

You can make another script that will execute your updating script every 10 minutes:

import os
import time

while True: 
  os.system("my_script.py")
  time.sleep(10*60)

Alternatively you can use CRON on linux for executing your script every 10 mins

answered Mar 27 '17 at 12:19

Václav Struhár

1,692
10
20

Thank you for your reply, you're awesome! On the console it says :'my_script.py' is not recognized as an internal or external command, operable program or batch file. – Elite_Doge Mar 27 '17 at 12:23
Thanks :) Would you mind to mark the answer as solution? @Elite_Doge – Václav Struhár Mar 27 '17 at 12:24
my_script.py is your updating script. Rename it accordingly. (@Elite_Doge) – Václav Struhár Mar 27 '17 at 12:25
4 more minutes before I can accept your comment as an answer, I'll do it as soon as I can, thank you very much. – Elite_Doge Mar 27 '17 at 12:26

How to automate a web scraping script with Python?

1 Answers1