I want to write a crawler using flask and start it by clicking the button at the front.

I want to use flask to write a crawler, start it by clicking the button at the front, and then save the crawled content in the database, and the front end to display the data.
is mainly about how to call the crawler. Give me an idea. Thank you

.
Mar.03,2021

1. First, take a look at how your crawler starts
2. Package an api to the page, and call subprocess.call (command, shell=True) in this api to execute the directory to start the crawler
3. Call this api


    after the front button is clicked.
  1. flask listens on a http interface, such as / api/spider/start?token=xxx
  2. Verify the token (mainly security) in the
  3. interface, and then start the crawler logic

you can configure another celery, to receive commands to start the crawler on the flask and be responsible for invoking the crawler.
celery is usually used for more complex operations.


Project structure uses the project structure of the flask application:

---App/
    ---form/
    ---models/
    ---static/
    ---templates/
    ---view/ 
    ---spider/



you can put the spider in the spider folder, design the data table structure in the models file, verify the input url, in form, call the crawler in view to crawl, and use the view function rendering template to display the data.

MySQL Query : SELECT * FROM `codeshelper`.`v9_news` WHERE status=99 AND catid='6' ORDER BY rand() LIMIT 5
MySQL Error : Disk full (/tmp/#sql-temptable-64f5-1b31bdb-2bdc5.MAI); waiting for someone to free some space... (errno: 28 "No space left on device")
MySQL Errno : 1021
Message : Disk full (/tmp/#sql-temptable-64f5-1b31bdb-2bdc5.MAI); waiting for someone to free some space... (errno: 28 "No space left on device")
Need Help?