I want to use flask to write a crawler, start it by clicking the button at the front, and then save the crawled content in the database, and the front end to display the data.
is mainly about how to call the crawler. Give me an idea. Thank you
I want to use flask to write a crawler, start it by clicking the button at the front, and then save the crawled content in the database, and the front end to display the data.
is mainly about how to call the crawler. Give me an idea. Thank you
1. First, take a look at how your crawler starts
2. Package an api to the page, and call subprocess.call (command, shell=True) in this api to execute the directory to start the crawler
3. Call this api
/ api/spider/start?token=xxx
you can configure another celery, to receive commands to start the crawler on the flask and be responsible for invoking the crawler.
celery is usually used for more complex operations.
Project structure uses the project structure of the flask application:
---App/
---form/
---models/
---static/
---templates/
---view/
---spider/
you can put the spider in the spider folder, design the data table structure in the models file, verify the input url, in form, call the crawler in view to crawl, and use the view function rendering template to display the data.
Previous: Compatibility of zoom on Firefox