ask for God"s guidance. I have just come into contact with python crawler. I have some questions. Thank you very much.
I want to crawl some English news headlines and store them in a csv file
my code is as follows
import csv, requests, re
from bs4 import BeautifulSoup
urls = ["https://www.defense.gov/News/Archive/?Page={}".format(str(i)) for i in range(1,10)]
def get_titles(urls,data = None):
html = requests.get(urls).text
soup = BeautifulSoup(html, "html.parser")
articles = []
for article in soup.find_all(class_="info"):
Label = "Archive"
News = article.find(class_="title").get_text()
articles.append([Label,News])
with open(r"1.csv","a", newline="") as f:
writer = csv.writer(f)
writer.writerow(["Label","News"])
for row in articles:
writer.writerow(row)
for titles in urls:
get_titles(titles)
I want to crawl 1-9 pages of news headlines like this, but the final result is this
each addition of a news title repeats the previous title to the csv.
ask for God"s guidance!