crawler literature website, get the number of chapters, the number of clicks, the number of chapter words, want to write a csv file, but climb out of the content and I set the colum does not match, the number of chapter words appear in front of the number of chapters and clicks. I want it to appear under the colum of wordcount. The
code is as follows:
import requests
import re
import json
import csv
from bs4 import BeautifulSoup as bs
start_url = "http://www.jjwxc.net/onebook.php?novelid=3601"
res = requests.get(start_url)
res.encoding = "gb2312"
soup = bs(res.text, "html.parser")
headers = {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 \
(KHTML, like Gecko) Chrome/67.0.3396.62 Safari/537.36"}
url = "http://s8-static.jjwxc.net/getnovelclick.php?novelid=3601&jsonpcall\
back=novelclick"
web_data = requests.get(url, headers=headers)
web_data.encoding = "gzip"
result = web_data.content.decode()
string = re.findall(r"({.*?})", result)[0]
tmp_dict = json.loads(string)
wordcount = soup.find_all("td", {"itemprop": "wordCount"})
with open("JJWXC Scraping.csv","w") as csvfile:
writer = csv.writer(csvfile)
writer.writerow(["Chapters","Views", "Wordcount"])
for w in wordcount:
writer.writerow(w)
for k in tmp_dict.items():
writer.writerow(k) it is printed and displayed in csv file as follows:
Chapters,Views,Wordcount
2592
819
1720
2862
4862
1988
1559
1069
2570
1812
2441
3549
2841
6222
1485
5002
2330
1795
3620
3969
5120
4943
4892
4818
wordcount appears in front of all articles and clicks.