you can also say this: change the IP, in the option and change the UA without shutting it down?
does not use crawler architecture, but directly uses python+selenium to write fixed-behavior crawlers. Now you need to integrate multithreading and IP proxy pool, and you don"t want new selenium to open a new browser.
you can dynamically change IP by js injection
the following is the Python code of Firefox:
def change_ip (driver):
ip = get_ip()-sharpip"127.0.0.1:80"
driver.get("about:config");
-sharpjs
setupScript = '''var prefs = Components.classes["@mozilla.org/preferences-service;1"].getService(Components.interfaces.nsIPrefBranch);
prefs.setIntPref("network.proxy.type", 1);
prefs.setCharPref("network.proxy.http", "%s");
prefs.setIntPref("network.proxy.http_port", "%s");
prefs.setCharPref("network.proxy.ssl", "%s");
prefs.setIntPref("network.proxy.ssl_port", "%s");
prefs.setCharPref("network.proxy.ftp", "$%s");
prefs.setIntPref("network.proxy.ftp_port", "%s");
''' % (ip.split(':')[0], ip.split(':')[1], ip.split(':')[0], ip.split(':')[1], ip.split(':')[0], ip.split(':')[1])
-sharpjs
driver.execute_script(setupScript);how to use ajax to interface with the interface provided by the backend for python. Anyone who knows about it can help you. ...
how can implement the following case in python? i just cant figure out how to solve this problem CASE-1 [Enter Correct user Name and Password] CASE-2 [Both Email and Password Fields are blank.] CASE-3 [Email field is filled and Password field ...
I am writing a web page panel of selenium crawler, using flask to control some behavior of selenium. The target web page needs to scan the QR code to log in, and now the card is at this stage after scanning the QR code, because I can t figure out how t...
my project code is mainly to use phantomJSDriver, to use Selenium-related packages use the mvn install command to package. However, the typed jar package cannot be run by clicking on it. Using the command to run the problem is as follows: the foll...
firefox can capture the complete picture of a single element normally, and chrome will WebDriverException: Message: unknown command: session 93696d1e0412953043ebd5024870aada element 0.24383677148808935-1 screenshot how to solve this problem? ...
this is my code I don t know how to open file or it may not be generated at all busy busy from selenium import webdriver import unittest, time import HTMLTestRunner class MyTest(unittest.TestCase): def setUp(self): self.driver = webdriver.Chro...
want to achieve selenium login, but how can not navigate to the account password input box, tried a lot of methods did not work. even this iframe is dynamic ...
alert = self.browser.switch_to_alert (); sleep (2); alert.accept (); Click OK using this get pop-up box, sometimes it will report that UnexpectedAlertPresentException, sometimes does not have it, and I don t know why. ...
do not need to click to allow, automatically run ...
Test a login and logout. I already instantiated a webdriver, when I logged in. I want to reference this driver. in the logout case. How can I refer to this driver? Why should I refer to the driver, in the login case instead of instantiating another dr...
after I log in to the website through selenium, I want to start automatically clicking some buttons on the web page. Through xpath positioning, I can t find . The code is as follows (account password is not important, you need to log in to enter the...
develop an automated test case of the system in a page-oriented development mode; the existing login login_case, will develop a logout_case. because the prerequisite for logout_case is that the user has logged in, you must first call login_case in the...
https: stooq.com t ?i=521&v=0 I try to crawl some of the data with the python crawler, but sometimes the browser shows that it has been loaded, but the display is still blank. then I need to refresh multiple times to recover. What s even weirder is...
question: after using selenium to request url ( https: www.toutiao.com c use.), the page will continue to initiate ajax requests to dynamically load data the code is as follows: option = webdriver.ChromeOptions() option.add_argument("h...
< H1 > coding=utf-8 < H1 > import requests from selenium import webdriver import time class JzSpider: def __init__(self,): self.start_url = "http: radar.itjuzi.com company" self.headers={"User-Agent":"Mozilla 5...
In the selenium+protractor automated test, when the attachment is uploaded, the pop-up box for the selected file uploaded by the attachment is opened, but there is no way to close the system pop-up box. How to solve this problem? var path = require( p...
Click to save driver.findElement (By.xpath ( " * [@ id= " btn-submit "] ")). Click (); ) and then there is a pop-up box. You need to click to confirm I thought this was alter . Alert javascriptConfirm = driver.switchTo (). Alert (); ja...
I m trying to cram data on https: finance.yahoo.com . I found that if you type a few letters in the search bar, there will be a result that suggests popping up. Similar to Google and Baidu. I want to get down on my stomach with this suggestion. I f...
ask for help! +v[ name ]+ < option> ; << ...
Why do these url jump back to the selenium of middleware via selenium jump to the url request crawled down the page in scrapy, instead of calling back to the following def def parse(self, response): contents = response.xpath( *[@id="...