question
when developing a crawler with scrapy, grab the package with fiddler and find that scrapy will automatically capitalize the key of the requested header, such as accept-encoding into Accept-Encoding , and accept into Accept . The problem is that I don"t want to convert uppercase for some header due to validation problems.
methods that have been tried
I tried to change key.title () in the normkey method in scrapy/http/headers.py to key.lower ()
class Headers(CaselessDict):
...
def normkey(self, key):
"""Normalize key to bytes"""
return self._tobytes(key.title()) -sharp key.lower()seems to work, but in the end, what you see in fiddler is capitalized and verification fails
in addition, if you use requests to send requests, this will not happen. Packet grabbing is normal
, so you can basically rule out software and other environmental problems
I would like to ask you, in which step does scrapy convert the key of header into the capitalized
of the word?