Всем доброго дня. Нужно спарсить данные с сайта https://www.pinnacle.com/en/, доступ с территории РФ ограничен -> пытаюсь использовать прокси в Scrapy. Пробовал и в одиночные запросы вписывать прокси и устанавливать вот это https://github.com/aivarsk/scrapy-proxies
Все время ода и та же ошибка:
2020-07-16 16:50:44 [scrapy.core.scraper] ERROR: Error downloading <GET https://www.pinnacle.com/en/>
Traceback (most recent call last):
File "/home/n30n/Run/DFS_math/venv/lib/python3.8/site-packages/scrapy/core/downloader/middleware.py", line 44, in process_request
return (yield download_func(request=request, spider=spider))
twisted.web._newclient.ResponseNeverReceived:
[<twisted.python.failure.Failure twisted.internet.error.ConnectionLost: Connection to the other side was lost in a non-clean fashion: Connection lost.>]
settings.py:
BOT_NAME = 'pinnacle_parser'
SPIDER_MODULES = ['pinnacle_parser.spiders']
NEWSPIDER_MODULE = 'pinnacle_parser.spiders'
# Crawl responsibly by identifying yourself (and your website) on the user-agent
USER_AGENT = 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.97 Safari/537.36'
# Obey robots.txt rules
ROBOTSTXT_OBEY = False
# Retry many times since proxies often fail
RETRY_TIMES = 10
# Retry on most error codes since proxies fail for different reasons
RETRY_HTTP_CODES = [500, 503, 504, 400, 403, 404, 408]
DOWNLOADER_MIDDLEWARES = {
'scrapy.downloadermiddlewares.retry.RetryMiddleware': 90,
'scrapy_proxies.RandomProxy': 100,
'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware': 110,
'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware': 105,
# 'scrapy_useragents.downloadermiddlewares.useragents.UserAgentsMiddleware': 500,
}
# Proxy list containing entries like
# http://host1:port
PROXY_LIST = '/home//Run/pinnacle_parser/proxy'
# Proxy mode
# 0 = Every requests have different proxy
# 1 = Take only one proxy from the list and assign it to every requests
# 2 = Put a custom proxy to use in the settings
PROXY_MODE = 2
# # If proxy mode is 2 uncomment this sentence :
CUSTOM_PROXY = "http://217.172.190.254:5836"
С чем связана данная ошибка?Каким образом сайт может дополнительно препятствовать скраппингу?