scrapy / scrapy

@@ -144,6 +144,7 @@
Loading
144 144
        self.max_retry_times = settings.getint('RETRY_TIMES')
145 145
        self.retry_http_codes = set(int(x) for x in settings.getlist('RETRY_HTTP_CODES'))
146 146
        self.priority_adjust = settings.getint('RETRY_PRIORITY_ADJUST')
147 +
        self.logger_give_up_level = settings.getint('RETRY_LOG_GIVE_UP_LEVEL')
147 148
148 149
    @classmethod
149 150
    def from_crawler(cls, crawler):
@@ -167,10 +168,12 @@
Loading
167 168
    def _retry(self, request, reason, spider):
168 169
        max_retry_times = request.meta.get('max_retry_times', self.max_retry_times)
169 170
        priority_adjust = request.meta.get('priority_adjust', self.priority_adjust)
171 +
        logger_give_up_level = request.meta.get('logger_give_up_level', self.logger_give_up_level)
170 172
        return get_retry_request(
171 173
            request,
172 174
            reason=reason,
173 175
            spider=spider,
174 176
            max_retry_times=max_retry_times,
175 177
            priority_adjust=priority_adjust,
178 +
            logger_give_up_level=logger_give_up_level,
176 179
        )

@@ -254,6 +254,7 @@
Loading
254 254
RETRY_TIMES = 2  # initial response + 2 retries = 3 requests
255 255
RETRY_HTTP_CODES = [500, 502, 503, 504, 522, 524, 408, 429]
256 256
RETRY_PRIORITY_ADJUST = -1
257 +
RETRY_LOG_GIVE_UP_LEVEL = 40
257 258
258 259
ROBOTSTXT_OBEY = False
259 260
ROBOTSTXT_PARSER = 'scrapy.robotstxt.ProtegoRobotParser'
Files Coverage
scrapy 88.65%
Project Totals (162 files) 88.65%
1
comment:
2
  layout: "header, diff, tree"
3

4
coverage:
5
  status:
6
    project: false
Sunburst
The inner-most circle is the entire project, moving away from the center are folders then, finally, a single file. The size and color of each slice is representing the number of statements and the coverage, respectively.
Icicle
The top section represents the entire project. Proceeding with folders and finally individual files. The size and color of each slice is representing the number of statements and the coverage, respectively.
Grid
Each block represents a single file in the project. The size and color of each block is represented by the number of statements and the coverage, respectively.
Loading