scrapy / scrapy
Showing 1 of 1 files from the diff.

@@ -1,7 +1,19 @@
Loading
1 1
from scrapy.exceptions import NotConfigured
2 -
from scrapy.utils.request import request_httprepr
3 -
from scrapy.utils.response import response_httprepr
4 2
from scrapy.utils.python import global_object_name
3 +
from scrapy.utils.request import request_httprepr
4 +
5 +
6 +
def get_header_size(headers):
7 +
    if headers is None:
8 +
        return None
9 +
    size = 0
10 +
    for key, value in headers.items():
11 +
        if isinstance(value, bytes):
12 +
            size += len(b": ") + len(key) + len(value)
13 +
        elif isinstance(value, (list, tuple)):
14 +
            for v in value:
15 +
                size += len(b": ") + len(key) + len(v)
16 +
    return size + len(b'\r\n') * (len(headers.keys()) - 1)
5 17
6 18
7 19
class DownloaderStats:
@@ -24,7 +36,7 @@
Loading
24 36
    def process_response(self, request, response, spider):
25 37
        self.stats.inc_value('downloader/response_count', spider=spider)
26 38
        self.stats.inc_value(f'downloader/response_status_count/{response.status}', spider=spider)
27 -
        reslen = len(response_httprepr(response))
39 +
        reslen = len(response.body) + get_header_size(response.headers)
28 40
        self.stats.inc_value('downloader/response_bytes', reslen, spider=spider)
29 41
        return response
30 42
Files Coverage
scrapy 88.15%
Project Totals (162 files) 88.15%
1
comment:
2
  layout: "header, diff, tree"
3

4
coverage:
5
  status:
6
    project: false
Sunburst
The inner-most circle is the entire project, moving away from the center are folders then, finally, a single file. The size and color of each slice is representing the number of statements and the coverage, respectively.
Icicle
The top section represents the entire project. Proceeding with folders and finally individual files. The size and color of each slice is representing the number of statements and the coverage, respectively.
Grid
Each block represents a single file in the project. The size and color of each block is represented by the number of statements and the coverage, respectively.
Loading