#4844 improve the log configuration with using RotatingFileHandler.

Open PyCatFish
Coverage Reach
utils/python.py utils/misc.py utils/conf.py utils/log.py utils/iterators.py utils/defer.py utils/datatypes.py utils/deprecate.py utils/test.py utils/console.py utils/project.py utils/reactor.py utils/curl.py utils/url.py utils/signal.py utils/request.py utils/response.py utils/ssl.py utils/testproc.py utils/reqser.py utils/spider.py utils/trackref.py utils/benchserver.py utils/gz.py utils/testsite.py utils/serialize.py utils/decorators.py utils/display.py utils/sitemap.py utils/ftp.py utils/engine.py utils/boto.py utils/http.py utils/template.py utils/ossignal.py utils/versions.py utils/httpobj.py utils/job.py utils/py36.py utils/markup.py utils/multipart.py core/downloader/handlers/http11.py core/downloader/handlers/ftp.py core/downloader/handlers/__init__.py core/downloader/handlers/s3.py core/downloader/handlers/http10.py core/downloader/handlers/datauri.py core/downloader/handlers/file.py core/downloader/handlers/http.py core/downloader/__init__.py core/downloader/webclient.py core/downloader/middleware.py core/downloader/tls.py core/downloader/contextfactory.py core/engine.py core/scraper.py core/scheduler.py core/spidermw.py extensions/feedexport.py extensions/httpcache.py extensions/memusage.py extensions/telnet.py extensions/throttle.py extensions/closespider.py extensions/debug.py extensions/logstats.py extensions/corestats.py extensions/spiderstate.py extensions/statsmailer.py extensions/memdebug.py commands/parse.py commands/genspider.py commands/startproject.py commands/__init__.py commands/check.py commands/runspider.py commands/fetch.py commands/shell.py commands/bench.py commands/settings.py commands/edit.py commands/crawl.py commands/version.py commands/view.py commands/list.py http/request/form.py http/request/__init__.py http/request/json_request.py http/request/rpc.py http/response/text.py http/response/__init__.py http/response/html.py http/response/xml.py http/cookies.py http/headers.py http/__init__.py http/common.py downloadermiddlewares/httpcache.py downloadermiddlewares/cookies.py downloadermiddlewares/robotstxt.py downloadermiddlewares/redirect.py downloadermiddlewares/decompression.py downloadermiddlewares/httpcompression.py downloadermiddlewares/httpproxy.py downloadermiddlewares/retry.py downloadermiddlewares/ajaxcrawl.py downloadermiddlewares/stats.py downloadermiddlewares/httpauth.py downloadermiddlewares/useragent.py downloadermiddlewares/downloadtimeout.py downloadermiddlewares/defaultheaders.py pipelines/files.py pipelines/media.py pipelines/images.py pipelines/__init__.py settings/__init__.py settings/default_settings.py spidermiddlewares/referer.py spidermiddlewares/offsite.py spidermiddlewares/depth.py spidermiddlewares/httperror.py spidermiddlewares/urllength.py spiders/crawl.py spiders/__init__.py spiders/feed.py spiders/sitemap.py spiders/init.py exporters.py contracts/__init__.py contracts/default.py crawler.py linkextractors/lxmlhtml.py linkextractors/__init__.py shell.py cmdline.py pqueues.py robotstxt.py mail.py item.py responsetypes.py resolver.py squeues.py dupefilters.py middleware.py statscollectors.py spiderloader.py logformatter.py selector/unified.py selector/__init__.py exceptions.py loader/__init__.py loader/processors.py loader/common.py signals.py signalmanager.py __init__.py link.py extension.py interfaces.py __main__.py

No flags found

Use flags to group coverage reports by test type, project and/or folders.
Then setup custom commit statuses and notifications for each flag.

e.g., #unittest #integration

#production #enterprise

#frontend #backend

Learn more about Codecov Flags here.

Showing 1 of 2 files from the diff.
Other files ignored by Codecov

@@ -2,7 +2,7 @@
Loading
2 2
import sys
3 3
import warnings
4 4
from logging.config import dictConfig
5 -
5 +
from logging.handlers import RotatingFileHandler
6 6
from twisted.python import log as twisted_log
7 7
from twisted.python.failure import Failure
8 8
@@ -122,7 +122,14 @@
Loading
122 122
    filename = settings.get('LOG_FILE')
123 123
    if filename:
124 124
        encoding = settings.get('LOG_ENCODING')
125 -
        handler = logging.FileHandler(filename, encoding=encoding)
125 +
        if settings.get("LOG_ROTATING") is True:
126 +
            max_bytes = settings.get('LOG_MAX_BYTES', 0)
127 +
            log_backup_count = settings.get('LOG_BACKUP_COUNT', 0)
128 +
            handler = RotatingFileHandler(filename, maxBytes=max_bytes,
129 +
                                          backupCount=log_backup_count,
130 +
                                          encoding=encoding)
131 +
        else:
132 +
            handler = logging.FileHandler(filename, encoding=encoding)
126 133
    elif settings.getbool('LOG_ENABLED'):
127 134
        handler = logging.StreamHandler()
128 135
    else:

Everything is accounted for!

No changes detected that need to be reviewed.
What changes does Codecov check for?
Lines, not adjusted in diff, that have changed coverage data.
Files that introduced coverage data that had none before.
Files that have missing coverage data that once were tracked.
Files Coverage
scrapy -0.03% 87.82%
Project Totals (160 files) 87.82%
Loading