Using proxy through http fails (https works) enhancement good first issue

#4505 opened Apr 16, 2020 by teodoroanca

11

Use f-strings cleanup good first issue

#4307 opened Feb 6, 2020 by Gallaecio

19

SCRAPY_CHECK is not set while running contract contracts enhancement good first issue

#4222 opened Dec 8, 2019 by Lightjohn

11

Missing the Scrapy entry in Wikipedia in many languages docs enhancement good first issue

#4233 opened Dec 15, 2019 by noviluni

21

Cover arbitrary filtering in the Scrapy logging documentation docs enhancement good first issue

#4216 opened Dec 5, 2019 by Gallaecio

7

SitemapSpider throws lxml.etree.XMLSyntaxError when trying to scrape a blank sitemap page enhancement good first issue

#4211 opened Dec 4, 2019 by Endi1

Scrapy with Puppeteer and/or Playwright? docs enhancement

#4484 opened Apr 11, 2020 by osmenia

3

Make documentation about contributing beginner friendly docs enhancement

#4451 opened Mar 24, 2020 by joybh98

2

defer.inlineCallbacks in spider ? docs enhancement

#4263 opened Jan 1, 2020 by HelloEdit

2

process_spider_output called twice when exception occurs docs enhancement

#4266 opened Jan 3, 2020 by StasDeep

3

DOWNLOAD_MAXSIZE logger level shouldn't be Error enhancement good first issue logging

#3874 opened Jul 13, 2019 by mohmad-null

8

Settings, multiple concurrent spiders, and middlewares docs enhancement

#4253 opened Dec 23, 2019 by mredaelli

4

Persistence is not fully persistent when using JOBDIR setting docs enhancement

#4106 opened Oct 25, 2019 by JSPromisel

1

Allow some redirects and deny some others docs enhancement

#4009 opened Sep 12, 2019 by casertap

2

Establish a policy on the use of Sphinx directives for deprecations and version changes discuss docs enhancement

#3971 opened Aug 21, 2019 by Gallaecio

TWISTED_REACTOR setting not honored from Spider.custom_settings bug

#4485 opened Apr 12, 2020 by elacuesta

2

Crawler logs are cut by day docs enhancement good first issue help wanted

#3628 opened Feb 15, 2019 by Lingshuirong

15

Bind request to response only if not bound already discuss

#4529 opened May 1, 2020 by elacuesta

3

is_generator_with_return_value raises IndentationError with a flush left doc string bug

#4477 opened Apr 8, 2020 by mdaniel

3

scrapy command line tools fail on custom schemes enhancement

#4530 opened May 1, 2020 by sylvinus

[DOC] Making it a documented ("officially supported") feature to return deferred from request callbacks. docs enhancement

#3850 opened Jul 2, 2019 by starrify

2

Small improvement to the docs for setting ITEM_PIPELINES discuss docs enhancement

#3775 opened May 14, 2019 by intotecho

1

Do you think about supporting Scrapy cheatsheet docs enhancement

#3765 opened May 5, 2019 by MarcSteven

9

Exceptions in middleware don't return exit code 1 in `scrapy crawl` & `scrapy check` contracts

#4292 opened Jan 28, 2020 by dpfeif

2

Add a scrapy wrapper for `twisted.internet.reactor` imports enhancement

#4435 opened Mar 17, 2020 by nyov

4

Sitemap spider does not resume though JOBDIR is set bug

#4479 opened Apr 9, 2020 by Arregator

2

ImportError: No module named _util docs enhancement

#3803 opened Jun 1, 2019 by merrisco

1

Support for Elastic APM for sending logs to elasticsearch and kibana stack enhancement

#4441 opened Mar 18, 2020 by codehia

1

Should we delete the deprecated code in Scrapy 2.0? cleanup good first issue

#4356 opened Feb 20, 2020 by noviluni

6

Deprecate environment variables being loaded as settings cleanup

#4379 opened Feb 27, 2020 by Gallaecio

6

Can not get certificate information when http body is empty. bug

#4466 opened Mar 31, 2020 by imfht

8

Wrong type(response) for binary responses bug gsoc-candidate

#4240 opened Dec 17, 2019 by ejulio

20

raise CloseSpider("Error Message") is not saving error message enhancement good first issue

#3463 opened Oct 15, 2018 by parlays

6

scrapy parse emits ANSI color sequences in the Windows terminal Windows bug

#4393 opened Mar 2, 2020 by A-hoy

7

FilesPipeline.file_path always getting response=None enhancement

#4457 opened Mar 27, 2020 by scratchmex

5

Apply priority in Rule docs enhancement

#3613 opened Feb 2, 2019 by phongtnit

4

Make scrapy interact with java script directly enhancement

#4432 opened Mar 15, 2020 by EbotFabien

1

General Message Queues as Storage for Requests enhancement gsoc-candidate

#4326 opened Feb 10, 2020 by whalebot-helmsman

36

Scrapy fails to crawl emoji domains, raises idna.core.InvalidCodepoint bug upstream issue

#4330 opened Feb 13, 2020 by john-parton

7

Provide built-in SFTP support, discourage FTP discuss enhancement

#4180 opened Nov 20, 2019 by Gallaecio

1

request.meta['download_slot'] is not documented docs enhancement

#3585 opened Jan 16, 2019 by kmike

3

Do not recommend Google Cache in the documentation docs enhancement

#3582 opened Jan 14, 2019 by Gallaecio

1

Adjust throttling for 429 response codes enhancement

#4424 opened Mar 13, 2020 by ejulio

8

Merge Slot classes back? cleanup discuss

#4296 opened Jan 31, 2020 by wRAR

5

Implement static checks for Scrapy projects enhancement gsoc-candidate

#4421 opened Mar 12, 2020 by Gallaecio

11

Uses ipdb instead pdb when package installed enhancement

#4284 opened Jan 23, 2020 by jeanmask

2

Create scrapy command to get the equivalent Request from a cURL command discuss enhancement

#3991 opened Aug 30, 2019 by noviluni

12

httpproxy remove creds of proxy after processing request enhancement

#4029 opened Sep 20, 2019 by NewUserHa

5

Scrapy pause/resume Bug ?

#4509 opened Apr 21, 2020 by ghost

4

Document Scheduler API docs enhancement

#3537 opened Dec 20, 2018 by elacuesta

Warning messages to add in the CSV exporter when there's any field dropped enhancement help wanted

#4002 opened Sep 5, 2019 by starrify

11

Importance of Requests enhancement

#4156 opened Nov 13, 2019 by oscarrobertson

3

Expose _find_method and _get_method from scrapy.utils.reqser enhancement

#4125 opened Nov 4, 2019 by Gallaecio

3

Scrapy not honoring the Retry-After header when given a 429 enhancement

#3849 opened Jul 2, 2019 by ddebernardy

2

Adding Type Hints to Scrapy and its Modules enhancement

#4041 opened Sep 27, 2019 by royahsan

13

Incorrect deprecation warning on S3FeedStorage bug

#3938 opened Aug 5, 2019 by hermit-crab

3

On distributing some blocking tasks to threads enhancement

#3751 opened Apr 18, 2019 by starrify

3

DOWNLOAD_FAIL_ON_DATALOSS logged for all requests bug

#4217 opened Dec 5, 2019 by Gallaecio

2

[bug] pillow will always recode images in imagepipieline enhancement good first issue

#3055 opened Dec 31, 2017 by NewUserHa

11

strip trailing backslashes from domain when redirecting to avoid invalid domains enhancement

#3854 opened Jul 4, 2019 by ddebernardy

4

Spider finshed normally despite an error in start_requests discuss enhancement

#4182 opened Nov 20, 2019 by lopuhin

8

Allow to pass objects in Settings? enhancement

#3870 opened Jul 12, 2019 by nyov

4

Scrapy download middelware should pause on Connection loss and Resume when internet connection is available. enhancement

#4043 opened Sep 28, 2019 by royahsan

5

Make it possible to update settings in `__init__` or `from_crawler` enhancement

#3663 opened Mar 7, 2019 by ejulio

7

HttpErrorMiddleware not honoring handle_httpstatus_all meta as documented bug

#3851 opened Jul 3, 2019 by ddebernardy

1

scrapy should have own base exception class discuss enhancement

#3487 opened Nov 12, 2018 by baby5

4

#3798 LinkExtractor with Unique = False doesn't extract fully identical Links bug

#3798 opened May 27, 2019 by Ksianka

1

exit code propagation in case of CLOSESPIDER_ERRORCOUNT extension? enhancement

#3653 opened Mar 5, 2019 by HyunTruth

3

Progress bar for large downloads discuss enhancement

#3793 opened May 24, 2019 by nemec

5

Return more detailed error message in scrapy check contracts enhancement

#3716 opened Apr 2, 2019 by Gravellent

4

Exporters should support nested scrapy.Item objects enhancement help wanted

#3888 opened Jul 19, 2019 by mjpieters

2

Better handling (and docs) of multiple spiders docs enhancement

#3191 opened Mar 28, 2018 by mohmad-null

4

Using request callback in pipeline does not seem to work docs enhancement

#3185 opened Mar 25, 2018 by fabrepe

4

Using twisted threads inside parse function docs enhancement

#3236 opened Apr 24, 2018 by richa-saraf

3

Catching VerificationError warning in scrapy.core.downloader.tls enhancement

#3573 opened Jan 10, 2019 by dkajtoch

Translating the docs enhancement

#3511 opened Nov 28, 2018 by Maransatto

21

Request priority not guaranteed with the default scheduler configurations due to different queues (memory/disk) used. bug

#3666 opened Mar 8, 2019 by starrify

1

Extend ItemLoader processors enhancement item loaders

#3576 opened Jan 11, 2019 by Matthijsy

6

_tunnelReadyDeferred need canceller to close connection like twisted do discuss

#3614 opened Feb 3, 2019 by baby5

Type hints and Response.selector enhancement

#3618 opened Feb 7, 2019 by mohmad-null

8

Genspider prepend `http://` without checking it in the `domain` enhancement

#3553 opened Dec 28, 2018 by ThunderMind2019

3

bindaddress of meta never work right. enhancement

#3565 opened Jan 4, 2019 by NewUserHa

10

scrapy check command exports output contracts enhancement

#3385 opened Aug 13, 2018 by StasDeep

SSLv3 method is no longer available docs enhancement

#2951 opened Oct 3, 2017 by kmike

3

should scrapy catch the exception that timeout raised by itself to "ignore response" or something instead Traceback (most recent call last):? enhancement logging

#3510 opened Nov 28, 2018 by NewUserHa

13

[offsite middleware] allow to dynamically add new entries to allowed domains discuss enhancement

#3257 opened May 11, 2018 by pawelmhm

sslv3 alert handshake failure docs enhancement

#2944 opened Sep 26, 2017 by Dainius-P

3

OffsiteMiddleware silently filtering (invalid) URLs bug

#3246 opened May 4, 2018 by jesuslosada

1

Set DNS Servers to use enhancement

#3396 opened Aug 19, 2018 by mohmad-null

1

[question] Why adding `dont_filter=True` for requests made from `start_urls` enhancement

#3276 opened May 28, 2018 by starrify

7

url_is_from_spider fails if allowed_domains defined with @property decorator enhancement

#3119 opened Feb 12, 2018 by pawelmhm

1

SSL website. `twisted.internet.error.ConnectionLost` docs enhancement

#2916 opened Sep 7, 2017 by unk2k

12

Memusage extension AssertionError bug

#3343 opened Jul 19, 2018 by nadzimo

1

How to create JOBDIR setting from Spider __init__ method ? or dynamically enhancement

#3416 opened Sep 7, 2018 by lxmn

1

Error when install scrapy in window by using pip install scrapy docs enhancement install

#2881 opened Aug 11, 2017 by lhkthomas

31

Allow Scrapy to send e-mails using Amazon SES enhancement

#3345 opened Jul 19, 2018 by rennerocha

1

Scrapy fails to fetch request with invalid hostname bug upstream issue

#3321 opened Jul 6, 2018 by pawelmhm

9

Import Request in the Template file docs enhancement help wanted

#2733 opened May 9, 2017 by zhongdai

3

Cannot retry requests in `scrapy parse` bug

#3095 opened Jan 26, 2018 by jdemaeyer

4

"Easy" Content type detection enhancement

#3205 opened Apr 8, 2018 by mohmad-null

10

Feature: Report Queue Length enhancement logging

#3174 opened Mar 15, 2018 by mohmad-null

6

Scrapy under Python 3 is slower than under Python 2 performance

#3134 opened Feb 21, 2018 by lopuhin

22

scrapy shell - bug in processing escaped URLs docs enhancement

#2624 opened Mar 6, 2017 by harshasrinivas

6

Deep Copy meta on copy() or replace() Response methods enhancement

#3194 opened Mar 31, 2018 by mohmad-null

3

JsonItemExporter puts lone comma in the output if encoder fails bug

#3090 opened Jan 25, 2018 by tlinhart

3

KeyError with the initialization of an Item Field defined with None using ItemLoader discuss item loaders

#3272 opened May 23, 2018 by NadirRisc

7

Deprecation Warning of Twisted cleanup

#3288 opened Jun 11, 2018 by Aqua-Dream

9

Growing download latency of a CPU-heavy spider performance

#3197 opened Apr 2, 2018 by lopuhin

16

Download Middleware HTTPCache pickle error enhancement

#3295 opened Jun 14, 2018 by yvmarques

3

name collision of spider with existing module docs enhancement patch available

#2680 opened Mar 23, 2017 by swoertz

DOC Add (short) explanation of return vs. yield in callbacks docs enhancement

#2546 opened Feb 8, 2017 by redapple

2

Entire HTML is not checked for finding base tag bug

#3017 opened Nov 23, 2017 by Vineeth-Mohan

14

Unable to retreive http return code from ImagesPipeline (or MediaPipeline) in scrapy docs enhancement

#2504 opened Jan 19, 2017 by manisoftwartist

4

improve documents: global variables and noblock ways enhancement

#3250 opened May 8, 2018 by NewUserHa

13

Some scrapy request don't show up in Wireshark bug

#3209 opened Apr 10, 2018 by maximevdk

3

Jupyter run error ReactorNotRestartable docs enhancement

#2594 opened Feb 23, 2017 by liveresume

8

exceptions.AttributeError: '_SIGCHLDWaker' object has no attribute 'doWrite' docs enhancement

#2499 opened Jan 16, 2017 by cp2587

6

Spider logging not optimal enhancement logging

#3163 opened Mar 10, 2018 by thernstig

2

Scrapy: how to rename project docs enhancement

#2484 opened Jan 6, 2017 by astrung

17

Issue with running scrapy spider from script. docs enhancement

#2473 opened Jan 2, 2017 by tituskex

40

FormRequest.formdata with GET method duplicates same key in query string bug help wanted

#2919 opened Sep 10, 2017 by talhashraf

2

Extend `load_object` to work with attributes discuss enhancement

#3086 opened Jan 20, 2018 by iAnanich

5

scrapy selector fails when large lines are present response bug upstream issue

#3077 opened Jan 15, 2018 by shanmuga-cv

6

Downloader middleware httpproxy enhancement enhancement

#3195 opened Mar 31, 2018 by grammy-jiang

Pipelines documentation limited docs enhancement help wanted

#2350 opened Oct 21, 2016 by mohmad-null

7

Response and other context is not passed to nested loader bug item loaders

#3159 opened Mar 8, 2018 by migr1

"Content-Encoding" header gets stripped from response headers enhancement good first issue

#1988 opened May 12, 2016 by mborho

7

Improve response.css speed performance

#3133 opened Feb 21, 2018 by lopuhin

2

Make public attribute formdata for FormRequest enhancement

#3145 opened Feb 27, 2018 by kalombos

6

Derived spider with overwritten custom_settings ignores custom_settings of the base spider enhancement

#3001 opened Nov 10, 2017 by woxcab

4

I ran into a problem when I used ipython in a scrapy shell logging upstream issue

#3064 opened Jan 8, 2018 by reaCodes

8

XMLFeedSpider iternodes iterator does not work on XML document with namespace bug help wanted

#2842 opened Jul 20, 2017 by redapple

5

get_project_settings of doesn't reflect per-spider settings docs enhancement

#2378 opened Nov 5, 2016 by mohmad-null

4

Add FAQ entry on how to deal with a very long "allowed_domains" docs enhancement

#2263 opened Sep 19, 2016 by redapple

3

Order of calling close_spider in pipelines discuss

#2645 opened Mar 12, 2017 by debosmit

7

add response.submit or Selector(List?).submit shortcut discuss

#2583 opened Feb 22, 2017 by kmike

1

Allow to create spider with custom templates docs enhancement

#2046 opened Jun 9, 2016 by DrJackilD

6

Mac os Python pip3 install Scrapy or Twisted gives“Command ”python setup.py egg_info“ failed with error code 1 in /private/tmp/pip-build-51x6vr4n/Twisted/” install upstream issue

#2808 opened Jul 1, 2017 by ShayChris

13

scrapy runspider in case of several spiders in a file discuss

#2608 opened Mar 1, 2017 by lopuhin

3

Items Pipeline output is documented to be dict or Item subclass, but can be anything discuss

#2358 opened Oct 24, 2016 by chekunkov

2

Make Exceptions Handled by Retry Middleware Configurable enhancement help wanted

#2701 opened Apr 5, 2017 by tad3j

2

Raising CloseSpider from DownloaderMiddleware doesn't work enhancement

#2578 opened Feb 20, 2017 by philonor

7

error at telnet shutdown in Python 3 bug

#2702 opened Apr 5, 2017 by kmike

2

Base classes for the item pipeline and middleware discuss

#2633 opened Mar 8, 2017 by jorenham

5

Invalid commands in example enhancement help wanted

#2349 opened Oct 21, 2016 by mohmad-null

5

Improving headers values decoding (utf-8 vs latin1) http

#2592 opened Feb 23, 2017 by rmax

1

CLOSESPIDER_PAGECOUNT don't work as should docs enhancement

#1651 opened Dec 29, 2015 by aktywnitu

1

Make logging configuration (more) customizable discuss logging

#2231 opened Sep 13, 2016 by redapple

4

robots.txt requests don't use errback discuss

#2443 opened Dec 11, 2016 by mohmad-null

5

add params kwarg to scrapy.Request() discuss enhancement

#2607 opened Mar 1, 2017 by pawelmhm

3

Duplicate log logging

#2228 opened Sep 11, 2016 by gabsn

1

make offsite middleware downloader middleware instead of spider middleware backward-incompatible discuss

#2241 opened Sep 15, 2016 by pawelmhm

8

engine_started vs spider_opened docs enhancement

#1523 opened Oct 3, 2015 by kmike

2

S3FilesStore should not disable SSL S3 patch available security

#2083 opened Jun 30, 2016 by wearpants

1

Add a page which explains important Scrapy concepts in a single page docs enhancement

#1569 opened Oct 29, 2015 by kmike

4

ImportError: No module named 'twisted.persisted' install upstream issue

#2580 opened Feb 21, 2017 by kelein

11

S3FilesStore: add option for AWS Signature v4 support S3 help wanted

#2448 opened Dec 14, 2016 by vshlapakov

5

Cookies not set when dont_merge_cookies is True bug

#2124 opened Jul 13, 2016 by LEChaney

13

Scrapy loses trailing ? discuss

#2472 opened Jan 1, 2017 by mohmad-null

2

Add item_failed signal discuss enhancement

#2295 opened Sep 29, 2016 by ilovenwd

Load settings dynamically on a per-spider basis enhancement

#2392 opened Nov 10, 2016 by mohmad-null

2

Improve traceback when "proxy" cannot be parsed correctly (e.g. missing scheme) enhancement logging

#2127 opened Jul 15, 2016 by redapple

2

RedirectMiddleware does not respect spider's crawling rules discuss

#2303 opened Oct 3, 2016 by djunzu

1

Date/Time handling in scrapy discuss

#2087 opened Jul 4, 2016 by nyov

2

defect of documentions: provide a list of deprecations docs enhancement

#1385 opened Jul 25, 2015 by SirbitoX

12

Order of HttpCacheMiddleware vs. RetryMiddleware backward-incompatible discuss

#2297 opened Sep 30, 2016 by ilovenwd

2

Move scrapy.utils.deprecate to separate package enhancement

#2194 opened Aug 23, 2016 by Digenis

1

Incorrect traceback in downloader middleware with @inlineCallbacks logging

#1948 opened Apr 21, 2016 by lopuhin

Redirected request uses same download slot when domain changes bug

#2141 opened Jul 22, 2016 by ArturGaspar

3

Break down the tutorial, present new topics more gradually docs enhancement

#1184 opened Apr 21, 2015 by eliasdorneles

Deprecate Request's AJAX escaping discuss

#1876 opened Mar 23, 2016 by redapple

1

FilesPipeline does not work with S3FilesStore with botocore bug

#1882 opened Mar 27, 2016 by lopuhin

run spider from within scrapy shell (when shell is activated from within project folder path) enhancement

#1845 opened Mar 4, 2016 by rosnk

6

Move from boto2/botocore to boto3 S3 help wanted

#1866 opened Mar 16, 2016 by jersub

2

In Scrapy 1.1.0 --output as xml appending to existing file when spider re-executed resulting in invalid xml file with duplicate declaration enhancement

#2071 opened Jun 22, 2016 by mobcdi

2

Expose cookiejars discuss

#1878 opened Mar 24, 2016 by kmike

9

IPv6 addresses not correctly recognized bug

#1832 opened Mar 1, 2016 by nyov

5

logger missing spider name logging

#1576 opened Nov 1, 2015 by jschilling1

9

Tracebacks are wrongfully written to stderr logging

#1609 opened Nov 20, 2015 by brunsgaard

5

Download delay does not work as documented when CONCURRNT_REQUESTS_PER_IP > 0 bug

#1659 opened Jan 5, 2016 by orangain

5

scrapy logging with python logging module logging

#1977 opened May 11, 2016 by sergei-sh

1

Prevent URL encoding option enhancement good first issue help wanted

#833 opened Aug 1, 2014 by DanMcInerney

16

ipython 4.0.1 / python 2.7.11 "scrapy shell" breaks with "ValueError: fallback required, but not specified" bug help wanted logging

#1646 opened Dec 21, 2015 by jschilling1

2

Multiple crawls with the same crawler discuss

#1587 opened Nov 9, 2015 by jdemaeyer

1

scraping https via proxy: remove custom proxy headers after tunnel connection enhancement security

#1543 opened Oct 15, 2015 by hugojanruiter

1

File is not downloading when response.status is 201 bug

#1615 opened Nov 24, 2015 by kurkop

11

Allow custom attributes to be added to responses in middlewares enhancement

#1453 opened Aug 24, 2015 by vincent-ferotin

5

Deprecate `download_delay` spider attribute cleanup

#1363 opened Jul 16, 2015 by barraponto

Use priority queues for Downloader slot queues enhancement

#1371 opened Jul 18, 2015 by kmike

8

parse command should use init_request(), make_requests_from_url() from CrawlSpider discuss

#1510 opened Sep 27, 2015 by chihchun

3

Allow copying existing cookiejar for request.meta['cookiejar'] enhancement

#1448 opened Aug 21, 2015 by vincent-ferotin

4

Headers object serializes to json differently depending on the indent value bug

#1459 opened Aug 26, 2015 by dvdbng

2

`trackref` ordering is non-deterministic on Windows bug

#1758 opened Feb 4, 2016 by jdemaeyer

1

Extensions, spider and settings initialization order enhancement

#1305 opened Jun 15, 2015 by chekunkov

9

Scrapy should handle "invalid" relative URLs better bug help wanted link extraction

#1304 opened Jun 15, 2015 by kmike

15

(Feature Request) Throttling number of spiders running concurrently in the same process docs enhancement

#869 opened Aug 30, 2014 by madvas

6

Adding support to FTP file listing enhancement

#1478 opened Sep 3, 2015 by bernardotorres

Speedup & fix URL parsing discuss performance

#1306 opened Jun 15, 2015 by kmike

38

Bring docstrings back docs enhancement

#713 opened May 7, 2014 by kmike

17

API to retrieve items from execution enhancement

#1395 opened Jul 31, 2015 by eltermann

11

Allow to start crawling with just an URL and a callback function enhancement

#1355 opened Jul 12, 2015 by rmax

2

CrawlerProcess settings have no effect on spider settings if Crawler is passed to crawl method bug

#1280 opened Jun 3, 2015 by kmike

1

Exception in LxmLinkExtractor.extract_links 'ascii' codec can't encode character bug

#1405 opened Aug 2, 2015 by aldarund

3