Issues

• Support <base> field for relative urls

  featurehtml5needs info
  postmodern/spidr#94 · Filter by author zealot128 opened on Aug 18, 2025 · 0.8.0
  5

  

• Switch to Addressable::URI for URI parsing

  feature
  postmodern/spidr#86 · Filter by author postmodern opened on Dec 22, 2022

• Switch to using async-http

  httpimprovement
  postmodern/spidr#76 · Filter by author postmodern opened on Jan 29, 2022
  1

  

• Add Logging

  feature
  postmodern/spidr#72 · Filter by author postmodern opened on Oct 25, 2019 · 0.8.0
  

• Add low-level HTTP request methods

  feature
  postmodern/spidr#62 · Filter by author postmodern opened on May 25, 2017
  

• Add ignore_paths and ignore_paths_like

  feature
  postmodern/spidr#61 · Filter by author postmodern opened on May 25, 2017 · 0.8.0
  

• Following redirects

  postmodern/spidr#56 · Filter by author ZackMattor opened on Nov 29, 2016
  4

  

• Infinite path loop

  needs info
  postmodern/spidr#31 · Filter by author ethicalhack3r opened on Feb 1, 2012
  2

• SSL session reuse may fail

  postmodern/spidr#30 · Filter by author nirvdrum opened on Jan 19, 2012
  1

• Multithreading

  feature
  postmodern/spidr#26 · Filter by author ethicalhack3r opened on Jun 6, 2011
  13

• Automatically detect and parse sitemap.xml

  feature
  postmodern/spidr#19 · Filter by author postmodern opened on Oct 16, 2010
  7

  

• Store history queue in Hash of host:port and paths.

  feature
  postmodern/spidr#18 · Filter by author postmodern opened on Oct 16, 2010
  

• Redirect fails silently - need debug messages

  feature
  postmodern/spidr#13 · Filter by author perplexes opened on Aug 19, 2010
  4

• handle css @import spidering

  feature
  postmodern/spidr#8 · Filter by author zapnap opened on Mar 17, 2010