Sitelet https://web.archive.org/web/20200710032116/https://github.com/freeCodeCamp/devdocs/issues/1219
Skip to content

/ devdocs

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Root cause of MDN documentation issues #1219

Open
jgarber623 opened this issue Apr 28, 2020 · 1 comment
Open

Root cause of MDN documentation issues #1219

jgarber623 opened this issue Apr 28, 2020 · 1 comment

Comments

@jgarber623
Copy link

@jgarber623 jgarber623 commented Apr 28, 2020

I spent a bunch of time today and yesterday tracing this down. I'm not sure how to proceed with a fix, so posting what I've found so far to see how best to solve the issue.

Current Behavior

Attempting to generate documentation using any of the MDN scrapers (e.g. thor docs:generate html --debug, thor docs:generate css --debug) throws an error:

bundler: failed to load command: thor (/Users/jason/.rbenv/versions/2.6.5/bin/thor)
TypeError: no implicit conversion from nil to integer
View stack trace 
Building internal urls...                                                                                                                   
Queue:   developer.mozilla.org/en-US/docs/Web/HTML                                                                                          
Ignore:  developer.mozilla.org/en-US/docs/Web/HTML                                                                                     [200]
Continuing...                                                                                                                               
bundler: failed to load command: thor (/Users/jason/.rbenv/versions/2.6.5/bin/thor)
TypeError: no implicit conversion from nil to integer
  /Users/jason/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/progress_bar-1.3.0/lib/progress_bar.rb:47:in `ratio'
  /Users/jason/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/progress_bar-1.3.0/lib/progress_bar.rb:101:in `render_bar'
  /Users/jason/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/progress_bar-1.3.0/lib/progress_bar.rb:92:in `render'
  /Users/jason/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/progress_bar-1.3.0/lib/progress_bar.rb:77:in `block in to_s'
  /Users/jason/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/progress_bar-1.3.0/lib/progress_bar.rb:76:in `each'
  /Users/jason/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/progress_bar-1.3.0/lib/progress_bar.rb:76:in `inject'
  /Users/jason/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/progress_bar-1.3.0/lib/progress_bar.rb:76:in `to_s'
  /Users/jason/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/progress_bar-1.3.0/lib/progress_bar.rb:39:in `write'
  /Users/jason/Developer/freeCodeCamp/devdocs/lib/docs/subscribers/progress_bar_subscriber.rb:11:in `running'
  /Users/jason/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/activesupport-5.2.3/lib/active_support/subscriber.rb:101:in `finish'
  /Users/jason/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/activesupport-5.2.3/lib/active_support/notifications/fanout.rb:104:in `finish'
  /Users/jason/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/activesupport-5.2.3/lib/active_support/notifications/fanout.rb:48:in `block in finish'
  /Users/jason/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/activesupport-5.2.3/lib/active_support/notifications/fanout.rb:48:in `each'
  /Users/jason/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/activesupport-5.2.3/lib/active_support/notifications/fanout.rb:48:in `finish'
  /Users/jason/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/activesupport-5.2.3/lib/active_support/notifications/instrumenter.rb:44:in `finish_with_state'
  /Users/jason/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/activesupport-5.2.3/lib/active_support/notifications/instrumenter.rb:29:in `instrument'
  /Users/jason/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/activesupport-5.2.3/lib/active_support/notifications.rb:168:in `instrument'
  /Users/jason/Developer/freeCodeCamp/devdocs/lib/docs/core/instrumentable.rb:15:in `instrument'
  /Users/jason/Developer/freeCodeCamp/devdocs/lib/docs/core/scraper.rb:74:in `build_pages'
  /Users/jason/Developer/freeCodeCamp/devdocs/lib/docs/core/doc.rb:108:in `block in store_pages'
  /Users/jason/Developer/freeCodeCamp/devdocs/lib/docs/storage/abstract_store.rb:87:in `block (2 levels) in replace'
  /Users/jason/Developer/freeCodeCamp/devdocs/lib/docs/storage/abstract_store.rb:182:in `track_touched'
  /Users/jason/Developer/freeCodeCamp/devdocs/lib/docs/storage/abstract_store.rb:87:in `block in replace'
  /Users/jason/Developer/freeCodeCamp/devdocs/lib/docs/storage/abstract_store.rb:170:in `lock'
  /Users/jason/Developer/freeCodeCamp/devdocs/lib/docs/storage/abstract_store.rb:87:in `replace'
  /Users/jason/Developer/freeCodeCamp/devdocs/lib/docs/storage/abstract_store.rb:85:in `block in replace'
  /Users/jason/Developer/freeCodeCamp/devdocs/lib/docs/storage/abstract_store.rb:144:in `open_yield_close'
  /Users/jason/Developer/freeCodeCamp/devdocs/lib/docs/storage/abstract_store.rb:30:in `open'
  /Users/jason/Developer/freeCodeCamp/devdocs/lib/docs/storage/abstract_store.rb:85:in `replace'
  /Users/jason/Developer/freeCodeCamp/devdocs/lib/docs/core/doc.rb:107:in `store_pages'
  /Users/jason/Developer/freeCodeCamp/devdocs/lib/docs/core/scraper.rb:225:in `block in store_pages'
  /Users/jason/Developer/freeCodeCamp/devdocs/lib/docs/core/scraper.rb:233:in `with_internal_urls'
  /Users/jason/Developer/freeCodeCamp/devdocs/lib/docs/core/scraper.rb:223:in `store_pages'
  /Users/jason/Developer/freeCodeCamp/devdocs/lib/docs.rb:100:in `generate'
  /Users/jason/Developer/freeCodeCamp/devdocs/lib/tasks/docs.thor:300:in `generate_doc'
  /Users/jason/Developer/freeCodeCamp/devdocs/lib/tasks/docs.thor:106:in `generate'
  /Users/jason/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/thor-0.20.3/lib/thor/command.rb:27:in `run'
  /Users/jason/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/thor-0.20.3/lib/thor/invocation.rb:126:in `invoke_command'
  /Users/jason/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/thor-0.20.3/lib/thor.rb:387:in `dispatch'
  /Users/jason/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/thor-0.20.3/lib/thor/base.rb:466:in `start'
  /Users/jason/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/thor-0.20.3/lib/thor/runner.rb:42:in `method_missing'
  /Users/jason/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/thor-0.20.3/lib/thor/command.rb:29:in `run'
  /Users/jason/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/thor-0.20.3/lib/thor/command.rb:128:in `run'
  /Users/jason/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/thor-0.20.3/lib/thor/invocation.rb:126:in `invoke_command'
  /Users/jason/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/thor-0.20.3/lib/thor.rb:387:in `dispatch'
  /Users/jason/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/thor-0.20.3/lib/thor/base.rb:466:in `start'
  /Users/jason/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/thor-0.20.3/bin/thor:6:in `<top (required)>'
  /Users/jason/.rbenv/versions/2.6.5/bin/thor:23:in `load'
  /Users/jason/.rbenv/versions/2.6.5/bin/thor:23:in `<top (required)>'

Root Cause

The error is a bit misleading, but here's what I found after digging deep into the weeds:

  1. The MDN HTML scraper sets base_url to https://developer.mozilla.org/en-US/docs/Web/HTML.
  2. Its parent class (Docs::Mdn) sets two params.
  3. This results in a URL of https://developer.mozilla.org/en-US/docs/Web/HTML?raw=1&macros=1 being crawled.
  4. That URL returns a 301 redirect to https://wiki.developer.mozilla.org/en-US/docs/Web/HTML?raw=1&macros=1.
  5. This becomes the response's effective_url which will fail in the process_response? method on line 58 of lib/docs/core/scraper.rb (which itself calls process_url? and fails on line 63).

The failure is in the comparison of base_url (https://developer.mozilla.org/…) with the response's effective_url (https://wiki.developer.mozilla.org/…).

Solutions

https://developer.mozilla.org is the canonical reference and should still be linked from the footer of scraped pages, etc. There's good reason to continue scraping pages from https://wiki.developer.mozilla.org with the two URL params (see https://wiki.developer.mozilla.org/en-US/docs/Web/HTML?raw=1&macros=1, for example).

So, I'm not sure what a solution might be here. It'd be great to get this resolved, though. The MDN documentation on devdocs.io is increasingly out of date. I'm happy to help out with some guidance and ideas.

Thanks for your time and for making and maintaining this awesome tool!
@NotWearingPants
Copy link

@NotWearingPants NotWearingPants commented May 22, 2020 •
edited

Maybe this will solve #1127
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
2 participants
@jgarber623 @NotWearingPants
You can’t perform that action at this time.