List of Links that aren't working #51

calebpeffer · 2024-04-23T16:34:55Z

calebpeffer
Apr 23, 2024
Maintainer

Hi all, Caleb from @mendableai here.

We will start collecting links that have issues here so we can fix them. If you'd like to add a link to the list, please follow this format.

Link: https://news.ycombinator.com/
Endpoint: /scrape, /crawl
Issue: Tables don't seem to be parsed correctly. The tables return as HTML instead of as a formatted table.

We will fix them as quickly as possible.

calebpeffer · 2024-04-23T20:34:50Z

calebpeffer
Apr 23, 2024
Maintainer Author

Link: https://privacy.patreon.com/policies
Endpoint: /scrape /crawl
Issue: fails all fallbacks

3 replies

rafaelsideguide Apr 24, 2024
Maintainer

I tested the system once and it failed, but it worked when I tried again. Non-deterministic behavior may be related to the fallback conditions?

nickscamara Apr 28, 2024
Maintainer

Yep. It seems to be a bit better now. Tried a few times and even triggering most fallbacks, it was still able to scrape the content.

rafaelsideguide May 9, 2024
Maintainer

Works with /scrape. "No pages found" for /crawl though

calebpeffer · 2024-04-23T20:37:21Z

calebpeffer
Apr 23, 2024
Maintainer Author

Link: https://scholarship.law.uci.edu/cgi/viewcontent.cgi?article=1584&context=ucilr
Endpoint: /scrape /crawl
Issue: All content is Null

1 reply

nickscamara Apr 28, 2024
Maintainer

#29 fixes this.

calebpeffer · 2024-04-24T18:12:03Z

calebpeffer
Apr 24, 2024
Maintainer Author

Link: https://docs.streamlit.io/
Endpoint: /crawl
Issue:

1 reply

nickscamara Apr 28, 2024
Maintainer

@calebpeffer hmm interesting. I believe we could add a dynamic routes and dynamic paths like [route]/path/path2/[dnyamic].cache in the excludes and includes to be able to filter out unwanted paths/files.

igalmarino · 2024-04-24T19:40:03Z

igalmarino
Apr 24, 2024

Link: https://support.greenpay.me/portal/es/kb/support
Endpoint: /crawl
Issue: It only gets the links on the header, but not the rest on the body of the page.

8 replies

igalmarino Apr 30, 2024

i do not know if im doing this right.

Do i need a new version of firecrawl-py to test these fixes.

https://pypi.org/project/firecrawl-py/

Or is there any other way to test it?

Tnanks

nickscamara Apr 30, 2024
Maintainer

No need for a new version, we pushed a server change so it should just work with what you currently had.

igalmarino May 2, 2024

I have been testing it and still only get 7 results (there are mora than 40 links on the page)

nickscamara May 10, 2024
Maintainer

@igalmarino Pushing a change that might help with this... Will be up in 1 hour. Can you let me know if it works? Thank you!

igalmarino May 10, 2024

It is working now. Thank you !

calebpeffer · 2024-04-30T21:31:10Z

calebpeffer
Apr 30, 2024
Maintainer Author

Link: https://apps.dos.ny.gov/publicinquiry/
Endpoint: /crawl
Issue: this site displays with a filter and javascript, this is a nasty one.

Note: some other websites that may or may not work in the same space:

Alabama - https://www.sos.alabama.gov/
Alaska - https://www.commerce.alaska.gov/web/cbpl/Corporations.aspx
Arizona - https://azsos.gov/business
Arkansas - https://www.sos.arkansas.gov/business-commercial-services-bcs
California - https://www.sos.ca.gov/business-programs/business-entities
Colorado - https://www.sos.state.co.us/

Note: Email michael@smobi.com once complete

1 reply

nickscamara May 10, 2024
Maintainer

Yea, first url is really weird...

calebpeffer · 2024-05-02T21:11:11Z

calebpeffer
May 2, 2024
Maintainer Author

Links: https://investors.snapone.com/static-files/c0d11e9f-f69b-48e2-b9a1-3cdb4992ed38
https://investors.snapone.com/static-files/365445cc-1182-4707-ab07-552c8c0a0bdf
https://investor.shutterstock.com/static-files/f5b52aee-0070-4922-acb6-9886feee789d
https://investor.shutterstock.com/static-files/79474632-2f5f-46a7-b481-e5cbd5274915

Endpoint: /scrape

returning
%PDF-1.7 %�� 1 0 obj <>/Metadata 115 0 R/ViewerPreferences 116 0 R>> endobj 2 0 obj <> endobj 3 0 obj <>/ExtGState<>/ProcSet[/PDF/Text/ImageB/ImageC/ImageI] >>/MediaBox[ 0 0 612 792] /Contents 4 0 R/Group<>/Tabs/S/StructParents 0>> endobj 4 0 obj <> stream x��][S��~~w��C=v�0EI%�T��a��abf�C��g��ж��~~��h�u�]U*]2S�_�R��z�iq�n��v��k��_�g?o��/��鷏k��rq��o_5__��6]#��0��n��'�!��Xs��
id rather it fail gracefully

Extra context:

1 reply

nickscamara May 10, 2024
Maintainer

#29 fixes this.

calebpeffer · 2024-05-05T16:52:57Z

calebpeffer
May 5, 2024
Maintainer Author

Link: https://www.rareskills.io/zk-book
Endpoint: /scrape, /crawl
receive a 502, could you please help? from Dmitry Ukhanov on discord

2 replies

nickscamara May 10, 2024
Maintainer

Fixed ✅

nickscamara May 10, 2024
Maintainer

DM'd him!

calebpeffer · 2024-05-06T16:21:01Z

calebpeffer
May 6, 2024
Maintainer Author

Link: https://developers.notion.com/reference/post-page
endpoint: /scrape /crawl

1 reply

nickscamara May 10, 2024
Maintainer

Fixed ✅ @calebpeffer can you let them know? and see if it works better now for them?

garrettfrohman · 2024-05-06T16:33:52Z

garrettfrohman
May 6, 2024

Link: https://docs2.hubitat.com/
Endpoint: /crawl
Error: No pages found

1 reply

nickscamara May 10, 2024
Maintainer

@garrettfrohman Fixed ✅

lodex23 · 2024-08-20T18:10:56Z

lodex23
Aug 20, 2024

Link: https://www.bax-shop.be/nl/speakers/speakerset
Endpoint: /scrape, /crawl
Issue: Request Time out everytime i try it, also when i try other urls that belong to the same website

0 replies

immartian · 2024-08-20T20:28:27Z

immartian
Aug 20, 2024

link: https://www.wsj.com/tech/elon-musks-twitter-takeover-is-now-the-worst-buyout-for-banks-since-the-financial-crisis-3f4272cb
endpoint: /scrape, /crawl
Issue: Request Time out always, same to every url from the website

0 replies

wukimidaire · 2024-08-21T09:48:39Z

wukimidaire
Aug 21, 2024

Link: https://mendable.ai
endpoint: /scrape, /crawl
Issue: No pages found, Please try a different URL or check for compatibility issues with the current website.

0 replies

talaus-tadata · 2024-08-22T07:42:27Z

talaus-tadata
Aug 22, 2024

Link: https://docs.temporal.io/
Endpoint: /crawl
Issue: No content found. This could be due to an incorrect URL or a compatibility issue on certain websites. Please verify the URL, and if correct, help us improve by reporting this issue.

0 replies

NioCarSan · 2024-08-25T14:17:57Z

NioCarSan
Aug 25, 2024

https://www.idealista.com/sala-de-prensa/informes-precio-vivienda/venta/cataluna/barcelona-provincia/castelldefels/

issue: No pages found

Please try a different URL or check for compatibility issues with the current website.

0 replies

wukimidaire · 2024-08-28T10:47:16Z

wukimidaire
Aug 28, 2024

https://www.elmos.be

issue: no pages found

Please try a different URL or check for compatibility issues with the current website.

0 replies

ad-angelo · 2024-09-01T07:32:05Z

ad-angelo
Sep 1, 2024

Hi guys !

Link : https://www.laredoute.fr/ppdp/prod-542712900.aspx
Endpoint : /scrape
Issue : Blocked by robot checking

Returned content :

Un cœur bat en vous.
====================

Merci de confirmer que vous n’êtes pas un robot.
------------------------------------------------

Verifying you are human. This may take a few seconds.

Waiting for www.laredoute.fr to respond...

Please enable Cookies and reload the page.

*   Ray ID : 8bc3885abeaa0a0d
*   Client IP : 193.142.36.20
*   Error reason : Country challenge

0 replies

jeremyssocial · 2024-09-24T07:26:18Z

jeremyssocial
Sep 24, 2024

Link: https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.utility/add-type?view=powershell-7.4
Endpoint: /scrape
Issue: No content found This could be due to an incorrect URL or a compatibility issue on certain websites. Please verify the URL, and if correct, help us improve by reporting this issue.

0 replies

jeremyssocial · 2024-09-26T11:08:24Z

jeremyssocial
Sep 26, 2024

Link: https://support.microsoft.com/de-de/windows/tastenkombinationen-in-windows-dcc61a57-8ff0-cffe-9796-cb9706c75eec
Endpoint: /scrape
Issue: No content found This could be due to an incorrect URL or a compatibility issue on certain websites. Please verify the URL, and if correct, help us improve by reporting this issue.

0 replies

ad-angelo · 2024-10-13T19:40:26Z

ad-angelo
Oct 13, 2024

Link : https://www.footlocker.fr/fr/product/nike-air-max-tuned-1-drift-homme-chaussures/314216761404.html
End-point : /scrape
Issue : No content - code 403

0 replies

bjoaquinc · 2024-10-25T05:34:27Z

bjoaquinc
Oct 25, 2024

Link : https://learn.microsoft.com/en-us/javascript/api/office-scripts/overview?view=office-scripts
End-point : /scrape
Issue : Failed to scrape URL. Error: {'success': True, 'warning': 'No page found'}. I added a waitFor: 2000 as well

0 replies

ericciarla · 2024-10-31T17:06:48Z

ericciarla
Oct 31, 2024
Maintainer

Link: https://insider.fitt.co/newsletter/
Endpoint: /crawl
Wont gather pages - notify rexhunter8@gmail.com

0 replies

ericciarla · 2024-11-04T16:38:57Z

ericciarla
Nov 4, 2024
Maintainer

Link: https://www.welcometothejungle.com/fr/companies/veesion-1/jobs
Endpoint: /crawl
I'm working on trying to crawl all job posting in the list of this website (https://www.welcometothejungle.com/fr/companies/veesion-1/jobs), but the output of the crawler is only 1 page. Email arthur@repio.co when done

0 replies

wonhs91 · 2024-11-20T14:00:31Z

wonhs91
Nov 20, 2024

Link: https://www.wsj.com/politics
Endpoint: /scrape /crawl
The page returned an error while being scraped. Unauthorized

0 replies

rincidium · 2024-12-17T15:44:02Z

rincidium
Dec 17, 2024

Link: https://opencorporates.com/companies/us_or/138287693, https://opencorporates.com/companies/us_tx/0802555441, https://opencorporates.com/companies/us_ca/202464211766
Endpoint: /scrape
Issue: Scraping OpenCorporates pages fails intermittently with the following error: Expecting value: line 2 column 1 (char 1)

0 replies

This comment has been hidden.

Sign in to view

This comment has been hidden.

Sign in to view

This comment has been hidden.

Sign in to view

This comment has been hidden.

Sign in to view