this post was submitted on 03 Jul 2024
3 points (57.9% liked)

Programming

17433 readers
223 users here now

Welcome to the main community in programming.dev! Feel free to post anything relating to programming here!

Cross posting is strongly encouraged in the instance. If you feel your post or another person's post makes sense in another community cross post into it.

Hope you enjoy the instance!

Rules

Rules

  • Follow the programming.dev instance rules
  • Keep content related to programming in some way
  • If you're posting long videos try to add in some form of tldr for those who don't want to watch videos

Wormhole

Follow the wormhole through a path of communities !webdev@programming.dev



founded 1 year ago
MODERATORS
 

The site seems very locked down lol, i guess they really want me to pay $300 for semi high resolution images, so i want to scrape the previews instead. it's probably some sort of script since with noscript on the site doesn't even load. It's even beaten my 'absolute enable right click' Extension , and while i can still get the right click going and take a screenshot, i have no option to open the image itself in a new window.

Next up was a simple scraping extension, one i use regularly is webscraper but it's a huge process to use and can snag super easily, so i tried this one called Download All images

That one didn't grab anything besides headers and icons and seemed to have gotten me IP banned. Thankfully i have a vpn and they didn't even revoke my gallery access and i'm back at it again.

I have tried commercial scraping software before but the problem is, afaik these are very big on following robots.txt and that makes a lot of sites unscrapable.

So you've all heard my dilemma, and i'm curious, cause at this point it's a game. How would you all approach this? what software would you use?

you are viewing a single comment's thread
view the rest of the comments
[–] 8263ksbr@lemmy.ml 3 points 4 months ago

Puppeteer and playwright were not mentioned yet