this post was submitted on 19 Jun 2024
79 points (93.4% liked)
Privacy
32109 readers
853 users here now
A place to discuss privacy and freedom in the digital world.
Privacy has become a very important issue in modern society, with companies and governments constantly abusing their power, more and more people are waking up to the importance of digital privacy.
In this community everyone is welcome to post links and discuss topics related to privacy.
Some Rules
- Posting a link to a website containing tracking isn't great, if contents of the website are behind a paywall maybe copy them into the post
- Don't promote proprietary software
- Try to keep things on topic
- If you have a question, please try searching for previous discussions, maybe it has already been answered
- Reposts are fine, but should have at least a couple of weeks in between so that the post can reach a new audience
- Be nice :)
Related communities
much thanks to @gary_host_laptop for the logo design :)
founded 5 years ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
Speaking from experience, be careful you don't become over-zealous in your anti-scraping efforts.
I often buy parts and equipment from a particular online supplier. I also use custom inventory software to catalog my parts. In the past, I could use cURL to pull from their website, and my software would parse the website and save part specifications to my local database.
They have since enacted intense anti-scraping measures, to the point that cURL no longer works. I've had to resort to having the software launch Firefox to load the web page, then the software extracts the HTML from Firefox.
I doubt that their goal was to block customers from accessing data for items they purchased, but that's exactly what they did in my case. I've bought thousands of dollars of product from them in the past, but this was enough of a pain in the ass to make me consider switching to a supplier with a decent API or at least a less restrictive website.
Simple rate limiting may have been a better choice.
Try using “curl -A” to specify a User-Agent string that matches Chrome or Firefox.
I probably should have specified I'm using libcurl, but I did try the equivalent of what you suggested. I even tried setting a list of user agents and having it cycle through. None of them work. A lot of anti-scraping methods use much more complex schemes than just validating the user agent. In some cases, even a headless browser will be blocked.
Mouser?