this post was submitted on 03 Jul 2023
105 points (100.0% liked)
Technology
23 readers
2 users here now
This magazine is dedicated to discussions on the latest developments, trends, and innovations in the world of technology. Whether you are a tech enthusiast, a developer, or simply curious about the latest gadgets and software, this is the place for you. Here you can share your knowledge, ask questions, and engage in discussions on topics such as artificial intelligence, robotics, cloud computing, cybersecurity, and more. From the impact of technology on society to the ethical considerations of new technologies, this category covers a wide range of topics related to technology. Join the conversation and let's explore the ever-evolving world of technology together!
founded 2 years ago
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
There has always been a symbiotic relationship between search engines and content site owners. The deal being "l (site owner) will let you index my site to make it easy for people to find my content. In exchange, you (Google) can make money by building user profiles and selling targeted advertising." Conceptually this is no different, except that Google is now using the data to build new applications and businesses - AI rather than ads.
I believe that Google does respect robots.txt (though these need to be well specified and located), so it's relatively easy for site owners to opt out of being indexed. Whether being indexed should be on an opt-out basis (as opposed to opt-in basis) in the first place is perhaps the key question, one I'd argue should have been discussed 20, 30 years ago.
or they could maliciously opt-in by including lots of hidden garbage text to poison the scrapers