Why would publishers allow you to crawl their sites if you're not sending them any traffic?
The big publishers certainly won't let you do that as they are selling their data to Google, Microsoft, Facebook and whoever else has the money to train a fully fledged LLM, which is certainly not everyone.
Because it lets them sell data to other parties than Google and Facebook? That's actually pretty great. Only having a 1 or 2 customers kinda sucks.
A search engine partnering with an answer engine may not send traffic, but the answer engine is a potential customer for the websites the search engines direct them to.
>Because it lets them sell data to other parties than Google and Facebook?
Only indirectly by charging search engines for access to content. It would be an entirely different business model that requires a complex set of agreements between publishers, search engines and LLM providers.
Granted it's not impossible and certainly worth considering if you have search engine expertise but no money to train an LLM.
