Tollbit was the one that pioneered the idea of fixing the robot.txt and scraping issue by creating a toll for agents. The founders saw the problem first hand when they were at Toast. Tollbit has been publishing quarterly reports tracking the evolution of agentic traffic. It started as learning traffic and now RAG is exploding. Cloudflare does have broader distribution and a bigger bullhorn than a small startup. So, they are getting the press. The problem I see with Cloudflare is that they are not the only CDN nor do they even have close to a plurality of the internet traffic. My guess is that the other CDNs such as Fastly and Akamai are not going to work with Cloudflare and will choose an independent marketplace provider.
Thanks David! You're absolutely right - Cloudflare has c. 8% market share in CDN and growing the fastest, but not a sure thing that their marketplace will win by any means.
Tollbit was the one that pioneered the idea of fixing the robot.txt and scraping issue by creating a toll for agents. The founders saw the problem first hand when they were at Toast. Tollbit has been publishing quarterly reports tracking the evolution of agentic traffic. It started as learning traffic and now RAG is exploding. Cloudflare does have broader distribution and a bigger bullhorn than a small startup. So, they are getting the press. The problem I see with Cloudflare is that they are not the only CDN nor do they even have close to a plurality of the internet traffic. My guess is that the other CDNs such as Fastly and Akamai are not going to work with Cloudflare and will choose an independent marketplace provider.
Check it out:
https://tollbit.com/bots/25q1/
Thanks David! You're absolutely right - Cloudflare has c. 8% market share in CDN and growing the fastest, but not a sure thing that their marketplace will win by any means.