I recently commented on a LinkedIn post by Sarah Kopit, Editor-in-Chief for Skift.
She posted a link to an article on Skift.com by Adriana Lee called Dear Robot (How Travel is and isn’t talking to AI).
It is an extensive look at what different travel companies are allowing (or not) agents to do on their website and whether they have set any rules in place at all.
This prompted me to leave a comment:
I’m waiting for a robots.txt that says
bookings:yes
complaints: no
How much agency does an agent have?
And then the subject started working its way through my cerebral cortex, and I started digging. The article might have been about travel, but the agent/human paradox applies everywhere you can access by URL.
This led me to Agents aren’t humans (but they kinda are) on the Cloudflare blog..
What Cloudflare have already done is take my flippant comment deadly seriously. Their argument runs like this: when an agent visits a website on your behalf, the site can no longer tell who it is dealing with. One agent can carry the intentions of millions of users, and act at a scale no human ever could. So the old signals break. Their answer is machinery: signed requests, identity, reputation, accountability, and a brand new industry coalition to write the rules. In other words, the people who run a sizeable chunk of the internet’s plumbing have decided that agents acting for humans need an entirely new layer of permission. And, because they are Cloudflare, they are building it.
Which raises a rather more interesting question than plumbing.
Agents can be you on the web. But they are not you. They are something else. This article reflects on what that might actually mean, and of course, I will fold in how we see this at meed.
Keep off the grass (signed, the Internet, 1994)
In 1994, Dutch software engineer Martijn Koster gave the world the Robots Exclusion Protocol, better known to all of us as robots.txt.
In the early days of the web, the spiders were misbehaving and quite literally overloading websites. Out of a mailing list discussion came a simple text file that told spiders, or at least the good ones, where they could and could not go.
For twenty-eight years, it was not even a standard. It became one in 2022, and the document that made it official still says, in terms, that these rules are not a form of access authorisation. It was, and is, a handshake that survives on good manners.
Robots.txt is also entirely about reading. You may crawl this page, or you may not. It says nothing about what a visitor can do, and that is now a problem, because the visitors are no longer just spiders.
The file works for Google and friends, who announce themselves and choose to comply. An unnamed bot, or an agent wearing a human’s browser, can walk straight past it. And an agent with your login does not look like a bot at all; it looks like you.
That flips the cost. A crawler that ignores the sign costs you bandwidth. An agent that acts costs you real things: bookings, cancellations, account changes and money.
Travel shows the vacuum in practice. Skift reviewed the robots.txt files of fifty major travel sites and found that only ten say anything to AI crawlers at all. I doubt the other forty decided to stay silent; nobody had asked them to decide.
So here is the scene: a thirty-year-old notice on the door, written for guests who read, now guarding a building full of guests who act.
Your custom is important to us (your robot’s, less so)
My joke file does not exist yet. Cloudflare says as much: robots.txt can say where a crawler may go, but it cannot express what an agent may do, and no standards group is yet focused on that gap.
What does exist, as of this month, is the first half. Cloudflare customers can now set access per agent, and the pitch is to welcome the agents that bring you business and restrict the ones that don’t. You can let an agent roam the whole site, or block it only on certain pages.
You can see where this goes. Every business will be tempted by a one-way door: the agent may come in to spend, but not to compare prices, cancel or complain.
In fairness, the temptation is not pure cynicism. Cloudflare’s list of what agents break is long, and in nearly every case, the site bears the cost: disputed purchases nobody can prove were authorised, duplicate orders, free trials drained, support chat hammered around the clock.
There is a deeper point, too. Complaints have always been rationed by friction. Twenty minutes on hold is a filter, and plenty of business models quietly depend on it. An agent never tires and will put up with friction that no person would; it will happily hold all day.
Have your agent call my agent
Now look at it from the other side of the door.
An agent that can book for me but cannot complain for me is not my agent. It is a vending machine with my credit card.
If I delegate the booking, I am delegating the whole relationship: the change of date, the refund, the awkward conversation about the room that looked nothing like the photos. A business that accepts the first and refuses the rest has told me how it sees me.
Agents will learn this because learning is their job. The one that books my trip will remember which hotel let it sort out a problem and which one made me ring a call centre myself. I suspect it will route accordingly, and that the businesses on the wrong end will never know why the bookings slowed.
Blocking is not a long-term answer either. Cloudflare’s own argument is that it stops working once most of your traffic is agentic.
The usual, sir?
Cloudflare illustrates the problem with a corner grocer. He used to know Amy; he knew what she bought and set aside what he thought she would like. Now Amy sends an assistant who shops for four people, and the grocer cannot tell whose preferences are whose.
That is a loyalty story, and Cloudflare lists loyalty programmes among the casualties. Their fix is to help the shop see past the agent to the person behind it.
At meed we come at it from the other end. If the shop can no longer see the customer, the memory has to travel with the customer: one wallet, across every shop, holding where I am a regular and how close I am to my next reward. The retailer sees behaviour, not identity.
Agency needs memory. An agent deciding where to book needs to know where I am already loyal and who has treated me well. A merchant who holds the agent at the checkout keeps the transaction and loses the relationship, because the relationship is now stored on my side of the door.
Please wipe your feet
Which brings me back to my flippant question: how much agency does an agent have?
As much as the person behind it chooses to give, and as much as the business in front of it is prepared to honour. For thirty years, the sign on the door was written by the site owner. I think the next one gets written by consumers, in the form of which agents they trust and which businesses those agents are willing to visit.
Bookings: yes. Complaints: yes. That is the only place I would let my agent go.


