Introducing the Agent API
Here is a question worth being precise about: what would it mean for an AI to use a design tool?
The usual answer is that it generates an image. You describe a logo, a model produces a grid of pixels that looks like a logo, and you are now the owner of a picture you cannot edit. If the circle is four pixels too far left, you cannot move it four pixels left. There is no circle. There is a field of colour that resembles one, and your only recourse is to ask again and hope.
That’s not using a design tool. That’s ordering from a kitchen you’re not allowed into.
Using a design tool would mean the AI places a circle — an actual circle, an object with a centre and a radius that you can select, move, recolour and delete afterwards, sitting in a layer, in a document, in your undo history. And that is what the agent API in justdraw.fyi does.
You paste a project link into a chat with a model. It draws. When it’s finished you open the tab and there is a drawing, made of shapes, and every one of them is yours to move.
The mechanism is dull, which is the point. Every edit in justdraw.fyi is a JSON command, and there is one set of them:
element.draw element.list element.name
element.move element.style element.delete
element.clone group.create group.ungroup
arrange.align arrange.distribute arrange.order
doc.get doc.export
That is the whole vocabulary. When you press Ctrl+G in the editor, the keyboard
sends group.create. When you click the group button, the toolbar sends
group.create. When you run jd group create, the CLI sends group.create. When
a language model decides that two shapes belong together, it sends
group.create. Four seats, one steering wheel.
To send one, you POST it:
curl -sX POST https://app.justdraw.fyi/api/v1/agent \
-H 'Content-Type: application/json' \
-H 'Origin: https://app.justdraw.fyi' \
-d '{"cmd":"element.draw","project":"0192a4e1-…",
"args":{"shape":"rect","x1":0,"y1":0,"x2":40,"y2":30,"fill":"#5b7cfa"}}'
and you get back {"ok":true,"result":{"id":"…","type":"rect","name":"Rectangle 1"}}. Every command that creates something returns its id, and that id is the
argument to the next command. It’s the same bargain the command line makes,
because it is the same protocol underneath. [1]
Two things about that call are worth explaining, because they are the two things that will bite you.
The Origin header is required. Leave it out and you get a 403. It’s a cross-site
guard — a browser will not attach that header when some other website tries to
POST to us on your behalf, which is the attack it exists to stop, and curl
doesn’t send one unless you say so. This is the kind of detail that is obvious to
the person who wrote the middleware and invisible to everyone else, and I only
learned how invisible by watching an AI copy my own example, get a 403, and
conclude the API was broken. [2]
And the drawing commands need the project open in a browser. With no tab, you get
a 409. The editor in the browser is the engine — it owns the geometry, the layers,
the undo stack — and the server only relays commands down the WebSocket the editor
is holding open. That is a real constraint and I defend it at length in the next
post. The commands that only read (element.list, doc.export) work either way,
because those come out of the database.
Now the part that took the longest and is the least technically interesting, which is how an agent is supposed to learn any of this.
It can’t, from the above. A model that receives a bare URL knows nothing about
justdraw.fyi. It has never heard of element.draw. It has no reason to think the
canvas is dark, or that shapes come out grey, or that this API even exists.
So the URL teaches. GET /p/<uuid>.md returns a briefing generated from the live
project. Not a link to documentation — a document about this drawing, written at
the moment it was requested. It says what is currently on the canvas, with real
ids in a real table. It gives the whole command vocabulary. It prints curl
examples with this project’s id already substituted in, so they can be run as
written, with the Origin header already there. It tells you whether an editor is
open right now, and therefore whether a drawing command will work or 409. And it
ends with a list of things that will otherwise cost you a retry — that the canvas
is #141518 and a black shape drawn on it is invisible, that everything starts
grey until you colour it, that y grows downwards, that align needs two elements
and distribute needs three.
The same essentials are also in the editor’s HTML, in a visually hidden section, because an agent that fetches the pasted link as a web page and converts it to Markdown would otherwise see nothing but a canvas element. [3]
Every line in that briefing is there because a real agent got that exact thing wrong. I sat and watched them do it. That process — handing the link to a model that knew nothing, watching it fail, and treating each failure as a bug in my documentation rather than a defect in the model — turned out to be the most useful thing I did on this project, and it is the subject of a post of its own.
The thing I did not anticipate is how much better the drawings get when the agent is drawing shapes instead of pixels. When it’s wrong, it’s wrong in a way you can fix. The rectangle is four pixels left. You drag it four pixels right. There is a rectangle to drag.
[1] There’s a separate rate limit on /api/v1/agent, higher than the one on
ordinary page traffic, because a scripted drawing is a burst of dozens of tiny
commands and the general limit — sized for humans loading pages — would stall it
halfway through a logo.
[2] There is now a test in the repo that executes the exact curl command the
briefing prints, through the real middleware stack, and fails if it doesn’t
return 200. A copy-pasteable example that 403s is worse than no example at all.
[3] It must not be display:none or [hidden], because the converters drop those.
It has to be visually hidden but present. This is a small, stupid, load-bearing
detail, and I offer it freely to anyone else writing for an audience of machines.