hi im like a game of telephone except using ṇ́ê̜u̙̇r͖͘a̞̋ĺ̰ ͚̽n̺͝e͓͊t̨͐s͕ and ᙢᗩᙅᖺᓮﬡᗴ Ⅴᓮᔕᓰᗝﬡ. toot at me!
Let’s make a Mastodon bot! We’ll name them Tootsee. Here’s the idea:
You toot at the bot.
The bot grabs the first image off DuckDuckGo that matches your toot.
The bot uses Microsoft’s machine vision API to caption that image.
The bot replies to your toot with Microsoft’s caption.
It’s like a game of telephone but with neural nets.
Setting Up For Development
You will need the following before setting up the project locally on your computer.
Mastodon account Access Token (Can be found in your desired Mastodon instance's Settings > Development > New Application)
- Enable read, write, and push permissions.
One of the two keys from your Microsoft Azure Computer Vision resource (Free plan available)
The URL to call the Microsoft Azure Computer Vision resource
- Usually in the format of
- Usually in the format of
To set up the project, you will need to install Crystal, the programming language used in this project. You can follow the instructions found here to install it on your system.
Once that is installed, set up the project and set up the dependencies by running
Next you will need to set up environmental variables.
export AZURE_KEY="[REPLACE WITH YOUR AZURE KEY HERE]" export AZURE_URL="[REPLACE WITH YOUR AZURE COMPUTER VISION SERVER URL]/analyze" export PORT=[PICK YOUR FAVORITE PORT] export ACCESS_TOKEN="[REPLACE WITH YOUR MASTODON ACCESS TOKEN]" export MASTO_URL="[REPlACE WITH YOUR MASTODON INSTANCE URL WITHOUT PROTOCOL]"
as an example it might look something like this:
export AZURE_KEY="ab3813dae9b8fe09fa09b3e70fb3bfae70fb3e70fb3bf78" export AZURE_URL="https://eastus.api.cognitive.microsoft.com/vision/v1.0/analyze" export PORT=7007 export ACCESS_TOKEN="8febab38ba0704fa070fb370f9f013dae9bf09f8" export MASTO_URL="botsin.space"
- Next, start the webserver with
crystal run app.crand tootsee away~
Retrieving Mentions from Mastodon
Tootsee subscribes to push notifications. This way, Mastodon will call our webhook whenever a user toots at tootsee.
To subscribe, we post once to
/api/v1/push/subscription. We specify the following form data:
subscription[endpoint]is the URL for the webhook to post to. We’ll use
/pushon our Heroku server.
subscription[keys][p256dh]is our public key. We’ll decrypt the webhooks sent to us with this key. That way, we verify that they were sent by Mastodon and not some third party.
subscription[keys][auth]is a throw-away random value. Not sure why we need this \o/
data[alerts][mention]is true. This tells Mastodon that we only care about mentions.
Then we listen for posts to our
/push endpoint. These requests will contain the triggering notification. In our case, it’s always a “mention” notification. We care about these fields:
typeshould always be
mention. If not, we’ll ignore the notification.
account[acct]is the handle of the user who mentioned tootsee.
status[id]is the ID of the toot. We’ll need it to reply to the toot.
status[content]is the HTML content of the toot. We’ll strip the HTML before using it.
Replying to Mentions
To send a reply toot, we post to
/api/v1/statuses and specify:
statusis the text of our toot.
in_reply_to_idis the ID of the toot we’re replying to.
- over 1 year ago
- August 8, 2018
Tue, 26 May 2020 13:08:28 GMT