Commanding Chaos for Coworking, Open Source and Creative Communities

Howto: Yahoo Pipes Auto-retweet recipe for Twitter

Tue, 02/03/2009 - 08:45 -- rprice

I have 5 or 6 twitter accounts, and some of them are only meant to serve as a discovery point for a blog, podcast, or community. That's why I cooked up this little pipe recipe to auto-retweet around two possible keywords, but NOT to retweet my own retweets if I did so from another account. Get it? Well, I'll explain:

Yahoo! Pipes is a free, web-based, visual programming environment. Those of you familiar with node-based editors like Shake or some other graphics and sound programs will feel right at home. Each "module" represents an input or output, or most importantly, ways to process and filter data as it travels through the pipeline. Pipes takes as input just about any feed or structured data - in this case, the river of tweets from a particular user's friends.

It's pretty easy to register for Pipes and create a new pipe. When you're done, you get a big blank piece of graph paper, and the list of sources, operators and other junk you can drop in on the left side.

Fetch Feed The first thing you need to do is go ahead and grab the twitter feed. In the left sidebar you want to find Sources > Fetch Feed and drag it on to the working area. In addition to the url to your friends list, you have to provide your username and password, because Twitter's API requires authentication.

Like this:
http://user:pass@twitter.com/statuses/friends_timeline.atom

At this point, if you take the output from the Fetch Feed module to the Pipe Output module, you have a working Pipe. But what you want to do here is filter out results based on certain keywords, so the output can be used to help you track a conversation.

Filter keyword 1 and yourself The next module you'll want to take a look at is under Operators > Filter. Drag this one out and take a look - this one reminds me of the Smart Mailbox rules in Mail.app, and it works in much the same way.

In this case we are going to:

Permit items that match all of the following Rules
item.title Contains fringe
item.title Does not contain bloggingfringe:

What does this mean?

  1. The first rule just pulls out tweets that contain a single word or phrase you're looking for - in this case, "fringe".
  2. The second rule filters out tweets by a particular user. In this case, the username contains the word you're filtering on, so you try to take that into account.

Again, you could connect the Fetch Feed to the Filter, and then to the Pipe Output and have a working version of the Pipe. In fact, I ran the pipe like this for a long time without anything else. But this is a two-keyword tutorial, and there are some other tricks I'd like to show you.

The next phase of your retweeting script is going to prepare the text of the tweets so they give a link back to the original author. You'll also remove a few precious characters by removing your own username from an @reply.

RSS and Atom documents coming out of your twitter friends feed give the title and body of each item the following format:
liberatr: @bloggingfringe I think it's really cool how you're retweeting our posts about fringe automatically!
This is the format you're going to pay attention to when you apply some simple regular expression replacement to the titles.

To give all of the retweets a more trational RT @user message format, you're going to grab the Operators > Regex module next. There are two rules you'll use:

In item.title replace : @bloggingfringe with :
In item.title replace ^ with RT @

Regex Rules

  1. On the first rule, we are just cleaning things up in the case of a reply. Check i to ignore capitalization, so the rule matches @bloggingFringe or @BloggingFringe as well.
  2. The second rule just adds a few letters to the beginning of the outgoing tweet.

Adding the second keyword

At any given time, there are other cultural events happening, not just fringe, so you might like to make two pipes, one pipe to track each event. If both keywords are going to the same account, you can use the Split and Union operators to apply different kinds of filters.

You need to use Split and Union in this case because your filtering has too many AND and OR questions:
If someone is tweeting about "fringe" AND it wasn't written by @bloggingfringe, OR
If someone is tweeting about "arts fest" OR "artsfest" AND not "fringe"

It's important to add the "and not fringe" to the end, because we don't want tweets appearing twice if someone mentions artsfest and fringe in the same tweet.

I'm going to gloss over a few steps here, because they are simple, and you've already done a few just like these: Filters for the second keyword

  1. Create two new Filter modules
  2. The first filter looks for your keywords, in my example the word could have spaces in it, so I needed a whole filter to spell out each variation
  3. The second filter makes sure you're not duplicating tweets in the final pipe output
  4. Put the Split operator at the very top, just after Fetch Feed. attach one of the outputs to the very first Filter and the second to the new keyword Filter.
  5. Put the Union operator just above your Regex operator from before, with your two filtering decision trees going into the top, and the output from the Union going to the Regex, and that module's output the the Pipe Output.

Retweet Filter The last rule you're going to apply is one final Filter module. Since you're retweeting any tweet that mentions these keywords you care about, you may end up catching some retweets yourself, if your friends decide to share what you wrote. In the example, I have this filter appearing in between the Union and Regex modules, but you could also put this at the very top, above the Split.

Block items that match any of the following Rules
item.title Contains RT @bloggingfringe
item.title Contains Retweeting @bloggingfringe

Finishing Up

Once you get all your filters connected, splits and unions, regular expressions, feeds and outputs in order, you should have something that looks like this:
Pipe Recipe for Twitter: Auto-retweet
Please note the output (in the gray area at the bottom), and how it took the original tweet and just added 4 text characters to the beginning, adding a bit of context.

Now you have to make sure you Save your pipe, and finally click the link at the top saying Run Pipe... The only thing I want to warn you about is Publishing your pipe. Please remember that your twitter password is in there!

Pipes will format your output as RSS or JSON (or serialized PHP objects), as well as hooking into your favorite RSS aggregator and Yahoo! Alerts. As if that wasn't enough, think about all the RSS-to-Twitter applications that exist. My personal favorite is Twitterfeed.com - which is a free service, but I encourage you to donate a few bucks via PayPal, because it's just a fantastic thing to do!
Edit: also check out the excellent TweetBots, another way to get feeds to twitter, but it also has auto-follow and a cool way to allow multiple people to update the account using direct messages.

Gotcha: Make sure you select "Title only" in twitterfeed's interface, or the first few words of short tweets may be repeated!

I'm sure there are lots of other ways to filter your friends' tweets, and tons of ways to create a retweeting service, but this is one that works for me, and I just wanted to share with anyone looking for a little help.

Update: Double-retweeting
Since retweeting is so widespread, you may come across several instances where you are retweeting a retweet. Most times this will be pretty innocuous, but if you get into a situation where several of the people you're following retweet the same message, it's probably best to avoid those.

Example of the bad behavior:

RT @Obsessionful: I have been retweeted by @VociDance and @bloggingfringe. I'm awesome. xP: Obsessionful: I have.. http://tinyurl.com/bewhqw

RT @VociDance: RT @Obsessionful: "I think the thing getting me through this mountain of homework is the fact tha.. http://tinyurl.com/dx3x4g

RT @Obsessionful: I think the thing getting me through this mountain of homework is the fact that Arts Fest is t.. http://tinyurl.com/br65e5

No need to retweet it a second time.

Therefore, add one last Filter module just before the Pipe Output.
Double Retweet

Block items that match any of the following Rules
item.title Matches regex RT.*RT

That should take care of most double retweets.

After I first made this script, I was thinking about setting up two accounts that just retweet each other all day, and call them "echo" and "chamber"...

RT @echo RT @chamber RT @echo RT @chamber RT @echo RT @chamber RT @echo RT @chamber RT @echo RT @chamber RT @echo RT @chamber RT @echo RT...

You get the idea...

Categories: 

Commenting on this Blog post is closed.

Comments

Submitted by Lance (not verified) on

Ryan - just a quick thank-you for this post, it really helped with something I was working on.

[...] MySpace, etc.), measuring will be just as difficult as with most social media campaigns. My Blogging Fringe twitter bot helps a bit, and Google Alerts help a bit more, but the only (simple) way to be able to see all [...]

[...] Yahoo! Pipes Auto-retweet recipe for Twitter Ryan Price documents his Auto-retweet pipe. So many interesting possibilities here. (tags: Yahoo twitter pipes tools) « Should we ask employees to tweet client stories? [...]

[...] been getting lots of good karma and feedback from people about my retweeting recipe using Yahoo Pipes. In truth, the tutorial can be a bit hard to swallow if you aren’t familiar with pipes or a [...]

This was BRILLIANT. I love it.
But, can you answer one question.
Is it automatic, or do I have to keep coming back and running it???
Thanks.

Submitted by Rob P (not verified) on

I'm setting up a twitter account for a group. I want it to auto RT any @ replies to the account and not restrict it to just friends feeds. How can I do that?

For example. A specific city traffic conditions. call it CityUSA. to anyone can post: @CityUSA I-5 eastbound is stopped due to crash.

I want to find that and RT to my CityUSA twitter account to that any subscribers can view. I hope that makes sense.

May I recommend a re-tweeting service? http://tweetbots.com/ - they may be able to help. Unless twitter provides you a feed with all of your @mentions in it, then there is no way to do this without a 3rd party script.

Instead of using the friends feed, you could try using the output of twitter search - any RSS feed.

Hi Ryan. Thanks for your article, it's been immensely helpful. One thing: I took the advice from your previous comment and based my feed on the output of a twitter search for my keyword. Unfortunately, unlike the friends feed, the search feed does not return results in the form of
@agarderik: Ryan is one cool dude
but in the form of
Ryan is one cool dude
so once formatting is done in the pipes, the output reads
RT @Ryan is one cool dude

Do you know of any way of formatting it so it mentions (@) the original tweeter (agarderik in this case) instead of the first word in the tweet?

Thanks a lot.
Erik

Sorry, I'm quite new to Yahoo Pipes (and programming in general). How would I implement that? I tried "In item.title replace ^ with RT @item.author.name" but it replaced it with the text, not the property.

Thanks!

Great! Thanks Nev! Only one more problem, item.author.name outputs as username (FirstName LastName) instead of just username. So a sample retweet would be

RT @agarderik (Erik Agard)Ryan and Nev are cool dudes

How do I truncate that?

Thanks a lot for your help, guys.

Submitted by Nev (not verified) on

Erik,

I had that in my comment too but it seems it got truncated...

Add a regex before the loop that replaces \(.*\) with nothing in the item.author.name.

Hopefully, thats everything

Submitted by chris (not verified) on

This is awesome thanks!

But I've got this string at the end of each of the items in my pipe output:
#SEARCHTERM

How do I get rid of that?

For Twitter automation i've got to say Twimbo from iq143.com was one of the easiest to set up and one of the more intelligent bots.

It only took a few minutes to set up and it's currently priced low for early adopters. My traffic has doubled since using it and I now have a great following on Twitter. I've even had new paying customers as a result! Winner.

This is great!!! I was wondering however if it was possible to send an auto reply (Not retweet) to someone in a particular area of the country who mentions a certain phrase. I can run a search on twitter to find people in my area that said they need a job but how can I set up a auto responce so my twitter account will automatically send them an @ reply saying that I may have some openings for them??? any thoughts????