Archive for March, 2008

Feedlr十八般用法之:让国内用户读到Feedburner烧制地址的Feed

Sunday, March 30th, 2008

国内喜欢订阅Feed的朋友肯定都有一个头痛的问题,由于Feedburner的撞墙,一些用Feedburner烧制的Feed在阅读器里的经过Feedburner重定向的链接就不能直接打开读全文了。

比如我自己在Google Reader里80%经常读的Feed都是用Feedburner的。如果Feed内容包含了全文,可以直接在Reader里看,那么没问题。但要是想点Feed文章的链接去网站看全文,就撞墙了。而且讨厌的是,我要看的网站本身往往都不会撞墙,只是因为要经过Feedburner重定向才出的问题。

所以,我自己非常喜欢Feedlr的“把FeedBurner地址转成原地址”这个功能。这个小小的功能,能在把Feed内容播发到迷你博客之前,把Feedburner重定向地址转换成网站的原地址,再播发出去。这样一来,迷你博客的读者就能直接通过接收到的链接读到Feed全文网页了,不会再撞墙了!

进一步的用法,可以把Feedburner的feed播发到迷你博客上,然后在阅读器里直接订阅迷你博客帐号本身的Feed,这样在阅读器里就能看到原文的文章链接,绕过Feedburner重定向了。

使用Feedlr的这个实用功能非常简单,只要在新建向导的第二页,“输入Bot资料”里,选上这个选项就行了:image

Feedlr十八般用法之:把自己的博客同步到叽歪,饭否,和Twitter

Friday, March 28th, 2008

一眨眼3月初起公开测试的Feedlr马上要满月了!feedlr日志之前的文章基本都是记录feedlr开发中的一些经验和技巧,但是还没有中文介绍。从今天起这里会有一系列feedlr使用技巧的中文文章,希望通过这些介绍能让feedlr为各位朋友提供更多的用处,成为各位在网络上的好工具好朋友:)

首先,feedlr是什么呢?一句话来说,feedlr是一个能把任何网上的feed内容同步播发到迷你博客上的在线工具。

什么是feed呢?

大家注意到没有,你在浏览网络的时候,现在能越来越多的发现这样一个图标 每当你看到这个图标的时候,你就知道,现在浏览的这个网站有feed可以提供。feed其实是多种xml格式文件的统称,feed的用处是把网站的内容以xml格式抽取出来,这样其他程序就能方便的得到网站的内容,做出各种有用的应用出来了。

如果你已经是Google Reader,抓虾,鲜果等等Feed阅读器的用户,那么应该对Feed已经很熟悉了吧!

什么是迷你博客呢?

迷你博客的概念是由国外的Twitter建立的,简单的说就是能方便的发布一两句话到网上,让朋友能马上知道你在做什么的一种服务。国内也有不少迷你博客服务,最常用的有叽歪饭否

迷你博客虽然也叫做博客,但其实用处和传统的博客有很大不同,更多的其实是一种实时沟通平台。如果你注册了一个迷你博客,只是用来在网页上发表几句话的话,那么你大约只用到了10%的功能。而当你把自己的各种IM帐号,手机,等等联系工具都绑定到迷你博客上以后,你会发觉你和朋友之间的沟通方式彻底改变了。

那么,为什么要把自己的博客同步播发到迷你博客上呢?

有了自己的博客以后,最需要做的事情是什么?就是让更多的朋友知道你的博客,访问你的博客。这样做的方法也有不少。但是,进一步来说,怎么能让你的读者第一时间读到你博客的更新呢?读者阅读的方式一直都是“拉”的方式,也就是说,读者需要时不时地想起来在网上看一下你的博客有没有更新。如果一直忘了,或者上不了网,那就读不到你的最新大作了。

而迷你博客对你的关注者来说是一种可选的“推”的形式。只要关注者打开了通知,你的更新就能马上被送到关注者那里。可以通过电子邮件,IM,甚至离线的时候能通过短信发送到读者的手机。

那么如果能把博客文章自动发布到迷你博客上,读者不就能随时随地第一时间受到博客的更新了吗?这就是feedlr的基本功能!

怎么让feedlr把我的博客同步播发到我的迷你博客呢?

过程很简单,总结起来只需要3步:

  1. 登录或者注册以后,点击页面顶上的“新建”选项进入向导: image
  2. 输入Feed的地址,然后按“验证Feed”按钮:image
    Feedlr会很快的连接一下Feed来验证你填的地址是不是ok,验证没问题的话feedlr会帮你把标题和描述自动用Feed的信息填上。其余选项都是可选的,可以直接按image 继续
  3. 在第二页也是最后一页上,填好用于bot(机器人)的迷你博客帐号,选择一个迷你博客类型,然后同样按image让feedlr检查一下帐号确认没问题,最后按下提交按钮,你的feed机器人就搞定了!

问题1:”bot”,“机器人”到底指的是什么?

bot就是机器人的意思,特指一些网络机器人。在feedlr,一个bot帐号指的就是你用来播发Feed内容的迷你博客帐号。它本身就是一个普通的迷你博客帐号,但在登记到feedlr以后,就会自动播发feed更新,所以就可以称作一个迷你博客机器人(bot)了。

问题2:我该用什么帐号做bot呢?

由于目前迷你博客普遍的帐号验证方式,feedlr需要知道bot帐号的用户名和密码才能自动播发消息过去。feedlr在服务器上尽可能保证帐号的安全,但还是建议大家为bot专门安排一个迷你博客帐号,不要用自己的主帐号,给机器人一个“马甲”。不过当然,任何帐号都是可以的。

问题3:我填的帐号为什么总是验证出错?

有几种可能。一是你可能填错了用户名。一般来说,一个用户在迷你博客上有email地址,用户名,和完整姓名或者screen name这3种标识,在feedlr上填的应该是用户名,而不是其他两种。可以到迷你博客帐号设置里查一下就清楚了。

另外是中文用户名。目前为止feedlr还不能验证中文用户名。这个问题会尽快解决的,但是现在还是请用英文用户名作为bot帐号。

更新:feedlr可以验证中文用户名的帐号了,叽歪和饭否都支持中文用户名,但是Twitter本身是不支持中文名的。 

好了,现在你的第一个Feed广播机器人已经诞生,开始广播啦!

有想法就有可能!

Feedlr的功能尽可能做到通用,提供实用的选项,鼓励DIY,让大家自己灵活使用,来做出许多实用的机器人。同步自己的博客只是最基本的用法,只要你想的到,feedlr可以有更多有趣的用法!

今天先举一个feedlr的朋友在“可能吧”提出的利用feedlr让手机免费接受天气预报短信的文章,就是一个有趣而且实用的用法。已经有不少朋友在feedlr建了天气预报机器人,可以从这里这里看到。当然,除了文章介绍的饭否,同样可以在叽歪和Twitter上做同样的事情,而且据我经验叽歪的短信支持最好,可能更适合收天气预报。

叽歪还在官方博客上介绍了Feedlr,提到一些适合叽歪的特定功能。

好了,今天就介绍这些吧,以后还会介绍更多新鲜有趣的用法。大家如果想给我留言的话,不管是碰到问题还是建议意见,可以直接到这里发言噢!我马上就能收到你的消息了:)

How to create an interactive Twitter bot with Grails in 10 minutes

Tuesday, March 18th, 2008

A while ago I read this interesting post from Glen Smith about connecting to Google Talk with Grails. And I thought, “Wow, that’s really easy!” With merely a dozen lines of code you can already send messages via XMPP to Google Talk clients! And the other day I stumbled upon the Timer bot on Twitter. I thought, with Grails I really can do that in 10 minutes! So here it is, and you can do it in 10 minutes as well!

The way a twitter bot works

If you are already a Twitter geek, you can well skip to the next part. But if not, let’s first understand how an interactive bot on Twitter like the timer bot works.

Timer is a bot that can remind you with your message in a pre-defined amount of time. e.g. You send a direct message over Twitter to Timer like “d timer 45 call mom”, and it will send you a direct message back 45 minutes later, telling you “call mom”.

There are actually two ways to interact with Twitter.

  1. Using the REST API
  2. Talking to the Twitter IM bot directly over one of the supported IM networks.

We are talking about “interactive” bots here, which means that you can send some commands to it, and it reacts to your command. So we need to use the second approach. And since one can connect to Google Talk using the open XMPP protocol, talking to Twitter over Google Talk is apparently the better choice.

And here are the steps for some paperwork.

  1. Register a new user on Twitter to act as the bot account. Let’s name it “mytimer”.
  2. Log on as “mytimer”, and in the settings, add the Google Talk account which you are going to use for the bot to connect to Google Talk and “chat” with the Twitter IM bot. You need to log on to GTalk using this account and send the supplied verification code to the Twitter IM bot and get your GTalk account confirmed.

OK. Now we have our GTalk account bound to the Twitter bot account. We are going to write the code to make the bot alive.

The plan

Here’s how we are gonna do it.

  1. Connect to Google Talk service using the Smack XMPP library
  2. Listen to direct messages coming from the Twitter GTalk account twitter@twitter.com.
  3. When a direct message comes in, parse it and send the message back in the pre-defined amount of time.

A Twitter direct message sent over Google Talk looks like this:

Direct from {Sender_User_Name}:
{Message Body}

Reply with ‘d {Sender_User_Name} hi.’

A Twitter user name can only consist of letters, numbers, and “_”. So we can safely tokenize the message using spaces and newlines as separators. Also, the first and last line of the message are of the fixed format.

Our timer command spec is, described in regular expression:

^\d+(s|m|h)*\s.*$

e.g. 5m buy the milk

This command means to remind in 5 minutes with the reminder “buy the milk”. Accordingly, appended after the first number, “s” means seconds and “h” means hours.

The code monkey job

Now let’s see the code. First define the service settings in Config.groovy.

//File: Config.groovy
chat {
    serviceName = "gmail.com"
    host = "talk.google.com"
    port = 5222
    username = "bot@your.domain" //This is the Google Talk account you prepare for the bot
    password = "your_password" //The Google Talk password
}

Then create a service to hold the code, say, TwitBotService. It looks like this.

//File: TwitBotService.groovy
import org.codehaus.groovy.grails.commons.ConfigurationHolder as C
import org.jivesoftware.smack.Chat
import org.jivesoftware.smack.ConnectionConfiguration
import org.jivesoftware.smack.Roster
import org.jivesoftware.smack.XMPPConnection
import org.jivesoftware.smack.packet.Message
import org.jivesoftware.smack.PacketListener
import org.jivesoftware.smack.filter.PacketFilter
import org.jivesoftware.smack.filter.PacketTypeFilter
import org.jivesoftware.smack.packet.Message
import org.jivesoftware.smack.packet.Packet
import org.jivesoftware.smack.PacketListenerclass

TwitBotService {
    boolean transactional = false
    XMPPConnection connection

    def connect() {
        ConnectionConfiguration cc = new ConnectionConfiguration(
        C.config.chat.host,
        C.config.chat.port,
        C.config.chat.serviceName)
        connection = new XMPPConnection(cc)
        PacketFilter msgFilter = new PacketTypeFilter(Message.class)

        def myListener = [processPacket:{ packet ->
            log.debug "Received message from ${packet.from}, subject: ${packet.subject}, body: ${packet.body}"
            def msg = packet.body
            if(!msg) return
            def words = msg.tokenize(' \n')
            //Direct message body
            def body = words[3..-6]
            //The Twitter user who sent the command, and we are going to reply to
            def to = words[2][0..-2]
            //The amount of time delay
            def delay = words[3]
            switch(delay[-1]){
                case 's':
 	       log.debug "Delay in seconds"
 	       delay = delay[0..-2].toInteger()
 	       break
               case 'm':
 	       log.debug "Delay in minutes"
 	       delay = delay[0..-2].toInteger()*60
 	       break
 	  case 'h':
 	       log.debug "Delay in hours"
 	       delay = delay[0..-2].toInteger()*3600
 	       break
 	  default:
 	       //default unit is minute
 	       delay = delay.toInteger()*60
            }
            //The reminder text to send back
            def reminder = body[1..-1].join(' ')
            if(delay instanceof Integer){
                new Timer().runAfter(delay*1000){
 	       //Send back direct message 'd user message'
 	       def cmd = 'd ' << to << ' ' << reminder
 	       sendChat(packet.from, cmd.toString())
 	   }
            }
       }] as PacketListener
       try {
            log.debug "Connecting to server..."
            connection.connect()
            connection.login(C.config.chat.username,C.config.chat.password)
            connection.addPacketListener(myListener, msgFilter)
            log.debug "Connected to server"
       } catch (Exception e){
        	log.error "Connection failed: $e.message"
       }
    }

    def disconnect(){
        connection.disconnect()
    }

    def sendChat(String to, String msg) {
        try{
            Chat chat = connection.chatManager.createChat(to, null)
            def msgObj = new Message(to, Message.Type.chat)
            msgObj.body = msg
            chat.sendMessage(msgObj)
        } catch (Exception e) {
            log.error "Failed to send message"
        }
    }
}

And finally, to start the bot with your grails app, simply add a call to TwitBotService#connect() in BootStrap.groovy.

//File:BootStrap.groovy
    def gTalkService
    def init = { servletContext ->
        gTalkService.connect()
    }
    def destroy = {
        gTalkService.disconnect()
    }

So here it is, your own timer bot recreated (with a bit more features) in 10 minutes, quick and dirty. And with the same approach, you can create all sorts of interactive Twitter bots.

P.S. Why would you create a Twitter bot while you can create a Google Talk bot in the same way?

Well, a GTalk bot can only interact with online GTalk users. But by leveraging the Twitter platform, users can interact with your bot either online with GTalk, or offline with mobile SMS. And that all comes with no cost at all. Ain’t that cool?

Deploying Grails - Things I learned from Feedlr

Monday, March 10th, 2008

When I was about to deploy Feedlr on my VPS server there wasn’t much information about the deployment of Grails out there. I know that deploying a Grails war is just like a Java war. But are there going to be any differences? How much memory do I need? Will it eat any more hardware than a plain old Java app? I really have no idea. So this post is about the server stuff I’ve learned by doing the deployment for Feedlr.

I started by deploying my Grails app on Tomcat 6 on a 256M RAM VPS. And since my blogs are also hosted on this VPS, I use mod_proxy_ajp to connect Tomcat to Apache for Feedlr.

My first attempt on the 256 VPS turned out very badly. When I started Tomcat with Feedlr deployed, the server simply becomes very unresponsive and swap activity started flying. That’s a bad sign for “low memory”, I think. So I upgraded the server to 512M RAM and things began to look normal.

Then comes the optimization part. I really don’t have much server tuning experience, especially with a limited resource VPS. So all this is new and interesting to me, solving issues one by one.

The first bad thing happened was that whenever Tomcat ran for a while, like a couple hours, the whole VPS box began to eat a low of swap, and connection will often timeout. I suspected Tomcat, but it turned out to be Apache. The default installation on Ubuntu for Apache + PHP5 uses the Apache mpm_prefork module. The problem is that mpm_prefork spawns multiple Apache instances to handle requests to the server. And the default config is like

<IfModule mpm_prefork_module>
StartServers 5
MinSpareServers 5
MaxSpareServers 10
MaxClients 150
MaxRequestsPerChild 0
</IfModule>

The default config worked fine when I was only hosting a couple blogs on the 256 box. But with a Grails app on Tomcat running, memory becomes critical. So then I changed the settings to smaller numbers and it has been working fine now.

<IfModule mpm_prefork_module>
StartServers 2
MinSpareServers 2
MaxSpareServers 4
MaxClients 150
MaxRequestsPerChild 0
</IfModule>

I will replace mpm_prefork with mpm_worker module since the latter is more efficient for concurrency. Aptitude on Ubuntu can’t figure out for you how to put mpm_worker and PHP5 together so there are some config work to do. Currently I’m running mpm_worker on my development/staging server and it’s working fine but I just don’t know how it’s gonna behave on a low memory environment. BTW. If you have anything to say about this I’m all ears.

Then it’s about Tomcat. Actually Tomcat is working great with Grails out of the box. The tuning is all about memory and efficiency. Here’s my Tomcat startup opts:

CATALINA_OPTS=”-server -Xmx420m -Xms420m -XX:MaxPermSize=128m -Dcom.sun.management.jmxremote”

I set the heap size to 420M since I want to leave some room for my blogs. And so far this setting has worked fine for me. The “-Dcom.sun.management.jmxremote” part is for LambdaProbe which I’m using as the server management app.

Besides, for the production environment, it’s better to set reloadable=”false” for the Tomcat Context of the app. It should be false by default. In this way new code won’t be reloaded on the fly, but it saves quite some server overhead which can be notable on a low memory environment.

And finally, server logging. Here’s my logging settings in Config.groovy. Log files will be saved daily for stdout and stacktrace.

log4j {
    appender.stdout = "org.apache.log4j.ConsoleAppender"
    appender.'stdout.layout'="org.apache.log4j.PatternLayout"
    appender.'stdout.layout.ConversionPattern'='[%r] %c{2} %m%n'

//    	rolling file logger
    appender.logfile = "org.apache.log4j.DailyRollingFileAppender"
    appender.'logfile.File' = "/path/to/public_html/feedlr.com/logs/feedlr_grails.log"
    appender.'logfile.layout' = "org.apache.log4j.PatternLayout"
    appender.'logfile.layout.ConversionPattern' = '%d{[ dd.MM.yy HH:mm:ss.SSS]} [%t] %-5p %c %x - %m%n'

    appender.errors = "org.apache.log4j.DailyRollingFileAppender"
    appender.'errors.layout'="org.apache.log4j.PatternLayout"
    appender.'errors.layout.ConversionPattern'='%d{[ dd.MM.yy HH:mm:ss.SSS]} [%t] %-5p %c %x - %m%n'
    appender.'errors.File'="/path/to/public_html/feedlr.com/logs/feedlr_grails_stacktrace.log" //prod setting
    rootLogger="error,stdout"
    logger {
        grails="info,stdout,logfile" //prod settings
        StackTrace="error,errors"
        org {
            codehaus.groovy.grails.web.servlet="info,stdout,logfile"  //  controllers
            codehaus.groovy.grails.web.pages="info,stdout,logfile" //  GSP
            codehaus.groovy.grails.web.sitemesh="info,stdout,logfile" //  layouts
            codehaus.groovy.grails."web.mapping.filter"="info,stdout,logfile" // URL mapping
            codehaus.groovy.grails."web.mapping"="info,stdout,logfile" // URL mapping
            codehaus.groovy.grails.commons="info,stdout,logfile" // core / classloading
            codehaus.groovy.grails.plugins="info,stdout,logfile" // plugins
            codehaus.groovy.grails.orm.hibernate="info,stdout,logfile" // hibernate integration
            springframework="off,stdout,logfile"
            hibernate="off,stdout,logfile"
        }
    }
    additivity.StackTrace=false
}

P.S. I’m using SliceHost’s VPS and it’s great. I especially love the on-demand slice resizing which will automatically resize your VPS in minutes, with data completely intact. So you can totally start with a frugal setup and grow as needed. And did I mention completely full control over your server? I won’t say more but I know your inner geek would love it once you use it. So if you are seeking some good server space for your project why not give it a try with my referral link :)

Feedlr is now alive!

Saturday, March 1st, 2008

Hi friend, my latest pet project feedlr has gone alive! After testing the initial version with friends, I’ve decided to make Feedlr open to the public now.

Feedlr is a service inspired by twitterfeed, built with Grails. It is both an automatic feed tweeting service like twitterfeed, and a bot showcase for miniblog users to browse, find, and follow interesting bots on popular miniblog services. Currently, Feedlr supports miniblogs including Twitter, JiWai, and FanFou. The idea is to make Feedlr a thriving farm of miniblog bots where anyone can create or adopt cute bots.

You can use Feedlr to broadcast feeds to multiple miniblogs

To do this, please first sign in or sign up for an account, and then click the “New” tab on the top and follow the simple wizard.

The wizard will ask for “bot accounts”, which means miniblog accounts used to broadcast the feeds. You should have already registered an account on a supported miniblog and provide the account information to the wizard.

A note on privacy here. Since the way miniblog API’s are designed, Feedlr has to have your bot login information in order to do the posting. So it’s strongly recommended that bot creators don’t use their personal accounts as bot accounts, but create dedicated accounts for the bot service only.

Feed creators can choose to attach a feed up to 2 bots at a time, choosing among the 3 currently supported services.

Feedlr is also a bot showcase

In addition, you can simply browse for bots on Feedlr and find interesting ones to follow. Once you want to follow a certain bot, there are two ways to follow it and both should take less than 10 seconds.

First, you can simply click the bot avatar and visit the bot’s homepage on the miniblog. Then just follow the bot as how you do it with the miniblog service.

Second, you can click the “Follow” hover menu. You will be prompted to enter your own login for the miniblog service where the bot is hosted. To ensure privacy and security, Feedlr does not record you login in any way. It’s strictly used to follow the bot for this one time.

Feedlr loves feeds, miniblogs, and you!

So start by feeding it with more feeds:)

If you have any comments, find any bugs, want any new features, or just want to say something, please leave a word or two by replying to this blog post. Feedlr is a new born baby. Love and caring makes it grow healthily and fast!