In response to some complaints from my hosting company and registrar that we were using too much CPU and bandwidth, I’ve moved feedhaus from a virtual dedicated server to a dedicated server.
I think Lucene was the culprit — as FUD was indexing feeds’ articles, Lucene was consuming WAY too much CPU. Anyway, after a 24 FUDless hours, FUD is happily chugging away to bring you new stories from the 70+ feeds we now have in feedhaus.
I’m very happy to say that I was able to move feedhaus and FUD over to the new hardware with only 30 minutes of web site downtime. W00t!
This incident led me to thinking a bit about scaling feedhaus. My fears about the scalability of cometd/long-polling are probably unwarranted. What I should really worry about is FUD. I think ultimately FUD needs to be separated out from the web server machines so that it doesn’t interfere with web site performance. Furthermore, I think the feed table should probably be broken into segments and there should be a new FUD process instantiated for every 100 (or so) feeds.
I guess FUD and I will cross that bridge when we get to it . . . but for now, performance is snappy as ever AND you’re getting your news in near-realtime. Enjoy!