___It was with some loss of productivity yesterday that I found out that if a Yahoo Pipes construct will not run, for whatever reason, you cannot save the pipe. This is not intuitive. In any programming language, word processing or other user initiated computer work, one can save interim work, regardless of correctness or not. Please imagine while working on an essay the word processor counts your spelling errors (or what it feels is a spelling error), determines that there are too many, and thus deny you the ability to save your interim work. “You must fix these errors right now” it would scream. I felt like whupping Pipes in the butt yesterday, but unfortunately one hole looks the same as the other. Indecision ensued.

___It is only in juggling that interim work cannot be saved. You either finish gracefully, or you, eh, dropped the ball, so to speak. Maybe Pipes developers also juggle.

___Not only can you not save your work if a construct does not work, but the error message is quite misleading. “Oops: System error. Problem parsing response.” Talk about your stereotypically misleading error message. I thought we IT people had evolved much further than this. An error message should be displayed as a response to a user action. The disconnect here is that my action was saving my Pipe, and the error is not directly connected to this act. Is it too much to ask that the error message indicate the origin of the error? How about displaying this message when I run the Pipe and it does not work? And even if something is not correct, I’ll fix it later. Just let me save my work. But I digress.

___As I continued to work on my Pipes I kept running them and looking at the Really Simple Syndication (rss) output feeds. For some reason, some news articles in my feeds were not coming out with summaries, but with the complete article, some multiple pages long. It’s not only counterproductive but annoying. I research rss 2.0 feeds and find the article summary should be in a field called item.description. I therefore put in a command that truncates item.description to no more than 400 characters. This does not work, and full length articles still appeared.

___After a couple more hours trying to crack this nut, I try to find a tool on the internet that can display rss record by record, field by field. I cannot. It turns out that within Pipes there is a display at the bottom of the screen. If you click on an individual news item, all their field names and contents are displayed. This function is as useful as it is impressive.

___I find out that, to my chagrin, all rss 2.0 feeds are not created equal. Rss feeds also may not adhere to the supposed standard. Looking feed by feed, I found out that some feeds put their summary into item.description, while others use the variable item.content:encoded. Those feeds that use item.description are written according to the rss standard, have no item.content:encoded field. Conversely those that used item.content:encoded have no variable called item.description. I was following the standard and truncating item.description for all feeds. If there was no such variable in a feed, a Pipes error occurred, thus not allowing me to save my interim work. To complicate matters, some feeds had both variables, but only used one.

___I therefore needed two different rules for the summary. For those that use item.description, limit this field to to 400 characters. Similarly, for those that use item.content:encoded, limit this field. Finally, both Pipes and I were satisfied, and I was able to save my work.

___While I’ve progressed greatly, there still are issues I am working on. My DonTai.com cron job seems to fail when pulling in news feeds from my China Japan News Yahoo Pipes. It seems I need to try a couple of times before the cron job can suck in all the news. It’s possible that this is an issue because I’m playing with the feeds and then sucking in the completely aggregated data. Once initially sucked in there should be less residual load. There is also the setting of the frequency of which news aggregates and how this synchronizes with the cron jobs. A news feed that is not yet scheduled to run will not run on a cron job. A news feed that is scheduled to run is then dependent on the next cron job.

___I question the robustness of Pipes. Will it finish a query and run to completion, or will it time out and leave me hanging. Should I cut up a large Pipes query into smaller ones? Maybe I need to research how to get pipes to run as efficiently as possible. On my Pipes diagrams I’ve already eliminated any redundancy I could find. I’ll let it run for a while and see.

___I happen to enjoy keeping up with news on China and Japan, but with so many sources, checking them individually is madness, and going to Google will drive you to use psychotic drugs. News about China and Japan in the local papers is scarce. There is always more space devoted to local issues, so world news gets pushed to the back of the queue, often not making the papers. By gathering news from around the world you gain different perspectives on news. I do like the objectivity of the British press. Both the Canadian and American press can warp stories one way or another.

___My search to filter rss feeds stopped at the door of Yahoo Pipes. After a brief video it was pretty easy to use and relatively intuitive. I threw together 26 rss feeds, filtered by keyword, eliminated duplicates, and sorted by descending order of date. My China Japan News feed was finished.

___My sources include: Google, NY Times, CNN, BBC, Danwei, Washington Post, Wall Street Journal, Times of India, Financial Times, International Herald Tribune, Asia Times, Asahi, Toronto Star, Globe and Mail, Japan Times, LA Times, The Straits Times, Channel News Asia (Singapore), The Star (Malaysia), Jakarta Post. Are there any other Asian flavours I should add?

___Some of my feeds are not cooperating and gave me errors:

warning error fetching http://feeds2.feedburner.com/latimes/asia (502 Bad Gateway)
warning error fetching http://feeds.washingtonpost.com/wp-dyn/rss/world/asia/index_xml (502 Bad Gateway)
warning error fetching http://rss.cnn.com/rss/cnn_world.rss (502 Bad Gateway)


___Oddly, some online newspapers do not provide rss, though they are mostly in Asia. These include:

• People’s Daily: RSS feed does not work
• China Daily, Yomiuri Shumbun: have no RSS feed. The Yomiuri Shimbun subscribe link is for a delivered paper copy subscription
• Asahi Shimbun’s english section: gives an RSS in Japanese
• Chicago Tribune: has no RSS. Maybe Obama will fix this

___I’ve fed this new feed into DonTai.com‘s news aggregator and it displays very nicely. I’ll play with it for a couple of days and see if the pipe breaks. Overall I’d give Yahoo Pipes a B+ for user interface and usability, and a B for bugs. Often the news preview and the save functions were not working, but you wait long enough and it fixes itself. The bottom line is that the functions of Yahoo Pipes work and work very well.

P.S. I just added The Straits Times, Channel News Asia (Singapore), The Star (Malaysia), Jakarta Post. I also notice that I must be logged into Yahoo mail or I cannot access Yahoo Pipes. Yahoo Pipes have their own login, but it won’t let me authenticate with my valid Yahoo account. Logged into Yahoo Mail I bypass Yahoo Pipes’ authentication.