Al Jazeera covers news on China. Nice!

The year 2009 was marked by significant news coverage of Al Qaeda attacks, soldiers and civilians getting blown up in Iraq and Afghanistan, much stricter airport security and, here in Canada, increased wariness of those from the Middle East. In the middle of this morass was the news agency Al Jazeera. Most people here in Canada actually did not recognize the agency until tapes from Osama Bin Laden were released only to Al Jazeera.

We have people of Middle Eastern heritage here in Toronto, but so does many other parts of the world. With all the negative publicity coming out of the Middle East, it is not unreasonable that Canadians would be wary of the news agency. When they applied for a TV station license there was an uproar, saying that Al Jazeera was the mouth piece of Al Qaeda and Osama Bin Laden. I don’t know much about Al Jazeera, the Middle East or Al Qaeda, but I am glad they received their CRTC license. News coverage from the Middle Eastern perspective can only enlighten Canadians.

Here is what Wikipedia explains on Al Jazeera:

Al Jazeera (Arabic: ???????? al-?az?ra IPA: [ald?azi:ra]), meaning “the Island” in Arabic, is a Middle Eastern news network. The name refers to the network’s geographic origins[1].Al Jazeera is a television network headquartered in Doha, Qatar.

Here is a news article from Al Jazeera covering Chinese New Year in China. They write nothing extraordinary but I appreciate their coverage none the less. A Middle Eastern perspective on China would also be great, but I am having difficulty finding reputable news sources. At least I recognize Al Jazeera.

New perspectives are always good. It is interesting to see “Mecca Time” on Al Jazeera’s web site. And welcome to Canada, Al Jazeera.

CTV News helicopter buzzes my house at 07:00am. They are LOUD.

Today I woke up at 06:30 to a morning of lovely sunshine and moderate 16°C temperatures. At 07:00 a helicopter begins hovering over my house, and does this for 30-40 minutes. Helicopters in my locale, means there’s trouble in the neighbourhood.

CTV News helicopter buzzes my house at 07:00am. They are LOUD.

Helicopters, even if they seem far up in the air, are quite noisy. It was noisy enough for me to go outside with my binoculars to try to ID the critter, and then to grab my camera and shoot a few off. The CTV helicopter call sign was “C-FCTV”. The helicopter has a very large camera and swivel on the front of the bird. This setup reminds me of the heat seeking or night vision FLIR (Forward looking infrared) systems used on military and police helicopters and airplanes.

The last time a helicopter was in the neighbourhood was when a guy was killed one block to the east. That was a police helicopter keeping an eye out over the neighbourhood.

A couple of hours later on the news I learned some guy got shot twice a block west and a little north. At 06:30 in the morning. I figure that people should calm themselves down and enjoy the beautiful day. Wishful thinking.

I still remember when I was in Beijing during Tiananmen Square in 1989, an army helicopter dropped leaflets over my university telling students to not protest and to behave themselves. That tact certainly did not work, but it was sure rare to see a helicopter then. All the Chinese students dropped everything, including their lectures and ran outside. Class dismissed!

Drupal 6's Feed Aggregator works well

I really love Drupal 6′s feed aggregator. It has the ability to aggregate numerous RSS (Really Simple Sybndication) news or blog feeds, categorize them, and keep them current, all in the effort to save you the trouble of going to each news source and hunting for relevant news. All you have to do is chose a category, read the summary and click the link for the article you what you want.

___What the feed aggregator will not do is filter your feeds by keyword. As well, when the RSS feeds are displayed, they can have short descriptions or full article descriptions. These full article descriptions clutter your screen, which I did not want. All I wanted was a link, a short description, and no more. When you are looking at hundreds of articles, less is indeed more.

___My solution was to use Yahoo Pipes. My current Pipes help aggregate 40 blogs (mostly from China), 40 news feeds (world wide), as well as aggregate about 100 other blog sites fed from yet another Drupal site (mostly from Asia). Yes, you can daisy chain Drupal news aggregators! With Yahoo Pipes I was able to filter by keyword, as well as go down to the RSS variable level and truncate full article descriptions. I was also able to eliminate duplicate entries and sort by article date.

___The difficulty was that my cron jobs were not completing and therefore not updating my feeds. I would get a MySQL server has gone away error, which means that my Yahoo Pipes were taking too long to complete. My Site5 host provider said that MySQL has a timeout of 15 seconds. One timeout will kill the cron job, leaving the other news feeds not updated. The Cron jobs were returning 1.2MB to 2.5MB error logs of not very helpful information. Web searches yielded very little on the MySQL error message, how to get my cron jobs to complete, and on increasing the efficiency of Yahoo Pipes.

___My original cron strategy was that I only needed to update my feeds every 3 hours, so I scheduled cron as such. What happens is that all the feeds need to be updated at once, the cron job gets overwhelmed, dies, and most of the feeds do not update.

___An alternative cron strategy proved to be the solution. I scheduled cron to run hourly, even though my feeds did not need to be updated no more than every 3 hours. A staggering pattern organically emerged. Those jobs that were easily completed were quickly done, and did not need to be updated for the next 2 cron jobs. Those jobs that took longer initially timed out. During the next cron job, because the easy feeds were already completed, there were far fewer feeds to run, so the cron jobs could concentrate on these stubborn few. In the end they were also completed.

___I used some other Yahoo Pipes strategies. I broke up one larger Yahoo Pipe into two, though I think I will try merging them together later. These larger Pipes should be run hourly, even though they may only change every 3 hours. If run hourly they will return with fewer updates but more quickly and therefore will have less of a tendency to time out.

___Take a look at my feed aggregator at DonTai.com and my Yahoo Pipes if you’re interested.

___I tried other alternate RSS solutions but they did not fulfill my requirements. Google Reader will aggregate RSS feeds but does not provide any filtering capability. Feedrinse will aggregate RSS feeds and filter, but there is no way to truncate full length RSS feeds. I could use a feed reader, but then would not be able to share my aggregated feeds with multiple people on the internet. A feed reader may be better for those who WANT the full article anyway.

___It was with some loss of productivity yesterday that I found out that if a Yahoo Pipes construct will not run, for whatever reason, you cannot save the pipe. This is not intuitive. In any programming language, word processing or other user initiated computer work, one can save interim work, regardless of correctness or not. Please imagine while working on an essay the word processor counts your spelling errors (or what it feels is a spelling error), determines that there are too many, and thus deny you the ability to save your interim work. “You must fix these errors right now” it would scream. I felt like whupping Pipes in the butt yesterday, but unfortunately one hole looks the same as the other. Indecision ensued.

___It is only in juggling that interim work cannot be saved. You either finish gracefully, or you, eh, dropped the ball, so to speak. Maybe Pipes developers also juggle.

___Not only can you not save your work if a construct does not work, but the error message is quite misleading. “Oops: System error. Problem parsing response.” Talk about your stereotypically misleading error message. I thought we IT people had evolved much further than this. An error message should be displayed as a response to a user action. The disconnect here is that my action was saving my Pipe, and the error is not directly connected to this act. Is it too much to ask that the error message indicate the origin of the error? How about displaying this message when I run the Pipe and it does not work? And even if something is not correct, I’ll fix it later. Just let me save my work. But I digress.

___As I continued to work on my Pipes I kept running them and looking at the Really Simple Syndication (rss) output feeds. For some reason, some news articles in my feeds were not coming out with summaries, but with the complete article, some multiple pages long. It’s not only counterproductive but annoying. I research rss 2.0 feeds and find the article summary should be in a field called item.description. I therefore put in a command that truncates item.description to no more than 400 characters. This does not work, and full length articles still appeared.

___After a couple more hours trying to crack this nut, I try to find a tool on the internet that can display rss record by record, field by field. I cannot. It turns out that within Pipes there is a display at the bottom of the screen. If you click on an individual news item, all their field names and contents are displayed. This function is as useful as it is impressive.

___I find out that, to my chagrin, all rss 2.0 feeds are not created equal. Rss feeds also may not adhere to the supposed standard. Looking feed by feed, I found out that some feeds put their summary into item.description, while others use the variable item.content:encoded. Those feeds that use item.description are written according to the rss standard, have no item.content:encoded field. Conversely those that used item.content:encoded have no variable called item.description. I was following the standard and truncating item.description for all feeds. If there was no such variable in a feed, a Pipes error occurred, thus not allowing me to save my interim work. To complicate matters, some feeds had both variables, but only used one.

___I therefore needed two different rules for the summary. For those that use item.description, limit this field to to 400 characters. Similarly, for those that use item.content:encoded, limit this field. Finally, both Pipes and I were satisfied, and I was able to save my work.

___While I’ve progressed greatly, there still are issues I am working on. My DonTai.com cron job seems to fail when pulling in news feeds from my China Japan News Yahoo Pipes. It seems I need to try a couple of times before the cron job can suck in all the news. It’s possible that this is an issue because I’m playing with the feeds and then sucking in the completely aggregated data. Once initially sucked in there should be less residual load. There is also the setting of the frequency of which news aggregates and how this synchronizes with the cron jobs. A news feed that is not yet scheduled to run will not run on a cron job. A news feed that is scheduled to run is then dependent on the next cron job.

___I question the robustness of Pipes. Will it finish a query and run to completion, or will it time out and leave me hanging. Should I cut up a large Pipes query into smaller ones? Maybe I need to research how to get pipes to run as efficiently as possible. On my Pipes diagrams I’ve already eliminated any redundancy I could find. I’ll let it run for a while and see.

___I happen to enjoy keeping up with news on China and Japan, but with so many sources, checking them individually is madness, and going to Google will drive you to use psychotic drugs. News about China and Japan in the local papers is scarce. There is always more space devoted to local issues, so world news gets pushed to the back of the queue, often not making the papers. By gathering news from around the world you gain different perspectives on news. I do like the objectivity of the British press. Both the Canadian and American press can warp stories one way or another.

___My search to filter rss feeds stopped at the door of Yahoo Pipes. After a brief video it was pretty easy to use and relatively intuitive. I threw together 26 rss feeds, filtered by keyword, eliminated duplicates, and sorted by descending order of date. My China Japan News feed was finished.

___My sources include: Google, NY Times, CNN, BBC, Danwei, Washington Post, Wall Street Journal, Times of India, Financial Times, International Herald Tribune, Asia Times, Asahi, Toronto Star, Globe and Mail, Japan Times, LA Times, The Straits Times, Channel News Asia (Singapore), The Star (Malaysia), Jakarta Post. Are there any other Asian flavours I should add?

___Some of my feeds are not cooperating and gave me errors:

warning error fetching http://feeds2.feedburner.com/latimes/asia (502 Bad Gateway)
warning error fetching http://feeds.washingtonpost.com/wp-dyn/rss/world/asia/index_xml (502 Bad Gateway)
warning error fetching http://rss.cnn.com/rss/cnn_world.rss (502 Bad Gateway)


___Oddly, some online newspapers do not provide rss, though they are mostly in Asia. These include:

• People’s Daily: RSS feed does not work
• China Daily, Yomiuri Shumbun: have no RSS feed. The Yomiuri Shimbun subscribe link is for a delivered paper copy subscription
• Asahi Shimbun’s english section: gives an RSS in Japanese
• Chicago Tribune: has no RSS. Maybe Obama will fix this

___I’ve fed this new feed into DonTai.com‘s news aggregator and it displays very nicely. I’ll play with it for a couple of days and see if the pipe breaks. Overall I’d give Yahoo Pipes a B+ for user interface and usability, and a B for bugs. Often the news preview and the save functions were not working, but you wait long enough and it fixes itself. The bottom line is that the functions of Yahoo Pipes work and work very well.

P.S. I just added The Straits Times, Channel News Asia (Singapore), The Star (Malaysia), Jakarta Post. I also notice that I must be logged into Yahoo mail or I cannot access Yahoo Pipes. Yahoo Pipes have their own login, but it won’t let me authenticate with my valid Yahoo account. Logged into Yahoo Mail I bypass Yahoo Pipes’ authentication.