Archive for the Project Category

Reading Less Is Reading More

| October 7th, 2009

If information is what drives you to the internet, like me, you might be spending roughly 60-70% of your time online reading blogs, news and feeds (not to forget twitter). For me at least, reading online has superseded email (and updating social networks) as the most time consuming activity. And yet everyone is busy generating more content rather than finding a solution to consume all this information. We are trying to tackle this problem precisely with Dygest. At its core Dygest is a summarization engine that tries to sift through all the noise and present only the *real* content/news contained in any (news) article/text. Recently, we released an experimental version of a feed summarizer that uses the Dygest engine to summarize blogposts/news for any RSS/ATOM feed. This summarized feed can be subscribed in any feed reader like Bloglines, Google Reader etc.

NOTE: A feed that has not been encountered by our system ever before should be summarized in a couple of minutes.

Feed Summarizer

On the whole with Dygest, reading blogs has now become much faster, much more concise and consuming information has become a great deal easier. Imagine the time saved reading the summarized version as compared to the original post (also you are not overwhelmed with useless information). See for yourself below:

Original Post

Original Post

Summarized Post

Summarized Post

While you might have the urge to head over to Dygest and summarize your entire subscription list on Google Reader, I would recommend reading this post a bit further for some real cool stuff we have in store. If you must though – click here to Dygest.

Summarizing Your Twitter Links

Readtwit is a really cool service launched recently, which extracts links from your twitter feed and packages them in a clean RSS format. The awesome combination of Readtwit along with Dygest yields a summarized twitter feed delivered to your favorite feed reader.

Steps to get a summarized twitter feed:

(1) Sign into Readtwit.
(2) Copy the link on the ‘Get me the feed’ button:

(3) Paste this link into the Dygest interface and subscribe to the summarized feed returned in your favorite feed reader.

More To Come

This is just an experimental release of Dygest and so do send in your feedback on the summaries and help us improve. In the coming months we are working on improving the algorithms and churning out other great applications of Dygest (there is something really cool in the works). So while we are busy teaching computers to read, Dygest your feeds – because reading less is reading more.

Follow us on twitter – @dygest

Dygest Your Search

| March 19th, 2009

Update: This hack won the coveted ‘Search’ category award.

For the last couple of days, I and @sudheer_624 have been busy working on this hack for a Yahoo! Hackday. Although still a prototype, the hack has turned out to be interesting so we thought of putting it out for others to play around with.

Dygest (pronounced as ‘digest’ – thanks to @bluesmoon) is aimed at changing the conventional way of displaying search context via a snippet to a more informative, machine generated document summary. There two kinds of relevance for evaluating search results:

  • Vertical relevance: determined by the ranking algorithms.
  • Horizontal relevance: the contextual information made available to the user about the result – Searchmonkey is a good initiative on this front.

The current way of displaying this context is via a snippet of text under every result. This snippet shows the neighborhood of the occurrence of the query terms. Usually this information is not rich enough for a searcher to make the right judgement about the result. This causes the searcher to switch back and forth between the documents and the search results if the the page is not relevant. This can be frustrating at times.

Dygest aims to solve this by either replacing or enhancing the current search snippet with a summary of the result page. At its core lies a summarization engine which figures out what the *real* content of the page is (distinguishing it from the other junk like surrounding text, navigational text, comments etc) and then performs text summarization on this content. The summary of the page is then displayed to the user via the appropriate interface. How cool is that?

The user no longer needs to click on irrelevant links. He/She can perceive the theme/important facts of the page from right within the results page. The other advantage of this is that it gives the user a good overview of the query topic – he no longer needs to spend time reading many long documents but rather read a few summaries from the top results to get a good overview of the subject. This is particularly well suited for mobile devices where its frustrating to switch back and forth between pages and the search results. This is also fit for news articles where we just need the important facts about the story.

Well, here is an example to convince you. A search for ‘Carol Bartz’ yields the following result which at the first glance is not at all informative.

Enhancing the existing view with an abstract of the page helps gauge the content and theme of the document. This would now look like:

Dygest outputs the following summaries for the query ‘Iran‘ restricted to Yahoo! News:

And following for ‘Obama stimulus plan‘:

Currently, Dygest has two interfaces – (1) a search interface powered by yahoo boss and (2) a searchmonkey plugin. Its just a prototype so be kind and don’t be too judgmental.

Start dygesting here.

I started working on my second weekend project, guess I’ll do something small every week. This one is an extension to LifeLogger. The aim is to analyze ones daily and weekly browsing history and extract themes which could aid in recommendations. It is still a ‘work in progress’ – currently I have been able to generate the following visualizations:

The following visualization depicts the dominant keywords/topics for one day (the terms are stemmed):

I had been reading a couple of Yahoo! related articles and visualization blogs. This is captured by the above visualization – but there is still alot of noise which I need to get rid of.

The next visualization depicts the linkages and clusters for the keywords. There exists a link between two terms if they occur in the same document. [may take sometime to load - you'll need to zoom in to get a better look - click on 'compute layout' if the clusters don't show]

Both the above visualizations depict important metrics that could be used to extract dominant themes from the browsing history. Dominance should not be just inferred from frequency but also from the prevalent of a term across multiple pages. I still need to work on removing noise and running this on larger datasets like browsing history for a week or so. If you have any ideas or good papers to recommend that would be nice.

Although I started this project as an experimental weekend thingy (to play around with Google App Engine), the project has shaped up well. Before you surf over to another blog, wondering what the hell I’m talking about, let me introduce you to “Personalized ARTICLE” aggregator (read as PARTICLE). The aim is to personalize a users online reading (just like what Findory did). Findory was an excellent service and I’ll be glad if I can achieve even an iota of what Greg created. This project is at very rudimetary and experimental stage. Rather than tapping into the users reading history on the site (monitored by the links clicked), the idea is to study how a users *interests*, scattered around at various “databases of interest” like, could be used to personalize online reading (news articles, blogs and more). This way the user could merrily browse the world wide web, bookmarking pages, doing his usual stuff and let PARTICLE worry about making this data useful.

Click here to try PARTICLE

Presently you need to provide PARTICLE with your username, which it uses to analyze your *interests* and present you with recent news stories you may like. It works well if you have a decent number of bookmarks in As I mentioned, the project is at a very rudimentary stage, so don’t feel disappointed by the results (ah! the unlucky few). I encourage you to play around with the app and recommend it to others to try. I’ll be making many changes/additions in the coming weeks.

Test drive PARTICLE at Kindly leave your feedback/comments/suggestions in the comments or send me an email at ‘anand at’.

[UPDATE] Yahoo! Research has a similar project called Garçon.