Archive for August, 2009

Web Content Extraction Dataset

| August 22nd, 2009

For a recent project, we (sudheer_624 and I) have had to deal with developing algorithms to extract the true content from any given web page. By true content I mean the text excluding the ads, navigational links/text, etc even excluding comments (if any). Thus, given a blog post we are interested in extracting just the content of the post and not the comments and other surrounding text. We did not come across any dataset for the given task that would let us evaluate our algorithms. We recently generated our own dataset for this purpose and would like to share it with anyone tackling a similar problem.

The dataset contains the html source and text content (true content) for around ~4000 webpages. One metric to measure your algorithm against this dataset could be the edit distance. If you do use this dataset, it would be great if you could share the results of your algorithms for benchmarks to compare against. I’ll be updating this post with the accuracy of our algorithm soon enough.

Download the dataset here (gzipped)

`Fact`orize Your Search

| August 14th, 2009

Dygest and a hackday later, @sudheer_624 and I (@semanticvoid) are back with ‘dfacto’, codename for our latest search hack for Yahoo! Hackday Summer 2009.

I think that search is undergoing a paradigm shift – its no longer about who presents the best ten blue links but now more about presenting the answers upfront. Dfacto (pronounced as ‘de facto‘, Latin for ‘by [the] fact‘) is aimed at addressing this issue. A large percentage (nearly 68%) of queries are informational queries – one where the searcher knows what she’d like to do or find but does not know how this can be achieved. Dfacto is aimed primarily at addressing this class of queries by presenting a set of facts associated with the query/topic to the searcher. It uses natural language algorithms to get facts that are most “semantically” related to the query. In lay terms, it literally tries to understand your query and the results. I’ll save the algorithmic details for another post. The few examples below show how it works:

Disclaimer: This is a work in progress, so you might notice a few ‘facts’ that are irrelevant to the query.

Lets say the searcher is (losing hair and) looking for causes of hair loss. Normally he/she would need to click through a bunch of links to get an overview on the causes. This hack on the other hand makes life a bit easier by presenting the causes upfront (click to enlarge):

click to enlarge
'hair loss cause'

Along with the facts, we also list the source from where it was extracted. Alternatively, the searcher can also select a bunch of facts he/she thinks are relevant and refine the search. This in turn would yield a new set of ‘web results’ along with new refined and related ‘facts’.

Another example (one which I particularly like) is a query about ‘table manners’. This precisely lists a set of etiquette’s to follow at the table (click to enlarge).

click to enlarge
'table manners'

Alternatively, Dfacto also serves well as a product research tool. A query for ‘iphone 3gs’ yeilds (click to enlarge):

click to enlarge
'iphone 3gs'

On another note, if you have a date in the coming weeks you might be interested in reading the list below (:

click to enlarge
'first date tips'

Happy hacking!