Скачать или смотреть Jeff Larson - On the resemblance and containment of documents

Meetup: http://www.meetup.com/papers-we-love/...
Paper: http://gatekeeper.dec.com/ftp/pub/dec...
Audio: http://www.mixcloud.com/paperswelove/...
---------------------------------------------------------------------------------------------
Sponsored by KissPatent and The Ladders (@TheLaddersDev)
---------------------------------------------------------------------------------------------

Description
---------------
Increasingly Journalists are dealing with ever larger document dumps, and in order to find interesting stories in these troves, they have to cluster the documents to separate the wheat from the chaff. The size of these dumps often means that traditional algorithms either are too complex and take too long, or they rely on apriori constants like the number of clusters to search for.

Jeff Larson will present a novel algorithm called minhashing that was invented at AltaVista in order to loosely cluster similar documents. The paper " On the resemblance and containment of documents" relies on a hash collisions to create document fingerprints and shows that documents can be clustered in linear time without knowledge of the entire document corpus.

This algorithm has been a key tool in some of ProPublica's biggest investigations, and has allowed reporters to shine light on topics such as political astroturfing and international money laundering.

Bio
----
Jeff Larson (@thejefflarson) is ProPublica's data editor and winner of the 2011 Livingston Award for the series "Redistricting: how Powerful Interests are Drawing You Out of a Vote." He was on the team reporting on the Snowden files in 2013 with the Guardian and the New York times, and was the lead reporter behind the NSA stories "The NSA’s Secret Campaign to Crack, Undermine Internet Security" and "Spy Agencies Probe Angry Birds and Other Apps for Personal Data."