Victus Spiritus


Cool Breezes, the Messenger of Change, and Fantastic Network Challenges

10 Sep 2010

In just the past week, morning temperatures in the north east have dropped 15-20 degrees Fahrenheit. The huge drop in humidity and rise in air quality have transformed sweaty morning marches into refreshing strolls. These are welcome changes which help reinforce my attention to time. The cooler weather signals an end to summer, and a time of transition as leaves fall to the earthen floor below.

Life changes mirroring seasons

The seasonal switch coincides with a professional life transition. It is the end of an era for me. For much of my adult life I worked at a single company in one industry. As I grew to know myself better, my inclination for open collaboration, new technology, and direct customer feedback diverged greatly from my closed off and aging industry. Most importantly, I never earned the independence to work on challenging problems of my choosing.

Seeking the right market, Relevance

My professional home has become, and likely always will be, tackling open network problems. Whether on the web or the broader internet, there is a growing need for improving personal relevance as a means of filtering out noise.

Relevance is an overloaded word. Each individual has their own definition for it, and it's subject to change without notice. Today's filter will be different than yesterday's which will be different than tomorrow's. The transient nature of relevance frustrates algorithm developers that seek convergent solutions. It is an ongoing tracking problem that is related to understanding the styles, needs and passions of individuals.

I have grown to rely on the leadership of Mahendra Palsule, Louis Gray, Robert Scoble and others as they seek to articulate a common definition for relevance. The above links refer to DataSift and My6sense, two tools highly focused on solving the challenges of personal relevance. Joining them are Hunch, Google, Facebook, Disqus, and many other Internet businesses that wish to provide visitors with engaging content and information.

Part of the challenge is mining massive data stores for signals of personal meaning and updating them all in real time in a client server RESTful architecture. The solution should be available at each user interface with data specific to the browser. But finding correlations between a single person's most pressing concerns and the rest of the changing world network of information is nontrivial.

Some have tackled this issue by segmenting tastes into clusters depending on many parameters/decision boundaries (Hunch). Other algorithms leverage connections between subscriptions (follow/friends) with how colleagues and friends judge content (social filtering/retweets, sentiment/likes, Google Magic, my6sense).

Search is Evolving

What we're witnessing is the segmentation and evolution of search from a broad scale to a hyperfine, personal level. Accepting that I can't know everything I'd like to, there's more written and observed and studied than I'll ever have time to consume or remember. What do I really need to know?