Victus Spiritus


Real Time Search Over Federated Networks

21 Aug 2009

Been doing a lot of thinking about open social media the past week or so since friendfeed, my favorite social site, got sold to facebook, one of my least favorite sites. One of the problems brought up in the OpenFF group I've been contributing to (ideas so far) is addressing the need of users for real time search.

Thus far businesses have solved this issue by keeping their pipelines centrally located for rapid searching. But this bottlenecks both bandwidth and total information flow to one location/limit. Users want to decouple their social connections from any one business so they can rely on their data portability and social connectivity regardless of liquidity events or mergers. OpenFF is an example of a user fueled desire to make this a reality, initially by supporting version 2 of the friendfeed API.

My solution to real time search over a federated network, or any distributed real time information system, is called "search like status" or SLS for short. SLS's implementation is to broadcast search in much the same way as a status messages by mimicking optimal information data response protocols. In terms of architecture, the messaging system can be implemented in any manner desired. Ultimately patterns matched by local hub or users are sent back to the searching user or host in the form of messages. These receipt messages are only sent if a match is found and are aggregated in an easy to display format at the end user. Local search caches allow for rapid return of recent searches, with additional searches being required for newly updated data.

A great implementation of a fast protocol is pubsubhubbub. The PSHB information system is a multi-tiered architecture that allows for servers, hubs and users to rapidly propagate information in a distributed manner. For more info please consult