StanleyWong.org » Search

Real Time Streams becoming more relevant

SWong — Mon, 18 May 2009 00:24:46 +0000

Great article on TechCrunch today about “Jump Into the Stream” and how we are in an interesting shift to real time streamed information. The biggest challenge is to make sense of this deluge of information and to figure out what is meaningful and what is meaningless.

With Google making some rumbling about real time search this is going to get really interesting. Much like SEO is used to drive traffic to web 1.0 websites, social bookmarking and social discovery was used to drive traffic to websites in Web 2.0. Web 3.0 may be where we see another source of traffic generation through different real time feeds and streams. An indication of this is Mark Cuban’s recent blog post about how he is getting more and more traffic from twitter and facebook than Google. In his case it is definitely Streams (Facebook + Twitter) > SEO (Google).

Very intent to see where this leads us.

The Price of Internet Content SPAM

SWong — Tue, 03 Apr 2007 06:25:16 +0000

A recent research report from Microsoft (NY Times article) claimed that approximately 11% of search results and as much as 30% of some competitive search results contain Internet Content SPAM. Spammers create these pages for the sole purpose of profiting from advertisements by fooling the search engines (i.e. Google, Yahoo!,etc.) to rank their pages well. Typically these SPAM sites are computer generated to create huge numbers of pages each targeted to rank well with a specific keyword.

There are many strategies, but typically these SPAM sites are computer generated to produce a huge numbers of pages, each targeted to rank well with a specific keyword. I’ve found several SPAM content generator scripts (YACG, RSSGM, WP AutoBlog, & MyGen) on the web that take content snippets from a variety of articles cribbed from across the web. These scripts auto generate content by jumbling content snippets into search engine readable pages using statistical algorithms such as Markov Chains. Add a custom template to make the site unique and create an internal link network between pages within the site.

Once the site is built, Spammers create huge numbers of bogus blogs (Blogger) and webpages (from free hosting sites such as Yahoo!â€™s GeoCities) to build links directed towards the Spam site. This confirmed by the Microsoft study where most free web hosting and free blog sites have over 70% Internet Content SPAM.

Internet Content SPAM is a huge problem for Search Engines because it degrades the quality their search results. As Albert Einstein famously said “We can’t solve problems by using the same kind of thinking we used when we created them.”, it is tough for Search Engines to identify Internet Content SPAM. Typical approaches entail usage of filters and/or blacklists when a SPAM signature or domain is discovered. This makes Internet Content SPAM one of the most vexing issues for this generation of Search Engines.

My Yahoo! Buzz Index Patent Has Been Granted

SWong — Fri, 12 Jan 2007 19:51:44 +0000

Last month, I found out the patent for the Yahoo! Buzz Index (US Patent 7,146,416) was granted. I had a great time working with Janice Yoo, Elliot Yasnovsky, and KT Lim on building this project. From Search Engine Land:

Yahoo’s Buzz

This patent had me wondering how Yahoo presently measures trends in topics searched for on their search engine and portal, selected in their directory, and from people’s usage of the many services they offer; and how the company might be analyzing and using that information.

Web site activity monitoring system with tracking by categories and terms
Invented by Janet Yoo, Kian-Tat Lim, Stanley Ben Wong, and Elliott Yasnokvsky
Assigned to Yahoo
US Patent 7,146,416
Granted December 5, 2006
Filed September 1, 2000

Abstract

A traffic monitor provides statistics of traffic using an activity input for receiving data related to activity on a server system. Events being monitored are binned by topic or term, where the terms are associated with categories. The categories can be a hierarchy of categories and subcategories, with terms being in one or more categories. The categorized events include page views and search requests and the results might be normalized over a field of events and a result output for outputting results of the normalizer as the statistical analyses of traffic.

The Yahoo! Buzz Index goes way beyond what you see on the public version of the site. Back when I was at Yahoo! we conceived of this product as a marketing dashboard that would give you all types of insight on arguably one of the largest online panels in the world.
The client version of the Buzz Index empowers marketers to slice and dice data to find out aggregate information (without personally identifiable information) about who was engaging with a brand, concept, or search term. These insights leverages Yahoo!’s unparalleled number of consumer profiles (over 250 Million when I last looked). Google’s Zeigeist and Google Trends are similar but lacks the insights contained by the behavioral patterns enhanced with the sheer number of profiles. For example, a brand manager can find out whether or not Pepsi’s brand profile on the web engaged with younger female audiences than Coca-Cola’s and how they compare.
It was a lot of fun dreaming up that product when I was at Yahoo! It was fun to work at Yahoo! during those times when we were routinely innovating with new concepts and technology. In fact, the Yahoo! Buzz Index predated Google’s Zeigeist and Google Trends by over 4 years. Glad to finally see the patent has been granted.

Google Take All?

SWong — Wed, 03 Jan 2007 20:55:07 +0000

I just read Rick Skrenta’s great blog post,
“Winner-Take-All: Google and the Third Age of Computing”

Rick is right on the money with a lot of his observations, especially the fact that Google has built their huge lead on the backs of the Search and Advertising dominance.

One thing I’d like to add is zero switching costs also has a downside for Google if a new entrant is **significantly better** than the incumbent. So far, the efforts of Yahoo!, Ask, and Microsoft in the search space are building on essentially the same technology platform as Google. The best they can do is incrementally better than Google and therefore have a huge challenge to overcome the brand gap and technology refinement from Google’s band of top notched engineers.

Significantly better requires one to make a dramatic leap (technology or business model) beyond what Google is doing today. When Google came into the market they introduced PageRank (link based algorithm) which was significantly better than the prevailing keyword based ranking technologies employed by the incumbents (AltaVista, Excite, Infoseek, Inktomi, etc). PageRank introduced, for that time, a relatively SPAM free search environment that plagued the keyword based search engines. So for users comparing searches performed on Google vs. the incumbents the results were startling.
The biggest challenge for search continues to be search SPAM. SEO practitioners (white hat and black hat) have continued to arm themselves with tools to SPAM the current generation of search engines. These tools include keyword based semantic page generators (using Markov chain models) while linking them together into artificially generated networks to improve ranking.

Having worked in the Internet space for over 10 years, I personally think that Google is not invincible. I’ve seen too many dominant players come and go during my career to be bold enough to make that claim. However for the potential competitors, it requires one to have a talented team that can approach this space from a completely different vantage point and offer up a new technology paradigm to advance the state of the art towards the next generation. This requires a healthy combination of Smarts (Innovation) + Money (to Scale the business) + Flawless Execution to achieve.