Problem with Enterprise Search- new veriosn of GSA launched

Google has launched new version of GSA (Google Search Appliance) meant for deployment within the firewall to do enterprise searches. There are some improvements built inside yellow box which is how standard GSA is shipped. Google claims that GSA can now index many million more pages, can conduct searches faster than ever and show role based search results. So for so good. The problem is that enterprise search is not about searching millions of pages or spitting query results a bit faster. The problem is that enterprise search engines don't work. The real problem is that it is not easy to find anything relevant. And the problem is not with search engine like GSA. The real problem is in very nature of enterprise and its content management and intranet systems.

I have worked a bit with GSA sometime back. It is extremely easy to install and use. Mostly, you point it to sites and sources you want to index, trigger its crawler, let it build its index and there you start searching with familiar Google UI, all out of box. Google has spent a decade perfecting search algorithms. It uses complex rules to bubble up the most relevant results for a user query on top of result pages. Page rank or relevancy of a page is calculated based on how page is cited by other pages, their interlinking and metadata which crawlers extracts from page elements like title, header and other structural elements (typical SEO domain). And it works so well on web because web has huge scale. Web is huge mass of interlinking web pages where cross referencing through links is a norm. This Interlinking of pages is most important factor for determining relevancy of result pages for a query.

When I say that enterprise search doesn’t work, what I mean is that results which search engines throw are hardly relevant. Unlike on web search where first few results are almost always most relevant to a query, in enterprise search, it is hardly true. You can get results which are no where relevant to your query and most important and relevant documents might be buried somewhere in huge pile of results, where you would never reach. The reason is simple, that enterprise systems and intranets don’t use cross-linking of pages and sources. There no almost non existent cross references. For example how many times would you find a "Banking Industry opportunity PoV" document or page cross referencing to another document which could be "Banking Industry Trend Analysis"!! In fact, many enterprise systems are like document storage systems where all the documents in form of excels, ppts, pdfs are dumped as equal. Search engine has no way to find which most relevant document for a query is. On top of that, most enterprise systems and intranets are not optimized for search engines. How many times, have you seen web pages on intranet which have no title or use proper Meta data!! The content creators don’t use some basic practices to make their content “findable”. So another vital source for search engine to determine page relevancy is lost. And what we get in effect is a sputtering and struggling search engine trying hard to fish out that most important document for you.

In a way, it is not problem with technology but with very nature and realities of enterprise. IMHO, an effective intranet search engine has to provide more than "out-of- box" features; it should be "tweak-able". It has to work with understanding that:

1. Enterprise content is not cross linked and cross referenced, so relevancy logic successful on web wouldn't be much useful.
2. Scale of content is limited, unlike web where million of pages are cross-linking, enterprise content is not so vast.
3. Pages are not optimized for search engines. (But that should be fixed by a company's policy)
4. Some documents written by some "experts" or “communities” in enterprise become naturally more important or relevant! The relevancy logic has to account for that, but how! Engine administrators should be able to feed new relevancy rules into the engine.
5. User ratings of documents and META -TAGS on web pages should be given more weightage in calculating relevancy. On web, these are mostly ignored due to their misuse for search engine spamming, but this is not the case in enterprise where problem is opposite.
   

From another perspective, a human edited search engine could be more useful and effective within firewall. Automated search engine can still be used to find out what are users searching most of time (trends), and experts, knowledge managers or users can contribute to index and ranking of pages manually. New version of GSA also seems to have a similar feature for Do It Yourself (DIY) key-match and some features for administrators to influence search results.

Prayer for K2 Climbers

Ever since I have been on trek to ABC in Nepalese Himalayas, I have become fascinated with mountain climbing. It is perhaps toughest challenge any man can go through, both physically and mentally. Those tall irreverent himalyan mountains allow only the fittest and strongest of men to climb them and I have nothing but respect for men who indulge in this dangerous activity. Mountain climbing community is very small and plugging to it gives a different perspective on life. It is a different world altogether. Ever Since I read Maurice Herzog's classic book Annapurna detailing Frenchman's first and successful ascent of a 8000 meter peek, I have peeked into fascinating world of Louis Lachenal, Lionel Terray, Gaston Rebuffat and many such legendry climbers. The community has its own hierarchy with toughest, high altitude climbers on top. There were climbers like Anatoli Boukreev, who was daredevil to the extent of being mad, climbing without acclimatization or without oxygen. He was also climbing during 1996's Everest tragedy when 8 climbers were killed in one day. The expedition created lots of controversy with Jon kraukauer , a journalist and climber partly blaming Boukreev for not saving some of climbers trapped in snow storm. An Indian climber who died on same tragic day on Everest, was still lying dead just beneath the peak until recently. For almost 10 years his dead body was used by climbers as a milestone (called greenboots) on way to Summit. In recent times, many controversies and debates have arisen where some people within community have decried the attitude of climbers to reach the summit at any cost and not helping or rescuing other climbers who might be in distress. In 2006, David Sharp, a British Climber laid dying on the way to Everest peak, even while 40 other climbers passed him without offering any help or rescue. There is someting in us, men, which is tested only when faced with extreme adversity like in case of Joe Simpson and Simon Yates's disasterous climb on a Peruvian mountain so brilliantly captured in book and movie Touching the Void.


Even though men would often in their follies term their successful climbs and summits as conquests, these mountains invariably keep reminding them who the real lord is .Freak weather, falling stones, sudden avalanches, extreme weather, anything can kill even most experienced and trained climber as happened on K2 yesterday. In deadliest day in K2's history ,11 climbers reportedly died while scaling K2. K2 is said to be the most difficult of mountains to climb with its razor sharp ridges and unpredictable weather. The deadly mountain is also said to be cursed for women as only 5 women could reach the top, 3 of them died on decent and rest 2 died later on other mountains. Regardless of obvious danger, K2 keeps attracting climbers like bees to the flame. It is said to be a handsome and mighty mountain which is also equally unforgiving. Here is little silent prayer for those who died scaling it yesterday.

About Me

Some questions used to have simple answers......(roll mouse over picture)

Fixed Price projects - Inconvenient truths by Scot Ambler

In recent times, for some reason, call it bad karma perhaps, I have been part of projects which have had contractual and cost issues. Projects followed BRUF (Big Requirements Up Front) method to accumulate requirements, estimates and pricing were done and contract signed with Customer. The fixed price project was started. And the problems which ensued were typical of so many such projects in industry. They are well documented variousuly, so I am not going to add to that. But I have often wondered why customers insist on agreeing to a fixed price in beginning of a big project when most of times scope can hardly be known. What it does is that, it makes both the parties, the customer and the IT Supplier draw the lines like in a battle and start on a war footing with Sr. Management on both the sides acting like Generals engaged in high strategy. Supplier implements strict change control procedures and become fixated by change management and repelling anything not in original scope. As initial estimates done during contract signing phase can't be relied upon for accuracy, big contingencies are built into estimates but without any surety of its finality. Customer in meanwhile practicaly disengages during project execution waiting for the final delivery to happen as most of his risks are offloaded to supplier. The whole dynamics of a fixed price project creates an enviornment where both the customer and vendor are playing games to protect themselves and not working together to solve the problem for which project is being executed.

Scot Ambler in this articles has articulated what he calls Inconvenient truths about fixed price software development. In fact he goes even further and calls it an unethical practice. Most of points he has highlighted are valid and right on target. Customers have to realise that if they really care about ROI, they need to engage and work with suppliers and not look to offload thier risks in form of fixed price projects. IT vendors or providers on other hand , have to relook into thier own practices which they have been selling to customer for years . As Scot Ambler says "For several decades traditional software engineering theory has told us to do formal, up-front cost estimates. In turn, we have taught our customers that this is the way things are done " . So it is time we teach them that we were wrong.

cuil a google killer

Twitter was on fire yesterday after cuil.com a new search engine was launched after much hype on techcrunch. Everybody seems to be waiting for google killer, maybe scared of its monopoly. Only if it was as easy as that. Google can't be beaten by mere hype and PR. Cuil claims that "...we searches more pages on the Web than anyone else—three times as many as Google and ten times as many as Microsoft." Really!! But I can't find my blog through it, it returns 0 records. what is the deal about this large index!!

Wiki adoption in enterprise - Participation inequality

There have been few online discussions about how emails and word documents are bad and their use in collaborative environments and organisations should be reduced. I mostly agree. Email within enterprise has become most used tool and often work comes to standstill if email is down. So regimented have we become with use of emails that we don’t want to use tools which are far better suited for today's changing times of increased collaboration where participants are often sitting across continents. Email is a great tool but like any other tool it is meant to do a certain task and not everything. While championing the use of wiki for team collaboration, I had few observations:

1. No learning new tool: Many people don’t use wikis because they don’t want to learn a new tool. Even though using wiki is as simple as typing in plain English with some simple mark up, it appears too much to learn for many.
2. Old Habits Difficult to Change: Many just don’t get it why they should not use MS Excel and MS word for certain tasks and instead should be asked to use the wiki. They think the wiki is an overhead and they dont have time to go to a site and put content there. They dont consider wiki as a tool but as some fancy site. Old habits die hard.
3. Email Regimentation: Heavy use of email has regimented us. I have seen many people doing their wiki contributions starting with "hellos" and ending with "regards" like they do in emails.
4. Scarcity is power: Many feel threatened by wiki culture as it takes away the control. Information and knowledge is the power. Scarcity of information is what makes certain people derive their power and authority. They wouldn't want to let it go. They want to keep the information scarce.
5. Openness is scary: Wikis are open and content can be created/edited and deleted by user. That scares lots of people. Many years of corporate life where every tool is controlled and every usage need some sort of approval by someone, this openness and control sounds too good and too scary.
6. No incentive to contribute: Wiki pages are result of collaboration and author names are not displayed on top of page. This puts off people as they are used to announce their names on top of word documents. Word documents can be sent over emails with CC to all those people who would do your year end performance reviews. Personal efforts are recognised and not the collaborative so there is little incentive in contributing to wiki.
7. Volunteering is unfamiliar: Wikis are flat, unstructured and non-hierarchical. That is exactly opposite of what an enterprise is. People who are too accustomed to hierarchies and being directed, volunteering to wiki remains an unfamiliar territory.

All This though is hardly surprising as most of multi-users communities suffer from this pattern of behaviour where very few people contribute and most just lurk around. This is also called as participation inequality and is excellently explained by Jacob Nielsen. As per this theory :

In most online communities, 90% of users are lurkers who never contribute, 9% of users contribute a little, and 1% of users account for almost all the action

The benefits which a simple, social and collaboration tool like wiki can bring behind firewall are immense but there remain few roadblocks to overcome to make wikis a success. I stumbled upon a site called wikipatterns , which does a very nice job of documenting patterns on wiki adoption. It s many patterns some of which I have personally come across, like wiki phobia (somewhat similar to my point 4 above). Reading the patterns, I just realised that I was about to become a Wiki Bully.

Enterprise2.0 - Let conversations happen

Transformation into Enterprise2.0 has to begin with enabling social software. Let employees connect, network , share and have conversations :) You got to know the change agents, the idea generators, the transmistors. The long tail of enterprise.