When I started using the Internet in the 1970s, it didn’t look anything like it does today, and our search tools were primitive. But when all you have is stone knives and bearskins, you make do.
Before I began writing for a living, I put myself through graduate school by doing research on the very first online database systems: NASA RECON, Dialog, and OCLC. These systems, which are still around, are part of what’s called the Matrix, and, no, I don’t mean the movies. The Matrix, as defined by Carl Malamud, is the superset of all interconnected networks. Today, unlike back then, you can get to these networks over the Internet, but you’ll be blocked from venturing deeply into them without permission.
As for the pre-Web Internet itself, it didn’t have search tools at first. It wasn’t until the late 1980s that the Internet became searchable. When I started, we had to go through ftp file directories screen by screen and hope that the file was in there somewhere.
The first major search advance was Archie, which beginning in 1990 made it possible to search through a site’s file directories. Archie was painful to use, but compared to what we had been dealing with, it was wonderful. Archie was quickly followed by the University of Nevada System Computing Services' Veronica, which tried to provide Archie-style searches for plain text files.
An even bigger advance was Gopher, which made it possible to search through online databases and text files. With Archie, you really first had to have a clue that a file was somewhere on a given site. With Gopher, you could simply search and let the server worry about finding which site had the information you wanted.
While Gopher was being built in 1991, the Web was also being created. By 1993, just as Gopher reached its maturity, I thought the first real Internet search engine, WAIS (Wide Area Information System) was going to be more important then the Web! I was putting the cart before the horse. WAIS, like Archie, Veronica, and Gopher, exists now only as Internet historical trivia.
In a way, though, I was onto something. Yes, we use the Web for everything, but it's the search engines that make it useful. After all, we say, “We Google for information,” not “We webbed for the answer.”
The Web quickly knocked out those earlier Internet search programs and online services like Prodigy, CompuServe, and GEnie. Now, only bits and pieces of them remain, and they're really only interesting to digital archaeologists.
Fortunately, for early Web users, the first Web designers set to work creating search engines. Indeed, the first search engine, the still functioning WWW Virtual Library, was created by the Web's creator, Tim Berners-Lee. The Virtual Library, and its most well-known clone, 1994's Yahoo!, weren't really search engines by today’s standards. They were human-assembled catalogs of useful Web links. You could (and did) submit your own webpage to Yahoo!, for instance, and suggest in which category it should be included. (Why yes, search engine optimization was very different back then.)
These directories still exist, and category-based sites that allow users to vote on which specific Web pages are interesting, such as Reddit, are still popular. However, most of the early ones, such as EINet Galaxy (1994), and Open Directory Project (1998), either died off or are little used. Others, such as Yahoo!, switched to a search engine model.
And what is that? The very first search engines, JumpStation, the World Wide Web Worm, and the Repository-Based Software Engineering (RBSE) spider, used automatic programs, called robots or spiders, to request webpages and then report what they found to a database. These first spiders weren't that useful. It took Archie-Like Indexing of the Web (ALIWEB) and Excite in 1993 before we had search engines that most people would recognize as useful tools.
“Useful” is being kind, though. These first search engines were only useful if you knew the exact name of the website for which you were searching. Searching inside webpages would have to wait for 1994's WebCrawler. WebCrawler is now largely forgotten, but in its first year it was so popular that it became a victim of its own success. The site was so slammed it often couldn't be used.
That gave Lycos its chance. By pouring more servers into this project, Lycos was almost certainly the first search engine to have full page search for more than a million pages. That's nothing by today's standards, but it was remarkable in mid-1994. Lycos was also the first search engine to introduce proximity searching.
The mid-90s became a hotbed of new Web search engines. InfoSeek, whose only claim to fame was somehow convincing Netscape to make it the Netscape's browser default search engine, came, and unlike the others, left no real trace behind. Inktomi, which appeared in 1996, briefly enjoyed a burst of popularity. Eventually, in 2003, it was bought out by Yahoo.
The most significant Web search engines to arrive were Altavista, my personal favorite in those days, and Ask Jeeves, now Ask.com. AltaVista was the first really fast search engine that covered much of the Web. It also gave users the first really successful Boolean search options. So, for example, you could search for “New York baseball history NOT Yankees” and get results about the New York Giants and Dodgers, but not the New York Yankees. Ask's claim to fame was its attempt to support search by natural language. So you could ask it, “Tell me about New York baseball history, but I don't want to hear about the Yankees.” [Because, really, who does? –Ed.] Whether you'd get useful results is another matter, but Ask Jeeves did better with this question style than its competition.
So why haven't you heard or most of these? Well, each had its own problems. Alta-Vista, for example, shared in the woes of its parent companies, DEC and then Compaq. Yahoo, which ended up with the ownership of Alta-Vista, would finally kill off this once great Web search engine in 2010.
But what really happened to all of them was Google. Google simply did a much better job than its rivals of making search easy and comprehensive. The key to Google’s overwhelming success was PageRank. With PageRank, Google rates the relevancy of webpages to queries, based not only on whether the pages contain the search terms (the technique used by all search engines) but also by how many relevant pages link to it.
Sounds simple. Well, the basic concept is simple. Back then, it was a groundbreaking advance. The company’s efforts to perfect Google Search and PageRank continues to this day.
Google has never rested on its laurels, either, even though no company has managed to give Google any serious competition in more than ten years. Today, Google is used for more than 65% of all searches. Frankly, I'm surprised it's that little. Bing, despite Microsoft pouring billions of dollars into the project, still has just over 15% of the market. Yahoo continues to decline with only 13.8% of searches, and Ask barely hangs on with 3% of the market.
Had AltaVista managed to avoid its parent companies' troubles, it might have been a different story. None of the other Before Google Web search engines had what it took to compete with Google's genius. Today, as a result, we really do live in A.G. (After Google) days. I doubt that will change anytime soon. Google certainly continues to work hard to make sure that will be the case.
See also:
- 30 Years Later: an Unfair Comparison between an IBM PC and an Apple iPhone4 (Infographic)
- 7 Awesome Bits of Tech That Just Freakin’ Disappeared
- Go Back In Time: How 10 Big Websites Looked 15 Years Ago (makeuseof.com)
- Exploring Google Search By Image
- 6 Products That Became Technology Roadkill
- 9 Tech Analogies That No Longer Mean Anything To Those Young Whippersnappers
