So, what do these search engine companies do?

If you're interested in how the wizards do their work, check out Behind the Scenes at Yahoo Labs parts 1, 2 and 3. There's lots here on the Philosophy of Search (if I can call it that) as well as what's coming from Yahoo in the future. In the articles, Gary Price interviews Dr. Gary Flake who is Principal Scientist & Head of Yahoo Research Labs.