Our web developer team created a large search engine using php5 and mysql5.
This project consist of 2 parts the backend php spider and the frontend search engine.

spider using php:

We created a php daemon which constantly scours the web and spiders urls based on our niche criteria. This php script was created so that it uses little memory footprint and is very robust.

The spider works in the background and populates the Mysql tables continuously. This php script starts with some specified see urls and then goes on to spider links and links of links from web pages very efficiently. The pages are downloaded to the server and gzipped.

Another background php script parses the downloaded gzipped files and puts them into the forward index table. This is called the indexing script. The indexing scripts parses html, document and pdf files. This scripts is also very fast.

Now the reverse index table is created with a lexicon. This lexicon is used for speeding up searches.

The ability of this php script to process millions of records is notable and stands testimony to the skills of our software developers.

front end search panel

The search panel scripts are also highly optimized scripts. The mysql tables are normalized and optimized completely. Also special techniques are used to show the results much faster than conventional search engines. Special mysql indexing is used.

Efficient Mysql table joins are used judiciously throughout the project. Search results can be divided into tags or categories.
Page wise breakup of search results are done too.