Updates (isoHunt v3?), advanced boolean search HOWTO

Posted: Sat Sep 17, 2005 1:58 pm

Firstly, there have been quite a lot of downtime lately. Our database suffered disk failures (twice), and some new software broke. Everything should be back to normal now (faster actually, with some code optimization).

After a little break from major coding besides continual small tweaks here and there, I've done a fair bit of code changes. A lot of minor fixes, and here are some notable fixes and new features:

- New default torrent search results sorting (REL), using combination of search terms relevance, seeds and leechers, so you get the most relevant and popular torrents at the top of the lists
- Improved query parsing for more accurate search terms matching (more below in comments)
- New tracker scraping algorithms, so you'll get more updated and accurate seeds/leechers stats in torrent search results
- More convenient single column search results sorting
- Fixed "undefined" links in search results and release pages
- RSS feeds now return more results and in more timely manner, for all searches as before

Together with more relevant search results and updated stats, our search index have also expanded to new record of over 4.5 million files (over 150 TB worth) contained in over 240,000 torrents, indexed from all major BitTorrent sites across the internet. This represents the largest repository of BitTorrent resources available anywhere, with the most advanced interface (sort on any date, seeds, etc. columns in web search, and RSS feeds on the latest in all searches).

With these foundation work out of the way, I'll be starting work on brand new features for isoHunt v3. Be prepared to be surprised Wink

UPDATE: Fixed bug with phrase search ("double quotes") not returning results.

Posted: Sat Sep 17, 2005 2:21 pm

Below are boolean operators for advanced search. By default, you can just type in search words without these and isoHunt.com will return you pretty good results (more on how isoHunt tweaks your query below)

mysql.com wrote:

Search Operators

+
A leading plus sign indicates that this word must be present in every row returned.

-
A leading minus sign indicates that this word must not be present in any row returned.

(no operator)
By default (when neither + nor - is specified) the word is optional, but the rows that contain it will be rated higher.

*
An asterisk is the truncation operator. Unlike the other operators, it should be appended to the word.

"
A phrase that is enclosed within double quote (`"') characters matches only rows that contain the phrase literally, as it was typed.

> <
These two operators are used to change a word's contribution to the relevance value that is assigned to a row. The > operator increases the contribution and the < operator decreases it. See the example below.

( )
Parentheses are used to group words into subexpressions. Parenthesized groups can be nested.

~
A leading tilde acts as a negation operator, causing the word's contribution to the row relevance to be negative. It's useful for marking noise words. A row that contains such a word will be rated lower than others, but will not be excluded altogether, as it would be with the - operator.

The following examples demonstrate some search strings that use boolean full-text operators:

apple banana
Find rows that contain at least one of the two words.

+apple +juice
Find rows that contain both words.

+apple macintosh
Find rows that contain the word ``apple'', but rank rows higher if they also contain ``macintosh''.

+apple -macintosh
Find rows that contain the word ``apple'' but not ``macintosh''.

apple*
Find rows that contain words such as ``apple'', ``apples'', ``applesauce'', or ``applet''.

"some words"
Find rows that contain the exact phrase ``some words'' (for example, rows that contain ``some words of wisdom'' but not ``some noise words''). Note that the `"' characters that surround the phrase are operator characters that delimit the phrase. They are not the quotes that surround the search string itself.

+apple +(>turnover <strudel)
Find rows that contain the words ``apple'' and ``turnover'', or ``apple'' and ``strudel'' (in any order), but rank ``apple turnover'' higher than ``apple strudel''.

With so many search items in our index, we've found it necessary that in most cases, an AND search of all terms will return a narrower but far more accurate results. However, an AND search on too many terms usually yield 0 results. Therefore as a compromise, our new query parser now optimizes your query in the following way:

1) The first 3 words without an operator are appended with a + sign (AND search), which you can override by appending the | operator in front of search terms
2) Subsequent words are left alone as optional keywords

For example, search on isoHunt for:

one two three four five six seven
would be modifed to actually
+one +two +three four five six seven

|one |two |three four five six seven
would be modifed to actually
one two three +four +five +six seven

|one |two |three four +five six seven
would be modifed to actually
one two three +four +five +six +seven

So for normal searching using long queries without boolean operators, just search with more important words at the beginning. Also note that any search word less than 2 characters in length will be ignored in the search. Have fun power searching!

Posted: Tue Mar 14, 2006 12:17 am

After the major upgrade, searches now have all terms considered (complete AND search by default). You can still override it by adding "|" in front of terms for making it optional.

These are boolean operators that can be used:

+ (already assumed), -, |, "", ()
(other boolean operators are now deprecated)

ex. Search for (|word1 |word2) searches for EITHER of the 2 words, when the default behaviour without any boolean operators would be (+word1 +word2), both words required in the results.
(word1 || word2) and (word1 OR word2) are boolean word variants, and are the same as (|word1 |word2).

Wildcard operators:
*
ex. Search for (gent*) would search for "gentle", "gentoo", and other terms that start with gent.
?
ex. Search for (te?t) would search for "test", "text", etc. Or terms that replaces ? with one character.

Also, note the following list of stopwords, which are words too common and are not used in searching nor are included in the index:

Code:

"a", "an", "and", "are", "as", "at", "be", "but", "by",
"for", "if", "in", "into", "is", "it",
"no", "not", "of", "on", "or", "s", "such",
"t", "that", "the", "their", "then", "there", "these",
"they", "this", "to", "was", "will", "with",
"from",
"files",
"directory",
"nbsp",
"com",
"www",
"file",
"http",
"visit"

These words are however INCLUDED within phrase searches.