Sphinx 0.9.8 reference manual

Prototype: function GetLastError()

Returns last error message, as a string, in human readable format. If there were no errors during the previous API call, empty string is returned.

You should call it when any other function (such as Section 5.6.1, “Query”) fails (typically, the failing function returns false). The returned string will contain the error description.

The error message is not reset by this call; so you can safely call it several times if needed.

Prototype: function GetLastWarning ()

Returns last warning message, as a string, in human readable format. If there were no warnings during the previous API call, empty string is returned.

You should call it to verify whether your request (such as Section 5.6.1, “Query”) was completed but with warnings. For instance, search query against a distributed index might complete succesfully even if several remote agents timed out. In that case, a warning message would be produced.

The warning message is not reset by this call; so you can safely call it several times if needed.

Prototype: function SetServer ( $host, $port )

Sets searchd host name and TCP port. All subsequent requests will use the new host and port settings. Default host and port are "localhost" and 3312, respectively.

Prototype: function SetArrayResult ( $arrayresult )

PHP specific. Controls matches format in the search results set (whether matches should be returned as an array or a hash).

$arrayresult argument must be boolean. If $arrayresult is false (the default mode), matches will returned in PHP hash format with document IDs as keys, and other information (weight, attributes) as values. If $arrayresult is true, matches will be returned as a plain array with complete per-match information including document ID.

Introduced along with GROUP BY support on MVA attributes. Group-by-MVA result sets may contain duplicate document IDs. Thus they need to be returned as plain arrays, because hashes will only keep one entry per document ID.

Prototype: function SetLimits ( $offset, $limit, $max_matches=0, $cutoff=0 )

Sets offset into server-side result set ($offset) and amount of matches to return to client starting from that offset ($limit). Can additionally control maximum server-side result set size for current query ($max_matches) and the threshold amount of matches to stop searching at ($cutoff). All parameters must be non-negative integers.

First two parameters to SetLimits() are identical in behaviour to MySQL LIMIT clause. They instruct searchd to return at most $limit matches starting from match number $offset. The default offset and limit settings are 0 and 20, that is, to return first 20 matches.

max_matches setting controls how much matches searchd will keep in RAM while searching. All matching documents will be normally processed, ranked, filtered, and sorted even if max_matches is set to 1. But only best N documents are stored in memory at any given moment for performance and RAM usage reasons, and this setting controls that N. Note that there are two places where max_matches limit is enforced. Per-query limit is controlled by this API call, but there also is per-server limit controlled by max_matches setting in the config file. To prevent RAM usage abuse, server will not allow to set per-query limit higher than the per-server limit.

You can't retrieve more than max_matches matches to the client application. The default limit is set to 1000. Normally, you must not have to go over this limit. One thousand records is enough to present to the end user. And if you're thinking about pulling the results to application for further sorting or filtering, that would be much more efficient if performed on Sphinx side.

$cutoff setting is intended for advanced performance control. It tells searchd to forcibly stop search query once $cutoff matches had been found and processed.

Prototype: function SetMaxQueryTime ( $max_query_time )

Sets maximum search query time, in milliseconds. Parameter must be a non-negative integer. Default valus is 0 which means "do not limit".

Similar to $cutoff setting from Section 5.2.1, “SetLimits”, but limits elapsed query time instead of processed matches count. Local search queries will be stopped once that much time has elapsed. Note that if you're performing a search which queries several local indexes, this limit applies to each index separately.

Prototype: function SetMatchMode ( $mode )

Sets full-text query matching mode, as described in Section 4.1, “Matching modes”. Parameter must be a constant specifying one of the known modes.

WARNING: (PHP specific) you must not take the matching mode constant name in quotes, that syntax specifies a string and is incorrect:

$cl->SetMatchMode ( "SPH_MATCH_ANY" ); // INCORRECT! will not work as expected
$cl->SetMatchMode ( SPH_MATCH_ANY ); // correct, works OK

Prototype: function SetRankingMode ( $ranker )

Sets ranking mode. Only available in SPH_MATCH_EXTENDED2 matching mode at the time of this writing. Parameter must be a constant specifying one of the known modes.

By default, Sphinx computes two factors which contribute to the final match weight. The major part is query phrase proximity to document text. The minor part is so-called BM25 statistical function, which varies from 0 to 1 depending on the keyword frequency within document (more occurrences yield higher weight) and within the whole index (more rare keywords yield higher weight).

However, in some cases you'd want to compute weight differently - or maybe avoid computing it at all for performance reasons because you're sorting the result set by something else anyway. This can be accomplished by setting the appropriate ranking mode.

Currently implemented modes are:

Prototype: function SetSortMode ( $mode, $sortby="" )

Set matches sorting mode, as described in Section 4.5, “Sorting modes”. Parameter must be a constant specifying one of the known modes.

WARNING: (PHP specific) you must not take the matching mode constant name in quotes, that syntax specifies a string and is incorrect:

$cl->SetSortMode ( "SPH_SORT_ATTR_DESC" ); // INCORRECT! will not work as expected
$cl->SetSortMode ( SPH_SORT_ATTR_ASC ); // correct, works OK

Prototype: function SetWeights ( $weights )

Binds per-field weights in the order of appearance in the index. DEPRECATED, use Section 5.3.5, “SetFieldWeights” instead.

Prototype: function SetFieldWeights ( $weights )

Binds per-field weights by name. Parameter must be a hash (associative array) mapping string field names to integer weights.

Match ranking can be affected by per-field weights. For instance, see Section 4.4, “Weighting” for an explanation how phrase proximity ranking is affected. This call lets you specify what non-default weights to assign to different full-text fields.

The weights must be positive 32-bit integers. The final weight will be a 32-bit integer too. Default weight value is 1. Unknown field names will be silently ignored.

There is no enforced limit on the maximum weight value at the moment. However, beware that if you set it too high you can start hitting 32-bit wraparound issues. For instance, if you set a weight of 10,000,000 and search in extended mode, then maximum possible weight will be equal to 10 million (your weight) by 1 thousand (internal BM25 scaling factor, see Section 4.4, “Weighting”) by 1 or more (phrase proximity rank). The result is at least 10 billion that does not fit in 32 bits and will be wrapped around, producing unexpected results.

Prototype: function SetIndexWeights ( $weights )

Sets per-index weights, and enables weighted summing of match weights across different indexes. Parameter must be a hash (associative array) mapping string index names to integer weights. Default is empty array that means to disable weighting summing.

When a match with the same document ID is found in several different local indexes, by default Sphinx simply chooses the match from the index specified last in the query. This is to support searching through partially overlapping index partitions.

However in some cases the indexes are not just partitions, and you might want to sum the weights across the indexes instead of picking one. SetIndexWeights() lets you do that. With summing enabled, final match weight in result set will be computed as a sum of match weight coming from the given index multiplied by respective per-index weight specified in this call. Ie. if the document 123 is found in index A with the weight of 2, and also in index B with the weight of 3, and you called SetIndexWeights ( array ( "A"=>100, "B"=>10 ) ), the final weight return to the client will be 2*100+3*10 = 230.

Prototype: function SetIDRange ( $min, $max )

Sets an accepted range of document IDs. Parameters must be integers. Defaults are 0 and 0; that combination means to not limit by range.

After this call, only those records that have document ID between $min and $max (including IDs exactly equal to $min or $max) will be matched.

Prototype: function SetFilter ( $attribute, $values, $exclude=false )

Adds new integer values set filter.

On this call, additional new filter is added to the existing list of filters. $attribute must be a string with attribute name. $values must be a plain array containing integer values. $exclude must be a boolean value; it controls whether to accept the matching documents (default mode, when $exclude is false) or reject them.

Only those documents where $attribute column value stored in the index matches any of the values from $values array will be matched (or rejected, if $exclude is true).

Prototype: function SetFilterRange ( $attribute, $min, $max, $exclude=false )

Adds new integer range filter.

On this call, additional new filter is added to the existing list of filters. $attribute must be a string with attribute name. $min and $max must be integers that define the acceptable attribute values range (including the boundaries). $exclude must be a boolean value; it controls whether to accept the matching documents (default mode, when $exclude is false) or reject them.

Only those documents where $attribute column value stored in the index is between $min and $max (including values that are exactly equal to $min or $max) will be matched (or rejected, if $exclude is true).

Prototype: function SetFilterFloatRange ( $attribute, $min, $max, $exclude=false )

Adds new float range filter.

On this call, additional new filter is added to the existing list of filters. $attribute must be a string with attribute name. $min and $max must be floats that define the acceptable attribute values range (including the boundaries). $exclude must be a boolean value; it controls whether to accept the matching documents (default mode, when $exclude is false) or reject them.

Only those documents where $attribute column value stored in the index is between $min and $max (including values that are exactly equal to $min or $max) will be matched (or rejected, if $exclude is true).

Prototype: function SetGeoAnchor ( $attrlat, $attrlong, $lat, $long )

Sets anchor point for and geosphere distance (geodistance) calculations, and enable them.

$attrlat and $attrlong must be strings that contain the names of latitude and longitude attributes, respectively. $lat and $long are floats that specify anchor point latitude and longitude, in radians.

Once an anchor point is set, you can use magic "@geodist" attribute name in your filters and/or sorting expressions. Sphinx will compute geosphere distance between the given anchor point and a point specified by latitude and lognitude attributes from each full-text match, and attach this value to the resulting match. The latitude and longitude values both in SetGeoAnchor and the index attribute data are expected to be in radians. The result will be returned in meters, so geodistance value of 1000.0 means 1 km. 1 mile is approximately 1609.344 meters.

Prototype: function SetGroupBy ( $attribute, $func, $groupsort="@group desc" )

Sets grouping attribute, function, and groups sorting mode; and enables grouping (as described in Section 4.6, “Grouping (clustering) search results ”).

$attribute is a string that contains group-by attribute name. $func is a constant that chooses a function applied to the attribute value in order to compute group-by key. $groupsort is a clause that controls how the groups will be sorted. Its syntax is similar to that described in Section 4.5, “SPH_SORT_EXTENDED mode”.

Grouping feature is very similar in nature to GROUP BY clause from SQL. Results produces by this function call are going to be the same as produced by the following pseudo code:

SELECT ... GROUP BY $func($attribute) ORDER BY $groupsort

Note that it's $groupsort that affects the order of matches in the final result set. Sorting mode (Section 5.3.3, “SetSortMode”) affect the ordering of matches within group, ie. what match will be selected as the best one from the group. So you can for instance order the groups by matches count and select the most relevant match within each group at the same time.

Prototype: function SetGroupDistinct ( $attribute )

Sets attribute name for per-group distinct values count calculations. Only available for grouping queries.

$attribute is a string that contains the attribute name. For each group, all values of this attribute will be stored (as RAM limits permit), then the amount of distinct values will be calculated and returned to the client. This feature is similar to COUNT(DISTINCT) clause in standard SQL; so these Sphinx calls:

$cl->SetGroupBy ( "category", SPH_GROUPBY_ATTR, "@count desc" );
$cl->SetGroupDistinct ( "vendor" );

can be expressed using the following SQL clauses:

SELECT id, weight, all-attributes,
	COUNT(DISTINCT vendor) AS @distinct,
	COUNT(*) AS @count
FROM products
GROUP BY category
ORDER BY @count DESC

In the sample pseudo code shown just above, SetGroupDistinct() call corresponds to COUNT(DISINCT vendor) clause only. GROUP BY, ORDER BY, and COUNT(*) clauses are all an equivalent of SetGroupBy() settings. Both queries will return one matching row for each category. In addition to indexed attributes, matches will also contain total per-category matches count, and the count of distinct vendor IDs within each category.

Prototype: function Query ( $query, $index="*" )

Connects to searchd server, runs given search query with current settings, obtains and returns the result set.

$query is a query string. $index is an index name (or names) string. Returns false and sets GetLastError() message on general error. Returns search result set on success.

Default value for $index is "*" that means to query all local indexes. Characters allowed in index names include Latin letters (a-z), numbers (0-9), minus sign (-), and underscore (_); everything else is considered a separator. Therefore, all of the following samples calls are valid and will search the same two indexes:

$cl->Query ( "test query", "main delta" );
$cl->Query ( "test query", "main;delta" );
$cl->Query ( "test query", "main, delta" );

Index specification order matters. If document with identical IDs are found in two or more indexes, weight and attribute values from the very last matching index will be used for sorting and returning to client (unless explicitly overridden with Section 5.3.6, “SetIndexWeights”). Therefore, in the example above, matches from "delta" index will always win over matches from "main".

On success, Query() returns a result set that contains some of the found matches (as requested by Section 5.2.1, “SetLimits”) and additional general per-query statistics. The result set is a hash (PHP specific; other languages might utilize other structures instead of hash) with the following keys and values:

Prototype: function AddQuery ( $query, $index="*" )

Adds additional query with current settings to multi-query batch. $query is a query string. $index is an index name (or names) string. Returns index to results array returned from Section 5.6.3, “RunQueries”.

Batch queries (or multi-queries) enable searchd to perform internal optimizations if possible. They also reduce network connection overheads and search process creation overheads in all cases. They do not result in any additional overheads compared to simple queries. Thus, if you run several different queries from your web page, you should always consider using multi-queries.

For instance, running the same full-text query but with different sorting or group-by settings will enable searchd to perform expensive full-text search and ranking operation only once, but compute multiple group-by results from its output.

This can be a big saver when you need to display not just plain search results but also some per-category counts, such as the amount of products grouped by vendor. Without multi-query, you would have to run several queries which perform essentially the same search and retrieve the same matches, but create result sets differently. With multi-query, you simply pass all these querys in a single batch and Sphinx optimizes the redundant full-text search internally.

AddQuery() internally saves full current settings state along with the query, and you can safely change them afterwards for subsequent AddQuery() calls. Already added queries will not be affected; there's actually no way to change them at all. Here's an example:

$cl->SetSortMode ( SPH_SORT_RELEVANCE );
$cl->AddQuery ( "hello world", "documents" );

$cl->SetSortMode ( SPH_SORT_ATTR_DESC, "price" );
$cl->AddQuery ( "ipod", "products" );

$cl->AddQuery ( "harry potter", "books" );

$results = $cl->RunQueries ();

With the code above, 1st query will search for "hello world" in "documents" index and sort results by relevance, 2nd query will search for "ipod" in "products" index and sort results by price, and 3rd query will search for "harry potter" in "books" index while still sorting by price. Note that 2nd SetSortMode() call does not affect the first query (because it's already added) but affects both other subsequent queries.

AddQuery() does not modify the current state. That is, all current sorting, filtering, and grouping settings will not be affected by this call; so subsequent queries can easily reuse current query settings.

AddQuery() returns an index into an array of results that will be returned from RunQueries() call. It is simply a sequentially increasing 0-based integer, ie. first call will return 0, second will return 1, and so on. Just a small helper so you won't have to track the indexes manualy if you need then.

Prototype: function RunQueries ()

Connect to searchd, runs a batch of all queries added using AddQuery(), obtains and returns the result sets. Returns false and sets GetLastError() message on general error (such as network I/O failure). Returns a plain array of result sets on success.

Each result set in the returned array is exactly the same as the result set returned from Query().

Note that the batch query request itself almost always succeds - unless there's a network error, blocking index rotation in progress, or another general failure which prevents the whole request from being processed.

However individual queries within the batch might very well fail. In this case their respective result sets will contain non-empty "error" message, but no matches or query statistics. In the extreme case all queries within the batch could fail. There still will be no general error reported, because API was able to succesfully connect to searchd, submit the batch, and receive the results - but every result set will have a specific error message.

Prototype: function ResetFilters ()

Clears all currently set filters.

This call is only normally required when using multi-queries. You might want to set different filters for different queries in the batch. To do that, you should call ResetFilters() and add new filters using the respective calls.

Prototype: function ResetGroupBy ()

Clears all currently group-by settings, and disables group-by.

This call is only normally required when using multi-queries. You can change individual group-by settings using SetGroupBy() and SetGroupDistinct() calls, but you can not disable group-by using those calls. ResetGroupBy() fully resets previous group-by settings and disables group-by mode in the current state, so that subsequent AddQuery() calls can perform non-grouping searches.

Prototype: function BuildExcerpts ( $docs, $index, $words, $opts=array() )

Excerpts (snippets) builder function. Connects to searchd, asks it to generate excerpts (snippets) from given documents, and returns the results.

$docs is a plain array of strings that carry the documents' contents. $index is an index name string. Different settings (such as charset, morphology, wordforms) from given index will be used. $words is a string that contains the keywords to highlight. They will be processed with respect to index settings. For instance, if English stemming is enabled in the index, "shoes" will be highlighted even if keyword is "shoe". $opts is a hash which contains additional optional highlighting parameters:

Returns false on failure. Returns a plain array of strings with excerpts (snippets) on success.

Prototype: function UpdateAttributes ( $index, $attrs, $values )

Instantly updates given attribute values in given documents. Returns number of actually updated documents (0 or more) on success, or -1 on failure.

$index is a name of the index (or indexes) to be updated. $attrs is a plain array with string attribute names, listing attributes that are updated. $values is a hash where key is document ID, and value is a plain array of new attribute values.

$index can be either a single index name or a list, like in Query(). Unlike Query(), wildcard is not allowed and all the indexes to update must be specified explicitly. The list of indexes can include distributed index names. Updates on distributed indexes will be pushed to all agents.

The updates only work with docinfo=extern storage strategy. They are very fast because they're working fully in RAM, but they can also be made persistent: updates are saved on disk on clean searchd shutdown initiated by SIGTERM signal.

Usage example:

$cl->UpdateAttributes ( "test1", array("group_id"), array(1=>array(456)) );
$cl->UpdateAttributes ( "products", array ( "price", "amount_in_stock" ),
	array ( 1001=>array(123,5), 1002=>array(37,11), 1003=>(25,129) ) );

The first sample statement will update document 1 in index "test1", setting "group_id" to 456. The second one will update documents 1001, 1002 and 1003 in index "products". For document 1001, the new price will be set to 123 and the new amount in stock to 5; for document 1002, the new price will be 37 and the new amount will be 11; etc.

Skips steps 1-3 if using already prepared delta tarball.

Skip steps 1-2 if using already prepared delta tarball.

     
mysql> show engines;
+------------+----------+----------------------------------------------------------------+
| Engine     | Support  | Comment                                                        |
+------------+----------+----------------------------------------------------------------+
| MyISAM     | DEFAULT  | Default engine as of MySQL 3.23 with great performance         |
  ...
| SPHINX     | YES      | Sphinx storage engine                                          |
  ...
+------------+----------+----------------------------------------------------------------+
13 rows in set (0.00 sec)    

Data source type. Available types are mysql, pgsql and xmlpipe.

This option is mandatory.

Example:

type = mysql

Whether to strip HTML formatting from incoming full-text data. 0 means that stripping should be disabled; 1 that it should be enabled.

Stripping currently works with mysql and pgsql source, and is not yet implemented for xmlpipe. It should work with properly formed HTML (such as well-formed XHTML) but MAY bug on malformed HTML (such as with stray <'s or unclosed >'s).

This option is optional. Default value is 0 (do not strip HTML). This option only applies to mysql and pgsql source types.

Example:

strip_html = 0

Specifies which HTML attributes' contents still should be indexed when stripping HTML. The format is per-tag enumeration of indexable attributes, as shown in the example below.

This option is optional. Default value is empty (do not index anything). This option only applies to mysql and pgsql source types.

Example:

index_html_attrs = img=alt,title; a=title;

SQL server host to connect to.

This option is mandatory. This option only applies to mysql and pgsql source types.

Example:

sql_host = localhost

SQL server IP port to connect to.

This option is optional. Default value is 3306 for mysql source type and 5432 for pgsql type. This option only applies to mysql and pgsql source types.

Example:

sql_port = 3306

SQL user to use on sql_host.

This option is mandatory. This option only applies to mysql and pgsql source types.

Example:

sql_user = test

SQL user password to use on sql_host.

This option is mandatory. This option only applies to mysql and pgsql source types.

Example:

sql_pass = mysecretpassword

SQL database (in MySQL terms) to use after connection and perform further queries in.

This option is mandatory. This option only applies to mysql and pgsql source types.

Example:

sql_db = test

UNIX socket name to connect to local MySQL server.

On Linux, it would typically be /var/lib/mysql/mysql.sock. On FreeBSD, it would typically be /tmp/mysql.sock.

This option is optional. This option only applies to mysql source type.

Example:

sql_sock = /tmp/mysql.sock

Pre-fetch query, or pre-query.

There might be multiple pre-queries specified. They are executed before the main fetch query in exactly the same order they were specified in config file. Pre-query results are ignored.

Pre-queries are useful to setup encoding, or mark records which are going to be indexed, or update internal counters, etc.

This option is optional. This option only applies to mysql and pgsql source types.

Example:

sql_query_pre = SET CHARACTER_SET_RESULTS=utf-8

Main document fetch query.

There can be only one main query. This is the query which is used to retrieve documents from SQL server.

You can specify up to 32 fields (formally, upto SPH_MAX_FIELDS from sphinx.h). All of the fields which are not document ID or attributes will be full-text indexed.

Document ID MUST be the very first field, and it MUST BE UNIQUE UNSIGNED NON-ZERO 32-BIT INTEGER NUMBER.

This option is mandatory. This option only applies to mysql and pgsql source types.

Example:

sql_query = \
	SELECT id, group_id, UNIX_TIMESTAMP(date_added) AS date_added, \
		title, content \
	FROM documents

Query which fetches min/max document IDs range to be used in ranged query (see Section 3.6, “Ranged queries”).

This option is optional. This option only applies to mysql and pgsql source types.

Example:

sql_query_range = SELECT MIN(id),MAX(id) FROM documents

How much records to index per one ranged query step (see Section 3.6, “Ranged queries”).

This option is optional. Default value is 1024. This option only applies to mysql and pgsql source types.

Example:

sql_range_step = 1000

Integer attribute column declaration. Specified column should be present among those fetched by Section 8.1.11, “sql_query”.

There might be multiple attributes specified.

This option is optional. This option only applies to mysql and pgsql source types.

Example:

sql_group_column = group_id    # declare 1st attribute
sql_group_column = author_id   # declare 2nd attribute

UNIX timestamp attribute column declaration. Specified column should be present among those fetched by Section 8.1.11, “sql_query”.

There might be multiple attributes specified.

This option is optional. This option only applies to mysql and pgsql source types.

Example:

sql_date_column = added_ts

Ordinal string number attribute column declaration. Specified column should be present among those fetched by Section 8.1.11, “sql_query”.

When indexing such attributes, string values are fetched from database, stored, sorted and then replaced by ordinal number (integer) in the sorted strings array. These integers could then be used when searching to sort by by string values lexicographically.

WARNING, all such string values are going to be stored in RAM while indexing!

WARNING, "C" locale will be used when sorting!

There might be multiple attributes specified.

This option is optional. This option only applies to mysql and pgsql source types.

Example:

sql_str2ordinal_column = author_name

Post-fetch query, executed immediately after main fetch query (Section 8.1.11, “sql_query”) ends. If this query produces errors, they are reported as warnings, but indexing is NOT terminated. It's result set is ignored.

Note that indexing is NOT completed at the point when post-query gets executed, and further indexing might fail.

This option is optional. This option only applies to mysql and pgsql source types.

Example:

sql_query_post = DROP TABLE my_tmp_table

Post-index query, executed when indexing is succesfully completed. If this query produces errors, they are reported as warnings, but indexing is NOT terminated. It's result set is ignored.

In this query, you can use $maxid macro, expanded to max document ID which was actually fetched from the database during indexing.

This option is optional. This option only applies to mysql and pgsql source types.

Example:

sql_query_post_index = REPLACE INTO counters ( id, val ) \
    VALUES ( 'max_indexed_id', $maxid )

Document info query. Only used by CLI search to fetch and display document information; and only intended for debugging purposes.

This query fetches info to be displayed by CLI search utility by document ID. Therefore, it must contain $id macro.

This option is optional. This option only applies to mysql and pgsql source types.

Example:

sql_query_info = SELECT * FROM documents WHERE id=$id

Command which will be executed in xmlpipe mode to obtain documents. See Section 3.7, “XMLpipe data source” for output format description.

This option is mandatory. This option only applies to xmlpipe source type.

Example:

xmlpipe_command = cat /home/sphinx/test.xml

This feature prevents short periods of searchd being inaccessible while rotating indexes with huge attribute and/or dictionary files (.spa and .spi respectively).

Normally, rotating an index works as follows:

However, if there's a lot of attribute or dictionary data, then preloading step could take noticeble time - up to several minutes in case of preloading 1-5+ GB files.

With seamless rotate enabled, rotating works as follows:

Seamless rotate comes at the cost of higher peak memory usage during the rotation (because both old and new copies of .spa and .spd data need to be in RAM while preloading new copy). Average usage stays the same.

This option is optional. Default values is 1, which means to enable seamless rotate.

Example:

seamless_rotate = 1