Search

Ferris provides some higher-level utilities on top of App Engine’s Search API. Notably ferris offers automatic translation of models into search documents via index_entity() and the Searchable behavior as well as a highler-level query interface via search().

Indexing models

You can easily add any entity to the full-text search index via index_entity().

ferris3.search.index_entity(instance, index, only=None, exclude=None, extra_converters=None, indexer=None, callback=None)[source]

Adds an Model instance into full-text search indexes.

Parameters:
  • instance – an instance of ndb.Model
  • only (list(string)) – If provided, will only index these fields
  • exclude (list(string)) – If provided, will not index any of these fields
  • extra_converters (dict) – Extra map of property names or types to converter functions.
  • indexer – A function that transforms properties into search index fields.
  • callback – A function that will recieve (instance, fields). Fields is a map of property names to search. Field instances generated by the indexer the callback can modify this dictionary to change how the item is indexed.

This is usually done in Model.after_put, for example:

def after_put(self):
    index(self)

Note

This function can not automatically index Computed, Key, or Blob properties. Use the search_callback to implement indexing for these fields.

Likewise, you can remove an entity via unindex_entity(). Note that this method only requires the key of the entity.

ferris3.search.unindex_entity(instance_or_key, index=None)[source]

Removes a document from the full-text search.

This is usually done in Model.after_delete, for example:

@classmethod
def after_delete(cls, key):
    unindex(key)

Automatic index management

The Searchable behavior will automatically add entities to the search index when saved and remove them when deleted:

from ferris import Model
from ferris.behaviors import searchable

class Post(Model):
    class Meta:
        behaviors = (searchable.Searchable,)

    title = ndb.StringProperty()
    context = ndb.TextProperty()
class ferris3.search.Searchable(Model)[source]

Automatically indexes models during after_put into the App Engine Text Search API.

This behavior can be configured using the Meta class:

class Meta:
    behaviors = (searchable.Searchable,)
    search_index = ('global', 'searchable:Post')
    search_exclude = ('thumbnail', 'likes')
SearchableMeta.search_index

Which search index to add the entity’s data to. By default this is searchable:[Model]. You can set it to a list or tuple to add the entity data to muliple indexes

SearchableMeta.search_fields

A list or tuple of field names to use when indexing. If not specified, all fields will be used.

SearchableMeta.search_exclude

A list or tuple of field names to exclude when indexing.

SearchableMeta.search_callback

A callback passed to index_entity(). This can be used to index additional fields:

from google.appengine.ext import ndb, search
from ferris.behaviors.searchable import Searchable

class Post(Model):
    class Meta:
        behaviors = (Searchable,)

        @static_method
        def search_callback(instance, fields):
            category = instance.category.get()
            fields.append(
                search.TextField(
                    name="category",
                    value=category.title
                )
            )

    title = ndb.StringProperty()
    category = ndb.KeyProperty(Category)

Performing searches

Ferris provides a slighly more pythonic wrapper for searching.

ferris3.search.search(index, query, sort=None, sort_default_values=None, limit=None, page_token=None, ids_only=True, options=None, per_document_cursor=False)[source]

Searches an index with the given query and returns a list of document ids or search documents.

To get the full search document pass ids_only = False.
Parameters:
  • index – The name of the index to search.
  • query – Query string as described in the App Engine documentation.
  • sort – A sort string, can be "field_name" for ascending or "-field_name" for descending. Can also be a list of sorts, such as ["price", "-rating"].
  • sort_default_values – The default value to use for sorting if there is no value in the document.
  • limit – Maximum number of results to return.
  • page_token – Cursor used to get a particular page of results.
  • ids_only – By default, this only returns document ids as the most common use case is to get the entity or database entries associated with the document. Pass False here to get the complete document.
  • options – Advanced options that are passed directly to index.search. See query options.
  • per_document_cursor – Whether to include a cursor for every document or not.
Returns:

a tuple of (items, error, next_page_token)

Examples:

# Search all pages for "policies"
results, error, next_page_token = search('searchable:Page', 'policies', limit=20)

# Search all products for "rake" sorted by price descending
results, error, next_page_token = search('searchable:Page', 'rake', sort='-price', limit=20)

Transforming results into entities

Search results alone are rarely what you want. Typically, we want to get the actual datastore entity we indexed. This is straightforward with to_entities().

ferris3.search.to_entities(results)[source]

Transform a list of search results into ndb.Model entities by using the document id as the urlsafe form of the key.

For example:

# Search all pages for "policies"
results, error, next_page_token = search.search('searchable:Page', 'policies', limit=20)

entities = search.to_entities(results)

Using search with endpoints

A search method can be added to an endpoint service easily using the building blocks above:

import ferris3
from google.appengine.ext import ndb

class Page(ferris3.Model):
    class Meta:
        behaviors = (ferris3.search.Searchable,)

    title = ndb.StringProperty()
    content = ndb.TextProperty()

PageMessage = ferris3.model_message(Page)
PageListMessage = ferris3.list_message(PageMessage)

@ferris3.auto_service
class PagesService(ferris3.Service):

    @ferris3.auto_method(returns=PageListMessage)
    def search(self, request, query=(str,), page_token=(str,None)):

        # Get the index to search, because this is a searchable model it'll be 'searchable:Page'
        index = ferris3.search.index_for(Page)

        # Perform the query
        results, error, next_page_token = ferris3.search.search(index, query, limit=20, page_token=page_token)

        # Check for errors
        if error:
            raise ferris3.BadRequestException("Search error: %s" % data.error)

        # Translate to entities
        entities = ferris3.search.to_entities(results)

        # Translate to list message
        msg = ferris3.messages.serialize_list(entities, ListMessage)

        # Set page token
        msg.next_page_token = next_page_token

        return msg

The hvild module has a generic implementation of this called searchable_list().