cloudant index leverages apache lucene libraries

Consider creating partial Cloudant indicies if you are only ever interested in querying a subset of your database (i.e., you could eliminate preliminary blog posts from your index and only include published posts in a partial index).This is good practice as you remove any ambiguity in index selection. Cloudant allows you to specify the index you want the database to use at query time.Use the POST /db/_explain to see how Cloudant picks an index for a specified query.It is then simple for the database to apply the second clause of the $and to this smaller data set. Our index on actor reduces a database of tens of thousands of records down to a few dozen. So an index on actor winnows the database to a smaller data set than an index on year.

In this case, I’d be inclined to create an index on actor because there may be thousands of movies in a year, but an actor might be credited with only a few dozen movies in their entire career.

the number of movies that star an actor.

the number of movies in that database for a given year.

Which field or fields should you create the index on to best serve this query? It depends on whether which is the smaller: Imagine you have a query like this for a database of movies: The job of an index is to help the database to reduce the volume of data it’s working with to a manageable size. (See the $text operator for matching arbitrary strings against blocks of text in your documents.)īoth index types have their subtleties, so make sure you’ve read the documentation and understand how it works before going into production! Indexing strategies This builds separate 'concordances' for the fields you supply, making it more universal for the database at query time. Indexes of type='text' are powered by Cloudant's Search technology. For example, if you index actor and dob then the index will be in actor/ dob order and useful for queries that involve selectors on those fields in that order. This creates single-use indexes in the order of the keys you supply. Indexes of type='json' are built using Cloudant's MapReduce technology. Index types - json & textĬloudant Query has two types of index - json and text. See CloudantStreaming.scala for examples.īy default, Spark Streaming will load all documents from a database.Our query is now using the index and you should see a performance improvement, especially with larger data sets. start () // run streaming for 120 secs Thread. Thus, a derived schema will always be: ) ssc. 1 scans all documentsįor fast loading, views are loaded without include_docs. The sample size used to discover the schema for this temp table. only used for load data with less than or equal to 200 results.Ĭloudant: as database name if database is not present

only used for load.Ĭloudant search index w/o the database name. If true, and a database with a provided name already exists, an error will be raised.Ĭonfiguration on Spark SQL Temporary Table or DataFrameīesides all the configurations passed to a temporary table or dataframe through SparkConf, it is also possible to set the following configurations in temporary table or dataframe using OPTIONS: NameĬloudant view w/o the database name. If false, a database should already exist. Whether to create a new database during save operation. 1 means we are using only first document for schema discovery -1 means all documents 0 will be treated as 1 any number N means min(N, total) docs The sample size for RDD schema discovery. The actual number is calculated based on total rows returned and satisfying maxInPartition and minInPartition The number of partitions intent used to drive JsonStoreRDD loading query result in parallel. The maximum number of results returned when querying the _find endpoint. When useQuery is enabled, _find endpoint will be used in place of _all_docs when query condition is not on primary key field (_id), so that query predicates may be driven into datastore. Protocol to use to transfer data: http or httpsīy default, _all_docs endpoint is used if configuration ‘view’ and ‘index’ (see below) are not set.

Configuration in nfĭefault values are defined in here. And configuration passed in spark-submit using –conf takes precedence over any setting in the code. Thus, configuration set using DataFrame option overrides what has beens set in SparkConf. Here each subsequent configuration overrides the previous one. “spark.”+key in the SparkConf (as they are treated as the one passed in through spark-submit using –conf option).key in the parameters, which is set in a dataframe or temporaty table options.key in the SparkConf, which is set in SparkConf.default in the Config, which is set in the nf.The configuration is obtained in the following sequence: This library is compiled for Scala 2.11 only, and intends to support Spark 2.0 onwards. Spark-submit -class '' -master local -jars

Cloudant index leverages apache lucene libraries