Module: Google::Cloud::Datastore

Defined in:
lib/google/cloud/datastore.rb,
lib/google/cloud/datastore/key.rb,
lib/google/cloud/datastore/query.rb,
lib/google/cloud/datastore/commit.rb,
lib/google/cloud/datastore/cursor.rb,
lib/google/cloud/datastore/entity.rb,
lib/google/cloud/datastore/errors.rb,
lib/google/cloud/datastore/convert.rb,
lib/google/cloud/datastore/dataset.rb,
lib/google/cloud/datastore/service.rb,
lib/google/cloud/datastore/version.rb,
lib/google/cloud/datastore/gql_query.rb,
lib/google/cloud/datastore/properties.rb,
lib/google/cloud/datastore/credentials.rb,
lib/google/cloud/datastore/transaction.rb,
lib/google/cloud/datastore/v1/credentials.rb,
lib/google/cloud/datastore/v1/datastore_client.rb,
lib/google/cloud/datastore/dataset/query_results.rb,
lib/google/cloud/datastore/read_only_transaction.rb,
lib/google/cloud/datastore/dataset/lookup_results.rb

Overview

Google Cloud Datastore

Google Cloud Datastore is a fully managed, schemaless database for storing non-relational data. You should feel at home if you are familiar with relational databases, but there are some key differences to be aware of to make the most of using Datastore.

The goal of google-cloud is to provide an API that is comfortable to Rubyists. Your authentication credentials are detected automatically in Google Cloud Platform environments such as Google Compute Engine, Google App Engine and Google Kubernetes Engine. In other environments you can configure authentication easily, either directly in your code or via environment variables. Read more about the options for connecting in the Authentication Guide.

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new(
  project_id: "my-todo-project",
  credentials: "/path/to/keyfile.json"
)

task = datastore.find "Task", "sampleTask"
task["priority"] = 5
datastore.save task

You can learn more about various options for connection on the Authentication Guide.

To learn more about Datastore, read the Google Cloud Datastore Concepts Overview .

Enabling Logging

To enable logging for this library, set the logger for the underlying gRPC library. The logger that you set may be a Ruby stdlib Logger as shown below, or a Google::Cloud::Logging::Logger that will write logs to Stackdriver Logging. See grpc/logconfig.rb and the gRPC spec_helper.rb for additional information.

Configuring a Ruby stdlib logger:

require "logger"

module MyLogger
  LOGGER = Logger.new $stderr, level: Logger::WARN
  def logger
    LOGGER
  end
end

# Define a gRPC module-level logger method before grpc/logconfig.rb loads.
module GRPC
  extend MyLogger
end

Retrieving records

Records, called "entities" in Datastore, are retrieved by using a key. The key is more than a numeric identifier, it is a complex data structure that can be used to model relationships. The simplest key has a string kind value and either a numeric id value or a string name value. A single record can be retrieved by calling Dataset#find and passing the parts of the key:

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new

task = datastore.find "Task", "sampleTask"

Optionally, Dataset#find can be given a key object:

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new

task_key = datastore.key "Task", 123456
task = datastore.find task_key

See Dataset#find

Querying records

Multiple records can be found that match criteria. (See Query#where)

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new

query = datastore.query("Task").
  where("done", "=", false)

tasks = datastore.run query

Records can also be ordered. (See Query#order)

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new

query = datastore.query("Task").
  order("created")

tasks = datastore.run query

The number of records returned can be specified. (See Query#limit)

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new

query = datastore.query("Task").
  limit(5)

tasks = datastore.run query

When using Datastore in a multitenant application, a query may be run within a namespace using the namespace option. (See Multitenancy)

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new

query = datastore.query("Task").
  where("done", "=", false)

tasks = datastore.run query, namespace: "example-ns"

Records' key structures can also be queried. (See Query#ancestor)

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new

task_list_key = datastore.key "TaskList", "default"

query = datastore.query("Task").
  ancestor(task_list_key)

tasks = datastore.run query

See Query and Dataset#run

Paginating records

All records may not return at once, but multiple calls can be made to Datastore to return them all.

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new

query = datastore.query("Task")
tasks = datastore.run query
tasks.all do |t|
  puts t["description"]
end

See Dataset::LookupResults and Dataset::QueryResults

Creating records

New entities can be created and persisted buy calling Dataset#save. The entity must have a key to be saved. If the key is incomplete then it will be completed when saved.

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new

task = datastore.entity "Task" do |t|
  t["type"] = "Personal"
  t["done"] = false
  t["priority"] = 4
  t["description"] = "Learn Cloud Datastore"
end
task.key.id #=> nil
datastore.save task
task.key.id #=> 123456

Multiple new entities may be created in a batch.

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new

task1 = datastore.entity "Task" do |t|
  t["type"] = "Personal"
  t["done"] = false
  t["priority"] = 4
  t["description"] = "Learn Cloud Datastore"
end

task2 = datastore.entity "Task" do |t|
  t["type"] = "Personal"
  t["done"] = false
  t["priority"] = 5
  t["description"] = "Integrate Cloud Datastore"
end

tasks = datastore.save(task1, task2)
task_key1 = tasks[0].key
task_key2 = tasks[1].key

Entities in Datastore form a hierarchically structured space similar to the directory structure of a file system. When you create an entity, you can optionally designate another entity as its parent; the new entity is a child of the parent entity.

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new

task_key = datastore.key "Task", "sampleTask"
task_key.parent = datastore.key "TaskList", "default"

task = datastore.entity task_key do |t|
  t["type"] = "Personal"
  t["done"] = false
  t["priority"] = 5
  t["description"] = "Integrate Cloud Datastore"
end

Setting properties

Entities hold properties. A property has a name that is a string or symbol, and a value that is an object. Most value objects are supported, including String, Integer, Date, Time, and even other entity or key objects. Changes to the entity's properties are persisted by calling Dataset#save.

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new

task = datastore.find "Task", "sampleTask"
# Read the priority property
task["priority"] #=> 4
# Write the priority property
task["priority"] = 5
# Persist the changes
datastore.save task

Array properties can be used to store more than one value.

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new

task = datastore.entity "Task", "sampleTask" do |t|
  t["tags"] = ["fun", "programming"]
  t["collaborators"] = ["alice", "bob"]
end

Deleting records

Entities can be removed from Datastore by calling Dataset#delete and passing the entity object or the entity's key object.

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new

task = datastore.find "Task", "sampleTask"
datastore.delete task

Multiple entities may be deleted in a batch.

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new

task_key1 = datastore.key "Task", "sampleTask1"
task_key2 = datastore.key "Task", "sampleTask2"
datastore.delete task_key1, task_key2

Transactions

Complex logic can be wrapped in a transaction. All queries and updates within the Dataset#transaction block are run within the transaction scope, and will be automatically committed when the block completes.

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new

task_key = datastore.key "Task", "sampleTask"

datastore.transaction do |tx|
  if tx.find(task_key).nil?
    task = datastore.entity task_key do |t|
      t["type"] = "Personal"
      t["done"] = false
      t["priority"] = 4
      t["description"] = "Learn Cloud Datastore"
    end
    tx.save task
  end
end

Alternatively, if no block is given the transaction object is returned allowing you to commit or rollback manually.

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new

task_key = datastore.key "Task", "sampleTask"

tx = datastore.transaction
begin
  if tx.find(task_key).nil?
    task = datastore.entity task_key do |t|
      t["type"] = "Personal"
      t["done"] = false
      t["priority"] = 4
      t["description"] = "Learn Cloud Datastore"
    end
    tx.save task
  end
  tx.commit
rescue
  tx.rollback
end

A read-only transaction cannot modify entities; in return they do not contend with other read-write or read-only transactions. Using a read-only transaction for transactions that only read data will potentially improve throughput.

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new

task_list_key = datastore.key "TaskList", "default"
query = datastore.query("Task").
  ancestor(task_list_key)

tasks = nil

datastore.transaction read_only: true do |tx|
  task_list = tx.find task_list_key
  if task_list
    tasks = tx.run query
  end
end

See Transaction and Dataset#transaction

Querying metadata

Datastore provides programmatic access to some of its metadata to support meta-programming, implementing backend administrative functions, simplify consistent caching, and similar purposes. The metadata available includes information about the entity groups, namespaces, entity kinds, and properties your application uses, as well as the property representations for each property.

The special entity kind __namespace__ can be used to find all the namespaces used in your application entities.

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new

query = datastore.query("__namespace__").
  select("__key__").
  where("__key__", ">=", datastore.key("__namespace__", "g")).
  where("__key__", "<", datastore.key("__namespace__", "h"))

namespaces = datastore.run(query).map do |entity|
  entity.key.name
end

The special entity kind __kind__ can be used to return all the kinds used in your application.

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new

query = datastore.query("__kind__").
  select("__key__")

kinds = datastore.run(query).map do |entity|
  entity.key.name
end

Property queries return entities of kind __property__ denoting the indexed properties associated with an entity kind. (Unindexed properties are not included.)

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new

query = datastore.query("__property__").
  select("__key__")

entities = datastore.run(query)
properties_by_kind = entities.each_with_object({}) do |entity, memo|
  kind = entity.key.parent.name
  prop = entity.key.name
  memo[kind] ||= []
  memo[kind] << prop
end

Property queries support ancestor filtering on a __kind__ or __property__ key, to limit the query results to a single kind or property. The property_representation property in the entity representing property p of kind k is an array containing all representations of p's value in any entity of kind k.

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new

ancestor_key = datastore.key "__kind__", "Task"
query = datastore.query("__property__").
  ancestor(ancestor_key)

entities = datastore.run(query)
representations = entities.each_with_object({}) do |entity, memo|
  property_name = entity.key.name
  property_types = entity["property_representation"]
  memo[property_name] = property_types
end

Property queries can also be filtered with a range over the pseudo-property __key__, where the keys denote either __kind__ or __property__ entities.

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new

start_key = datastore.key "__property__", "priority"
start_key.parent = datastore.key "__kind__", "Task"
query = datastore.query("__property__").
  select("__key__").
  where("__key__", ">=", start_key)

entities = datastore.run(query)
properties_by_kind = entities.each_with_object({}) do |entity, memo|
  kind = entity.key.parent.name
  prop = entity.key.name
  memo[kind] ||= []
  memo[kind] << prop
end

Configuring timeout

You can configure the request timeout value in seconds.

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new timeout: 120

The Cloud Datastore Emulator

As of this release, the Cloud Datastore emulator that is part of the gcloud SDK is no longer compatible with google-cloud. This is because the gcloud SDK's Cloud Datastore emulator does not yet support gRPC as a transport layer.

A gRPC-compatible emulator is available until the gcloud SDK Cloud Datastore emulator supports gRPC. To use it you must download the gRPC emulator and use the cloud_datastore_emulator script.

When you run the Cloud Datastore emulator you will see a message similar to the following printed:

If you are using a library that supports the DATASTORE_EMULATOR_HOST
environment variable, run:

export DATASTORE_EMULATOR_HOST=localhost:8978

Now you can connect to the emulator using the DATASTORE_EMULATOR_HOST environment variable:

require "google/cloud/datastore"

# Make Datastore use the emulator
ENV["DATASTORE_EMULATOR_HOST"] = "localhost:8978"

datastore = Google::Cloud::Datastore.new project: "emulator-project-id"

task = datastore.entity "Task", "emulatorTask" do |t|
  t["type"] = "Testing"
  t["done"] = false
  t["priority"] = 5
  t["description"] = "Use Datastore Emulator"
end

datastore.save task

Defined Under Namespace

Modules: V1 Classes: Commit, Credentials, Cursor, Dataset, Entity, GqlQuery, Key, KeyError, Properties, PropertyError, Query, ReadOnlyTransaction, Transaction, TransactionError

Constant Summary collapse

VERSION = 
"1.4.1".freeze

Class Method Summary collapse

Class Method Details

.configure {|Google::Cloud.configure.datastore| ... } ⇒ Google::Cloud::Config

Configure the Google Cloud Datastore library.

The following Datastore configuration parameters are supported:

  • project_id - (String) Identifier for a Datastore project. (The parameter project is considered deprecated, but may also be used.)
  • credentials - (String, Hash, Google::Auth::Credentials) The path to the keyfile as a String, the contents of the keyfile as a Hash, or a Google::Auth::Credentials object. (See Credentials) (The parameter keyfile is considered deprecated, but may also be used.)
  • scope - (String, Array) The OAuth 2.0 scopes controlling the set of resources and operations that the connection can access.
  • timeout - (Integer) Default timeout to use in requests.
  • client_config - (Hash) A hash of values to override the default behavior of the API client.
  • emulator_host - (String) Host name of the emulator. Defaults to ENV["DATASTORE_EMULATOR_HOST"]

Yields:

Returns:

  • (Google::Cloud::Config) —

    The configuration object the Google::Cloud::Datastore library uses.



706
707
708
709
710
 
# File 'lib/google/cloud/datastore.rb', line 706

def self.configure
  yield Google::Cloud.configure.datastore if block_given?

  Google::Cloud.configure.datastore
end

.new(project_id: nil, credentials: nil, scope: nil, timeout: nil, client_config: nil, emulator_host: nil, project: nil, keyfile: nil) ⇒ Google::Cloud::Datastore::Dataset

Creates a new object for connecting to the Datastore service. Each call creates a new connection.

For more information on connecting to Google Cloud see the Authentication Guide.

Examples:

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new(
  project_id: "my-todo-project",
  credentials: "/path/to/keyfile.json"
)

task = datastore.entity "Task", "sampleTask" do |t|
  t["type"] = "Personal"
  t["done"] = false
  t["priority"] = 4
  t["description"] = "Learn Cloud Datastore"
end

datastore.save task

Parameters:

  • project_id (String) —

    Identifier for a Datastore project. If not present, the default project for the credentials is used.

  • credentials (String, Hash, Google::Auth::Credentials) —

    The path to the keyfile as a String, the contents of the keyfile as a Hash, or a Google::Auth::Credentials object. (See Credentials)

  • scope (String, Array<String>) —

    The OAuth 2.0 scopes controlling the set of resources and operations that the connection can access. See Using OAuth 2.0 to Access Google APIs.

    The default scope is:

    • https://www.googleapis.com/auth/datastore
  • timeout (Integer) —

    Default timeout to use in requests. Optional.

  • client_config (Hash) —

    A hash of values to override the default behavior of the API client. See Google::Gax::CallSettings. Optional.

  • emulator_host (String) —

    Datastore emulator host. Optional. If the param is nil, uses the value of the emulator_host config.

  • project (String) —

    Alias for the project_id argument. Deprecated.

  • keyfile (String) —

    Alias for the credentials argument. Deprecated.

Returns:

Raises:

  • (ArgumentError) 


651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
 
# File 'lib/google/cloud/datastore.rb', line 651

def self.new project_id: nil, credentials: nil, scope: nil, timeout: nil,
             client_config: nil, emulator_host: nil, project: nil,
             keyfile: nil
  project_id ||= (project || default_project_id)
  project_id = project_id.to_s # Always cast to a string
  raise ArgumentError, "project_id is missing" if project_id.empty?

  scope ||= configure.scope
  timeout ||= configure.timeout
  client_config ||= configure.client_config
  emulator_host ||= configure.emulator_host
  if emulator_host
    return Datastore::Dataset.new(
      Datastore::Service.new(
        project_id, :this_channel_is_insecure,
        host: emulator_host, client_config: client_config
      )
    )
  end

  credentials ||= (keyfile || default_credentials(scope: scope))
  unless credentials.is_a? Google::Auth::Credentials
    credentials = Datastore::Credentials.new credentials, scope: scope
  end

  Datastore::Dataset.new(
    Datastore::Service.new(
      project_id, credentials,
      timeout: timeout, client_config: client_config
    )
  )
end