Class: Google::Cloud::Datastore::Dataset

Inherits:
Object
  • Object
show all
Defined in:
lib/google/cloud/datastore/dataset.rb,
lib/google/cloud/datastore/dataset/query_results.rb,
lib/google/cloud/datastore/dataset/lookup_results.rb

Overview

Dataset

Dataset is the data saved in a project's Datastore. Dataset is analogous to a database in relational database world.

Dataset is the main object for interacting with Google Datastore. Entity objects are created, read, updated, and deleted by Dataset.

See Google::Cloud#datastore

Examples:

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new

query = datastore.query("Task").
  where("done", "=", false)

tasks = datastore.run query

Direct Known Subclasses

Transaction

Defined Under Namespace

Classes: LookupResults, QueryResults

Instance Method Summary collapse

Instance Method Details

#allocate_ids(incomplete_key, count = 1) ⇒ Array<Google::Cloud::Datastore::Key>

Generate IDs for a Key before creating an entity.

Examples:

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new

task_key = datastore.key "Task"
task_keys = datastore.allocate_ids task_key, 5

Parameters:

  • incomplete_key (Key) —

    A Key without id or name set.

  • count (String) (defaults to: 1) —

    The number of new key IDs to create.

Returns:



102
103
104
105
106
107
108
109
110
111
 
# File 'lib/google/cloud/datastore/dataset.rb', line 102

def allocate_ids incomplete_key, count = 1
  if incomplete_key.complete?
    raise Datastore::KeyError, "An incomplete key must be provided."
  end

  ensure_service!
  incomplete_keys = Array.new(count) { incomplete_key.to_grpc }
  allocate_res = service.allocate_ids(*incomplete_keys)
  allocate_res.keys.map { |key| Key.from_grpc key }
end

#commit {|commit| ... } ⇒ Array<Google::Cloud::Datastore::Entity>

Make multiple changes in a single commit.

Examples:

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new

datastore.commit do |c|
  c.save task3, task4
  c.delete task1, task2
end

Yields:

  • (commit) —

    a block for making changes

Yield Parameters:

  • commit (Commit) —

    The object that changes are made on

Returns:



288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
 
# File 'lib/google/cloud/datastore/dataset.rb', line 288

def commit
  return unless block_given?
  c = Commit.new
  yield c

  ensure_service!
  commit_res = service.commit c.mutations
  entities = c.entities
  returned_keys = commit_res.mutation_results.map(&:key)
  returned_keys.each_with_index do |key, index|
    next if entities[index].nil?
    entities[index].key = Key.from_grpc(key) unless key.nil?
  end
  entities.each { |e| e.key.freeze unless e.persisted? }
  entities
end

#delete(*entities_or_keys) ⇒ Boolean

Remove entities from the Datastore.

Examples:

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new

datastore.delete task1, task2

Parameters:

  • entities_or_keys (Entity, Key) —

    One or more Entity or Key objects to remove.

Returns:

  • (Boolean) —

    Returns true if successful



264
265
266
267
 
# File 'lib/google/cloud/datastore/dataset.rb', line 264

def delete *entities_or_keys
  commit { |c| c.delete(*entities_or_keys) }
  true
end

#entity(*key_or_path, project: nil, namespace: nil) {|entity| ... } ⇒ Google::Cloud::Datastore::Entity

Create a new empty Entity instance. This is a convenience method to make the creation of Entity objects easier.

Examples:

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new

task = datastore.entity

The previous example is equivalent to:

require "google/cloud/datastore"

task = Google::Cloud::Datastore::Entity.new

The key can also be passed in as an object:

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new

task_key = datastore.key "Task", "sampleTask"
task = datastore.entity task_key

Or the key values can be passed in as parameters:

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new

task = datastore.entity "Task", "sampleTask"

The previous example is equivalent to:

require "google/cloud/datastore"

task_key = Google::Cloud::Datastore::Key.new "Task", "sampleTask"
task = Google::Cloud::Datastore::Entity.new
task.key = task_key

The newly created entity can also be configured using block:

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new

task = datastore.entity "Task", "sampleTask" do |t|
  t["type"] = "Personal"
  t["done"] = false
  t["priority"] = 4
  t["description"] = "Learn Cloud Datastore"
end

The previous example is equivalent to:

require "google/cloud/datastore"

task_key = Google::Cloud::Datastore::Key.new "Task", "sampleTask"
task = Google::Cloud::Datastore::Entity.new
task.key = task_key
task["type"] = "Personal"
task["done"] = false
task["priority"] = 4
task["description"] = "Learn Cloud Datastore"

Parameters:

  • key_or_path (Key, Array<Array(String,(String|Integer|nil))>) —

    An optional list of pairs for the key's path. Each pair may include the key's kind (String) and an id (Integer) or name (String). This is optional.

  • project (String) —

    The project of the Key. This is optional.

  • namespace (String) —

    namespace kind of the Key. This is optional.

Yields:

  • (entity) —

    a block yielding a new entity

Yield Parameters:

  • entity (Entity) —

    the newly created entity object

Returns:



855
856
857
858
859
860
861
862
863
864
865
866
867
868
 
# File 'lib/google/cloud/datastore/dataset.rb', line 855

def entity *key_or_path, project: nil, namespace: nil
  entity = Entity.new

  # Set the key
  entity.key = if key_or_path.flatten.first.is_a? Datastore::Key
                 key_or_path.flatten.first
               else
                 key key_or_path, project: project, namespace: namespace
               end

  yield entity if block_given?

  entity
end

#find(key_or_kind, id_or_name = nil, consistency: nil) ⇒ Google::Cloud::Datastore::Entity? Also known as: get

Retrieve an entity by key.

Examples:

Finding an entity with a key:

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new

task_key = datastore.key "Task", "sampleTask"
task = datastore.find task_key

Finding an entity with a kind and id/name:

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new

task = datastore.find "Task", "sampleTask"

Parameters:

  • key_or_kind (Key, String) —

    A Key object or kind string value.

  • id_or_name (Integer, String, nil) (defaults to: nil) —

    The Key's id or name value if a kind was provided in the first parameter.

  • consistency (Symbol) —

    The non-transactional read consistency to use. Cannot be set to :strong for global queries. Accepted values are :eventual and :strong.

    The default consistency depends on the type of lookup used. See Eventual Consistency in Google Cloud Datastore for more information.

Returns:



337
338
339
340
341
342
343
 
# File 'lib/google/cloud/datastore/dataset.rb', line 337

def find key_or_kind, id_or_name = nil, consistency: nil
  key = key_or_kind
  unless key.is_a? Google::Cloud::Datastore::Key
    key = Key.new key_or_kind, id_or_name
  end
  find_all(key, consistency: consistency).first
end

#find_all(*keys, consistency: nil) ⇒ Google::Cloud::Datastore::Dataset::LookupResults Also known as: lookup

Retrieve the entities for the provided keys. The order of results is undefined and has no relation to the order of keys arguments.

Examples:

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new

task_key1 = datastore.key "Task", "sampleTask1"
task_key2 = datastore.key "Task", "sampleTask2"
tasks = datastore.find_all task_key1, task_key2

Parameters:

  • keys (Key) —

    One or more Key objects to find records for.

  • consistency (Symbol) —

    The non-transactional read consistency to use. Cannot be set to :strong for global queries. Accepted values are :eventual and :strong.

    The default consistency depends on the type of lookup used. See Eventual Consistency in Google Cloud Datastore for more information.

Returns:



371
372
373
374
375
376
377
 
# File 'lib/google/cloud/datastore/dataset.rb', line 371

def find_all *keys, consistency: nil
  ensure_service!
  check_consistency! consistency
  lookup_res = service.lookup(*Array(keys).flatten.map(&:to_grpc),
                              consistency: consistency)
  LookupResults.from_grpc lookup_res, service, consistency
end

#gql(query, bindings = {}) ⇒ Google::Cloud::Datastore::GqlQuery

Create a new GqlQuery instance. This is a convenience method to make the creation of GqlQuery objects easier.

Examples:

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new

gql_query = datastore.gql "SELECT * FROM Task WHERE done = @done",
                          done: false
tasks = datastore.run gql_query

The previous example is equivalent to:

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new

gql_query = Google::Cloud::Datastore::GqlQuery.new
gql_query.query_string = "SELECT * FROM Task WHERE done = @done"
gql_query.named_bindings = {done: false}
tasks = datastore.run gql_query

Parameters:

  • query (String) —

    The GQL query string.

  • bindings (Hash) (defaults to: {}) —

    Named bindings for the GQL query string, each key must match regex [A-Za-z_$][A-Za-z_$0-9]*, must not match regex __.*__, and must not be "". The value must be an Object that can be stored as an Entity property value, or a Cursor.

Returns:



681
682
683
684
685
686
 
# File 'lib/google/cloud/datastore/dataset.rb', line 681

def gql query, bindings = {}
  gql = GqlQuery.new
  gql.query_string = query
  gql.named_bindings = bindings unless bindings.empty?
  gql
end

#insert(*entities) ⇒ Array<Google::Cloud::Datastore::Entity>

Insert one or more entities to the Datastore. An InvalidArgumentError will raised if the entities cannot be inserted.

Examples:

Insert a new entity:

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new

task = datastore.entity "Task" do |t|
  t["type"] = "Personal"
  t["done"] = false
  t["priority"] = 4
  t["description"] = "Learn Cloud Datastore"
end
task.key.id #=> nil
datastore.insert task
task.key.id #=> 123456

Insert multiple new entities in a batch:

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new

task1 = datastore.entity "Task" do |t|
  t["type"] = "Personal"
  t["done"] = false
  t["priority"] = 4
  t["description"] = "Learn Cloud Datastore"
end

task2 = datastore.entity "Task" do |t|
  t["type"] = "Personal"
  t["done"] = false
  t["priority"] = 5
  t["description"] = "Integrate Cloud Datastore"
end

task_key1, task_key2 = datastore.insert(task1, task2).map(&:key)

Parameters:

  • entities (Entity) —

    One or more entity objects to be inserted.

Returns:



214
215
216
 
# File 'lib/google/cloud/datastore/dataset.rb', line 214

def insert *entities
  commit { |c| c.insert(*entities) }
end

#key(*path, project: nil, namespace: nil) ⇒ Google::Cloud::Datastore::Key

Create a new Key instance. This is a convenience method to make the creation of Key objects easier.

Examples:

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new

task_key = datastore.key "Task", "sampleTask"

The previous example is equivalent to:

require "google/cloud/datastore"

task_key = Google::Cloud::Datastore::Key.new "Task", "sampleTask"

Create an empty key:

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new

key = datastore.key

Create an incomplete key:

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new

key = datastore.key "User"

Create a key with a parent:

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new

key = datastore.key [["TaskList", "default"],
                     ["Task", "sampleTask"]]
key.path #=> [["TaskList", "default"], ["Task", "sampleTask"]]

Create a key with multi-level ancestry:

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new

key = datastore.key([
  ["User", "alice"],
  ["TaskList", "default"],
  ["Task", "sampleTask"]
])
key.path.count #=> 3
key.path[0] #=> ["User", "alice"]
key.path[1] #=> ["TaskList", "default"]
key.path[2] #=> ["Task", "sampleTask"]

Create an incomplete key with a parent:

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new

key = datastore.key "TaskList", "default", "Task"
key.path #=> [["TaskList", "default"], ["Task", nil]]

Create a key with a project and namespace:

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new

key = datastore.key ["TaskList", "default"], ["Task", "sampleTask"],
                    project: "my-todo-project",
                    namespace: "example-ns"
key.path #=> [["TaskList", "default"], ["Task", "sampleTask"]]
key.project #=> "my-todo-project"
key.namespace #=> "example-ns"

Parameters:

  • path (Array<Array(String,(String|Integer|nil))>) —

    An optional list of pairs for the key's path. Each pair may include the key's kind (String) and an id (Integer) or name (String). This is optional.

  • project (String) —

    The project of the Key. This is optional.

  • namespace (String) —

    namespace kind of the Key. This is optional.

Returns:



771
772
773
774
775
776
777
778
779
780
781
 
# File 'lib/google/cloud/datastore/dataset.rb', line 771

def key *path, project: nil, namespace: nil
  path = path.flatten.each_slice(2).to_a # group in pairs
  kind, id_or_name = path.pop
  Key.new(kind, id_or_name).tap do |k|
    k.project = project
    k.namespace = namespace
    unless path.empty?
      k.parent = key path, project: project, namespace: namespace
    end
  end
end

#project_id ⇒ Object Also known as: project

The Datastore project connected to.

Examples:

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new(
  project_id: "my-todo-project",
  credentials: "/path/to/keyfile.json"
)

datastore.project_id #=> "my-todo-project"


81
82
83
 
# File 'lib/google/cloud/datastore/dataset.rb', line 81

def project_id
  service.project
end

#query(*kinds) ⇒ Google::Cloud::Datastore::Query

Create a new Query instance. This is a convenience method to make the creation of Query objects easier.

Examples:

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new

query = datastore.query("Task").
  where("done", "=", false)
tasks = datastore.run query

The previous example is equivalent to:

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new

query = Google::Cloud::Datastore::Query.new.
  kind("Task").
  where("done", "=", false)
tasks = datastore.run query

Parameters:

  • kinds (String) —

    The kind of entities to query. This is optional.

Returns:



644
645
646
647
648
 
# File 'lib/google/cloud/datastore/dataset.rb', line 644

def query *kinds
  query = Query.new
  query.kind(*kinds) unless kinds.empty?
  query
end

#read_only_transaction {|tx| ... } ⇒ Object Also known as: snapshot

Creates a read-only transaction that provides a consistent snapshot of Cloud Datastore. This can be useful when multiple reads are needed to render a page or export data that must be consistent.

A read-only transaction cannot modify entities; in return they do not contend with other read-write or read-only transactions. Using a read-only transaction for transactions that only read data will potentially improve throughput.

Read-only single-group transactions never fail due to concurrent modifications, so you don't have to implement retries upon failure. However, multi-entity-group transactions can fail due to concurrent modifications, so these should have retries.

Examples:

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new

task_list_key = datastore.key "TaskList", "default"
query = datastore.query("Task").
  ancestor(task_list_key)

tasks = nil

datastore.read_only_transaction do |tx|
  task_list = tx.find task_list_key
  if task_list
    tasks = tx.run query
  end
end

Yields:

  • (tx) —

    a block yielding a new transaction

Yield Parameters:

See Also:



598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
 
# File 'lib/google/cloud/datastore/dataset.rb', line 598

def read_only_transaction
  tx = ReadOnlyTransaction.new service
  return tx unless block_given?

  begin
    yield tx
    tx.commit
  rescue StandardError
    begin
      tx.rollback
    rescue StandardError
      raise TransactionError,
            "Transaction failed to commit and rollback."
    end
    raise TransactionError, "Transaction failed to commit."
  end
end

#run(query, namespace: nil, consistency: nil) ⇒ Google::Cloud::Datastore::Dataset::QueryResults Also known as: run_query

Retrieve entities specified by a Query.

Examples:

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new

query = datastore.query("Task").
  where("done", "=", false)
tasks = datastore.run query

Run an ancestor query with eventual consistency:

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new

task_list_key = datastore.key "TaskList", "default"
query = datastore.query.kind("Task").
  ancestor(task_list_key)

tasks = datastore.run query, consistency: :eventual

Run the query within a namespace with the namespace option:

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new

query = datastore.query("Task").
  where("done", "=", false)
tasks = datastore.run query, namespace: "example-ns"

Run the query with a GQL string.

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new

gql_query = datastore.gql "SELECT * FROM Task WHERE done = @done",
                          done: false
tasks = datastore.run gql_query

Run the GQL query within a namespace with namespace option:

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new

gql_query = datastore.gql "SELECT * FROM Task WHERE done = @done",
                          done: false
tasks = datastore.run gql_query, namespace: "example-ns"

Parameters:

  • query (Query, GqlQuery) —

    The object with the search criteria.

  • namespace (String) —

    The namespace the query is to run within.

  • consistency (Symbol) —

    The non-transactional read consistency to use. Cannot be set to :strong for global queries. Accepted values are :eventual and :strong.

    The default consistency depends on the type of query used. See Eventual Consistency in Google Cloud Datastore for more information.

Returns:



443
444
445
446
447
448
449
450
451
452
453
 
# File 'lib/google/cloud/datastore/dataset.rb', line 443

def run query, namespace: nil, consistency: nil
  ensure_service!
  unless query.is_a?(Query) || query.is_a?(GqlQuery)
    raise ArgumentError, "Cannot run a #{query.class} object."
  end
  check_consistency! consistency
  query_res = service.run_query query.to_grpc, namespace,
                                consistency: consistency
  QueryResults.from_grpc query_res, service, namespace,
                         query.to_grpc.dup
end

#save(*entities) ⇒ Array<Google::Cloud::Datastore::Entity> Also known as: upsert

Persist one or more entities to the Datastore.

Examples:

Insert a new entity:

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new

task = datastore.entity "Task" do |t|
  t["type"] = "Personal"
  t["done"] = false
  t["priority"] = 4
  t["description"] = "Learn Cloud Datastore"
end
task.key.id #=> nil
datastore.save task
task.key.id #=> 123456

Insert multiple new entities in a batch:

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new

task1 = datastore.entity "Task" do |t|
  t["type"] = "Personal"
  t["done"] = false
  t["priority"] = 4
  t["description"] = "Learn Cloud Datastore"
end

task2 = datastore.entity "Task" do |t|
  t["type"] = "Personal"
  t["done"] = false
  t["priority"] = 5
  t["description"] = "Integrate Cloud Datastore"
end

task_key1, task_key2 = datastore.save(task1, task2).map(&:key)

Update an existing entity:

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new

task = datastore.find "Task", "sampleTask"
task["priority"] = 5
datastore.save task

Parameters:

  • entities (Entity) —

    One or more entity objects to be saved.

Returns:



165
166
167
 
# File 'lib/google/cloud/datastore/dataset.rb', line 165

def save *entities
  commit { |c| c.save(*entities) }
end

#transaction(deadline: nil, previous_transaction: nil) {|tx| ... } ⇒ Object

Creates a Datastore Transaction.

Transactions using the block syntax are committed upon block completion and are automatically retried when known errors are raised during commit. All other errors will be passed on.

All changes are accumulated in memory until the block completes. Transactions will be automatically retried when possible, until deadline is reached. This operation makes separate API requests to begin and commit the transaction.

Examples:

Runs the given block in a database transaction:

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new

task = datastore.entity "Task", "sampleTask" do |t|
  t["type"] = "Personal"
  t["done"] = false
  t["priority"] = 4
  t["description"] = "Learn Cloud Datastore"
end

datastore.transaction do |tx|
  if tx.find(task.key).nil?
    tx.save task
  end
end

If no block is given, a Transaction object is returned:

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new

task = datastore.entity "Task", "sampleTask" do |t|
  t["type"] = "Personal"
  t["done"] = false
  t["priority"] = 4
  t["description"] = "Learn Cloud Datastore"
end

tx = datastore.transaction
begin
  if tx.find(task.key).nil?
    tx.save task
  end
  tx.commit
rescue
  tx.rollback
end

Parameters:

  • deadline (Numeric) —

    The total amount of time in seconds the transaction has to succeed. The default is 60.

  • previous_transaction (String) —

    The transaction identifier of a transaction that is being retried. Read-write transactions may fail due to contention. A read-write transaction can be retried by specifying previous_transaction when creating the new transaction.

    Specifying previous_transaction provides information that can be used to improve throughput. In particular, if transactional operations A and B conflict, specifying the previous_transaction can help to prevent livelock. (See Transaction#id)

Yields:

  • (tx) —

    a block yielding a new transaction

Yield Parameters:

See Also:



526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
 
# File 'lib/google/cloud/datastore/dataset.rb', line 526

def transaction deadline: nil, previous_transaction: nil
  deadline = validate_deadline deadline
  backoff = 1.0
  start_time = Time.now

  tx = Transaction.new \
    service, previous_transaction: previous_transaction
  return tx unless block_given?

  begin
    yield tx
    tx.commit
  rescue Google::Cloud::UnavailableError => err
    # Re-raise if deadline has passed
    raise err if Time.now - start_time > deadline

    # Sleep with incremental backoff
    sleep(backoff *= 1.3)

    # Create new transaction and retry the block
    tx = Transaction.new service, previous_transaction: tx.id
    retry
  rescue StandardError
    begin
      tx.rollback
    rescue StandardError
      raise TransactionError,
            "Transaction failed to commit and rollback."
    end
    raise TransactionError, "Transaction failed to commit."
  end
end

#update(*entities) ⇒ Array<Google::Cloud::Datastore::Entity>

Update one or more entities to the Datastore. An InvalidArgumentError will raised if the entities cannot be updated.

Examples:

Update an existing entity:

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new

task = datastore.find "Task", "sampleTask"
task["done"] = true
datastore.save task

Update multiple new entities in a batch:

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new

query = datastore.query("Task").where("done", "=", false)
tasks = datastore.run query
tasks.each { |t| t["done"] = true }
datastore.update tasks

Parameters:

  • entities (Entity) —

    One or more entity objects to be updated.

Returns:



245
246
247
 
# File 'lib/google/cloud/datastore/dataset.rb', line 245

def update *entities
  commit { |c| c.update(*entities) }
end