Class: Google::Cloud::Datastore::Dataset

Inherits:
Object
  • Object
show all
Defined in:
lib/google/cloud/datastore/dataset.rb,
lib/google/cloud/datastore/dataset/query_results.rb,
lib/google/cloud/datastore/dataset/lookup_results.rb

Overview

Dataset

Dataset is the data saved in a project's Datastore. Dataset is analogous to a database in relational database world.

Dataset is the main object for interacting with Google Datastore. Entity objects are created, read, updated, and deleted by Dataset.

See Google::Cloud#datastore

Examples:

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new

query = datastore.query("Task").
  where("done", "=", false)

tasks = datastore.run query

Direct Known Subclasses

Transaction

Defined Under Namespace

Classes: LookupResults, QueryResults

Instance Method Summary collapse

Instance Method Details

#allocate_ids(incomplete_key, count = 1) ⇒ Array<Google::Cloud::Datastore::Key>

Generate IDs for a Key before creating an entity.

Examples:

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new

task_key = datastore.key "Task"
task_keys = datastore.allocate_ids task_key, 5

Parameters:

  • incomplete_key (Key) —

    A Key without id or name set.

  • count (String) (defaults to: 1) —

    The number of new key IDs to create.

Returns:



113
114
115
116
117
118
119
120
121
122
123
 
# File 'lib/google/cloud/datastore/dataset.rb', line 113

def allocate_ids incomplete_key, count = 1
  if incomplete_key.complete?
    fail Google::Cloud::Datastore::KeyError,
         "An incomplete key must be provided."
  end

  ensure_service!
  incomplete_keys = count.times.map { incomplete_key.to_grpc }
  allocate_res = service.allocate_ids(*incomplete_keys)
  allocate_res.keys.map { |key| Key.from_grpc key }
end

#commit {|commit| ... } ⇒ Array<Google::Cloud::Datastore::Entity>

Make multiple changes in a single commit.

Examples:

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new

datastore.commit do |c|
  c.save task3, task4
  c.delete task1, task2
end

Yields:

  • (commit) —

    a block for making changes

Yield Parameters:

  • commit (Commit) —

    The object that changes are made on

Returns:



300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
 
# File 'lib/google/cloud/datastore/dataset.rb', line 300

def commit
  return unless block_given?
  c = Commit.new
  yield c

  ensure_service!
  commit_res = service.commit c.mutations
  entities = c.entities
  returned_keys = commit_res.mutation_results.map(&:key)
  returned_keys.each_with_index do |key, index|
    next if entities[index].nil?
    entities[index].key = Key.from_grpc(key) unless key.nil?
  end
  entities.each { |e| e.key.freeze unless e.persisted? }
  entities
end

#delete(*entities_or_keys) ⇒ Boolean

Remove entities from the Datastore.

Examples:

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new

datastore.delete task1, task2

Parameters:

  • entities_or_keys (Entity, Key) —

    One or more Entity or Key objects to remove.

Returns:

  • (Boolean) —

    Returns true if successful



276
277
278
279
 
# File 'lib/google/cloud/datastore/dataset.rb', line 276

def delete *entities_or_keys
  commit { |c| c.delete(*entities_or_keys) }
  true
end

#entity(*key_or_path, project: nil, namespace: nil) {|entity| ... } ⇒ Google::Cloud::Datastore::Entity

Create a new empty Entity instance. This is a convenience method to make the creation of Entity objects easier.

Examples:

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new

task = datastore.entity

The previous example is equivalent to:

require "google/cloud/datastore"

task = Google::Cloud::Datastore::Entity.new

The key can also be passed in as an object:

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new

task_key = datastore.key "Task", "sampleTask"
task = datastore.entity task_key

Or the key values can be passed in as parameters:

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new

task = datastore.entity "Task", "sampleTask"

The previous example is equivalent to:

require "google/cloud/datastore"

task_key = Google::Cloud::Datastore::Key.new "Task", "sampleTask"
task = Google::Cloud::Datastore::Entity.new
task.key = task_key

The newly created entity can also be configured using block:

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new

task = datastore.entity "Task", "sampleTask" do |t|
  t["type"] = "Personal"
  t["done"] = false
  t["priority"] = 4
  t["description"] = "Learn Cloud Datastore"
end

The previous example is equivalent to:

require "google/cloud/datastore"

task_key = Google::Cloud::Datastore::Key.new "Task", "sampleTask"
task = Google::Cloud::Datastore::Entity.new
task.key = task_key
task["type"] = "Personal"
task["done"] = false
task["priority"] = 4
task["description"] = "Learn Cloud Datastore"

Parameters:

  • key_or_path (Key, Array<Array(String,(String|Integer|nil))>) —

    An optional list of pairs for the key's path. Each pair may include the key's kind (String) and an id (Integer) or name (String). This is optional.

  • project (String) —

    The project of the Key. This is optional.

  • namespace (String) —

    namespace kind of the Key. This is optional.

Yields:

  • (entity) —

    a block yielding a new entity

Yield Parameters:

  • entity (Entity) —

    the newly created entity object

Returns:



867
868
869
870
871
872
873
874
875
876
877
878
879
880
 
# File 'lib/google/cloud/datastore/dataset.rb', line 867

def entity *key_or_path, project: nil, namespace: nil
  entity = Entity.new

  # Set the key
  if key_or_path.flatten.first.is_a? Google::Cloud::Datastore::Key
    entity.key = key_or_path.flatten.first
  else
    entity.key = key key_or_path, project: project, namespace: namespace
  end

  yield entity if block_given?

  entity
end

#find(key_or_kind, id_or_name = nil, consistency: nil) ⇒ Google::Cloud::Datastore::Entity? Also known as: get

Retrieve an entity by key.

Examples:

Finding an entity with a key:

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new

task_key = datastore.key "Task", "sampleTask"
task = datastore.find task_key

Finding an entity with a kind and id/name:

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new

task = datastore.find "Task", "sampleTask"

Parameters:

  • key_or_kind (Key, String) —

    A Key object or kind string value.

  • id_or_name (Integer, String, nil) (defaults to: nil) —

    The Key's id or name value if a kind was provided in the first parameter.

  • consistency (Symbol) —

    The non-transactional read consistency to use. Cannot be set to :strong for global queries. Accepted values are :eventual and :strong.

    The default consistency depends on the type of lookup used. See Eventual Consistency in Google Cloud Datastore for more information.

Returns:



349
350
351
352
353
354
355
 
# File 'lib/google/cloud/datastore/dataset.rb', line 349

def find key_or_kind, id_or_name = nil, consistency: nil
  key = key_or_kind
  unless key.is_a? Google::Cloud::Datastore::Key
    key = Key.new key_or_kind, id_or_name
  end
  find_all(key, consistency: consistency).first
end

#find_all(*keys, consistency: nil) ⇒ Google::Cloud::Datastore::Dataset::LookupResults Also known as: lookup

Retrieve the entities for the provided keys. The order of results is undefined and has no relation to the order of keys arguments.

Examples:

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new

task_key1 = datastore.key "Task", "sampleTask1"
task_key2 = datastore.key "Task", "sampleTask2"
tasks = datastore.find_all task_key1, task_key2

Parameters:

  • keys (Key) —

    One or more Key objects to find records for.

  • consistency (Symbol) —

    The non-transactional read consistency to use. Cannot be set to :strong for global queries. Accepted values are :eventual and :strong.

    The default consistency depends on the type of lookup used. See Eventual Consistency in Google Cloud Datastore for more information.

Returns:



383
384
385
386
387
388
389
 
# File 'lib/google/cloud/datastore/dataset.rb', line 383

def find_all *keys, consistency: nil
  ensure_service!
  check_consistency! consistency
  lookup_res = service.lookup(*Array(keys).flatten.map(&:to_grpc),
                              consistency: consistency)
  LookupResults.from_grpc lookup_res, service, consistency
end

#gql(query, bindings = {}) ⇒ Google::Cloud::Datastore::GqlQuery

Create a new GqlQuery instance. This is a convenience method to make the creation of GqlQuery objects easier.

Examples:

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new

gql_query = datastore.gql "SELECT * FROM Task WHERE done = @done",
                          done: false
tasks = datastore.run gql_query

The previous example is equivalent to:

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new

gql_query = Google::Cloud::Datastore::GqlQuery.new
gql_query.query_string = "SELECT * FROM Task WHERE done = @done"
gql_query.named_bindings = {done: false}
tasks = datastore.run gql_query

Parameters:

  • query (String) —

    The GQL query string.

  • bindings (Hash) (defaults to: {}) —

    Named bindings for the GQL query string, each key must match regex [A-Za-z_$][A-Za-z_$0-9]*, must not match regex __.*__, and must not be "". The value must be an Object that can be stored as an Entity property value, or a Cursor.

Returns:



693
694
695
696
697
698
 
# File 'lib/google/cloud/datastore/dataset.rb', line 693

def gql query, bindings = {}
  gql = GqlQuery.new
  gql.query_string = query
  gql.named_bindings = bindings unless bindings.empty?
  gql
end

#insert(*entities) ⇒ Array<Google::Cloud::Datastore::Entity>

Insert one or more entities to the Datastore. An InvalidArgumentError will raised if the entities cannot be inserted.

Examples:

Insert a new entity:

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new

task = datastore.entity "Task" do |t|
  t["type"] = "Personal"
  t["done"] = false
  t["priority"] = 4
  t["description"] = "Learn Cloud Datastore"
end
task.key.id #=> nil
datastore.insert task
task.key.id #=> 123456

Insert multiple new entities in a batch:

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new

task1 = datastore.entity "Task" do |t|
  t["type"] = "Personal"
  t["done"] = false
  t["priority"] = 4
  t["description"] = "Learn Cloud Datastore"
end

task2 = datastore.entity "Task" do |t|
  t["type"] = "Personal"
  t["done"] = false
  t["priority"] = 5
  t["description"] = "Integrate Cloud Datastore"
end

task_key1, task_key2 = datastore.insert(task1, task2).map(&:key)

Parameters:

  • entities (Entity) —

    One or more entity objects to be inserted.

Returns:



226
227
228
 
# File 'lib/google/cloud/datastore/dataset.rb', line 226

def insert *entities
  commit { |c| c.insert(*entities) }
end

#key(*path, project: nil, namespace: nil) ⇒ Google::Cloud::Datastore::Key

Create a new Key instance. This is a convenience method to make the creation of Key objects easier.

Examples:

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new

task_key = datastore.key "Task", "sampleTask"

The previous example is equivalent to:

require "google/cloud/datastore"

task_key = Google::Cloud::Datastore::Key.new "Task", "sampleTask"

Create an empty key:

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new

key = datastore.key

Create an incomplete key:

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new

key = datastore.key "User"

Create a key with a parent:

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new

key = datastore.key [["TaskList", "default"],
                     ["Task", "sampleTask"]]
key.path #=> [["TaskList", "default"], ["Task", "sampleTask"]]

Create a key with multi-level ancestry:

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new

key = datastore.key([
  ["User", "alice"],
  ["TaskList", "default"],
  ["Task", "sampleTask"]
])
key.path.count #=> 3
key.path[0] #=> ["User", "alice"]
key.path[1] #=> ["TaskList", "default"]
key.path[2] #=> ["Task", "sampleTask"]

Create an incomplete key with a parent:

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new

key = datastore.key "TaskList", "default", "Task"
key.path #=> [["TaskList", "default"], ["Task", nil]]

Create a key with a project and namespace:

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new

key = datastore.key ["TaskList", "default"], ["Task", "sampleTask"],
                    project: "my-todo-project",
                    namespace: "example-ns"
key.path #=> [["TaskList", "default"], ["Task", "sampleTask"]]
key.project #=> "my-todo-project"
key.namespace #=> "example-ns"

Parameters:

  • path (Array<Array(String,(String|Integer|nil))>) —

    An optional list of pairs for the key's path. Each pair may include the key's kind (String) and an id (Integer) or name (String). This is optional.

  • project (String) —

    The project of the Key. This is optional.

  • namespace (String) —

    namespace kind of the Key. This is optional.

Returns:



783
784
785
786
787
788
789
790
791
792
793
 
# File 'lib/google/cloud/datastore/dataset.rb', line 783

def key *path, project: nil, namespace: nil
  path = path.flatten.each_slice(2).to_a # group in pairs
  kind, id_or_name = path.pop
  Key.new(kind, id_or_name).tap do |k|
    k.project = project
    k.namespace = namespace
    unless path.empty?
      k.parent = key path, project: project, namespace: namespace
    end
  end
end

#project_id ⇒ Object Also known as: project

The Datastore project connected to.

Examples:

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new(
  project_id: "my-todo-project",
  credentials: "/path/to/keyfile.json"
)

datastore.project_id #=> "my-todo-project"


82
83
84
 
# File 'lib/google/cloud/datastore/dataset.rb', line 82

def project_id
  service.project
end

#query(*kinds) ⇒ Google::Cloud::Datastore::Query

Create a new Query instance. This is a convenience method to make the creation of Query objects easier.

Examples:

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new

query = datastore.query("Task").
  where("done", "=", false)
tasks = datastore.run query

The previous example is equivalent to:

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new

query = Google::Cloud::Datastore::Query.new.
  kind("Task").
  where("done", "=", false)
tasks = datastore.run query

Parameters:

  • kinds (String) —

    The kind of entities to query. This is optional.

Returns:



656
657
658
659
660
 
# File 'lib/google/cloud/datastore/dataset.rb', line 656

def query *kinds
  query = Query.new
  query.kind(*kinds) unless kinds.empty?
  query
end

#read_only_transaction {|tx| ... } ⇒ Object Also known as: snapshot

Creates a read-only transaction that provides a consistent snapshot of Cloud Datastore. This can be useful when multiple reads are needed to render a page or export data that must be consistent.

A read-only transaction cannot modify entities; in return they do not contend with other read-write or read-only transactions. Using a read-only transaction for transactions that only read data will potentially improve throughput.

Read-only single-group transactions never fail due to concurrent modifications, so you don't have to implement retries upon failure. However, multi-entity-group transactions can fail due to concurrent modifications, so these should have retries.

Examples:

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new

task_list_key = datastore.key "TaskList", "default"
query = datastore.query("Task").
  ancestor(task_list_key)

tasks = nil

datastore.read_only_transaction do |tx|
  task_list = tx.find task_list_key
  if task_list
    tasks = tx.run query
  end
end

Yields:

  • (tx) —

    a block yielding a new transaction

Yield Parameters:

See Also:



610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
 
# File 'lib/google/cloud/datastore/dataset.rb', line 610

def read_only_transaction
  tx = ReadOnlyTransaction.new service
  return tx unless block_given?

  begin
    yield tx
    tx.commit
  rescue
    begin
      tx.rollback
    rescue
      raise TransactionError,
            "Transaction failed to commit and rollback."
    end
    raise TransactionError, "Transaction failed to commit."
  end
end

#run(query, namespace: nil, consistency: nil) ⇒ Google::Cloud::Datastore::Dataset::QueryResults Also known as: run_query

Retrieve entities specified by a Query.

Examples:

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new

query = datastore.query("Task").
  where("done", "=", false)
tasks = datastore.run query

Run an ancestor query with eventual consistency:

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new

task_list_key = datastore.key "TaskList", "default"
query = datastore.query.kind("Task").
  ancestor(task_list_key)

tasks = datastore.run query, consistency: :eventual

Run the query within a namespace with the namespace option:

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new

query = datastore.query("Task").
  where("done", "=", false)
tasks = datastore.run query, namespace: "example-ns"

Run the query with a GQL string.

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new

gql_query = datastore.gql "SELECT * FROM Task WHERE done = @done",
                          done: false
tasks = datastore.run gql_query

Run the GQL query within a namespace with namespace option:

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new

gql_query = datastore.gql "SELECT * FROM Task WHERE done = @done",
                          done: false
tasks = datastore.run gql_query, namespace: "example-ns"

Parameters:

  • query (Query, GqlQuery) —

    The object with the search criteria.

  • namespace (String) —

    The namespace the query is to run within.

  • consistency (Symbol) —

    The non-transactional read consistency to use. Cannot be set to :strong for global queries. Accepted values are :eventual and :strong.

    The default consistency depends on the type of query used. See Eventual Consistency in Google Cloud Datastore for more information.

Returns:



455
456
457
458
459
460
461
462
463
464
465
 
# File 'lib/google/cloud/datastore/dataset.rb', line 455

def run query, namespace: nil, consistency: nil
  ensure_service!
  unless query.is_a?(Query) || query.is_a?(GqlQuery)
    fail ArgumentError, "Cannot run a #{query.class} object."
  end
  check_consistency! consistency
  query_res = service.run_query query.to_grpc, namespace,
                                consistency: consistency
  QueryResults.from_grpc query_res, service, namespace,
                         query.to_grpc.dup
end

#save(*entities) ⇒ Array<Google::Cloud::Datastore::Entity> Also known as: upsert

Persist one or more entities to the Datastore.

Examples:

Insert a new entity:

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new

task = datastore.entity "Task" do |t|
  t["type"] = "Personal"
  t["done"] = false
  t["priority"] = 4
  t["description"] = "Learn Cloud Datastore"
end
task.key.id #=> nil
datastore.save task
task.key.id #=> 123456

Insert multiple new entities in a batch:

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new

task1 = datastore.entity "Task" do |t|
  t["type"] = "Personal"
  t["done"] = false
  t["priority"] = 4
  t["description"] = "Learn Cloud Datastore"
end

task2 = datastore.entity "Task" do |t|
  t["type"] = "Personal"
  t["done"] = false
  t["priority"] = 5
  t["description"] = "Integrate Cloud Datastore"
end

task_key1, task_key2 = datastore.save(task1, task2).map(&:key)

Update an existing entity:

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new

task = datastore.find "Task", "sampleTask"
task["priority"] = 5
datastore.save task

Parameters:

  • entities (Entity) —

    One or more entity objects to be saved.

Returns:



177
178
179
 
# File 'lib/google/cloud/datastore/dataset.rb', line 177

def save *entities
  commit { |c| c.save(*entities) }
end

#transaction(deadline: nil, previous_transaction: nil) {|tx| ... } ⇒ Object

Creates a Datastore Transaction.

Transactions using the block syntax are committed upon block completion and are automatically retried when known errors are raised during commit. All other errors will be passed on.

All changes are accumulated in memory until the block completes. Transactions will be automatically retried when possible, until deadline is reached. This operation makes separate API requests to begin and commit the transaction.

Examples:

Runs the given block in a database transaction:

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new

task = datastore.entity "Task", "sampleTask" do |t|
  t["type"] = "Personal"
  t["done"] = false
  t["priority"] = 4
  t["description"] = "Learn Cloud Datastore"
end

datastore.transaction do |tx|
  if tx.find(task.key).nil?
    tx.save task
  end
end

If no block is given, a Transaction object is returned:

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new

task = datastore.entity "Task", "sampleTask" do |t|
  t["type"] = "Personal"
  t["done"] = false
  t["priority"] = 4
  t["description"] = "Learn Cloud Datastore"
end

tx = datastore.transaction
begin
  if tx.find(task.key).nil?
    tx.save task
  end
  tx.commit
rescue
  tx.rollback
end

Parameters:

  • deadline (Numeric) —

    The total amount of time in seconds the transaction has to succeed. The default is 60.

  • previous_transaction (String) —

    The transaction identifier of a transaction that is being retried. Read-write transactions may fail due to contention. A read-write transaction can be retried by specifying previous_transaction when creating the new transaction.

    Specifying previous_transaction provides information that can be used to improve throughput. In particular, if transactional operations A and B conflict, specifying the previous_transaction can help to prevent livelock. (See Transaction#id)

Yields:

  • (tx) —

    a block yielding a new transaction

Yield Parameters:

See Also:



538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
 
# File 'lib/google/cloud/datastore/dataset.rb', line 538

def transaction deadline: nil, previous_transaction: nil
  deadline = validate_deadline deadline
  backoff = 1.0
  start_time = Time.now

  tx = Transaction.new \
    service, previous_transaction: previous_transaction
  return tx unless block_given?

  begin
    yield tx
    tx.commit
  rescue Google::Cloud::UnavailableError => err
    # Re-raise if deadline has passed
    raise err if Time.now - start_time > deadline

    # Sleep with incremental backoff
    sleep(backoff *= 1.3)

    # Create new transaction and retry the block
    tx = Transaction.new service, previous_transaction: tx.id
    retry
  rescue
    begin
      tx.rollback
    rescue
      raise TransactionError,
            "Transaction failed to commit and rollback."
    end
    raise TransactionError, "Transaction failed to commit."
  end
end

#update(*entities) ⇒ Array<Google::Cloud::Datastore::Entity>

Update one or more entities to the Datastore. An InvalidArgumentError will raised if the entities cannot be updated.

Examples:

Update an existing entity:

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new

task = datastore.find "Task", "sampleTask"
task["done"] = true
datastore.save task

Update multiple new entities in a batch:

require "google/cloud/datastore"

datastore = Google::Cloud::Datastore.new

query = datastore.query("Task").where("done", "=", false)
tasks = datastore.run query
tasks.each { |t| t["done"] = true }
datastore.update tasks

Parameters:

  • entities (Entity) —

    One or more entity objects to be updated.

Returns:



257
258
259
 
# File 'lib/google/cloud/datastore/dataset.rb', line 257

def update *entities
  commit { |c| c.update(*entities) }
end