Class: Google::Cloud::Bigquery::External::BigtableSource

Inherits:
DataSource
  • Object
show all
Defined in:
lib/google/cloud/bigquery/external.rb

Overview

BigtableSource

BigtableSource is a subclass of DataSource and represents a Bigtable external data source that can be queried from directly, even though the data is not stored in BigQuery. Instead of loading or streaming the data, this object references the external data source.

Examples:

require "google/cloud/bigquery"

bigquery = Google::Cloud::Bigquery.new

bigtable_url = "https://googleapis.com/bigtable/projects/..."
bigtable_table = bigquery.external bigtable_url do |bt|
  bt.rowkey_as_string = true
  bt.add_family "user" do |u|
    u.add_string "name"
    u.add_string "email"
    u.add_integer "age"
    u.add_boolean "active"
  end
end

data = bigquery.query "SELECT * FROM my_ext_table",
                      external: { my_ext_table: bigtable_table }

data.each do |row|
  puts row[:name]
end

Defined Under Namespace

Classes: Column, ColumnFamily

Instance Method Summary collapse

Methods inherited from DataSource

#autodetect, #autodetect=, #avro?, #backup?, #bigtable?, #compression, #compression=, #csv?, #format, #ignore_unknown, #ignore_unknown=, #json?, #max_bad_records, #max_bad_records=, #sheets?, #urls

Instance Method Details

#add_family(family_id, encoding: nil, latest: nil, type: nil) {|family| ... } ⇒ BigtableSource::ColumnFamily

Add a column family to expose in the table schema along with its types. Columns belonging to the column family may also be exposed.

Examples:

require "google/cloud/bigquery"

bigquery = Google::Cloud::Bigquery.new

bigtable_url = "https://googleapis.com/bigtable/projects/..."
bigtable_table = bigquery.external bigtable_url do |bt|
  bt.rowkey_as_string = true
  bt.add_family "user" do |u|
    u.add_string "name"
    u.add_string "email"
    u.add_integer "age"
    u.add_boolean "active"
  end
end

Parameters:

Yields:

  • (family) —

    a block for setting the family

Yield Parameters:

Returns:



1339
1340
1341
1342
1343
1344
1345
1346
1347
1348
1349
 
# File 'lib/google/cloud/bigquery/external.rb', line 1339

def add_family family_id, encoding: nil, latest: nil, type: nil
  frozen_check!
  fam = BigtableSource::ColumnFamily.new
  fam.family_id = family_id
  fam.encoding = encoding if encoding
  fam.latest = latest if latest
  fam.type = type if type
  yield fam if block_given?
  @families << fam
  fam
end

#families ⇒ Array<BigtableSource::ColumnFamily>

List of column families to expose in the table schema along with their types. This list restricts the column families that can be referenced in queries and specifies their value types. You can use this list to do type conversions - see Google::Cloud::Bigquery::External::BigtableSource::ColumnFamily#type for more details. If you leave this list empty, all column families are present in the table schema and their values are read as BYTES. During a query only the column families referenced in that query are read from Bigtable.

Examples:

require "google/cloud/bigquery"

bigquery = Google::Cloud::Bigquery.new

bigtable_url = "https://googleapis.com/bigtable/projects/..."
bigtable_table = bigquery.external bigtable_url do |bt|
  bt.rowkey_as_string = true
  bt.add_family "user" do |u|
    u.add_string "name"
    u.add_string "email"
    u.add_integer "age"
    u.add_boolean "active"
  end
end

bigtable_table.families.count #=> 1

Returns:



1300
1301
1302
 
# File 'lib/google/cloud/bigquery/external.rb', line 1300

def families
  @families
end

#rowkey_as_string ⇒ Boolean

Whether the rowkey column families will be read and converted to string. Otherwise they are read with BYTES type values and users need to manually cast them with CAST if necessary. The default value is false.

Examples:

require "google/cloud/bigquery"

bigquery = Google::Cloud::Bigquery.new

bigtable_url = "https://googleapis.com/bigtable/projects/..."
bigtable_table = bigquery.external bigtable_url do |bt|
  bt.rowkey_as_string = true
end

bigtable_table.rowkey_as_string #=> true

Returns:

  • (Boolean) 


1371
1372
1373
 
# File 'lib/google/cloud/bigquery/external.rb', line 1371

def rowkey_as_string
  @gapi.bigtable_options.read_rowkey_as_string
end

#rowkey_as_string=(row_rowkey) ⇒ Object

Set the number of rows at the top of a sheet that BigQuery will skip when reading the data.

Examples:

require "google/cloud/bigquery"

bigquery = Google::Cloud::Bigquery.new

bigtable_url = "https://googleapis.com/bigtable/projects/..."
bigtable_table = bigquery.external bigtable_url do |bt|
  bt.rowkey_as_string = true
end

bigtable_table.rowkey_as_string #=> true

Parameters:

  • row_rowkey (Boolean) —

    New rowkey_as_string value



1393
1394
1395
1396
 
# File 'lib/google/cloud/bigquery/external.rb', line 1393

def rowkey_as_string= row_rowkey
  frozen_check!
  @gapi.bigtable_options.read_rowkey_as_string = row_rowkey
end