Module: Google::Cloud::Bigtable::RowFilter

Defined in:
lib/google/cloud/bigtable/row_filter.rb,
lib/google/cloud/bigtable/row_filter/chain_filter.rb,
lib/google/cloud/bigtable/row_filter/simple_filter.rb,
lib/google/cloud/bigtable/row_filter/condition_filter.rb,
lib/google/cloud/bigtable/row_filter/interleave_filter.rb

Overview

RowFilter

Takes a row as input and produces an alternate view of the row based on specified rules. For example, a RowFilter might trim down a row to include just the cells from columns matching a given regular expression, or might return all the cells of a row but not their values. More complicated filters can be composed out of these components to express requests such as, "within every column of a particular family, give just the two most recent cells which are older than timestamp X."

There are two broad categories of RowFilters (true filters and transformers), as well as two ways to compose simple filters into more complex ones (chains and interleaves). They work as follows:

  • True filters alter the input row by excluding some of its cells wholesale from the output row. An example of a true filter is the value_regex_filter, which excludes cells whose values don't match the specified pattern. All regex true filters use RE2 syntax (https:#github.com/google/re2/wiki/Syntax) in raw byte mode (RE2::Latin1), and are evaluated as full matches. An important point to keep in mind is that RE2(.) is equivalent by default to RE2([^\n]), meaning that it does not match newlines. When attempting to match an arbitrary byte, you should therefore use the escape sequence \C, which may need to be further escaped as \\C in your client language.

  • Transformers alter the input row by changing the values of some of its cells in the output, without excluding them completely. Currently, the only supported transformer is the strip_value_transformer, which replaces every cell's value with the empty string.

  • Chains and interleaves are described in more detail in the RowFilter.Chain and RowFilter.Interleave documentation.

The total serialized size of a RowFilter message must not exceed 4096 bytes, and RowFilters may not be nested within each other (in Chains or Interleaves) to a depth of more than 20.

ADVANCED USE:. Hook for introspection into the RowFilter. Outputs all cells directly to the output of the read rather than to any parent filter. Consider the following example:

Chain(
  FamilyRegex("A"),
  Interleave(
    All(),
    Chain(Label("foo"), Sink())
  ),
  QualifierRegex("B")
)

                    A,A,1,w
                    A,B,2,x
                    B,B,4,z
                       |
                FamilyRegex("A")
                       |
                    A,A,1,w
                    A,B,2,x
                       |
          +------------+-------------+
          |                          |
        All()                    Label(foo)
          |                          |
       A,A,1,w              A,A,1,w,labels:[foo]
       A,B,2,x              A,B,2,x,labels:[foo]
          |                          |
          |                        Sink() --------------+
          |                          |                  |
          +------------+      x------+          A,A,1,w,labels:[foo]
                       |                        A,B,2,x,labels:[foo]
                    A,A,1,w                             |
                    A,B,2,x                             |
                       |                                |
               QualifierRegex("B")                      |
                       |                                |
                    A,B,2,x                             |
                       |                                |
                       +--------------------------------+
                       |
                    A,A,1,w,labels:[foo]
                    A,B,2,x,labels:[foo]  # could be switched
                    A,B,2,x               # could be switched

Despite being excluded by the qualifier filter, a copy of every cell that reaches the sink is present in the final result.

As with an Interleave filter duplicate cells are possible, and appear in an unspecified mutual order. In this case we have a duplicate with column "A:B" and timestamp 2, because one copy passed through the all filter while the other was passed through the label and sink. Note that one copy has label "foo", while the other does not.

Examples:


# Pass filter
Google::Cloud::Bigtable::RowFilter.pass

# Key regex filter
Google::Cloud::Bigtable::RowFilter.key("user-*")

# Cell limit filter
Google::Cloud::Bigtable::RowFilter.cells_per_row(10)

Defined Under Namespace

Classes: ChainFilter, ConditionFilter, InterleaveFilter, SimpleFilter

Class Method Summary collapse

Class Method Details

.block ⇒ Google::Cloud::Bigtable::RowFilter::SimpleFilter

Create block all filter instance

Does not match any cells, regardless of input. Useful for temporarily disabling just part of a filter.

Examples:


filter = Google::Cloud::Bigtable::RowFilter.block

Returns:



299
300
301
 
# File 'lib/google/cloud/bigtable/row_filter.rb', line 299

def self.block
  BLOCK
end

.cells_per_column(limit) ⇒ Google::Cloud::Bigtable::RowFilter::SimpleFilter

Create cells per column filter instance

Matches only the most recent N cells within each column. For example, if N=2, this filter would match column foo:bar at timestamps 10 and 9, skip all earlier cells in foo:bar, and then begin matching again in column foo:bar2. If duplicate cells are present, as is possible when using an Interleave, each copy of the cell is counted separately.

Examples:


filter = Google::Cloud::Bigtable::RowFilter.cells_per_column(5)

Parameters:

  • limit (String) —

    Max cell match per column limit

Returns:



517
518
519
 
# File 'lib/google/cloud/bigtable/row_filter.rb', line 517

def self.cells_per_column limit
  SimpleFilter.new.cells_per_column(limit)
end

.cells_per_row(limit) ⇒ Google::Cloud::Bigtable::RowFilter::SimpleFilter

Create cells per row limit filter instance

Matches only the first N cells of each row. If duplicate cells are present, as is possible when using an Interleave, each copy of the cell is counted separately.

Examples:


filter = Google::Cloud::Bigtable::RowFilter.cells_per_row(5)

Parameters:

  • limit (String) —

    Max cell match per row limit

Returns:



497
498
499
 
# File 'lib/google/cloud/bigtable/row_filter.rb', line 497

def self.cells_per_row limit
  SimpleFilter.new.cells_per_row(limit)
end

.cells_per_row_offset(offset) ⇒ Google::Cloud::Bigtable::RowFilter::SimpleFilter

Create cell per row offset filter instance to skip first N cells.

Skips the first N cells of each row, matching all subsequent cells. If duplicate cells are present, as is possible when using an Interleave, each copy of the cell is counted separately.

Examples:


filter = Google::Cloud::Bigtable::RowFilter.cells_per_row_offset(3)

Parameters:

  • offset (Integer) —

    Offset value.

Returns:



480
481
482
 
# File 'lib/google/cloud/bigtable/row_filter.rb', line 480

def self.cells_per_row_offset offset
  SimpleFilter.new.cells_per_row_offset(offset)
end

.chain ⇒ Google::Cloud::Bigtable::RowFilter::ChainFilter

Create chain filter instance.

A Chain RowFilter which sends rows through several RowFilters in sequence.

See ChainFilter

The elements of "filters" are chained together to process the input row: in row -> f(0) -> intermediate row -> f(1) -> ... -> f(N) -> out row The full chain is executed atomically.

Examples:

Create chain filter with simple filter.


chain = Google::Cloud::Bigtable::RowFilter.chain

# Add filters to chain filter
chain.key("user-*")
chain.strip_value

# OR
chain.key("user-*).strip_value

Create complex chain filter.


chain = Google::Cloud::Bigtable::RowFilter.chain

chain_1 = Google::Cloud::Bigtable::RowFilter.chain
chain_1.label("users").qualifier("name").cells_per_row(5)

# Add to main chain filter
chain.chain(chain_1).value("xyz*).key("user-*")

Returns:



181
182
183
 
# File 'lib/google/cloud/bigtable/row_filter.rb', line 181

def self.chain
  ChainFilter.new
end

.column_range(range) ⇒ Google::Cloud::Bigtable::RowFilter::SimpleFilter

Create column range filter instance.

Matches only cells from columns within the given range.

Examples:


range = Google::Cloud::Bigtable::ColumnRange.new(cf).from("field0").to("field5")

filter = Google::Cloud::Bigtable::RowFilter.column_range(range)

Parameters:

Returns:



587
588
589
 
# File 'lib/google/cloud/bigtable/row_filter.rb', line 587

def self.column_range range
  SimpleFilter.new.column_range(range)
end

.condition(predicate) ⇒ Google::Cloud::Bigtable::RowFilter::ConditionFilter

Create condition filter instance

A RowFilter which evaluates one of two possible RowFilters, depending on whether or not a predicate RowFilter outputs any cells from the input row.

IMPORTANT NOTE: The predicate filter does not execute atomically with the true and false filters, which may lead to inconsistent or unexpected results. Additionally, Condition filters have poor performance, especially when filters are set for the false condition.

Cannot be used within the predicate_filter, true_filter, or false_filter

Examples:


predicate = Google::Cloud::Bigtable::RowFilter.key("user-*")
condition = Google::Cloud::Bigtable::RowFilter.condition(predicate)

label = Google::Cloud::Bigtable::RowFilter.label("user")
strip_value = Google::Cloud::Bigtable::RowFilter.strip_value

# On match apply lable else strip cell values
condition.on_match(label).otherwise(strip_value)

Parameters:

Returns:



269
270
271
 
# File 'lib/google/cloud/bigtable/row_filter.rb', line 269

def self.condition predicate
  ConditionFilter.new(predicate)
end

.family(regex) ⇒ Google::Cloud::Bigtable::RowFilter::SimpleFilter

Create family name match filter using regex

Matches only cells from columns whose families satisfy the given RE2 regex. For technical reasons, the regex must not contain the : character, even if it is not being used as a literal. Note that, since column families cannot contain the new line character \n, it is sufficient to use . as a full wildcard when matching column family names.

For Regex syntax:

Examples:


filter = Google::Cloud::Bigtable::RowFilter.family("cf-.*")

Parameters:

  • regex (String) —

    Regex to match family name.

Returns:

See Also:



391
392
393
 
# File 'lib/google/cloud/bigtable/row_filter.rb', line 391

def self.family regex
  SimpleFilter.new.family(regex)
end

.interleave ⇒ Google::Cloud::Bigtable::RowFilter::InterleaveFilter

Create interleave filter.

A RowFilter which sends each row to each of several component RowFilters and interleaves the results.

The elements of "filters" all process a copy of the input row, and the results are pooled, sorted, and combined into a single output row. If multiple cells are produced with the same column and timestamp, they will all appear in the output row in an unspecified mutual order. Consider the following example, with three filters:

                             input row
                                 |
       -----------------------------------------------------
       |                         |                         |
      f(0)                      f(1)                      f(2)
       |                         |                         |
1: foo,bar,10,x             foo,bar,10,z              far,bar,7,a
2: foo,blah,11,z            far,blah,5,x              far,blah,5,x
       |                         |                         |
       -----------------------------------------------------
                                 |
1:                      foo,bar,10,z   # could have switched with #2
2:                      foo,bar,10,x   # could have switched with #1
3:                      foo,blah,11,z
4:                      far,bar,7,a
5:                      far,blah,5,x   # identical to #6
6:                      far,blah,5,x   # identical to #5

All interleaved filters are executed atomically.

Examples:

Create interleave filter with simple filter.


interleave = Google::Cloud::Bigtable::RowFilter.interleave

# Add filters to interleave filter
interleave.key("user-*")
interleave.sink

# OR
interleave.key("user-*).sink

Create complex interleave filter.


interleave = Google::Cloud::Bigtable::RowFilter.interleave

chain_1 = Google::Cloud::Bigtable::RowFilter.chain
chain_1.label("users").qualifier("name").cells_per_row(5)

# Add to main chain filter
interleave.chain(chain_1).value("xyz*).key("user-*")

Returns:



239
240
241
 
# File 'lib/google/cloud/bigtable/row_filter.rb', line 239

def self.interleave
  InterleaveFilter.new
end

.key(regex) ⇒ Google::Cloud::Bigtable::RowFilter::SimpleFilter

Create key filter instance to match key using regular expression.

Matches only cells from rows whose keys satisfy the given RE2 regex. In other words, passes through the entire row when the key matches, and otherwise produces an empty row. Note that, since row keys can contain arbitrary bytes, the \C escape sequence must be used if a true wildcard is desired. The . character will not match the new line character \n, which may be present in a binary key.

For Regex syntax:

Examples:


filter = Google::Cloud::Bigtable::RowFilter.key("user-.*")

Parameters:

  • regex (String) —

    Regex to match row keys.

Returns:

See Also:



351
352
353
 
# File 'lib/google/cloud/bigtable/row_filter.rb', line 351

def self.key regex
  SimpleFilter.new.key(regex)
end

.label(value) ⇒ Google::Cloud::Bigtable::RowFilter::SimpleFilter

Create label filter instance to apply label on result of read rows.

Applies the given label to all cells in the output row. This allows the client to determine which results were produced from which part of the filter.

Values must be at most 15 characters in length, and match the RE2 pattern [a-z0-9\\-]+

Due to a technical limitation, it is not currently possible to apply multiple labels to a cell. As a result, a Chain may have no more than one sub-filter which contains a apply_label_transformer. It is okay for an Interleave to contain multiple apply_label_transformers, as they will be applied to separate copies of the input. This may be relaxed in the future.

Examples:


filter = Google::Cloud::Bigtable::RowFilter.label("user-detail")

Parameters:

  • value (String) —

    Label name

Returns:



463
464
465
 
# File 'lib/google/cloud/bigtable/row_filter.rb', line 463

def self.label value
  SimpleFilter.new.label(value)
end

.pass ⇒ Google::Cloud::Bigtable::RowFilter::SimpleFilter

Create pass filter instance

Matches all cells, regardless of input. Functionally equivalent to leaving filter unset, but included for completeness.

Examples:


filter = Google::Cloud::Bigtable::RowFilter.pass

Returns:



284
285
286
 
# File 'lib/google/cloud/bigtable/row_filter.rb', line 284

def self.pass
  PASS
end

.qualifier(regex) ⇒ Google::Cloud::Bigtable::RowFilter::SimpleFilter

Create column qualifier match filter using regex

Matches only cells from columns whose qualifiers satisfy the given RE2 regex. Note that, since column qualifiers can contain arbitrary bytes, the \C escape sequence must be used if a true wildcard is desired. The . character will not match the new line character \n, which may be present in a binary qualifier.

For Regex syntax:

Examples:


filter = Google::Cloud::Bigtable::RowFilter.qualifier("user-name.*")

Parameters:

  • regex (String) —

    Regex to match column qualifier name.

Returns:

See Also:



414
415
416
 
# File 'lib/google/cloud/bigtable/row_filter.rb', line 414

def self.qualifier regex
  SimpleFilter.new.qualifier(regex)
end

.sample(probability) ⇒ Google::Cloud::Bigtable::RowFilter::SimpleFilter

Create sample probability filter instance

Matches all cells from a row with probability p, and matches no cells from the row with probability 1-p.

Examples:


filter = Google::Cloud::Bigtable::RowFilter.sample(0.5)

Parameters:

  • probability (Float) —

    Probability value Probability must be greather then 0 and less then 1.0

Returns:



368
369
370
 
# File 'lib/google/cloud/bigtable/row_filter.rb', line 368

def self.sample probability
  SimpleFilter.new.sample(probability)
end

.sink ⇒ Google::Cloud::Bigtable::RowFilter::SimpleFilter

Create sink filter instance

Outputs all cells directly to the output of the read rather than to any parent filter

Examples:


filter = Google::Cloud::Bigtable::RowFilter.sink

Returns:



313
314
315
 
# File 'lib/google/cloud/bigtable/row_filter.rb', line 313

def self.sink
  SINK
end

.strip_value ⇒ Google::Cloud::Bigtable::RowFilter::SimpleFilter

Create strip value filter instance

Replaces each cell's value with the empty string.

Examples:


filter = Google::Cloud::Bigtable::RowFilter.strip_value

Returns:



327
328
329
 
# File 'lib/google/cloud/bigtable/row_filter.rb', line 327

def self.strip_value
  STRIP_VALUE
end

.timestamp_range(from: nil, to: nil) ⇒ Google::Cloud::Bigtable::RowFilter::SimpleFilter

Create timestamp range filter instance

Matches only cells with timestamps within the given range. Specified a contiguous range of timestamps.

Examples:


from = (Time.now - 300).to_i * 1000
to = Time.now.to_f * 1000

filter = Google::Cloud::Bigtable::RowFilter.timestamp_range(from: from, to: to)

# From to infinity
filter = Google::Cloud::Bigtable::RowFilter.timestamp_range(from: from)

# From 0 value to `to`
filter = Google::Cloud::Bigtable::RowFilter.timestamp_range(to: to)

Parameters:

  • from (Integer) —

    Inclusive lower bound. If left empty, interpreted as 0.

  • to (Integer) —

    Exclusive upper bound. If left empty, interpreted as infinity.

Returns:



543
544
545
 
# File 'lib/google/cloud/bigtable/row_filter.rb', line 543

def self.timestamp_range from: nil, to: nil
  SimpleFilter.new.timestamp_range(from, to)
end

.value(regex) ⇒ Google::Cloud::Bigtable::RowFilter::SimpleFilter

Create value match filter using regex

Matches only cells with values that satisfy the given regular expression. Note that, since cell values can contain arbitrary bytes, the \C escape sequence must be used if a true wildcard is desired. The . character will not match the new line character \n, which may be present in a binary value.

For Regex syntax:

Examples:


filter = Google::Cloud::Bigtable::RowFilter.value("abc.*")

Parameters:

  • regex (String) —

    Regex to match cell value.

Returns:

See Also:



436
437
438
 
# File 'lib/google/cloud/bigtable/row_filter.rb', line 436

def self.value regex
  SimpleFilter.new.value(regex)
end

.value_range(range) ⇒ Google::Cloud::Bigtable::RowFilter::SimpleFilter

Create value range filter instance

Matches only cells with values that fall within the given range.

See ValueRange#from and { Google::Cloud::Bigtable::ValueRange#to} for range option inclusive/exclusive options

  • The value at which to start the range.If neither field is set, interpreted as the empty string, inclusive.
  • The value at which to end the range. If neither field is set, interpreted as the infinite string, exclusive.

Examples:

Start to end range


range = Google::Cloud::Bigtable::ValueRange.from("abc").to("xyz")
filter = Google::Cloud::Bigtable::RowFilter.value_range(range)

Start exlusive to infinite end range


range = Google::Cloud::Bigtable::ValueRange.from("abc", inclusive: false)
filter = Google::Cloud::Bigtable::RowFilter.value_range(range)

Parameters:

Returns:



570
571
572
 
# File 'lib/google/cloud/bigtable/row_filter.rb', line 570

def self.value_range range
  SimpleFilter.new.value_range(range)
end