메뉴 건너뛰기

Bigdata, Semantic IoT, Hadoop, NoSQL

Bigdata, Hadoop ecosystem, Semantic IoT등의 프로젝트를 진행중에 습득한 내용을 정리하는 곳입니다.
필요한 분을 위해서 공개하고 있습니다. 문의사항은 gooper@gooper.com로 메일을 보내주세요.


hbase hbase shell 필드 검색 방법

총관리자 2015.05.24 13:09 조회 수 : 1724

0. regexstring검색은 =, !=만 지원된다.


1. regDate가 2015-05-23 17:20로 시작하는 값의 등록 역순으로 100개만 조회하는경우(부분문자열)

 scan 't1', {REVERSED=>true, LIMIT=>100, FILTER=>"SingleColumnValueFilter('cf','regDate',=,'regexstring:2015-05-23 17:20*')"}


2. regSeq가 1인 row를 등록된순으로 100개만 조회하는 경우

scan 't1', {LIMIT=>100, FILTER=>"SingleColumnValueFilter('cf','regSeq',=,'binary:1')"}


3 rowkey에서 부분문자열 조회
 예, rowkey가 row1-2-aaa-bbb-333인 경우 아래와 같이 검색하면 중간에 있는 aaa와 매칭되어 검색된다.
  scan 't1', {LIMIT=>100, FILTER=>"RowFilter(=,'regexstring:aaa')"}
  
4. rowkey에 'aaa'가 없는 것만 조회하는 경우
scan 't1', {LIMIT=>100, FILTER=>"RowFilter(!=,'regexstring:aaa')"}

5. rowkey갑중에 'aaa'가 있고 timestamp가 1432365378314 혹은 1432365378315인 row를 100개만 조회한다.
scan 't1', {LIMIT=>100, FILTER=>"RowFilter(=,'regexstring:aaa') AND TimestampsFilter ( 1432365378314, 1432365378315)"}


Use Case:

Currently, to use any of the filters, one has to explicitly add a scanner for the filter in the Thrift API making it messy and long. Thus, I am trying to add support for all the filters in a clean way.

Solution

The user specifies a filter via a string. The string is parsed on the server to construct the filter

General Filter String Syntax:

A simple filter expression is expressed as: “FilterName (argument, argument, … , argument)”

You must specify the name of the filter followed by the argument list in parenthesis. Commas separate the individual arguments

If the argument represents a string, it should be enclosed in single quotes.

If it represents a boolean, an integer or a comparison operator like <, >, != etc. it should not be enclosed in quotes

The filter name must be one word. All ASCII characters are allowed except for whitespace, single quotes and parenthesis.

The filter’s arguments can contain any ASCII character. If single quotes are present in the argument, they must be escaped by a preceding single quote

 

Compound Filters and Operators:

Currently, two binary operators – AND/OR and two unary operators – WHILE/SKIP are supported.

Note: the operators are all in uppercase

AND – as the name suggests, if this operator is used, the key-value must pass both the filters

OR – as the name suggests, if this operator is used, the key-value must pass at least one of the filters

SKIP – For a particular row, if any of the key-values don’t pass the filter condition, the entire row is skipped

WHILE – For a particular row, it continues to emit key-values until a key-value is reached that fails the filter condition

Compound Filters: Using these operators, a hierarchy of filters can be created. For example:

“(Filter1 AND Filter2) OR (Filter3 AND Filter4)”

   

Order of Evaluation

 Parenthesis have the highest precedence

The SKIP and WHILE operators are next and have the same precedence.
The AND operator has the next highest precedence followed by the OR operator

For example:
A filter string of the form: “Filter1 AND Filter2 OR Filter3” will be evaluated as: “(Filter1 AND Filter2) OR Filter3”
A filter string of the form: “Filter1 AND SKIP Filter2 OR Filter3” will be evaluated as: “(Filter1 AND (SKIP Filter2)) OR Filter3” 

Compare Operator and Comparators

Compare Operator

A compare operator can be any of the following:

  1. LESS (<)
  2. LESS_OR_EQUAL (<=)
  3. EQUAL (=)
  4. NOT_EQUAL (!=)
  5. GREATER_OR_EQUAL (>=)
  6. GREATER (>)
  7. NO_OP (no operation)

The client should use the symbols (<, <=, =, !=, >, >=) to express compare operators.

 

Comparators

A comparator can be any of the following:

  1. BinaryComparator – This lexicographically compares against the specified byte array using Bytes.compareTo(byte[], byte[])
  2. BinaryPrefixComparator – This lexicographically compares against a specified byte array. It only compares up to the length of this byte array.
  3. RegexStringComparator – This compares against the specified byte array using the given regular expression. Only EQUAL and NOT_EQUAL comparisons are valid with this comparator
  4. SubStringComparator – This tests if the given substring appears in a specified byte array. The comparison is case insensitive. Only EQUAL and NOT_EQUAL comparisons are valid with this comparator

The general syntax of a comparator is: ComparatorType:ComparatorValue
The ComparatorType for the various comparators is as follows:

  1. BinaryComparator – binary
  2. BinaryPrefixComparator – binaryprefix
  3. RegexStringComparator – regexstring
  4. SubStringComparator – substring

The ComparatorValue can be any value.
Example1: >, ‘binary:abc’ will match everything that is lexicographically greater than “abc”
Example2: =, ‘binaryprefix:abc’ will match everything whose first 3 characters are lexicographically equal to “abc”
Example3: !=, ‘regexstring:ab*yz’ will match everything that doesn’t begin with “ab” and ends with “yz”
Example4: =, ‘substring:abc123′ will match everything that begins with the substring “abc123″ 

 

Example PHP Client Program that uses Filter Strings

<? $_SERVER[‘PHP_ROOT’] = realpath(dirname(__FILE__).’/..’);

require_once $_SERVER[‘PHP_ROOT’].’/flib/__flib.php';

flib_init(FLIB_CONTEXT_SCRIPT);

require_module(‘storage/hbase’);

$hbase = new HBase(‘<server_name_running_thrift_server>’, <port on which thrift server is running>); $hbase->open(); $client = $hbase->getClient();

 

echo “Sample Program which uses the Filter Language”;

$result = $client->scannerOpenWithFilterString(‘table_name’, “(PrefixFilter (‘row2′) AND (QualifierFilter (>=, ‘binary:xyz’))) AND (TimestampsFilter ( 123, 456))”);

$to_print = $client->scannerGetList($result,1);

 while ($to_print) {

print_r($to_print);

$to_print = $client->scannerGetList($result,1);

}

 $client->scannerClose($result); ?>

 

Example Filter Strings

  •  “PrefixFilter (‘Row’) AND PageFilter (1) AND FirstKeyOnlyFilter ()”

will return all key-value pairs that match the following conditions:
1) The row containing the key-value should have prefix “Row”
2) The key-value must be located in the first row of the table
3) The key-value pair must be the first key-value in the row

 

  • “(RowFilter (=, ‘binary:Row 1’) AND TimeStampsFilter (74689, 89734)) OR ColumnRangeFilter (‘abc’, true, ‘xyz’, false))”

will return all key-value pairs that match both the following conditions:
1) The key-value is in a row having row key “Row 1”
2) The key-value must have a timestamp of either 74689 or 89734
Or it must match the following condition:
1) The key-value pair must be in a column that is lexicographically >= abc and < xyz 

  • “SKIP ValueFilter (0)”

will skip the entire row if any of the values in the row is not 0

 

Individual Filter Syntax  

  • KeyOnlyFilter 

Description: This filter doesn’t take any arguments. It returns only the key component of each key-value. 

Syntax: KeyOnlyFilter () 

Example: “KeyOnlyFilter ()”

  • FirstKeyOnlyFilter 

Description: This filter doesn’t take any arguments. It returns only the first key-value from each row. 

Syntax: FirstKeyOnlyFilter () 

Example: “FirstKeyOnlyFilter ()” 

  • PrefixFilter 

Description: This filter takes one argument – a prefix of a row key. It returns only those key-values present in a row that starts with the specified row prefix

Syntax: PrefixFilter (‘<row_prefix>’)

Example: “PrefixFilter (‘Row’)” 

  • ColumnPrefixFilter 

Description: This filter takes one argument – a column prefix. It returns only those key-values present in a column that starts with the specified column prefix. The column prefix must be of the form:

– “qualifier”

Syntax:    ColumnPrefixFilter (‘<column_prefix>’)

Example: “ColumnPrefixFilter (‘Col’)” 

  • MultipleColumnPrefixFilter 

Description: This filter takes a list of column prefixes. It returns key-values that are present in a column that starts with any of the specified column prefixes. The column prefix must be of the form:

– “qualifier” in which case it looks for key-values in columns whose qualifiers have the specified prefix

Syntax:    MultipleColumnPrefixFilter (‘<column_prefix>’, ‘<column_prefix>’, …, ‘<column_prefix>’)

Example: “MultipleColumnPrefixFilter (‘Col1’, ‘Col2’)” 

  • ColumnCountGetFilter 

Description: This filter takes one argument – a limit. It returns the first limit number of columns in the table

Syntax: ColumnCountGetFilter (‘<limit>’)

Example: “ColumnCountGetFilter (4)” 

  • PageFilter 

Description: This filter takes one argument – a page size. It returns page size number of rows from the table.

Syntax: PageFilter (‘<page_size>’)

Example: “PageFilter (2)” 

  • ColumnPaginationFilter

 Description: This filter takes two arguments – a limit and offset. It returns limit number of columns after offset number of columns. It does this for all the rows

Syntax: ColumnPaginationFilter (‘<limit>’, ‘<offest>’)

Example: “ColumnPaginationFilter (3, 5)” 

  • InclusiveStopFilter 

Description: This filter takes one argument – a row key on which to stop scanning. It returns all key-values present in rows up to and including the specified row

Syntax: InclusiveStopFilter (‘<stop_row_key>’)

Example: “InclusiveStopFilter (‘Row2’)” 

  • TimeStampsFilter 

Description: This filter takes a list of timestamps. It returns those key-values whose timestamps matches any of the specified timestamps

Syntax: TimeStampsFilter (<timestamp>, <timestamp>, … ,<timestamp>)

Example: “TimeStampsFilter (5985489, 48895495, 58489845945)” 

  • RowFilter 

Description: This filter takes a compare operator and a comparator. It compares each row key with the comparator using the compare operator and if the comparison returns true, it returns all the key-values in that row

Syntax: RowFilter (<compareOp>, ‘<row_comparator>’)

Example: “RowFilter (<=, ‘xyz)” 

  • Family Filter 

Description: This filter takes a compare operator and a comparator. It compares each family name with the comparator using the compare operator and if the comparison returns true, it returns all the key-values in that family

Syntax: FamilyFilter (<compareOp>, ‘<family_comparator>’)

Example: “FamilyFilter (>=, ‘FamilyB’)”

  • QualifierFilter 

Description: This filter takes a compare operator and a comparator. It compares each qualifier name with the comparator using the compare operator and if the comparison returns true, it returns all the key-values in that column

Syntax: QualifierFilter (<compareOp>, ‘<qualifier_comparator>’)

Example: “QualifierFilter (=, ‘Column1’)” 

  • ValueFilter 

Description: This filter takes a compare operator and a comparator. It compares each value with the comparator using the compare operator and if the comparison returns true, it returns that key-value

Syntax: ValueFilter (<compareOp>, ‘<value_comparator>’)

Example: “ValueFilter (!=, ‘Value’)” 

  • DependentColumnFilter

Description: This filter takes two arguments – a family and a qualifier. It tries to locate this column in each row and returns all key-values in that row that have the same timestamp. If the row doesn’t contain the specified column – none of the key-values in that row will be returned.

The filter can also take an optional boolean argument – dropDependentColumn. If set to true, the column we were depending on doesn’t get returned.

The filter can also take two more additional optional arguments – a compare operator and a value comparator, which are further checks in addition to the family and qualifier. If the dependent column is found, its value should also pass the value check and then only is its timestamp taken into consideration

Syntax: DependentColumnFilter (‘<family>’, ‘<qualifier>’, <boolean>, <compare operator>, ‘<value comparator’)

DependentColumnFilter (‘<family>’, ‘<qualifier>’, <boolean>)

DependentColumnFilter (‘<family>’, ‘<qualifier>’)

Example: “DependentColumnFilter (‘conf’, ‘blacklist’, false, >=, ‘zebra’)”

“DependentColumnFilter (‘conf’, ‘blacklist’, true)”

“DependentColumnFilter (‘conf’, ‘blacklist’)” 

  • SingleColumnValueFilter 

Description: This filter takes a column family, a qualifier, a compare operator and a comparator. If the specified column is not found – all the columns of that row will be emitted. If the column is found and the comparison with the comparator returns true, all the columns of the row will be emitted. If the condition fails, the row will not be emitted. 

This filter also takes two additional optional boolean arguments – filterIfColumnMissing and setLatestVersionOnly 

If the filterIfColumnMissing flag is set to true the columns of the row will not be emitted if the specified column to check is not found in the row. The default value is false. 

If the setLatestVersionOnly flag is set to false, it will test previous versions (timestamps) too. The default value is true. 

These flags are optional and if you must set neither or both 

Syntax: SingleColumnValueFilter (<compare operator>, ‘<comparator>’, ‘<family>’, ‘<qualifier>’, <filterIfColumnMissing_boolean>, <latest_version_boolean>)

Syntax: SingleColumnValueFilter (<compare operator>, ‘<comparator>’, ‘<family>’, ‘<qualifier>’)

Example: “SingleColumnValueFilter (<=, ‘abc’, ‘FamilyA’, ‘Column1’, true, false)”

Example: “SingleColumnValueFilter (<=, ‘abc’, ‘FamilyA’, ‘Column1’)”

  • SingleColumnValueExcludeFilter 

Description: This filter takes the same arguments and behaves same as SingleColumnValueFilter – however, if the column is found and the condition passes, all the columns of the row will be emitted except for the tested column value.

Syntax: SingleColumnValueExcludeFilter (<compare operators> <comparator> <family> <qualifier> <latest_version_boolean> <filterIfColumnMissing_boolean>)

Syntax: SingleColumnValueExcludeFilter (<compare operator> <comparator> <family> <qualifier>)

Example: “SingleColumnValueExcludeFilter (‘<=’, ‘abc’, ‘FamilyA’, ‘Column1’, ‘false’, ‘true’)”

Example: “SingleColumnValueExcludeFilter (‘<=’, ‘abc’, ‘FamilyA’, ‘Column1’)” 

  • ColumnRangeFilter 

Description: This filter is used for selecting only those keys with columns that are between minColumn and maxColumn. It also takes two boolean variables to indicate whether to include the minColumn and maxColumn or not.

If you don’t want to set the minColumn or the maxColumn – you can pass in an empty argument.

Syntax: ColumnRangeFilter (‘<minColumn >’, <minColumnInclusive_bool>, ‘<maxColumn>’, <maxColumnInclusive_bool>)

Example: “ColumnRangeFilter (‘abc’, true, ‘xyz’, false)”

번호 제목 글쓴이 날짜 조회 수
29 Hadoop의 Datanode를 Decommission하고 나서 HBase의 regionservers파일에 해당 노드명을 지웠는데 여전히 "Dead regionser"로 표시되는 경우 처리 총관리자 2018.01.25 55
28 Current heap configuration for MemStore and BlockCache exceeds the threshold required for successful cluster operation 총관리자 2017.07.18 32
27 HBase 설정 최적화하기(VCNC) file 총관리자 2017.07.18 87
26 HBase write 성능 튜닝 file 총관리자 2017.07.18 50
25 Hbase API를 이용하여 scan시 페이징을 고려하여 목록을 가져올때 사용할 수 있는 로직의 예시를 보여줌 총관리자 2017.04.26 162
24 테이블의 row수를 빠르게 카운트 하는 방법 총관리자 2017.01.26 58
23 hbase startrow와 endrow를 지정하여 검색하기 샘플 총관리자 2016.12.07 41
22 Flume을 이용한 데이타 수집시 HBase write 성능 튜닝 file 총관리자 2016.10.31 544
21 bin/start-hbase.sh실행시 org.apache.hadoop.hbase.util.FileSystemVersionException: HBase file layout needs to be upgraded오류가 발생하면 조치사항 총관리자 2016.08.01 142
20 org.apache.hadoop.hbase.ClockOutOfSyncException: org.apache.hadoop.hbase.ClockOutOfSyncException 오류시 조치사항 총관리자 2016.07.14 44
19 Master rejected startup because clock is out of sync 오류 해결방법 총관리자 2016.05.03 34
» hbase shell 필드 검색 방법 총관리자 2015.05.24 1724
17 secureCRT에서 backspace키가 작동하지 않는 경우 해결방법 총관리자 2015.05.11 518
16 hbase가 기동시키는 zookeeper에서 받아드리는 ip가 IPv6로 사용되는 경우가 있는데 이를 IPv4로 강제적용하는 방법 총관리자 2015.05.08 230
15 hbase CustomFilter만들기 (0.98.X이상) 총관리자 2015.05.08 150
14 HBase 0.98.12(1.2.5) for hadoop2 설치-5대에 완전분산모드 (HDFS HA상테) 총관리자 2015.04.29 888
13 scan의 startrow, stoprow지정하는 방법 총관리자 2015.04.08 220
12 SASL configuration failed: javax.security.auth.login.LoginException: java.lang.NullPointerException 오류 해결방법 총관리자 2015.04.02 433
11 hbase shell에서 컬럼값 검색하기(SingleColumnValueFilter이용) 총관리자 2014.04.25 2369
10 column family삭제시 Column family 'delete' does not exist오류 발생하는 경우 총관리자 2014.04.14 916

A personal place to organize information learned during the development of such Hadoop, Hive, Hbase, Semantic IoT, etc.
We are open to the required minutes. Please send inquiries to gooper@gooper.com.

위로