Skip to main content

types

ColumnsFilter​

Filter columns using lists of included and excluded columns.

includeColumns : string​

List of columns that will be included to transfer

excludeColumns : string​

List of columns that will be excluded to transfer

Endpoint​

id : string​
folderId : string​
name : string​
description : string​
labels : string​
settings : EndpointSettings​

EndpointSettings​

One of settings​

  • mysqlSource : endpoint.MysqlSource​
  • postgresSource : endpoint.PostgresSource​
  • ydbSource : endpoint.YdbSource​
  • ydsSource : endpoint.YDSSource​
  • kafkaSource : endpoint.KafkaSource​
  • mongoSource : endpoint.MongoSource​
  • clickhouseSource : endpoint.ClickhouseSource​
  • mysqlTarget : endpoint.MysqlTarget​
  • postgresTarget : endpoint.PostgresTarget​
  • clickhouseTarget : endpoint.ClickhouseTarget​
  • ydbTarget : endpoint.YdbTarget​
  • kafkaTarget : endpoint.KafkaTarget​
  • mongoTarget : endpoint.MongoTarget​
  • metrikaSource : endpoint.MetrikaSource​
  • ydsTarget : endpoint.YDSTarget​

FilterColumnsTransformer​

Set up a list of table columns to transfer

tables : TablesFilter​

List of the tables to filter using lists of included and excluded tables.

columns : ColumnsFilter​

List of the columns to transfer to the target tables using lists of included and excluded columns.

FilterRowsTransformer​

This filter only applies to transfers with queues (Logbroker or Apache Kafka®) as a data source. When running a transfer, only the strings meeting the specified criteria remain in a changefeed.

tables : TablesFilter​

List of included and excluded tables.

filter : string​

Filtering criterion. This can be comparison operators for numeric, string, and Boolean values, comparison to NULL, and checking whether a substring is part of a string. Details here: https://yandex.cloud/en-ru/docs/data-transfer/concepts/data-transformation#append-only-sources. Deprecated: Use filters instead.

filters : string​

Data is transported if it satisfies at least one of filters. Consider that there is OR statement between filters. Each filter can be comparison operators for numeric, string, and Boolean values, comparison to NULL, and checking whether a substring is part of a string. Details in docs: https://yandex.cloud/en-ru/docs/data-transfer/concepts/data-transformation#append-only-sources.

MaskFieldTransformer​

Mask field transformer allows you to hash data

tables : TablesFilter​

List of included and excluded tables

columns : string​

Specify the name of the column for data masking (a regular expression).

function : MaskFunction​

Mask function

MaskFunction​

Mask function

One of maskFunction​

  • maskFunctionHash : MaskFunctionHash​

    Hash mask function

MaskFunctionHash​

Hash data using HMAC

userDefinedSalt : string​

This string will be used in the HMAC(sha256, salt) function applied to the column data.

RenameTable​

Specify rule for renaming table

originalName : Table​

Specify the current names of the table in the source

newName : Table​

Specify the new names for this table in the target

RenameTablesTransformer​

Set rules for renaming tables by specifying the current names of the tables in the source and new names for these tables in the target.

renameTables : RenameTable​

List of renaming rules

ReplacePrimaryKeyTransformer​

Override primary keys

tables : TablesFilter​

List of included and excluded tables

keys : string​

List of columns to be used as primary keys

Runtime​

One of runtime​

  • ycRuntime : YcRuntime​

SharderTransformer​

Set the number of shards for particular tables and a list of columns whose values will be used for calculating a hash to determine a shard.

tables : TablesFilter​

List of included and excluded tables

columns : ColumnsFilter​

List of included and excluded columns

shardsCount : int64​

Number of shards

ShardingUploadParams​

jobCount : int64​
processCount : int64​

Table​

nameSpace : string​
name : string​

TableSplitterTransformer​

A transfer splits the X table into multiple tables (X_1, X_2, ..., X_n) based on data. If a row was located in the X table before it was split, it is now in the X_i table, where i is determined by the column list and split string parameters. Example: If the column list has two columns, month of birth and gender, specified and the split string states @, information about an employee whose name is John and who was born on February 11, 1984, from the Employees table will get to a new table named Employees@February@male.

tables : TablesFilter​

List of included and excluded tables

columns : string​

Specify the columns in the tables to be partitioned.

splitter : string​

Specify the split string to be used for merging components in a new table name.

TablesFilter​

Filter tables using lists of included and excluded tables.

includeTables : string​

List of tables that will be included to transfer

excludeTables : string​

List of tables that will be excluded to transfer

ToStringTransformer​

Convert column values to strings The values will be converted depending on the source type Conversion rules are described here: https://cloud.yandex.com/en/docs/data-transfer/concepts/data-transformation#convert-to-string

tables : TablesFilter​

List of included and excluded tables

columns : ColumnsFilter​

List of included and excluded columns

Transfer​

Transfer core entity

id : string​
folderId : string​
name : string​
description : string​
labels : string​
source : Endpoint​
target : Endpoint​
runtime : Runtime​
status : TransferStatus​
type : TransferType​
warning : string​
transformation : Transformation​
prestable : bool​

Transformation​

Transformation is converting data using special transformer functions. These functions are executed on a data stream, applied to each data change item, and transform them. A transformer can be run at both the metadata and data levels. Data can only be transformed if the source and target are of different types.

transformers : Transformer​

Transformers are set as a list. When activating a transfer, a transformation plan is made for the tables that match the specified criteria. Transformers are applied to the tables in the sequence specified in the list.

Transformer​

Some transformers may have limitations and only apply to some source-target pairs.

One of transformer​

  • maskField : MaskFieldTransformer​
  • filterColumns : FilterColumnsTransformer​
  • renameTables : RenameTablesTransformer​
  • replacePrimaryKey : ReplacePrimaryKeyTransformer​
  • convertToString : ToStringTransformer​
  • sharderTransformer : SharderTransformer​
  • tableSplitterTransformer : TableSplitterTransformer​
  • filterRows : FilterRowsTransformer​

YcRuntime​

jobCount : int64​
uploadShardParams : ShardingUploadParams​