Skip to main content

types

AltName​

fromName : string​

Source table name

toName : string​

Target table name

AuditTrailsV1Parser​

ClickhouseConnection​

One of connection​

  • connectionOptions : ClickhouseConnectionOptions​

ClickhouseConnectionOptions​

One of address​

  • onPremise : OnPremiseClickhouse​
  • mdbClusterId : string​
user : string​
password : Secret​
database : string​

Database

ClickhouseShard​

name : string​
hosts : string​

ClickhouseSharding​

ColumnValueHash​

columnName : string​

ColumnValueMapping​

ValueToShard​
columnValue : ColumnValue​
shardName : string​
columnName : string​
mapping : ValueToShard​

One of sharding​

  • columnValueHash : ColumnValueHash​
  • customMapping : ColumnValueMapping​
  • transferId : google.protobuf.Empty​
  • roundRobin : google.protobuf.Empty​

ClickhouseSource​

connection : ClickhouseConnection​
includeTables : string​

While list of tables for replication. If none or empty list is presented - will replicate all tables. Can contain * patterns.

excludeTables : string​

Exclude list of tables for replication. If none or empty list is presented - will replicate all tables. Can contain * patterns.

subnetId : string​
securityGroups : string​
clickhouseClusterName : string​

Name of the ClickHouse cluster. For Managed ClickHouse that is name of ShardGroup.

ClickhouseTarget​

connection : ClickhouseConnection​
subnetId : string​
altNames : AltName​

Alternative table names in target

cleanupPolicy : ClickhouseCleanupPolicy​
sharding : ClickhouseSharding​
clickhouseClusterName : string​

Name of the ClickHouse cluster. For Managed ClickHouse that is name of ShardGroup.

securityGroups : string​

CloudLoggingParser​

ColSchema​

name : string​
type : ColumnType​
key : bool​
required : bool​
path : string​

ColumnValue​

One of value​

  • stringValue : string​

DataSchema​

One of schema​

  • jsonFields : string​
  • fields : FieldList​

DataTransformationOptions​

cloudFunction : string​

Cloud function

numberOfRetries : int64​

Number of retries

bufferSize : string​

Buffer size for function

bufferFlushInterval : string​

Flush interval

invocationTimeout : string​

Invocation timeout

serviceAccountId : string​

Service account

DebeziumSerializerParameter​

key : string​

Name of the serializer parameter

value : string​

Value of the serializer parameter

FieldList​

fields : ColSchema​

Column schema

GenericParserCommon​

dataSchema : DataSchema​
nullKeysAllowed : bool​

Allow null keys, if no - null keys will be putted to unparsed data

addRestColumn : bool​

Will add _rest column for all unknown fields

unescapeStringValues : bool​

Unescape string values

KafkaAuth​

One of security​

  • sasl : KafkaSaslSecurity​

    Authentication with SASL

  • noAuth : NoAuth​

    No authentication

KafkaConnectionOptions​

  • clusterId : string​

    Managed Service for Kafka cluster ID

  • onPremise : OnPremiseKafka​

    Connection options for on-premise Kafka

KafkaSaslSecurity​

user : string​

User name

mechanism : KafkaMechanism​

SASL mechanism for authentication

password : Secret​

Password for user

KafkaSource​

connection : KafkaConnectionOptions​

Connection settings

auth : KafkaAuth​

Authentication settings

securityGroups : string​

Security groups

topicName : string​

Full source topic name Deprecated in favor of topic names

transformer : DataTransformationOptions​

Data transformation rules

parser : Parser​

Data parsing rules

topicNames : string​

List of topic names to read

KafkaTarget​

connection : KafkaConnectionOptions​

Connection settings

auth : KafkaAuth​

Authentication settings

securityGroups : string​

Security groups

topicSettings : KafkaTargetTopicSettings​

Target topic settings

serializer : Serializer​

Data serialization format settings

KafkaTargetTopic​

topicName : string​

Topic name

saveTxOrder : bool​

Save transactions order Not to split events queue into separate per-table queues.

KafkaTargetTopicSettings​

One of topicSettings​

  • topic : KafkaTargetTopic​

    Full topic name

  • topicPrefix : string​

    Topic prefix

Analogue of the Debezium setting database.server.name. Messages will be sent to topic with name <topic_prefix>.<schema>.<table_name>.

MetrikaSource​

counterIds : int64​
token : Secret​
streams : MetrikaStream​

MetrikaStream​

type : MetrikaStreamType​
columns : string​

MongoCollection​

databaseName : string​
collectionName : string​

MongoConnection​

  • connectionOptions : MongoConnectionOptions​

MongoConnectionOptions​

  • mdbClusterId : string​
  • onPremise : OnPremiseMongo​
user : string​

User name

password : Secret​

Password for user

authSource : string​

Database name associated with the credentials

MongoSource​

connection : MongoConnection​
subnetId : string​
collections : MongoCollection​

List of collections for replication. Empty list implies replication of all tables on the deployment. Allowed to use * as collection name.

excludedCollections : MongoCollection​

List of forbidden collections for replication. Allowed to use * as collection name for forbid all collections of concrete schema.

secondaryPreferredMode : bool​

Read mode for mongo client

securityGroups : string​

Security groups

MongoTarget​

connection : MongoConnection​
database : string​

Database name

cleanupPolicy : CleanupPolicy​
subnetId : string​
securityGroups : string​

Security groups

MysqlConnection​

  • mdbClusterId : string​

    Managed Service for MySQL cluster ID

  • onPremise : OnPremiseMysql​

    Connection options for on-premise MySQL

MysqlObjectTransferSettings​

view : ObjectTransferStage​

Views

CREATE VIEW ...

routine : ObjectTransferStage​

Routines

CREATE PROCEDURE ... ; CREATE FUNCTION ... ;

trigger : ObjectTransferStage​

Triggers

CREATE TRIGGER ...

tables : ObjectTransferStage​

MysqlSource​

connection : MysqlConnection​

Database connection settings

database : string​

Database name

You can leave it empty, then it will be possible to transfer tables from several databases at the same time from this source.

user : string​

User for database access.

password : Secret​

Password for database access.

timezone : string​

Database timezone

Is used for parsing timestamps for saving source timezones. Accepts values from IANA timezone database. Default: local timezone.

objectTransferSettings : MysqlObjectTransferSettings​

Schema migration

Select database objects to be transferred during activation or deactivation.

includeTablesRegex : string​
excludeTablesRegex : string​
securityGroups : string​

Security groups

serviceDatabase : string​

Database for service tables

Default: data source database. Here created technical tables (tm_keeper, tm_gtid_keeper).

MysqlTarget​

connection : MysqlConnection​

Database connection settings

database : string​

Database name

Allowed to leave it empty, then the tables will be created in databases with the same names as on the source. If this field is empty, then you must fill below db schema for service table.

user : string​

User for database access.

password : Secret​

Password for database access.

sqlMode : string​

Default: NO_AUTO_VALUE_ON_ZERO,NO_DIR_IN_CREATE,NO_ENGINE_SUBSTITUTION.

skipConstraintChecks : bool​

Disable constraints checks

Recommend to disable for increase replication speed, but if schema contain cascading operations we don't recommend to disable. This option set FOREIGN_KEY_CHECKS=0 and UNIQUE_CHECKS=0.

timezone : string​

Database timezone

Is used for parsing timestamps for saving source timezones. Accepts values from IANA timezone database. Default: local timezone.

cleanupPolicy : CleanupPolicy​

Cleanup policy

Cleanup policy for activate, reactivate and reupload processes. Default is DISABLED.

serviceDatabase : string​

Database schema for service table

Default: db name. Here created technical tables (tm_keeper, tm_gtid_keeper).

securityGroups : string​

Security groups

NoAuth​

No authentication

OnPremiseClickhouse​

shards : ClickhouseShard​
httpPort : int64​
nativePort : int64​
tlsMode : TLSMode​

OnPremiseKafka​

brokerUrls : string​

Kafka broker URLs

subnetId : string​

Network interface for endpoint. If none will assume public ipv4

tlsMode : TLSMode​

TLS settings for broker connection. Disabled by default.

OnPremiseMongo​

hosts : string​
port : int64​
replicaSet : string​
tlsMode : TLSMode​

OnPremiseMysql​

port : int64​

Database port

subnetId : string​

Network interface for endpoint. If none will assume public ipv4

hosts : string​
tlsMode : TLSMode​

TLS settings for server connection. Disabled by default.

OnPremisePostgres​

port : int64​

Will be used if the cluster ID is not specified.

subnetId : string​

Network interface for endpoint. If none will assume public ipv4

hosts : string​
tlsMode : TLSMode​

TLS settings for server connection. Disabled by default.

Parser​

One of parser​

  • jsonParser : GenericParserCommon​
  • auditTrailsV1Parser : AuditTrailsV1Parser​
  • cloudLoggingParser : CloudLoggingParser​
  • tskvParser : GenericParserCommon​

PostgresConnection​

  • mdbClusterId : string​

    Managed Service for PostgreSQL cluster ID

  • onPremise : OnPremisePostgres​

    Connection options for on-premise PostgreSQL

PostgresObjectTransferSettings​

sequence : ObjectTransferStage​

Sequences

CREATE SEQUENCE ...

sequenceOwnedBy : ObjectTransferStage​

Owned sequences

CREATE SEQUENCE ... OWNED BY ...

table : ObjectTransferStage​

Tables

CREATE TABLE ...

primaryKey : ObjectTransferStage​

Primary keys

ALTER TABLE ... ADD PRIMARY KEY ...

fkConstraint : ObjectTransferStage​

Foreign keys

ALTER TABLE ... ADD FOREIGN KEY ...

defaultValues : ObjectTransferStage​

Default values

ALTER TABLE ... ALTER COLUMN ... SET DEFAULT ...

constraint : ObjectTransferStage​

Constraints

ALTER TABLE ... ADD CONSTRAINT ...

index : ObjectTransferStage​

Indexes

CREATE INDEX ...

view : ObjectTransferStage​

Views

CREATE VIEW ...

function : ObjectTransferStage​

Functions

CREATE FUNCTION ...

trigger : ObjectTransferStage​

Triggers

CREATE TRIGGER ...

type : ObjectTransferStage​

Types

CREATE TYPE ...

rule : ObjectTransferStage​

Rules

CREATE RULE ...

collation : ObjectTransferStage​

Collations

CREATE COLLATION ...

policy : ObjectTransferStage​

Policies

CREATE POLICY ...

cast : ObjectTransferStage​

Casts

CREATE CAST ...

materializedView : ObjectTransferStage​

Materialized views

CREATE MATERIALIZED VIEW ...

sequenceSet : ObjectTransferStage​

PostgresSource​

connection : PostgresConnection​

Database connection settings

database : string​

Database name

user : string​

User for database access. not required as may be in connection

password : Secret​

Password for database access.

includeTables : string​

Included tables

If none or empty list is presented, all tables are replicated. Full table name with schema. Can contain schema_name.* patterns.

excludeTables : string​

Excluded tables

If none or empty list is presented, all tables are replicated. Full table name with schema. Can contain schema_name.* patterns.

slotByteLagLimit : int64​

Maximum lag of replication slot (in bytes); after exceeding this limit replication will be aborted.

serviceSchema : string​

Database schema for service tables (consumer_keeper, data_transfer_mole_finder). Default is public

objectTransferSettings : PostgresObjectTransferSettings​

Select database objects to be transferred during activation or deactivation.

securityGroups : string​

Security groups

PostgresTarget​

connection : PostgresConnection​

Database connection settings

database : string​

Database name

user : string​

User for database access. not required as may be in connection

password : Secret​

Password for database access.

cleanupPolicy : CleanupPolicy​

Cleanup policy for activate, reactivate and reupload processes. Default is truncate.

securityGroups : string​

Security groups

Secret​

  • raw : string​

    Raw secret value

Serializer​

Data serialization format

One of serializer​

  • serializerAuto : SerializerAuto​

    Select the serialization format automatically

  • serializerJson : SerializerJSON​

    Serialize data in json format

  • serializerDebezium : SerializerDebezium​

    Serialize data in debezium format

SerializerAuto​

SerializerDebezium​

serializerParameters : DebeziumSerializerParameter​

Settings of sterilization parameters as key-value pairs

SerializerJSON​

TLSConfig​

caCertificate : string​

CA certificate

X.509 certificate of the certificate authority which issued the server's certificate, in PEM format. When CA certificate is specified TLS is used to connect to the server.

TLSMode​

One of tlsMode​

  • disabled : google.protobuf.Empty​
  • enabled : TLSConfig​

YDSSource​

database : string​

Database

stream : string​

Stream

serviceAccountId : string​

SA which has read access to the stream.

supportedCodecs : YdsCompressionCodec​

Compression codec

parser : Parser​

Data parsing rules

allowTtlRewind : bool​

Should continue working, if consumer read lag exceed TTL of topic False: stop the transfer in error state, if detected lost data. True: continue working with losing part of data

endpoint : string​

for dedicated db

subnetId : string​

Network interface for endpoint. If none will assume public ipv4

securityGroups : string​

Security groups

consumer : string​

for important streams

YDSTarget​

database : string​

Database

stream : string​

Stream

serviceAccountId : string​

SA which has read access to the stream.

saveTxOrder : bool​

Save transaction order Not to split events queue into separate per-table queues. Incompatible with setting Topic prefix, only with Topic full name.

serializer : Serializer​

Data serialization format

endpoint : string​

for dedicated db

subnetId : string​

Network interface for endpoint. If none will assume public ipv4

securityGroups : string​

Security groups

YdbSource​

database : string​

Path in YDB where to store tables

instance : string​

Instance of YDB. example: ydb-ru-prestable.yandex.net:2135

paths : string​
serviceAccountId : string​
subnetId : string​

Network interface for endpoint. If none will assume public ipv4

saKeyContent : string​

Authorization Key

securityGroups : string​

Security groups

changefeedCustomName : string​

Pre-created change feed

YdbTarget​

database : string​

Path in YDB where to store tables

instance : string​

Instance of YDB. example: ydb-ru-prestable.yandex.net:2135

path : string​

Path extension for database, each table will be layouted into this path

serviceAccountId : string​
cleanupPolicy : YdbCleanupPolicy​

Cleanup policy

subnetId : string​

Network interface for endpoint. If none will assume public ipv4

saKeyContent : string​

SA content

securityGroups : string​

Security groups

isTableColumnOriented : bool​

Should create column-oriented table (OLAP). By default it creates row-oriented (OLTP)

defaultCompression : YdbDefaultCompression​

Compression that will be used for default columns family on YDB table creation