Skip to main content

Create

import {
  cloudApi,
  serviceClients,
  Session,
  waitForOperation,
} from "@yandex-cloud/nodejs-sdk";

const CreateTransferRequest =
  cloudApi.datatransfer.transfer_service.CreateTransferRequest;
const TransferType = cloudApi.datatransfer.transfer.TransferType;

(async () => {
  const authToken = process.env["YC_OAUTH_TOKEN"];
  const session = new Session({ oauthToken: authToken });
  const client = session.client(serviceClients.TransferServiceClient);

  const operation = await client.create(
    CreateTransferRequest.fromPartial({
      // sourceId: "sourceId",
      // targetId: "targetId",
      // description: "description",
      // folderId: "folderId",
      // runtime: {
      // ycRuntime: {
      // jobCount: 0,
      // uploadShardParams: {
      // jobCount: 0,
      // processCount: 0
      // }
      // }
      // },
      // type: TransferType.SNAPSHOT_AND_INCREMENT,
      // name: "name",
      // labels: {"key": "labels"},
      // transformation: {
      // transformers: [{
      // maskField: {
      // tables: {
      // includeTables: ["includeTables"],
      // excludeTables: ["excludeTables"]
      // },
      // columns: ["columns"],
      // function: {
      // maskFunctionHash: {
      // userDefinedSalt: "userDefinedSalt"
      // }
      // }
      // },
      // filterColumns: {
      // tables: {
      // includeTables: ["includeTables"],
      // excludeTables: ["excludeTables"]
      // },
      // columns: {
      // includeColumns: ["includeColumns"],
      // excludeColumns: ["excludeColumns"]
      // }
      // },
      // renameTables: {
      // renameTables: [{
      // originalName: {
      // nameSpace: "nameSpace",
      // name: "name"
      // },
      // newName: {
      // nameSpace: "nameSpace",
      // name: "name"
      // }
      // }]
      // },
      // replacePrimaryKey: {
      // tables: {
      // includeTables: ["includeTables"],
      // excludeTables: ["excludeTables"]
      // },
      // keys: ["keys"]
      // },
      // convertToString: {
      // tables: {
      // includeTables: ["includeTables"],
      // excludeTables: ["excludeTables"]
      // },
      // columns: {
      // includeColumns: ["includeColumns"],
      // excludeColumns: ["excludeColumns"]
      // }
      // },
      // sharderTransformer: {
      // tables: {
      // includeTables: ["includeTables"],
      // excludeTables: ["excludeTables"]
      // },
      // columns: {
      // includeColumns: ["includeColumns"],
      // excludeColumns: ["excludeColumns"]
      // },
      // shardsCount: 0
      // },
      // tableSplitterTransformer: {
      // tables: {
      // includeTables: ["includeTables"],
      // excludeTables: ["excludeTables"]
      // },
      // columns: ["columns"],
      // splitter: "splitter"
      // },
      // filterRows: {
      // tables: {
      // includeTables: ["includeTables"],
      // excludeTables: ["excludeTables"]
      // },
      // filter: "filter",
      // filters: ["filters"]
      // }
      // }]
      // }
    })
  );
  await waitForOperation(operation, session);
})();

CreateTransferRequest​

sourceId : string​
targetId : string​
description : string​
folderId : string​
runtime : Runtime​
type : TransferType​
name : string​
labels : string​
transformation : Transformation​

Runtime​

One of runtime​

  • ycRuntime : YcRuntime​

Transformation​

Transformation is converting data using special transformer functions. These functions are executed on a data stream, applied to each data change item, and transform them. A transformer can be run at both the metadata and data levels. Data can only be transformed if the source and target are of different types.

transformers : Transformer​

Transformers are set as a list. When activating a transfer, a transformation plan is made for the tables that match the specified criteria. Transformers are applied to the tables in the sequence specified in the list.

YcRuntime​

jobCount : int64​
uploadShardParams : ShardingUploadParams​

Transformer​

Some transformers may have limitations and only apply to some source-target pairs.

One of transformer​

  • maskField : MaskFieldTransformer​
  • filterColumns : FilterColumnsTransformer​
  • renameTables : RenameTablesTransformer​
  • replacePrimaryKey : ReplacePrimaryKeyTransformer​
  • convertToString : ToStringTransformer​
  • sharderTransformer : SharderTransformer​
  • tableSplitterTransformer : TableSplitterTransformer​
  • filterRows : FilterRowsTransformer​

ShardingUploadParams​

jobCount : int64​
processCount : int64​

MaskFieldTransformer​

Mask field transformer allows you to hash data

tables : TablesFilter​

List of included and excluded tables

columns : string​

Specify the name of the column for data masking (a regular expression).

function : MaskFunction​

Mask function

FilterColumnsTransformer​

Set up a list of table columns to transfer

tables : TablesFilter​

List of the tables to filter using lists of included and excluded tables.

columns : ColumnsFilter​

List of the columns to transfer to the target tables using lists of included and excluded columns.

RenameTablesTransformer​

Set rules for renaming tables by specifying the current names of the tables in the source and new names for these tables in the target.

renameTables : RenameTable​

List of renaming rules

ReplacePrimaryKeyTransformer​

Override primary keys

tables : TablesFilter​

List of included and excluded tables

keys : string​

List of columns to be used as primary keys

ToStringTransformer​

Convert column values to strings The values will be converted depending on the source type Conversion rules are described here: https://cloud.yandex.com/en/docs/data-transfer/concepts/data-transformation#convert-to-string

tables : TablesFilter​

List of included and excluded tables

columns : ColumnsFilter​

List of included and excluded columns

SharderTransformer​

Set the number of shards for particular tables and a list of columns whose values will be used for calculating a hash to determine a shard.

tables : TablesFilter​

List of included and excluded tables

columns : ColumnsFilter​

List of included and excluded columns

shardsCount : int64​

Number of shards

TableSplitterTransformer​

A transfer splits the X table into multiple tables (X_1, X_2, ..., X_n) based on data. If a row was located in the X table before it was split, it is now in the X_i table, where i is determined by the column list and split string parameters. Example: If the column list has two columns, month of birth and gender, specified and the split string states @, information about an employee whose name is John and who was born on February 11, 1984, from the Employees table will get to a new table named Employees@February@male.

tables : TablesFilter​

List of included and excluded tables

columns : string​

Specify the columns in the tables to be partitioned.

splitter : string​

Specify the split string to be used for merging components in a new table name.

FilterRowsTransformer​

This filter only applies to transfers with queues (Logbroker or Apache Kafka®) as a data source. When running a transfer, only the strings meeting the specified criteria remain in a changefeed.

tables : TablesFilter​

List of included and excluded tables.

filter : string​

Filtering criterion. This can be comparison operators for numeric, string, and Boolean values, comparison to NULL, and checking whether a substring is part of a string. Details here: https://yandex.cloud/en-ru/docs/data-transfer/concepts/data-transformation#append-only-sources. Deprecated: Use filters instead.

filters : string​

Data is transported if it satisfies at least one of filters. Consider that there is OR statement between filters. Each filter can be comparison operators for numeric, string, and Boolean values, comparison to NULL, and checking whether a substring is part of a string. Details in docs: https://yandex.cloud/en-ru/docs/data-transfer/concepts/data-transformation#append-only-sources.

TablesFilter​

Filter tables using lists of included and excluded tables.

includeTables : string​

List of tables that will be included to transfer

excludeTables : string​

List of tables that will be excluded to transfer

MaskFunction​

Mask function

One of maskFunction​

  • maskFunctionHash : MaskFunctionHash​

    Hash mask function

ColumnsFilter​

Filter columns using lists of included and excluded columns.

includeColumns : string​

List of columns that will be included to transfer

excludeColumns : string​

List of columns that will be excluded to transfer

RenameTable​

Specify rule for renaming table

originalName : Table​

Specify the current names of the table in the source

newName : Table​

Specify the new names for this table in the target

MaskFunctionHash​

Hash data using HMAC

userDefinedSalt : string​

This string will be used in the HMAC(sha256, salt) function applied to the column data.

Table​

nameSpace : string​
name : string​

Operation​

An Operation resource. For more information, see Operation.

id : string​

ID of the operation.

description : string​

Description of the operation. 0-256 characters long.

createdAt : google.protobuf.Timestamp​

Creation timestamp.

createdBy : string​

ID of the user or service account who initiated the operation.

modifiedAt : google.protobuf.Timestamp​

The time when the Operation resource was last modified.

done : bool​

If the value is false, it means the operation is still in progress. If true, the operation is completed, and either error or response is available.

metadata : google.protobuf.Any​

Service-specific metadata associated with the operation. It typically contains the ID of the target resource that the operation is performed on. Any method that returns a long-running operation should document the metadata type, if any.

One of result​

The operation result. If done == false and there was no failure detected, neither error nor response is set. If done == false and there was a failure detected, error is set. If done == true, exactly one of error or response is set.

  • error : google.rpc.Status​

    The error result of the operation in case of failure or cancellation.

  • response : google.protobuf.Any​
    The normal response of the operation in case of success.

    If the original method returns no data on success, such as Delete, the response is google.protobuf.Empty. If the original method is the standard Create/Update, the response should be the target resource of the operation. Any method that returns a long-running operation should document the response type, if any.