Skip to main content

Create

Creates a Greenplum® cluster in the specified folder.

import {
cloudApi,
decodeMessage,
serviceClients,
Session,
waitForOperation,
} from "@yandex-cloud/nodejs-sdk";

const Cluster = cloudApi.dataproc.cluster.Cluster;
const CreateClusterRequest =
cloudApi.dataproc.cluster_service.CreateClusterRequest;
const HadoopConfig_Service = cloudApi.dataproc.cluster.HadoopConfig_Service;
const Role = cloudApi.dataproc.subcluster.Role;

(async () => {
const authToken = process.env["YC_OAUTH_TOKEN"];
const session = new Session({ oauthToken: authToken });
const client = session.client(serviceClients.ClusterServiceClient);

const operation = await client.create(
CreateClusterRequest.fromPartial({
folderId: "folderId",
// name: "name",
// description: "description",
// labels: {"key": "labels"},
configSpec: {
// versionId: "versionId",
// hadoop: {
// services: [HadoopConfig_Service.HDFS],
// properties: {"key": "properties"},
// sshPublicKeys: ["sshPublicKeys"],
// initializationActions: [{
// uri: "uri",
// args: ["args"],
// timeout: 0
// }]
// },
// subclustersSpec: [{
// name: "name",
// role: Role.MASTERNODE,
// resources: {
// resourcePresetId: "resourcePresetId",
// diskTypeId: "diskTypeId",
// diskSize: 0
// },
// subnetId: "subnetId",
// hostsCount: 0,
// assignPublicIp: true,
// autoscalingConfig: {
// maxHostsCount: 0,
// preemptible: true,
// measurementDuration: {
// seconds: 0,
// nanos: 0
// },
// warmupDuration: {
// seconds: 0,
// nanos: 0
// },
// stabilizationDuration: {
// seconds: 0,
// nanos: 0
// },
// cpuUtilizationTarget: 0,
// decommissionTimeout: 0
// }
// }]
},
zoneId: "zoneId",
serviceAccountId: "serviceAccountId",
// bucket: "bucket",
// uiProxy: true,
// securityGroupIds: ["securityGroupIds"],
// hostGroupIds: ["hostGroupIds"],
// deletionProtection: true,
// logGroupId: "logGroupId"
})
);
const finishedOp = await waitForOperation(operation, session);

if (finishedOp.response) {
const result = decodeMessage<typeof Cluster>(finishedOp.response);
console.log(result);
}
})();

CreateClusterRequest

folderId : string

ID of the folder to create the Greenplum® cluster in.

name : string

Name of the Greenplum® cluster. The name must be unique within the folder.

description : string

Description of the Greenplum® cluster.

labels : string

Custom labels for the Greenplum® cluster as key:value pairs. For example, "project":"mvp" or "source":"dictionary".

environment : Cluster.Environment

Deployment environment of the Greenplum® cluster.

config : GreenplumConfig

Greenplum® cluster configuration.

masterConfig : MasterSubclusterConfigSpec

Configuration of the Greenplum® master subcluster.

segmentConfig : SegmentSubclusterConfigSpec

Configuration of the Greenplum® segment subcluster.

masterHostCount : int64

Number of hosts in the master subcluster.

segmentInHost : int64

Number of segments per host.

segmentHostCount : int64

Number of hosts in the segment subcluster.

userName : string

Owner user name.

userPassword : string

Owner user password.

networkId : string

ID of the network to create the cluster in.

securityGroupIds : string

User security groups.

deletionProtection : bool

Determines whether the cluster is protected from being deleted.

hostGroupIds : string

Host groups to place VMs of the cluster in.

maintenanceWindow : MaintenanceWindow

A Greenplum® cluster maintenance window. Should be defined by either one of the two options.

configSpec : ConfigSpec

Configuration of Greenplum® and Odyssey®.

cloudStorage : CloudStorage

Cloud storage settings

masterHostGroupIds : string

Host groups hosting VMs of the master subcluster.

segmentHostGroupIds : string

Host groups hosting VMs of the segment subcluster.

GreenplumConfig

version : string

Version of the Greenplum® server software.

backupWindowStart : google.type.TimeOfDay

Time to start the daily backup, in the UTC timezone.

backupRetainPeriodDays : google.protobuf.Int64Value

Retention policy of automated backups.

access : Access

Access policy for external services.

zoneId : string

ID of the availability zone the cluster belongs to. To get a list of available zones, use the yandex.cloud.compute.v1.ZoneService.List request.

subnetId : string

ID of the subnet the cluster belongs to. This subnet should be a part of the cloud network the cluster belongs to (see Cluster.network_id).

assignPublicIp : bool

Determines whether the cluster has a public IP address.

After the cluster has been created, this setting cannot be changed.

MasterSubclusterConfigSpec

resources : Resources

Resources allocated to Greenplum® master subcluster hosts.

SegmentSubclusterConfigSpec

resources : Resources

Resources allocated to Greenplum® segment subcluster hosts.

MaintenanceWindow

One of policy

  • anytime : AnytimeMaintenanceWindow

    An any-time maintenance window.

  • weeklyMaintenanceWindow : WeeklyMaintenanceWindow

    A weekly maintenance window.

ConfigSpec

One of greenplumConfig

  • greenplumConfig_6_17 : GreenplumConfig6_17
  • greenplumConfig_6_19 : GreenplumConfig6_19
  • greenplumConfig_6_21 : GreenplumConfig6_21
  • greenplumConfig_6_22 : GreenplumConfig6_22
  • greenplumConfig_6 : GreenplumConfig6
pool : ConnectionPoolerConfig

Odyssey® pool settings.

backgroundActivities : BackgroundActivitiesConfig

Managed Greenplum® background tasks configuration.

pxfConfig : PXFConfig

CloudStorage

Cloud Storage Settings

enable : bool

enable Cloud Storage for cluster

Access

dataLens : bool

Allows data export from the cluster to DataLens.

webSql : bool

Allows SQL queries to the cluster databases from the management console.

dataTransfer : bool

Allows access for DataTransfer.

yandexQuery : bool

Allow access for YandexQuery.

Resources

resourcePresetId : string

ID of the preset for computational resources allocated to a host.

Available presets are listed in the documentation.

diskSize : int64

Volume of the storage used by the host, in bytes.

diskTypeId : string

Type of the storage used by the host: network-hdd, network-ssd or local-ssd.

AnytimeMaintenanceWindow

An any-time maintenance window.

WeeklyMaintenanceWindow

A weekly maintenance window.

WeekDay
  • WEEK_DAY_UNSPECIFIED

  • MON

    Monday

  • TUE

    Tuesday

  • WED

    Wednesday

  • THU

    Thursday

  • FRI

    Friday

  • SAT

    Saturday

  • SUN

    Sunday

day : WeekDay

Day of the week.

hour : int64

Hour of the day in the UTC timezone.

GreenplumConfig6_17

maxConnections : google.protobuf.Int64Value

Maximum number of inbound connections on master segment.

maxSlotWalKeepSize : google.protobuf.Int64Value

The maximum size of WAL files that replication slots are allowed to retain in the pg_wal directory at checkpoint time.

More info in PostgreSQL® documentation.

gpWorkfileLimitPerSegment : google.protobuf.Int64Value

The maximum total disk size that all running queries are allowed to use for creating temporary spill files at each segment.

The default value is 0 (no limit).

More info in Greenplum® documentation.

gpWorkfileLimitPerQuery : google.protobuf.Int64Value

The maximum disk size that an individual query is allowed to use for creating temporary spill files at each segment.

The default value is 0 (no limit).

More info in Greenplum® documentation.

gpWorkfileLimitFilesPerQuery : google.protobuf.Int64Value

The maximum number of temporary spill files allowed per query at each segment.

Spill files, also known as workfiles, are created when a query requires more memory than there is allocated.

The current query is terminated if the limit is exceeded.

Set to zero to disable the limit.

Master session reloads if the parameter changes.

Default value is 10000.

More info in Greenplum® documentation.

maxPreparedTransactions : google.protobuf.Int64Value

The maximum number of transactions that can be in the prepared state simultaneously.

More info in PostgreSQL® documentation.

gpWorkfileCompression : google.protobuf.BoolValue

Whether the spill files are compressed or not.

More info in Greenplum® documentation.

GreenplumConfig6_19

maxConnections : google.protobuf.Int64Value

Maximum number of inbound connections on master segment.

maxSlotWalKeepSize : google.protobuf.Int64Value

The maximum size of WAL files that replication slots are allowed to retain in the pg_wal directory at checkpoint time.

More info in PostgreSQL® documentation.

gpWorkfileLimitPerSegment : google.protobuf.Int64Value

The maximum total disk size that all running queries are allowed to use for creating temporary spill files at each segment.

The default value is 0 (no limit).

More info in Greenplum® documentation.

gpWorkfileLimitPerQuery : google.protobuf.Int64Value

The maximum disk size that an individual query is allowed to use for creating temporary spill files at each segment.

The default value is 0 (no limit).

More info in Greenplum® documentation.

gpWorkfileLimitFilesPerQuery : google.protobuf.Int64Value

The maximum number of temporary spill files allowed per query at each segment.

Spill files, also known as workfiles, are created when a query requires more memory than there is allocated.

The current query is terminated if the limit is exceeded.

Set to zero to disable the limit.

Master session reloads if the parameter changes.

Default value is 10000.

More info in Greenplum® documentation.

maxPreparedTransactions : google.protobuf.Int64Value

The maximum number of transactions that can be in the prepared state simultaneously.

More info in PostgreSQL® documentation.

gpWorkfileCompression : google.protobuf.BoolValue

Whether the spill files are compressed or not.

More info in Greenplum® documentation.

maxStatementMem : google.protobuf.Int64Value

The maximum memory limit for a query, in bytes.

Helps to avoid out-of-memory errors on a segment host during query processing as a result of setting statement_mem too high.

Taking into account the configuration of a single segment host, calculate max_statement_mem as follows: seghost_physical_memory / average_number_concurrent_queries.

When changing both max_statement_mem and statement_mem, max_statement_mem must be changed first, or listed first in the postgresql.conf file.

Default value is 2097152000 (2000 MB).

More info in Greenplum® documentation.

logStatement : LogStatement

Logged SQL statements.

PREPARE and EXPLAIN ANALYZE statements are also logged if their contained command belongs to an appropriate type.

More info in Greenplum® documentation.

GreenplumConfig6_21

maxConnections : google.protobuf.Int64Value

Maximum number of inbound connections on master segment

maxSlotWalKeepSize : google.protobuf.Int64Value

Specify the maximum size of WAL files that replication slots are allowed to retain in the pg_wal directory at checkpoint time. https://www.postgresql.org/docs/current/runtime-config-replication.html

gpWorkfileLimitPerSegment : google.protobuf.Int64Value

Sets the maximum total disk size that all running queries are allowed to use for creating temporary spill files at each segment. The default value is 0, which means a limit is not enforced. https://docs.vmware.com/en/VMware-Greenplum/6/greenplum-database/ref_guide-config_params-guc-list.html#gp_workfile_limit_per_segment

gpWorkfileLimitPerQuery : google.protobuf.Int64Value

Sets the maximum disk size an individual query is allowed to use for creating temporary spill files at each segment. The default value is 0, which means a limit is not enforced. https://docs.vmware.com/en/VMware-Greenplum/6/greenplum-database/ref_guide-config_params-guc-list.html#gp_workfile_limit_per_query

gpWorkfileLimitFilesPerQuery : google.protobuf.Int64Value

Sets the maximum number of temporary spill files (also known as workfiles) allowed per query per segment. Spill files are created when executing a query that requires more memory than it is allocated. The current query is terminated when the limit is exceeded. Set the value to 0 (zero) to allow an unlimited number of spill files. master session reload https://docs.vmware.com/en/VMware-Greenplum/6/greenplum-database/ref_guide-config_params-guc-list.html#gp_workfile_limit_files_per_query Default value is 10000

maxPreparedTransactions : google.protobuf.Int64Value

Sets the maximum number of transactions that can be in the "prepared" state simultaneously https://www.postgresql.org/docs/9.6/runtime-config-resource.html

gpWorkfileCompression : google.protobuf.BoolValue

Specifies whether the temporary files created, when a hash aggregation or hash join operation spills to disk, are compressed. https://docs.vmware.com/en/VMware-Greenplum/6/greenplum-database/ref_guide-config_params-guc-list.html#gp_workfile_compression

maxStatementMem : google.protobuf.Int64Value

Sets the maximum memory limit for a query. Helps avoid out-of-memory errors on a segment host during query processing as a result of setting statement_mem too high. Taking into account the configuration of a single segment host, calculate max_statement_mem as follows: (seghost_physical_memory) / (average_number_concurrent_queries) When changing both max_statement_mem and statement_mem, max_statement_mem must be changed first, or listed first in the postgresql.conf file. https://greenplum.docs.pivotal.io/6-19/ref_guide/config_params/guc-list.html#max_statement_mem Default value is 2097152000 (2000MB)

logStatement : LogStatement

Controls which SQL statements are logged. DDL logs all data definition commands like CREATE, ALTER, and DROP commands. MOD logs all DDL statements, plus INSERT, UPDATE, DELETE, TRUNCATE, and COPY FROM. PREPARE and EXPLAIN ANALYZE statements are also logged if their contained command is of an appropriate type. https://docs.vmware.com/en/VMware-Greenplum/6/greenplum-database/ref_guide-config_params-guc-list.html#log_statement Default value is ddl

gpAddColumnInheritsTableSetting : google.protobuf.BoolValue

https://docs.vmware.com/en/VMware-Tanzu-Greenplum/6/greenplum-database/GUID-ref_guide-config_params-guc-list.html#gp_add_column_inherits_table_setting

GreenplumConfig6_22

maxConnections : google.protobuf.Int64Value

Maximum number of inbound connections on master segment

maxSlotWalKeepSize : google.protobuf.Int64Value

Specify the maximum size of WAL files that replication slots are allowed to retain in the pg_wal directory at checkpoint time. https://www.postgresql.org/docs/current/runtime-config-replication.html

gpWorkfileLimitPerSegment : google.protobuf.Int64Value

Sets the maximum total disk size that all running queries are allowed to use for creating temporary spill files at each segment. The default value is 0, which means a limit is not enforced. https://docs.vmware.com/en/VMware-Greenplum/6/greenplum-database/ref_guide-config_params-guc-list.html#gp_workfile_limit_per_segment

gpWorkfileLimitPerQuery : google.protobuf.Int64Value

Sets the maximum disk size an individual query is allowed to use for creating temporary spill files at each segment. The default value is 0, which means a limit is not enforced. https://docs.vmware.com/en/VMware-Greenplum/6/greenplum-database/ref_guide-config_params-guc-list.html#gp_workfile_limit_per_query

gpWorkfileLimitFilesPerQuery : google.protobuf.Int64Value

Sets the maximum number of temporary spill files (also known as workfiles) allowed per query per segment. Spill files are created when executing a query that requires more memory than it is allocated. The current query is terminated when the limit is exceeded. Set the value to 0 (zero) to allow an unlimited number of spill files. master session reload https://docs.vmware.com/en/VMware-Greenplum/6/greenplum-database/ref_guide-config_params-guc-list.html#gp_workfile_limit_files_per_query Default value is 10000

maxPreparedTransactions : google.protobuf.Int64Value

Sets the maximum number of transactions that can be in the "prepared" state simultaneously https://www.postgresql.org/docs/9.6/runtime-config-resource.html

gpWorkfileCompression : google.protobuf.BoolValue

Specifies whether the temporary files created, when a hash aggregation or hash join operation spills to disk, are compressed. https://docs.vmware.com/en/VMware-Greenplum/6/greenplum-database/ref_guide-config_params-guc-list.html#gp_workfile_compression

maxStatementMem : google.protobuf.Int64Value

Sets the maximum memory limit for a query. Helps avoid out-of-memory errors on a segment host during query processing as a result of setting statement_mem too high. Taking into account the configuration of a single segment host, calculate max_statement_mem as follows: (seghost_physical_memory) / (average_number_concurrent_queries) When changing both max_statement_mem and statement_mem, max_statement_mem must be changed first, or listed first in the postgresql.conf file. https://greenplum.docs.pivotal.io/6-19/ref_guide/config_params/guc-list.html#max_statement_mem Default value is 2097152000 (2000MB)

logStatement : LogStatement

Controls which SQL statements are logged. DDL logs all data definition commands like CREATE, ALTER, and DROP commands. MOD logs all DDL statements, plus INSERT, UPDATE, DELETE, TRUNCATE, and COPY FROM. PREPARE and EXPLAIN ANALYZE statements are also logged if their contained command is of an appropriate type. https://docs.vmware.com/en/VMware-Greenplum/6/greenplum-database/ref_guide-config_params-guc-list.html#log_statement Default value is ddl

gpAddColumnInheritsTableSetting : google.protobuf.BoolValue

https://docs.vmware.com/en/VMware-Tanzu-Greenplum/6/greenplum-database/GUID-ref_guide-config_params-guc-list.html#gp_add_column_inherits_table_setting

GreenplumConfig6

maxConnections : google.protobuf.Int64Value

Maximum number of inbound connections on master segment

maxSlotWalKeepSize : google.protobuf.Int64Value

Specify the maximum size of WAL files that replication slots are allowed to retain in the pg_wal directory at checkpoint time. https://www.postgresql.org/docs/current/runtime-config-replication.html

gpWorkfileLimitPerSegment : google.protobuf.Int64Value

Sets the maximum total disk size that all running queries are allowed to use for creating temporary spill files at each segment. The default value is 0, which means a limit is not enforced. https://docs.greenplum.org/6-5/ref_guide/config_params/guc-list.html#gp_workfile_limit_per_segment

gpWorkfileLimitPerQuery : google.protobuf.Int64Value

Sets the maximum disk size an individual query is allowed to use for creating temporary spill files at each segment. The default value is 0, which means a limit is not enforced. https://docs.greenplum.org/6-5/ref_guide/config_params/guc-list.html#gp_workfile_limit_per_query

gpWorkfileLimitFilesPerQuery : google.protobuf.Int64Value

Sets the maximum number of temporary spill files (also known as workfiles) allowed per query per segment. Spill files are created when executing a query that requires more memory than it is allocated. The current query is terminated when the limit is exceeded. Set the value to 0 (zero) to allow an unlimited number of spill files. master session reload https://docs.greenplum.org/6-5/ref_guide/config_params/guc-list.html#gp_workfile_limit_files_per_query Default value is 10000

maxPreparedTransactions : google.protobuf.Int64Value

Sets the maximum number of transactions that can be in the "prepared" state simultaneously https://www.postgresql.org/docs/9.6/runtime-config-resource.html

gpWorkfileCompression : google.protobuf.BoolValue

Specifies whether the temporary files created, when a hash aggregation or hash join operation spills to disk, are compressed. https://docs.greenplum.org/6-5/ref_guide/config_params/guc-list.html#gp_workfile_compression

maxStatementMem : google.protobuf.Int64Value

Sets the maximum memory limit for a query. Helps avoid out-of-memory errors on a segment host during query processing as a result of setting statement_mem too high. Taking into account the configuration of a single segment host, calculate max_statement_mem as follows: (seghost_physical_memory) / (average_number_concurrent_queries) When changing both max_statement_mem and statement_mem, max_statement_mem must be changed first, or listed first in the postgresql.conf file. https://greenplum.docs.pivotal.io/6-19/ref_guide/config_params/guc-list.html#max_statement_mem Default value is 2097152000 (2000MB)

logStatement : LogStatement

Controls which SQL statements are logged. DDL logs all data definition commands like CREATE, ALTER, and DROP commands. MOD logs all DDL statements, plus INSERT, UPDATE, DELETE, TRUNCATE, and COPY FROM. PREPARE and EXPLAIN ANALYZE statements are also logged if their contained command is of an appropriate type. https://docs.greenplum.org/6-5/ref_guide/config_params/guc-list.html#log_statement Default value is ddl

gpAddColumnInheritsTableSetting : google.protobuf.BoolValue

https://docs.vmware.com/en/VMware-Tanzu-Greenplum/6/greenplum-database/GUID-ref_guide-config_params-guc-list.html#gp_add_column_inherits_table_setting

ConnectionPoolerConfig

PoolMode
  • POOL_MODE_UNSPECIFIED

  • SESSION

    Assign server connection to a client until it disconnects. Default value.

  • TRANSACTION

    Assign server connection to a client for a transaction processing.

mode : PoolMode

Route server pool mode.

size : google.protobuf.Int64Value

The number of servers in the server pool. Clients are placed in a wait queue when all servers are busy.

Set to zero to disable the limit.

clientIdleTimeout : google.protobuf.Int64Value

Server pool idle timeout, in seconds.

A server connection closes after being idle for the specified time.

Set to zero to disable the limit.

BackgroundActivitiesConfig

tableSizes : TableSizes

Enables scripts that collects tables sizes to *_sizes tables in mdb_toolkit schema.

analyzeAndVacuum : AnalyzeAndVacuum

Configuration for ANALYZE and VACUUM operations.

queryKillerScripts : QueryKillerScripts

Configuration for long running queries killer.

PXFConfig

connectionTimeout : google.protobuf.Int64Value

Timeout for connection to the Apache Tomcat® server when making read requests.

Specify values in seconds.

uploadTimeout : google.protobuf.Int64Value

Timeout for connection to the Apache Tomcat® server when making write requests.

Specify the values in seconds.

maxThreads : google.protobuf.Int64Value

Maximum number of the Apache Tomcat® threads.

To prevent situations when requests get stuck or fail due to running out of memory or malfunctioning of the Java garbage collector, specify the number of the Apache Tomcat® threads. Learn more about adjusting the number of threads in the VMware Greenplum® Platform Extension Framework documentation.

poolAllowCoreThreadTimeout : google.protobuf.BoolValue

Determines whether the timeout for core streaming threads is permitted.

poolCoreSize : google.protobuf.Int64Value

Number of core streaming threads per pool.

poolQueueCapacity : google.protobuf.Int64Value

Maximum number of requests you can add to a pool queue for core streaming threads.

If 0, no pool queue is generated.

poolMaxSize : google.protobuf.Int64Value

Maximum allowed number of core streaming threads.

xmx : google.protobuf.Int64Value

Initial size, in megabytes, of the JVM heap for the PXF daemon.

xms : google.protobuf.Int64Value

Maximum size, in megabytes, of the JVM heap for the PXF daemon.

TableSizes

starts : BackgroundActivityStartAt

AnalyzeAndVacuum

start : BackgroundActivityStartAt
analyzeTimeout : google.protobuf.Int64Value

Maximum duration of the ANALYZE operation, in seconds. The default value is 36000. As soon as this period expires, the ANALYZE operation will be forced to terminate.

vacuumTimeout : google.protobuf.Int64Value

Maximum duration of the VACUUM operation, in seconds. The default value is 36000. As soon as this period expires, the VACUUM operation will be forced to terminate.

QueryKillerScripts

idle : QueryKiller

Configuration of script that kills long running queries that are in idle state.

idleInTransaction : QueryKiller

Configuration of script that kills long running queries that are in idle in transaction state.

longRunning : QueryKiller

Configuration of script that kills long running queries (in any state).

BackgroundActivityStartAt

hours : int64
minutes : int64

QueryKiller

enable : google.protobuf.BoolValue
maxAge : google.protobuf.Int64Value

Maximum duration for this type of queries (in seconds).

ignoreUsers : string

Ignore these users when considering queries to terminate

Operation

An Operation resource. For more information, see Operation.

id : string

ID of the operation.

description : string

Description of the operation. 0-256 characters long.

createdAt : google.protobuf.Timestamp

Creation timestamp.

createdBy : string

ID of the user or service account who initiated the operation.

modifiedAt : google.protobuf.Timestamp

The time when the Operation resource was last modified.

done : bool

If the value is false, it means the operation is still in progress. If true, the operation is completed, and either error or response is available.

metadata : google.protobuf.Any

Service-specific metadata associated with the operation. It typically contains the ID of the target resource that the operation is performed on. Any method that returns a long-running operation should document the metadata type, if any.

One of result

The operation result. If done == false and there was no failure detected, neither error nor response is set. If done == false and there was a failure detected, error is set. If done == true, exactly one of error or response is set.

  • error : google.rpc.Status

    The error result of the operation in case of failure or cancellation.

  • response : google.protobuf.Any
    The normal response of the operation in case of success.

    If the original method returns no data on success, such as Delete, the response is google.protobuf.Empty. If the original method is the standard Create/Update, the response should be the target resource of the operation. Any method that returns a long-running operation should document the response type, if any.