导航：首页 > 互联网科技 >

大数据之Apache Beam 使用Flink Runner管道参数

发表于：2024-11-16 作者：千家信息网编辑

千家信息网最后更新 2024年11月16日，例子Flink集群mvn package exec:java -Dexec.mainClass=org.apache.beam.examples.WordCount -Dexec.args="--ru

千家信息网最后更新 2024年11月16日大数据之Apache Beam 使用Flink Runner管道参数

例子

Flink集群

mvn package exec:java -Dexec.mainClass=org.apache.beam.examples.WordCount -Dexec.args="--runner=FlinkRunner --flinkMaster= --filesToStage=target/word-count-beam-bundled-0.1.jar --inputFile=/path/to/quickstart/pom.xml --output=/tmp/counts" -Pflink-runner

管道参数说明

Field	Description	Default Value
`runner`	The pipeline runner to use. This option allows you to determine the pipeline runner at runtime.	Set to `FlinkRunner`to run using Flink.
`streaming`	Whether streaming mode is enabled or disabled; `true` if enabled. Set to `true` if running pipelines with unbounded `PCollection`s.	`false`
`flinkMaster`	The url of the Flink JobManager on which to execute pipelines. This can either be the address of a cluster JobManager, in the form `"host:port"` or one of the special Strings `"[local]"` or `"[auto]"`. `"[local]"` will start a local Flink Cluster in the JVM while `"[auto]"` will let the system decide where to execute the pipeline based on the environment.	`[auto]`
`filesToStage`	Jar Files to send to all workers and put on the classpath. Here you have to put the fat jar that contains your program along with all dependencies.	empty
`parallelism`	The degree of parallelism to be used when distributing operations onto workers.	`1`
`checkpointingInterval`	The interval between consecutive checkpoints (i.e. snapshots of the current pipeline state used for fault tolerance).	`-1L`, i.e. disabled
`numberOfExecutionRetries`	Sets the number of times that failed tasks are re-executed. A value of `0` effectively disables fault tolerance. A value of `-1` indicates that the system default value (as defined in the configuration) should be used.	`-1`
`executionRetryDelay`	Sets the delay between executions. A value of `-1` indicates that the default value should be used.	`-1`
`stateBackend`	Sets the state backend to use in streaming mode. The default is to read this setting from the Flink config.	`empty`, i.e. read from Flink config