Releases: mattcasters/kettle-beam
1.0.3
You can download this plugin version over here: kettle-beam-1.0.3.zip
A full Kettle Beam Remix download version is available at www.kettle.be
Releasing the hundreds of MB in libraries for this release is not optimal for GitHub so you need to download the plugin with the link above. Please download kettle-beam-1.0.3.zip and unzip it in the /plugins/ folder.
Then patch your Kettle CE version 8.2 by unzipping pdi-engine-configuration-8.2.0.0-342.zip in and over your Kettle distribution root folder . Doing this adds "Beam" as a Run Configuration option besides "Pentaho" and "Spark". The source code for this new "Beam Run Configuration" can be found by getting branch 8.2.0.1 from pentaho-kettle in the engine-configuration plugin
Kafka Consumer improvements
You can download this plugin version over here: kettle-beam-1.0.2.zip
A full Kettle Beam Remix download version is available at www.kettle.be
Releasing the hundreds of MB in libraries for this release is not optimal for GitHub so you need to download the plugin with the link above. Please download kettle-beam-1.0.2.zip and unzip it in the /plugins/ folder.
Then patch your Kettle CE version 8.2 by unzipping pdi-engine-configuration-8.2.0.0-342.zip in and over your Kettle distribution root folder . Doing this adds "Beam" as a Run Configuration option besides "Pentaho" and "Spark". The source code for this new "Beam Run Configuration" can be found by getting branch 8.2.0.1 from pentaho-kettle in the engine-configuration plugin
Apache Beam 2.18.0
Update to the latest Beam version 2.18.0
Version set to 1.0.1 to differentiate properly.
Plugin dowload location: http://kettle-eu.s3.amazonaws.com/kettle-beam-1.0.1.zip
A full Kettle Beam Remix download version is available at www.kettle.be
Releasing the hundreds of MB in libraries for this release is not optimal for GitHub so you need to download the plugin with the link above. Please download kettle-beam-1.0.1.zip and unzip it in the /plugins/ folder.
Then patch your Kettle CE version 8.2 by unzipping pdi-engine-configuration-8.2.0.0-342.zip in and over your Kettle distribution root folder . Doing this adds "Beam" as a Run Configuration option besides "Pentaho" and "Spark". The source code for this new "Beam Run Configuration" can be found by getting branch 8.2.0.1 from pentaho-kettle in the engine-configuration plugin
For configuration and usage please see the README.md file in this project.
Apache Beam 2.15.0
With many improvements as well around fat jars, creation of fat jars, simplification of command line tools and so on.
Plugin download
Or download a pre-build version on the Kettle homepage.
Apache Beam 2.11.0 update
Issue #34
Issue #35
Issue #36
Issue #37
Releasing the hundreds of MB in libraries for this release is not optimal for GitHub so you need to download the plugin elsewhere. Please download kettle-beam-0.6.0.zip and unzip it in the <PDI>/plugins/ folder.
Then patch your Kettle CE version 8.2 by unzipping pdi-engine-configuration-8.2.0.0-342.zip in and over your Kettle distribution root folder . Doing this adds "Beam" as a Run Configuration option besides "Pentaho" and "Spark". The source code for this new "Beam Run Configuration" can be found by getting branch 8.2.0.1 from pentaho-kettle in the engine-configuration plugin
For configuration and usage please see the README.md file in this project.
Flink
New & Improved:
- Issue #32 : Flink support (Main Method)
- Issue #28 : Improved BigQuery output
- Issue #26 : Menu item to create kettle-beam-fat.jar / metastore.json
- Issue #33 : Apache Beam 2.10.0
- Issue #30 : Support for generic Kettle input steps
- Issue #31 : Batching up rows for output steps like Neo4j Output.
Releasing the hundreds of MB in libraries for this release is not optimal for GitHub so you need to download the plugin elsewhere. Please download kettle-beam-0.5.0.zip and unzip it in the <PDI>/plugins/ folder.
Then patch your Kettle CE version 8.2 by unzipping pdi-engine-configuration-8.2.0.0-342.zip in and over your Kettle distribution root folder . Doing this adds "Beam" as a Run Configuration option besides "Pentaho" and "Spark". The source code for this new "Beam Run Configuration" can be found by getting branch 8.2.0.1 from pentaho-kettle in the engine-configuration plugin
For configuration and usage please see the README.md file in this project.
Batching & Flink
New & Improved:
- Any Kettle input step loads data from anywhere (downside: data needs to fit in memory)
- Any Output step, now with batching (using "row set size" of transformation)
- Beam Job Config Dialog cleanup
- Beam Job Config: added Flink options
- Local Flink runner support
Releasing the hundreds of MB in libraries for this release is not optimal for GitHub so you need to download the plugin elsewhere. Please download this archive and unzip it in the <PDI>/plugins/ folder.
Then patch your Kettle CE version 8.2 by unzipping pdi-engine-configuration-8.2.0.0-342.zip in and over your Kettle distribution root folder . Doing this adds "Beam" as a Run Configuration option besides "Pentaho" and "Spark". The source code for this new "Beam Run Configuration" can be found by getting branch 8.2.0.1 from pentaho-kettle in the engine-configuration plugin
For configuration and usage please see the README.md file in this project.
Kafka & Spark
Releasing the hundreds of MB in libraries for this release is not optimal for GitHub so you need to download the plugin elsewhere. Please download this archive and unzip it in the <PDI>/plugins/ folder.
Then patch your Kettle CE version 8.2 by unzipping pdi-engine-configuration-8.2.0.0-342.zip in and over your Kettle distribution root folder . Doing this adds "Beam" as a Run Configuration option besides "Pentaho" and "Spark". The source code for this new "Beam Run Configuration" can be found by getting branch 8.2.0.1 from pentaho-kettle in the engine-configuration plugin
For configuration and usage please see the README.md file in this project.
Streaming with PubSub and BigQuery
Issue #20 : Google Pub/Sub Publish and Subscribe
Issue #21 : Goole BigQuery Input and Output
Issue #22 : Beam Timestamp (for streaming bounded data sources)
Releasing the hundreds of MB in libraries for this release is not optimal for GitHub so you need to download the plugin elsewhere. Please download this archive and unzip it in the <PDI>/plugins/ folder.
Then patch your Kettle CE version 8.2 by unzipping pdi-engine-configuration-8.2.0.0-342.zip in and over your Kettle distribution root folder <PDI>. Doing this adds "Beam" as a Run Configuration option besides "Pentaho" and "Spark". The source code for this new "Beam Run Configuration" can be found by getting branch 8.2.0.1 from pentaho-kettle in the engine-configuration plugin
For configuration and usage please see the README.md file in this project.
Streaming with Pub/Sub
Releasing the hundreds of MB in libraries for this release is not possible so you need to puzzle a bit.
Use this archive instead for this release.
Patch your Kettle CE version 8.2 by unzipping pdi-engine-configuration-8.2.0.0-342.zip in and over your Kettle distribution folder <PDI>.
Doing this adds Beam as a Run Configuration option besides "Pentaho" and "Spark".
Source code for the new Beam Run Configuration can be found by getting branch 8.2.0.1 from pentaho-kettle, the engine-configuration plugin