<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom">
  <channel>
    <title>Flink S3 F3 Multiple :: Ay Docs</title>
    <link>https://ops.docs.72602.space/demo/plugins/flink-s3-f3-multiple/index.html</link>
    <description>Normally, Flink only can access only one S3 endpoint during the runtime. But we need to process some files from multiple minio simultaneously.&#xA;So I modified the original flink-s3-fs-hadoop and enable flink to do so.&#xA;StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); env.enableCheckpointing(5000L, CheckpointingMode.EXACTLY_ONCE); env.setParallelism(1); env.setStateBackend(new HashMapStateBackend()); env.getCheckpointConfig().setCheckpointStorage(&#34;file:///./checkpoints&#34;); final FileSource&lt;String&gt; source = FileSource.forRecordStreamFormat( new TextLineInputFormat(), new Path( &#34;s3u://admin:ZrwpsezF1Lt85dxl@10.11.33.132:9000/user-data/home/conti/2024-02-08--10&#34;)) .build(); final FileSource&lt;String&gt; source2 = FileSource.forRecordStreamFormat( new TextLineInputFormat(), new Path( &#34;s3u://minioadmin:minioadmin@10.101.16.72:9000/user-data/home/conti&#34;)) .build(); env.fromSource(source, WatermarkStrategy.noWatermarks(), &#34;file-source&#34;) .union(env.fromSource(source2, WatermarkStrategy.noWatermarks(), &#34;file-source2&#34;)) .print(&#34;union-result&#34;); env.execute(); original usage example using default flink-s3-fs-hadoop, the configuration value will set into Hadoop configuration map. Only one value functioning at the same, there is no way for user to operate different in single one job context.</description>
    <generator>Hugo</generator>
    <language>en</language>
    <lastBuildDate></lastBuildDate>
    <atom:link href="https://ops.docs.72602.space/demo/plugins/flink-s3-f3-multiple/index.xml" rel="self" type="application/rss+xml" />
  </channel>
</rss>