Package org.apache.heron.spouts.kafka
Class KafkaSpout<K,V>
- java.lang.Object
-
- org.apache.heron.api.topology.BaseComponent
-
- org.apache.heron.api.spout.BaseRichSpout
-
- org.apache.heron.spouts.kafka.KafkaSpout<K,V>
-
- Type Parameters:
K
- the type of the key field of the Kafka recordV
- the type of the value field of the Kafka record
- All Implemented Interfaces:
Serializable
,org.apache.heron.api.spout.IRichSpout
,org.apache.heron.api.spout.ISpout
,org.apache.heron.api.topology.IComponent
,org.apache.heron.api.topology.IStatefulComponent<org.apache.kafka.common.TopicPartition,Long>
public class KafkaSpout<K,V> extends org.apache.heron.api.spout.BaseRichSpout implements org.apache.heron.api.topology.IStatefulComponent<org.apache.kafka.common.TopicPartition,Long>
Kafka spout to consume data from Kafka topic(s), each record is converted into a tuple viaConsumerRecordTransformer
, and emitted into a topology- See Also:
- Serialized Form
-
-
Nested Class Summary
Nested Classes Modifier and Type Class Description class
KafkaSpout.KafkaConsumerRebalanceListener
-
Constructor Summary
Constructors Constructor Description KafkaSpout(KafkaConsumerFactory<K,V> kafkaConsumerFactory, Collection<String> topicNames)
create a KafkaSpout instance that subscribes to a list of topicsKafkaSpout(KafkaConsumerFactory<K,V> kafkaConsumerFactory, TopicPatternProvider topicPatternProvider)
create a KafkaSpout instance that subscribe to all topics matching the topic pattern
-
Method Summary
All Methods Instance Methods Concrete Methods Modifier and Type Method Description void
ack(Object msgId)
void
activate()
void
close()
void
deactivate()
void
declareOutputFields(org.apache.heron.api.topology.OutputFieldsDeclarer declarer)
void
fail(Object msgId)
Map<String,Object>
getComponentConfiguration()
ConsumerRecordTransformer<K,V>
getConsumerRecordTransformer()
return the consumer record transformervoid
initState(org.apache.heron.api.state.State<org.apache.kafka.common.TopicPartition,Long> aState)
void
nextTuple()
void
open(Map<String,Object> conf, org.apache.heron.api.topology.TopologyContext context, org.apache.heron.api.spout.SpoutOutputCollector aCollector)
void
preSave(String checkpointId)
void
setConsumerRecordTransformer(ConsumerRecordTransformer<K,V> consumerRecordTransformer)
set the Kafka record transformer
-
-
-
Constructor Detail
-
KafkaSpout
public KafkaSpout(KafkaConsumerFactory<K,V> kafkaConsumerFactory, Collection<String> topicNames)
create a KafkaSpout instance that subscribes to a list of topics- Parameters:
kafkaConsumerFactory
- kafka consumer factorytopicNames
- list of topic names
-
KafkaSpout
public KafkaSpout(KafkaConsumerFactory<K,V> kafkaConsumerFactory, TopicPatternProvider topicPatternProvider)
create a KafkaSpout instance that subscribe to all topics matching the topic pattern- Parameters:
kafkaConsumerFactory
- kafka consumer factorytopicPatternProvider
- provider of the topic matching pattern
-
-
Method Detail
-
getConsumerRecordTransformer
public ConsumerRecordTransformer<K,V> getConsumerRecordTransformer()
return the consumer record transformer- Returns:
- the Kafka record transformer instance used by this Kafka Spout
-
setConsumerRecordTransformer
public void setConsumerRecordTransformer(ConsumerRecordTransformer<K,V> consumerRecordTransformer)
set the Kafka record transformer- Parameters:
consumerRecordTransformer
- kafka record transformer
-
open
public void open(Map<String,Object> conf, org.apache.heron.api.topology.TopologyContext context, org.apache.heron.api.spout.SpoutOutputCollector aCollector)
- Specified by:
open
in interfaceorg.apache.heron.api.spout.ISpout
-
initState
public void initState(org.apache.heron.api.state.State<org.apache.kafka.common.TopicPartition,Long> aState)
-
preSave
public void preSave(String checkpointId)
-
nextTuple
public void nextTuple()
- Specified by:
nextTuple
in interfaceorg.apache.heron.api.spout.ISpout
-
activate
public void activate()
- Specified by:
activate
in interfaceorg.apache.heron.api.spout.ISpout
- Overrides:
activate
in classorg.apache.heron.api.spout.BaseRichSpout
-
deactivate
public void deactivate()
- Specified by:
deactivate
in interfaceorg.apache.heron.api.spout.ISpout
- Overrides:
deactivate
in classorg.apache.heron.api.spout.BaseRichSpout
-
ack
public void ack(Object msgId)
- Specified by:
ack
in interfaceorg.apache.heron.api.spout.ISpout
- Overrides:
ack
in classorg.apache.heron.api.spout.BaseRichSpout
-
fail
public void fail(Object msgId)
- Specified by:
fail
in interfaceorg.apache.heron.api.spout.ISpout
- Overrides:
fail
in classorg.apache.heron.api.spout.BaseRichSpout
-
close
public void close()
- Specified by:
close
in interfaceorg.apache.heron.api.spout.ISpout
- Overrides:
close
in classorg.apache.heron.api.spout.BaseRichSpout
-
declareOutputFields
public void declareOutputFields(org.apache.heron.api.topology.OutputFieldsDeclarer declarer)
- Specified by:
declareOutputFields
in interfaceorg.apache.heron.api.topology.IComponent
-
-