Class KafkaSpout<K,​V>

  • Type Parameters:
    K - the type of the key field of the Kafka record
    V - the type of the value field of the Kafka record
    All Implemented Interfaces:
    Serializable, org.apache.heron.api.spout.IRichSpout, org.apache.heron.api.spout.ISpout, org.apache.heron.api.topology.IComponent, org.apache.heron.api.topology.IStatefulComponent<org.apache.kafka.common.TopicPartition,​Long>

    public class KafkaSpout<K,​V>
    extends org.apache.heron.api.spout.BaseRichSpout
    implements org.apache.heron.api.topology.IStatefulComponent<org.apache.kafka.common.TopicPartition,​Long>
    Kafka spout to consume data from Kafka topic(s), each record is converted into a tuple via ConsumerRecordTransformer, and emitted into a topology
    See Also:
    Serialized Form
    • Constructor Detail

      • KafkaSpout

        public KafkaSpout​(KafkaConsumerFactory<K,​V> kafkaConsumerFactory,
                          Collection<String> topicNames)
        create a KafkaSpout instance that subscribes to a list of topics
        Parameters:
        kafkaConsumerFactory - kafka consumer factory
        topicNames - list of topic names
      • KafkaSpout

        public KafkaSpout​(KafkaConsumerFactory<K,​V> kafkaConsumerFactory,
                          TopicPatternProvider topicPatternProvider)
        create a KafkaSpout instance that subscribe to all topics matching the topic pattern
        Parameters:
        kafkaConsumerFactory - kafka consumer factory
        topicPatternProvider - provider of the topic matching pattern
    • Method Detail

      • getConsumerRecordTransformer

        public ConsumerRecordTransformer<K,​V> getConsumerRecordTransformer()
        return the consumer record transformer
        Returns:
        the Kafka record transformer instance used by this Kafka Spout
      • setConsumerRecordTransformer

        public void setConsumerRecordTransformer​(ConsumerRecordTransformer<K,​V> consumerRecordTransformer)
        set the Kafka record transformer
        Parameters:
        consumerRecordTransformer - kafka record transformer
      • open

        public void open​(Map<String,​Object> conf,
                         org.apache.heron.api.topology.TopologyContext context,
                         org.apache.heron.api.spout.SpoutOutputCollector aCollector)
        Specified by:
        open in interface org.apache.heron.api.spout.ISpout
      • initState

        public void initState​(org.apache.heron.api.state.State<org.apache.kafka.common.TopicPartition,​Long> aState)
        Specified by:
        initState in interface org.apache.heron.api.topology.IStatefulComponent<K,​V>
      • preSave

        public void preSave​(String checkpointId)
        Specified by:
        preSave in interface org.apache.heron.api.topology.IStatefulComponent<K,​V>
      • nextTuple

        public void nextTuple()
        Specified by:
        nextTuple in interface org.apache.heron.api.spout.ISpout
      • activate

        public void activate()
        Specified by:
        activate in interface org.apache.heron.api.spout.ISpout
        Overrides:
        activate in class org.apache.heron.api.spout.BaseRichSpout
      • deactivate

        public void deactivate()
        Specified by:
        deactivate in interface org.apache.heron.api.spout.ISpout
        Overrides:
        deactivate in class org.apache.heron.api.spout.BaseRichSpout
      • ack

        public void ack​(Object msgId)
        Specified by:
        ack in interface org.apache.heron.api.spout.ISpout
        Overrides:
        ack in class org.apache.heron.api.spout.BaseRichSpout
      • fail

        public void fail​(Object msgId)
        Specified by:
        fail in interface org.apache.heron.api.spout.ISpout
        Overrides:
        fail in class org.apache.heron.api.spout.BaseRichSpout
      • close

        public void close()
        Specified by:
        close in interface org.apache.heron.api.spout.ISpout
        Overrides:
        close in class org.apache.heron.api.spout.BaseRichSpout
      • declareOutputFields

        public void declareOutputFields​(org.apache.heron.api.topology.OutputFieldsDeclarer declarer)
        Specified by:
        declareOutputFields in interface org.apache.heron.api.topology.IComponent
      • getComponentConfiguration

        public Map<String,​Object> getComponentConfiguration()
        Specified by:
        getComponentConfiguration in interface org.apache.heron.api.topology.IComponent
        Overrides:
        getComponentConfiguration in class org.apache.heron.api.topology.BaseComponent