Class StreamletBaseImpl<R>

  • All Implemented Interfaces:
    StreamletBase<R>
    Direct Known Subclasses:
    ConsumerStreamlet, LogStreamlet, SinkStreamlet, StreamletImpl

    public abstract class StreamletBaseImpl<R>
    extends Object
    implements StreamletBase<R>
    A Streamlet is a (potentially unbounded) ordered collection of tuples. Streamlets originate from pub/sub systems(such Pulsar/Kafka), or from static data(such as csv files, HDFS files), or for that matter any other source. They are also created by transforming existing Streamlets using operations such as map/flatMap, etc. Besides the tuples, a Streamlet has the following properties associated with it a) name. User assigned or system generated name to refer the streamlet b) nPartitions. Number of partitions that the streamlet is composed of. Thus the ordering of the tuples in a Streamlet is wrt the tuples within a partition. This allows the system to distribute each partition to different nodes across the cluster. A bunch of transformations can be done on Streamlets(like map/flatMap, etc.). Each of these transformations operate on every tuple of the Streamlet and produce a new Streamlet. One can think of a transformation attaching itself to the stream and processing each tuple as they go by. Thus the parallelism of any operator is implicitly determined by the number of partitions of the stream that it is operating on. If a particular transformation wants to operate at a different parallelism, one can repartition the Streamlet before doing the transformation.
    • Field Detail

      • nPartitions

        protected int nPartitions
    • Constructor Detail

      • StreamletBaseImpl

        protected StreamletBaseImpl()
        Only used by the implementors
    • Method Detail

      • setName

        public StreamletBase<R> setName​(String sName)
        Sets the name of the Streamlet.
        Specified by:
        setName in interface StreamletBase<R>
        Parameters:
        sName - The name given by the user for this streamlet
        Returns:
        Returns back the Streamlet with changed name
      • getName

        public String getName()
        Gets the name of the Streamlet.
        Specified by:
        getName in interface StreamletBase<R>
        Returns:
        Returns the name of the Streamlet
      • setDefaultNameIfNone

        protected void setDefaultNameIfNone​(StreamletBaseImpl.StreamletNamePrefix prefix,
                                            Set<String> stageNames)
        Sets a default unique name to the Streamlet by type if it is not set. Otherwise, just checks its uniqueness.
        Parameters:
        prefix - The name prefix of this streamlet
        stageNames - The collections of created streamlet/stage names
      • setNumPartitions

        public StreamletBase<R> setNumPartitions​(int numPartitions)
        Sets the number of partitions of the streamlet
        Specified by:
        setNumPartitions in interface StreamletBase<R>
        Parameters:
        numPartitions - The user assigned number of partitions
        Returns:
        Returns back the Streamlet with changed number of partitions
      • getNumPartitions

        public int getNumPartitions()
        Gets the number of partitions of this Streamlet.
        Specified by:
        getNumPartitions in interface StreamletBase<R>
        Returns:
        the number of partitions of this Streamlet
      • getChildren

        public List<StreamletBaseImpl<?>> getChildren()
        Gets all the children of this streamlet. Children of a streamlet are streamlets that are resulting from transformations of elements of this and potentially other streamlets.
        Returns:
        The kid streamlets
      • isBuilt

        public boolean isBuilt()
      • isFullyBuilt

        public boolean isFullyBuilt()