Package org.apache.heron.streamlet.impl
Class StreamletBaseImpl<R>
- java.lang.Object
-
- org.apache.heron.streamlet.impl.StreamletBaseImpl<R>
-
- All Implemented Interfaces:
StreamletBase<R>
- Direct Known Subclasses:
ConsumerStreamlet
,LogStreamlet
,SinkStreamlet
,StreamletImpl
public abstract class StreamletBaseImpl<R> extends Object implements StreamletBase<R>
A Streamlet is a (potentially unbounded) ordered collection of tuples. Streamlets originate from pub/sub systems(such Pulsar/Kafka), or from static data(such as csv files, HDFS files), or for that matter any other source. They are also created by transforming existing Streamlets using operations such as map/flatMap, etc. Besides the tuples, a Streamlet has the following properties associated with it a) name. User assigned or system generated name to refer the streamlet b) nPartitions. Number of partitions that the streamlet is composed of. Thus the ordering of the tuples in a Streamlet is wrt the tuples within a partition. This allows the system to distribute each partition to different nodes across the cluster. A bunch of transformations can be done on Streamlets(like map/flatMap, etc.). Each of these transformations operate on every tuple of the Streamlet and produce a new Streamlet. One can think of a transformation attaching itself to the stream and processing each tuple as they go by. Thus the parallelism of any operator is implicitly determined by the number of partitions of the stream that it is operating on. If a particular transformation wants to operate at a different parallelism, one can repartition the Streamlet before doing the transformation.
-
-
Nested Class Summary
Nested Classes Modifier and Type Class Description protected static class
StreamletBaseImpl.StreamletNamePrefix
-
Field Summary
Fields Modifier and Type Field Description protected String
name
protected int
nPartitions
-
Constructor Summary
Constructors Modifier Constructor Description protected
StreamletBaseImpl()
Only used by the implementors
-
Method Summary
All Methods Instance Methods Abstract Methods Concrete Methods Modifier and Type Method Description <T> void
addChild(StreamletBaseImpl<T> child)
void
build(TopologyBuilder bldr, Set<String> stageNames)
protected abstract boolean
doBuild(TopologyBuilder bldr, Set<String> stageNames)
List<StreamletBaseImpl<?>>
getChildren()
Gets all the children of this streamlet.String
getName()
Gets the name of the Streamlet.int
getNumPartitions()
Gets the number of partitions of this Streamlet.boolean
isBuilt()
boolean
isFullyBuilt()
protected void
setDefaultNameIfNone(StreamletBaseImpl.StreamletNamePrefix prefix, Set<String> stageNames)
Sets a default unique name to the Streamlet by type if it is not set.StreamletBase<R>
setName(String sName)
Sets the name of the Streamlet.StreamletBase<R>
setNumPartitions(int numPartitions)
Sets the number of partitions of the streamlet
-
-
-
Field Detail
-
name
protected String name
-
nPartitions
protected int nPartitions
-
-
Method Detail
-
setName
public StreamletBase<R> setName(String sName)
Sets the name of the Streamlet.- Specified by:
setName
in interfaceStreamletBase<R>
- Parameters:
sName
- The name given by the user for this streamlet- Returns:
- Returns back the Streamlet with changed name
-
getName
public String getName()
Gets the name of the Streamlet.- Specified by:
getName
in interfaceStreamletBase<R>
- Returns:
- Returns the name of the Streamlet
-
setDefaultNameIfNone
protected void setDefaultNameIfNone(StreamletBaseImpl.StreamletNamePrefix prefix, Set<String> stageNames)
Sets a default unique name to the Streamlet by type if it is not set. Otherwise, just checks its uniqueness.- Parameters:
prefix
- The name prefix of this streamletstageNames
- The collections of created streamlet/stage names
-
setNumPartitions
public StreamletBase<R> setNumPartitions(int numPartitions)
Sets the number of partitions of the streamlet- Specified by:
setNumPartitions
in interfaceStreamletBase<R>
- Parameters:
numPartitions
- The user assigned number of partitions- Returns:
- Returns back the Streamlet with changed number of partitions
-
getNumPartitions
public int getNumPartitions()
Gets the number of partitions of this Streamlet.- Specified by:
getNumPartitions
in interfaceStreamletBase<R>
- Returns:
- the number of partitions of this Streamlet
-
addChild
public <T> void addChild(StreamletBaseImpl<T> child)
-
getChildren
public List<StreamletBaseImpl<?>> getChildren()
Gets all the children of this streamlet. Children of a streamlet are streamlets that are resulting from transformations of elements of this and potentially other streamlets.- Returns:
- The kid streamlets
-
build
public void build(TopologyBuilder bldr, Set<String> stageNames)
-
isBuilt
public boolean isBuilt()
-
isFullyBuilt
public boolean isFullyBuilt()
-
doBuild
protected abstract boolean doBuild(TopologyBuilder bldr, Set<String> stageNames)
-
-