Class StreamletImpl<R>

    • Constructor Detail

      • StreamletImpl

        protected StreamletImpl()
        Only used by the implementors
    • Method Detail

      • withStream

        public Streamlet<R> withStream​(String streamId)
        Set the id of the stream to be used by the children nodes. Usage (assuming source is a Streamlet object with two output streams: stream1 and stream2): source.withStream("stream1").filter(...).log(); source.withStream("stream2").filter(...).log();
        Specified by:
        withStream in interface Streamlet<R>
        Parameters:
        streamId - The specified stream id
        Returns:
        Returns back the Streamlet with changed stream id
      • getAvailableStreamIds

        protected Set<String> getAvailableStreamIds()
        Get the available stream ids in the Streamlet. For most Streamlets, there is only one internal stream id, therefore the function returns a set of one single stream id.
        Returns:
        Returns a set of one single stream id.
      • getStreamId

        public String getStreamId()
        Gets the stream id of this Streamlet.
        Specified by:
        getStreamId in interface Streamlet<R>
        Returns:
        the stream id of this Streamlet`
      • map

        public <T> Streamlet<T> map​(SerializableFunction<R,​? extends T> mapFn)
        Return a new Streamlet by applying mapFn to each element of this Streamlet
        Specified by:
        map in interface Streamlet<R>
        Parameters:
        mapFn - The Map Function that should be applied to each element
      • flatMap

        public <T> Streamlet<T> flatMap​(SerializableFunction<R,​? extends Iterable<? extends T>> flatMapFn)
        Return a new Streamlet by applying flatMapFn to each element of this Streamlet and flattening the result
        Specified by:
        flatMap in interface Streamlet<R>
        Parameters:
        flatMapFn - The FlatMap Function that should be applied to each element
      • filter

        public Streamlet<R> filter​(SerializablePredicate<R> filterFn)
        Return a new Streamlet by applying the filterFn on each element of this streamlet and including only those elements that satisfy the filterFn
        Specified by:
        filter in interface Streamlet<R>
        Parameters:
        filterFn - The filter Function that should be applied to each element
      • repartition

        public Streamlet<R> repartition​(int numPartitions)
        Same as filter(Identity).setNumPartitions(nPartitions)
        Specified by:
        repartition in interface Streamlet<R>
      • clone

        public List<Streamlet<R>> clone​(int numClones)
        Clones the current Streamlet. It returns an array of numClones Streamlets where each Streamlet contains all the tuples of the current Streamlet
        Specified by:
        clone in interface Streamlet<R>
        Parameters:
        numClones - The number of clones to clone
      • join

        public <K,​S,​T> KVStreamlet<KeyedWindow<K>,​T> join​(Streamlet<S> otherStreamlet,
                                                                            SerializableFunction<R,​K> thisKeyExtractor,
                                                                            SerializableFunction<S,​K> otherKeyExtractor,
                                                                            WindowConfig windowCfg,
                                                                            SerializableBiFunction<R,​S,​? extends T> joinFunction)
        Return a new Streamlet by inner joining 'this streamlet with ‘other’ streamlet. The join is done over elements accumulated over a time window defined by windowCfg. The elements are compared using the thisKeyExtractor for this streamlet with the otherKeyExtractor for the other streamlet. On each matching pair, the joinFunction is applied.
        Specified by:
        join in interface Streamlet<R>
        Parameters:
        otherStreamlet - The Streamlet that we are joining with.
        thisKeyExtractor - The function applied to a tuple of this streamlet to get the key
        otherKeyExtractor - The function applied to a tuple of the other streamlet to get the key
        windowCfg - This is a specification of what kind of windowing strategy you like to have. Typical windowing strategies are sliding windows and tumbling windows
        joinFunction - The join function that needs to be applied
      • join

        public <K,​S,​T> KVStreamlet<KeyedWindow<K>,​T> join​(Streamlet<S> otherStreamlet,
                                                                            SerializableFunction<R,​K> thisKeyExtractor,
                                                                            SerializableFunction<S,​K> otherKeyExtractor,
                                                                            WindowConfig windowCfg,
                                                                            JoinType joinType,
                                                                            SerializableBiFunction<R,​S,​? extends T> joinFunction)
        Return a new KVStreamlet by joining 'this streamlet with ‘other’ streamlet. The type of joining is declared by the joinType parameter. The join is done over elements accumulated over a time window defined by windowCfg. The elements are compared using the thisKeyExtractor for this streamlet with the otherKeyExtractor for the other streamlet. On each matching pair, the joinFunction is applied. Types of joins JoinType
        Specified by:
        join in interface Streamlet<R>
        Parameters:
        otherStreamlet - The Streamlet that we are joining with.
        thisKeyExtractor - The function applied to a tuple of this streamlet to get the key
        otherKeyExtractor - The function applied to a tuple of the other streamlet to get the key
        windowCfg - This is a specification of what kind of windowing strategy you like to have. Typical windowing strategies are sliding windows and tumbling windows
        joinType - Type of Join. Options JoinType
        joinFunction - The join function that needs to be applied
      • reduceByKey

        public <K,​T> KVStreamlet<K,​T> reduceByKey​(SerializableFunction<R,​K> keyExtractor,
                                                              SerializableFunction<R,​T> valueExtractor,
                                                              SerializableBinaryOperator<T> reduceFn)
        Return a new Streamlet accumulating tuples of this streamlet and applying reduceFn on those tuples.
        Specified by:
        reduceByKey in interface Streamlet<R>
        Parameters:
        keyExtractor - The function applied to a tuple of this streamlet to get the key
        valueExtractor - The function applied to a tuple of this streamlet to extract the value to be reduced on
        reduceFn - The reduce function that you want to apply to all the values of a key.
      • reduceByKey

        public <K,​T> KVStreamlet<K,​T> reduceByKey​(SerializableFunction<R,​K> keyExtractor,
                                                              T identity,
                                                              SerializableBiFunction<T,​R,​? extends T> reduceFn)
        Return a new Streamlet accumulating tuples of this streamlet and applying reduceFn on those tuples.
        Specified by:
        reduceByKey in interface Streamlet<R>
        Parameters:
        keyExtractor - The function applied to a tuple of this streamlet to get the key
        identity - The identity element is the initial value for each key
        reduceFn - The reduce function that you want to apply to all the values of a key.
      • reduceByKeyAndWindow

        public <K,​T> KVStreamlet<KeyedWindow<K>,​T> reduceByKeyAndWindow​(SerializableFunction<R,​K> keyExtractor,
                                                                                    SerializableFunction<R,​T> valueExtractor,
                                                                                    WindowConfig windowCfg,
                                                                                    SerializableBinaryOperator<T> reduceFn)
        Return a new Streamlet accumulating tuples of this streamlet over a Window defined by windowCfg and applying reduceFn on those tuples.
        Specified by:
        reduceByKeyAndWindow in interface Streamlet<R>
        Parameters:
        keyExtractor - The function applied to a tuple of this streamlet to get the key
        valueExtractor - The function applied to a tuple of this streamlet to extract the value to be reduced on
        windowCfg - This is a specification of what kind of windowing strategy you like to have. Typical windowing strategies are sliding windows and tumbling windows
        reduceFn - The reduce function that you want to apply to all the values of a key.
      • reduceByKeyAndWindow

        public <K,​T> KVStreamlet<KeyedWindow<K>,​T> reduceByKeyAndWindow​(SerializableFunction<R,​K> keyExtractor,
                                                                                    WindowConfig windowCfg,
                                                                                    T identity,
                                                                                    SerializableBiFunction<T,​R,​? extends T> reduceFn)
        Return a new Streamlet accumulating tuples of this streamlet over a Window defined by windowCfg and applying reduceFn on those tuples. For each window, the value identity is used as a initial value. All the matching tuples are reduced using reduceFn starting from this initial value.
        Specified by:
        reduceByKeyAndWindow in interface Streamlet<R>
        Parameters:
        keyExtractor - The function applied to a tuple of this streamlet to get the key
        windowCfg - This is a specification of what kind of windowing strategy you like to have. Typical windowing strategies are sliding windows and tumbling windows
        identity - The identity element is both the initial value inside the reduction window and the default result if there are no elements in the window
        reduceFn - The reduce function takes two parameters: a partial result of the reduction and the next element of the stream. It returns a new partial result.
      • union

        public Streamlet<R> union​(Streamlet<? extends R> otherStreamlet)
        Returns a new Streamlet that is the union of this and the ‘other’ streamlet. Essentially the new streamlet will contain tuples belonging to both Streamlets
        Specified by:
        union in interface Streamlet<R>
      • log

        public StreamletBase<R> log()
        Logs every element of the streamlet using String.valueOf function Note that LogStreamlet is an empty streamlet. That is its a streamlet that does not contain any tuple. Thus this function returns void.
        Specified by:
        log in interface Streamlet<R>
      • consume

        public StreamletBase<R> consume​(SerializableConsumer<R> consumer)
        Applies the consumer function for every element of this streamlet
        Specified by:
        consume in interface Streamlet<R>
        Parameters:
        consumer - The user supplied consumer function that is invoked for each element
      • toSink

        public StreamletBase<R> toSink​(Sink<R> sink)
        Uses the sink to consume every element of this streamlet
        Specified by:
        toSink in interface Streamlet<R>
        Parameters:
        sink - The Sink that consumes
      • transform

        public <T> Streamlet<T> transform​(SerializableTransformer<R,​? extends T> serializableTransformer)
        Returns a new Streamlet by applying the transformFunction on each element of this streamlet. Before starting to cycle the transformFunction over the Streamlet, the open function is called. This allows the transform Function to do any kind of initialization/loading, etc.
        Specified by:
        transform in interface Streamlet<R>
        Type Parameters:
        T - The return type of the transform
        Parameters:
        serializableTransformer - The transformation function to be applied
        Returns:
        Streamlet containing the output of the transformFunction
      • applyOperator

        public <T> Streamlet<T> applyOperator​(IStreamletOperator<R,​T> operator)
        Returns a new Streamlet by applying the operator on each element of this streamlet.
        Specified by:
        applyOperator in interface Streamlet<R>
        Type Parameters:
        T - The return type of the transform
        Parameters:
        operator - The operator to be applied
        Returns:
        Streamlet containing the output of the operation
      • applyOperator

        public <T> Streamlet<T> applyOperator​(IStreamletOperator<R,​T> operator,
                                              StreamGrouping grouper)
        Returns a new Streamlet by applying the operator on each element of this streamlet.
        Specified by:
        applyOperator in interface Streamlet<R>
        Type Parameters:
        T - The return type of the transform
        Parameters:
        operator - The operator to be applied
        grouper - The grouper to be applied with the operator
        Returns:
        Streamlet containing the output of the operation
      • split

        public Streamlet<R> split​(Map<String,​SerializablePredicate<R>> splitFns)
        Returns multiple streams by splitting incoming stream.
        Specified by:
        split in interface Streamlet<R>
        Parameters:
        splitFns - The Split Functions that test if the tuple should be emitted into each stream Note that there could be 0 or multiple target stream ids
      • keyBy

        public <K> KVStreamlet<K,​R> keyBy​(SerializableFunction<R,​K> keyExtractor)
        Return a new KVStreamlet by applying key extractor to each element of this Streamlet
        Specified by:
        keyBy in interface Streamlet<R>
        Parameters:
        keyExtractor - The function applied to a tuple of this streamlet to get the key
      • keyBy

        public <K,​V> KVStreamlet<K,​V> keyBy​(SerializableFunction<R,​K> keyExtractor,
                                                        SerializableFunction<R,​V> valueExtractor)
        Return a new KVStreamlet by applying key and value extractor to each element of this Streamlet
        Specified by:
        keyBy in interface Streamlet<R>
        Parameters:
        keyExtractor - The function applied to a tuple of this streamlet to get the key
        valueExtractor - The function applied to a tuple of this streamlet to extract the value
      • countByKey

        public <K> KVStreamlet<K,​Long> countByKey​(SerializableFunction<R,​K> keyExtractor)
        Returns a new stream of by counting tuples in this stream on each key.
        Specified by:
        countByKey in interface Streamlet<R>
        Parameters:
        keyExtractor - The function applied to a tuple of this streamlet to get the key
      • countByKeyAndWindow

        public <K> KVStreamlet<KeyedWindow<K>,​Long> countByKeyAndWindow​(SerializableFunction<R,​K> keyExtractor,
                                                                              WindowConfig windowCfg)
        Returns a new stream of by counting tuples over a window in this stream on each key.
        Specified by:
        countByKeyAndWindow in interface Streamlet<R>
        Parameters:
        keyExtractor - The function applied to a tuple of this streamlet to get the key
        windowCfg - This is a specification of what kind of windowing strategy you like to have. Typical windowing strategies are sliding windows and tumbling windows Note that there could be 0 or multiple target stream ids