Class Config

  • All Implemented Interfaces:
    Serializable, Cloneable, Map<String,​Object>

    public class Config
    extends HashMap<String,​Object>
    Topology configs are specified as a plain old map. This class provides a convenient way to create a topology config map by providing setter methods for all the configs that can be set. It also makes it easier to do things like add serializations.

    This class also provides constants for all the configurations possible on a Storm cluster and Storm topology. Default values for these configs can be found in defaults.yaml.

    Note that you may put other configurations in any of the configs. Storm will ignore anything it doesn't recognize, but your topologies are free to make use of them by reading them in the prepare method of Bolts or the open method of Spouts. .

    See Also:
    Serialized Form
    • Field Detail

      • TOPOLOGY_ENABLE_MESSAGE_TIMEOUTS

        public static final String TOPOLOGY_ENABLE_MESSAGE_TIMEOUTS
        True if Storm should timeout messages or not. Defaults to true. This is meant to be used in unit tests to prevent tuples from being accidentally timed out during the test. Same functionality in Heron.
        See Also:
        Constant Field Values
      • TOPOLOGY_OPTIMIZE

        public static final String TOPOLOGY_OPTIMIZE
        Whether or not the master should optimize topologies by running multiple tasks in a single thread where appropriate. This has no impact in Heron
        See Also:
        Constant Field Values
      • TOPOLOGY_TASKS

        public static final String TOPOLOGY_TASKS
        How many instances to create for a spout/bolt. A task runs on a thread with zero or more other tasks for the same spout/bolt. The number of tasks for a spout/bolt is always the same throughout the lifetime of a topology, but the number of executors (threads) for a spout/bolt can change over time. This allows a topology to scale to more or less resources without redeploying the topology or violating the constraints of Storm (such as a fields grouping guaranteeing that the same value goes to the same task). This has no impact on Heron.
        See Also:
        Constant Field Values
      • TOPOLOGY_KRYO_REGISTER

        public static final String TOPOLOGY_KRYO_REGISTER
        A list of serialization registrations for Kryo ( http://code.google.com/p/kryo/ ), the underlying serialization framework for Storm. A serialization can either be the name of a class (in which case Kryo will automatically create a serializer for the class that saves all the object's fields), or an implementation of com.esotericsoftware.kryo.Serializer.

        See Kryo's documentation for more information about writing custom serializers. Same in Heron.

        See Also:
        Constant Field Values
      • TOPOLOGY_KRYO_DECORATORS

        public static final String TOPOLOGY_KRYO_DECORATORS
        A list of classes that customize storm's kryo instance during start-up. Each listed class name must implement IKryoDecorator. During start-up the listed class is instantiated with 0 arguments, then its 'decorate' method is called with storm's kryo instance as the only argument. Same in Heron.
        See Also:
        Constant Field Values
      • TOPOLOGY_KRYO_FACTORY

        public static final String TOPOLOGY_KRYO_FACTORY
        Class that specifies how to create a Kryo instance for serialization. Storm will then apply topology.kryo.register and topology.kryo.decorators on top of this. The default implementation implements topology.fall.back.on.java.serialization and turns references off. Same in Heron.
        See Also:
        Constant Field Values
      • TOPOLOGY_SKIP_MISSING_KRYO_REGISTRATIONS

        public static final String TOPOLOGY_SKIP_MISSING_KRYO_REGISTRATIONS
        Whether or not Storm should skip the loading of kryo registrations for which it does not know the class or have the serializer implementation. Otherwise, the task will fail to load and will throw an error at runtime. The use case of this is if you want to declare your serializations on the storm.yaml files on the cluster rather than every single time you submit a topology. Different applications may use different serializations and so a single application may not have the code for the other serializers used by other apps. By setting this config to true, Storm will ignore that it doesn't have those other serializations rather than throw an error. Same in Heron.
        See Also:
        Constant Field Values
      • TOPOLOGY_STATE_SYNCHRONIZATION_TIMEOUT_SECS

        public static final String TOPOLOGY_STATE_SYNCHRONIZATION_TIMEOUT_SECS
        The maximum amount of time a component gives a source of state to synchronize before it requests synchronization again. This is not implemented in Heron.
        See Also:
        Constant Field Values
      • TOPOLOGY_FALL_BACK_ON_JAVA_SERIALIZATION

        public static final String TOPOLOGY_FALL_BACK_ON_JAVA_SERIALIZATION
        Whether or not to use Java serialization in a topology. This has the same meaning in Heron.
        See Also:
        Constant Field Values
      • TOPOLOGY_WORKER_CHILDOPTS

        public static final String TOPOLOGY_WORKER_CHILDOPTS
        Topology-specific options for the worker child process. This is used in addition to WORKER_CHILDOPTS. Not yet implemented in Heron.
        See Also:
        Constant Field Values
      • TOPOLOGY_TRANSACTIONAL_ID

        public static final String TOPOLOGY_TRANSACTIONAL_ID
        This config is available for TransactionalSpouts, and contains the id ( a String) for the transactional topology. This id is used to store the state of the transactional topology in Zookeeper. This is not implemented in Heron.
        See Also:
        Constant Field Values
      • TOPOLOGY_TICK_TUPLE_FREQ_SECS

        public static final String TOPOLOGY_TICK_TUPLE_FREQ_SECS
        How often a tick tuple from the "__system" component and "__tick" stream should be sent to tasks. Meant to be used as a component-specific configuration. Same meaning in Heron.
        See Also:
        Constant Field Values
      • TOPOLOGY_ERROR_THROTTLE_INTERVAL_SECS

        public static final String TOPOLOGY_ERROR_THROTTLE_INTERVAL_SECS
        The interval in seconds to use for determining whether to throttle error reported to Zookeeper. For example, an interval of 10 seconds with topology.max.error.report.per.interval set to 5 will only allow 5 errors to be reported to Zookeeper per task for every 10 second interval of time. This is not supported in Heron.
        See Also:
        Constant Field Values
      • TOPOLOGY_MAX_ERROR_REPORT_PER_INTERVAL

        public static final String TOPOLOGY_MAX_ERROR_REPORT_PER_INTERVAL
        See doc for TOPOLOGY_ERROR_THROTTLE_INTERVAL_SECS. This is not supported in Heron
        See Also:
        Constant Field Values
      • TOPOLOGY_DEBUG

        public static final String TOPOLOGY_DEBUG
        When set to true, Storm will log every message that's emitted.
        See Also:
        Constant Field Values
      • TOPOLOGY_WORKERS

        public static final String TOPOLOGY_WORKERS
        This currently gets translated to TOPOLOGY_STMGRS. Please see the documentation for TOPOLOGY_STMGRS.
        See Also:
        Constant Field Values
      • TOPOLOGY_ACKER_EXECUTORS

        public static final String TOPOLOGY_ACKER_EXECUTORS
        How many executors to spawn for ackers.

        If this is set to 0, then Storm will immediately ack tuples as soon as they come off the spout, effectively disabling reliability. In Heron any values of > 0 means to enable acking.

        See Also:
        Constant Field Values
      • TOPOLOGY_MESSAGE_TIMEOUT_SECS

        public static final String TOPOLOGY_MESSAGE_TIMEOUT_SECS
        The maximum amount of time given to the topology to fully process a message emitted by a spout. If the message is not acked within this time frame, Storm will fail the message on the spout. Some spouts implementations will then replay the message at a later time. This has the same meaning in Heron.
        See Also:
        Constant Field Values
      • TOPOLOGY_MAX_TASK_PARALLELISM

        public static final String TOPOLOGY_MAX_TASK_PARALLELISM
        The maximum parallelism allowed for a component in this topology. This configuration is typically used in testing to limit the number of threads spawned in simulator. This does not have any impact in Heron
        See Also:
        Constant Field Values
      • TOPOLOGY_MAX_SPOUT_PENDING

        public static final String TOPOLOGY_MAX_SPOUT_PENDING
        The maximum number of tuples that can be pending on a spout task at any given time. This config applies to individual tasks, not to spouts or topologies as a whole.

        A pending tuple is one that has been emitted from a spout but has not been acked or failed yet. Note that this config parameter has no effect for unreliable spouts that don't tag their tuples with a message id. This has same meaning in Heron.

        See Also:
        Constant Field Values
      • TOPOLOGY_SPOUT_WAIT_STRATEGY

        public static final String TOPOLOGY_SPOUT_WAIT_STRATEGY
        A class that implements a strategy for what to do when a spout needs to wait. Waiting is triggered in one of two conditions:

        1. nextTuple emits no tuples 2. The spout has hit maxSpoutPending and can't emit any more tuples This is not yet implemented in Heron.

        See Also:
        Constant Field Values
      • TOPOLOGY_SLEEP_SPOUT_WAIT_STRATEGY_TIME_MS

        public static final String TOPOLOGY_SLEEP_SPOUT_WAIT_STRATEGY_TIME_MS
        The amount of milliseconds the SleepEmptyEmitStrategy should sleep for. This is not yet implemented in Heron
        See Also:
        Constant Field Values
      • TOPOLOGY_STATS_SAMPLE_RATE

        public static final String TOPOLOGY_STATS_SAMPLE_RATE
        The percentage of tuples to sample to produce stats for a task. This is not implemented in Heron.
        See Also:
        Constant Field Values
      • TOPOLOGY_AUTO_TASK_HOOKS

        public static final String TOPOLOGY_AUTO_TASK_HOOKS
        A list of task hooks that are automatically added to every spout and bolt in the topology. An example of when you'd do this is to add a hook that integrates with your internal monitoring system. These hooks are instantiated using the zero-arg constructor.
        See Also:
        Constant Field Values
      • TOPOLOGY_NAME

        public static final String TOPOLOGY_NAME
        Name of the topology. This config is automatically set by Storm when the topology is submitted. Same in Heron
        See Also:
        Constant Field Values
      • TOPOLOGY_TEAM_NAME

        public static final String TOPOLOGY_TEAM_NAME
        Name of the team which owns this topology. Same in Heron
        See Also:
        Constant Field Values
      • TOPOLOGY_TEAM_EMAIL

        public static final String TOPOLOGY_TEAM_EMAIL
        Email of the team which owns this topology. Same in Heron
        See Also:
        Constant Field Values
      • TOPOLOGY_CAP_TICKET

        public static final String TOPOLOGY_CAP_TICKET
        Cap ticket (if filed) for the topology. If the topology is in prod this has to be set or it cannot be deployed. Same in Heron
        See Also:
        Constant Field Values
      • TOPOLOGY_PROJECT_NAME

        public static final String TOPOLOGY_PROJECT_NAME
        Project name of the topology, to help us with tagging which topologies are part of which project. For example, if topology A and Topology B are part of the same project, we will like to aggregate them as part of the same project. This is required by Cap team. Same in Heron
        See Also:
        Constant Field Values
      • STORM_ZOOKEEPER_SERVERS

        public static final String STORM_ZOOKEEPER_SERVERS
        A list of hosts of ZooKeeper servers used to manage the cluster.
        See Also:
        Constant Field Values
      • STORM_ZOOKEEPER_PORT

        public static final String STORM_ZOOKEEPER_PORT
        The port Storm will use to connect to each of the ZooKeeper servers.
        See Also:
        Constant Field Values
      • TRANSACTIONAL_ZOOKEEPER_ROOT

        public static final String TRANSACTIONAL_ZOOKEEPER_ROOT
        The root directory in ZooKeeper for metadata about TransactionalSpouts.
        See Also:
        Constant Field Values
      • STORM_ZOOKEEPER_SESSION_TIMEOUT

        public static final String STORM_ZOOKEEPER_SESSION_TIMEOUT
        The session timeout for clients to ZooKeeper.
        See Also:
        Constant Field Values
      • STORM_ZOOKEEPER_CONNECTION_TIMEOUT

        public static final String STORM_ZOOKEEPER_CONNECTION_TIMEOUT
        The connection timeout for clients to ZooKeeper.
        See Also:
        Constant Field Values
      • STORM_ZOOKEEPER_RETRY_TIMES

        public static final String STORM_ZOOKEEPER_RETRY_TIMES
        The number of times to retry a Zookeeper operation.
        See Also:
        Constant Field Values
      • STORM_ZOOKEEPER_RETRY_INTERVAL

        public static final String STORM_ZOOKEEPER_RETRY_INTERVAL
        The interval between retries of a Zookeeper operation.
        See Also:
        Constant Field Values
      • TRANSACTIONAL_ZOOKEEPER_SERVERS

        public static final String TRANSACTIONAL_ZOOKEEPER_SERVERS
        The list of zookeeper servers in which to keep the transactional state. If null (which is default), will use storm.zookeeper.servers
        See Also:
        Constant Field Values
      • TRANSACTIONAL_ZOOKEEPER_PORT

        public static final String TRANSACTIONAL_ZOOKEEPER_PORT
        The port to use to connect to the transactional zookeeper servers. If null (which is default), will use storm.zookeeper.port
        See Also:
        Constant Field Values
      • TOPOLOGY_BOLTS_WINDOW_LENGTH_DURATION_MS

        public static final String TOPOLOGY_BOLTS_WINDOW_LENGTH_DURATION_MS
        See Also:
        Constant Field Values
      • TOPOLOGY_BOLTS_SLIDING_INTERVAL_COUNT

        public static final String TOPOLOGY_BOLTS_SLIDING_INTERVAL_COUNT
        See Also:
        Constant Field Values
      • TOPOLOGY_BOLTS_SLIDING_INTERVAL_DURATION_MS

        public static final String TOPOLOGY_BOLTS_SLIDING_INTERVAL_DURATION_MS
        See Also:
        Constant Field Values
      • TOPOLOGY_BOLTS_LATE_TUPLE_STREAM

        public static final String TOPOLOGY_BOLTS_LATE_TUPLE_STREAM
        Bolt-specific configuration for windowed bolts to specify the name of the stream on which late tuples are going to be emitted. This configuration should only be used from the BaseWindowedBolt.withLateTupleStream builder method, and not as global parameter, otherwise IllegalArgumentException is going to be thrown.
        See Also:
        Constant Field Values
      • TOPOLOGY_BOLTS_TUPLE_TIMESTAMP_MAX_LAG_MS

        public static final String TOPOLOGY_BOLTS_TUPLE_TIMESTAMP_MAX_LAG_MS
        Bolt-specific configuration for windowed bolts to specify the maximum time lag of the tuple timestamp in milliseconds. It means that the tuple timestamps cannot be out of order by more than this amount. This config will be effective only if TimestampExtractor is specified.
        See Also:
        Constant Field Values
      • TOPOLOGY_BOLTS_WATERMARK_EVENT_INTERVAL_MS

        public static final String TOPOLOGY_BOLTS_WATERMARK_EVENT_INTERVAL_MS
        See Also:
        Constant Field Values
      • TOPOLOGY_ENVIRONMENT

        public static final String TOPOLOGY_ENVIRONMENT
        Topology-specific environment variables for the worker child process. This is added to the existing environment (that of the supervisor)
        See Also:
        Constant Field Values
      • STORMCOMPAT_TOPOLOGY_AUTO_TASK_HOOKS

        public static final String STORMCOMPAT_TOPOLOGY_AUTO_TASK_HOOKS
        ---- DO NOT USE ----- This variable is used to rewrite the TOPOLOGY_AUTO_TASK_HOOKS variable. As such this is a strictly internal config variable that is not exposed the user
        See Also:
        Constant Field Values
    • Constructor Detail

      • Config

        public Config()
    • Method Detail

      • setDebug

        public static void setDebug​(Map conf,
                                    boolean isOn)
      • setTeamName

        public static void setTeamName​(Map conf,
                                       String teamName)
      • setTeamEmail

        public static void setTeamEmail​(Map conf,
                                        String teamEmail)
      • setTopologyCapTicket

        public static void setTopologyCapTicket​(Map conf,
                                                String ticket)
      • setTopologyProjectName

        public static void setTopologyProjectName​(Map conf,
                                                  String project)
      • setNumWorkers

        public static void setNumWorkers​(Map conf,
                                         int workers)
      • setNumAckers

        public static void setNumAckers​(Map conf,
                                        int numExecutors)
      • setMessageTimeoutSecs

        public static void setMessageTimeoutSecs​(Map conf,
                                                 int secs)
      • registerSerialization

        public static void registerSerialization​(Map conf,
                                                 Class klass)
      • registerSerialization

        public static void registerSerialization​(Map conf,
                                                 Class klass,
                                                 Class<? extends com.esotericsoftware.kryo.Serializer> serializerClass)
      • registerDecorator

        public static void registerDecorator​(Map conf,
                                             Class<? extends IKryoDecorator> klass)
      • setKryoFactory

        public static void setKryoFactory​(Map conf,
                                          Class<? extends IKryoFactory> klass)
      • setSkipMissingKryoRegistrations

        public static void setSkipMissingKryoRegistrations​(Map conf,
                                                           boolean skip)
      • setMaxTaskParallelism

        public static void setMaxTaskParallelism​(Map conf,
                                                 int max)
      • setMaxSpoutPending

        public static void setMaxSpoutPending​(Map conf,
                                              int max)
      • setStatsSampleRate

        public static void setStatsSampleRate​(Map conf,
                                              double rate)
      • setFallBackOnJavaSerialization

        public static void setFallBackOnJavaSerialization​(Map conf,
                                                          boolean fallback)
      • setEnvironment

        public static void setEnvironment​(Map<String,​Object> conf,
                                          Map env)
      • setDebug

        public void setDebug​(boolean isOn)
      • setTeamName

        public void setTeamName​(String teamName)
      • setTeamEmail

        public void setTeamEmail​(String teamEmail)
      • setTopologyCapTicket

        public void setTopologyCapTicket​(String ticket)
      • setTopologyProjectName

        public void setTopologyProjectName​(String project)
      • setOptimize

        @Deprecated
        public void setOptimize​(boolean isOn)
        Deprecated.
        we don't have optimization
        Set topology optimization
        Parameters:
        isOn -
      • setNumWorkers

        public void setNumWorkers​(int workers)
      • setNumAckers

        public void setNumAckers​(int numExecutors)
      • setMessageTimeoutSecs

        public void setMessageTimeoutSecs​(int secs)
      • registerSerialization

        public void registerSerialization​(Class klass)
      • registerSerialization

        public void registerSerialization​(Class klass,
                                          Class<? extends com.esotericsoftware.kryo.Serializer> serializerClass)
      • registerMetricsConsumer

        public void registerMetricsConsumer​(Class klass,
                                            Object argument,
                                            long parallelismHint)
      • registerMetricsConsumer

        public void registerMetricsConsumer​(Class klass,
                                            long parallelismHint)
      • registerMetricsConsumer

        public void registerMetricsConsumer​(Class klass)
      • setKryoFactory

        public void setKryoFactory​(Class<? extends IKryoFactory> klass)
      • setSkipMissingKryoRegistrations

        public void setSkipMissingKryoRegistrations​(boolean skip)
      • setMaxTaskParallelism

        public void setMaxTaskParallelism​(int max)
      • setMaxSpoutPending

        public void setMaxSpoutPending​(int max)
      • setStatsSampleRate

        public void setStatsSampleRate​(double rate)
      • setFallBackOnJavaSerialization

        public void setFallBackOnJavaSerialization​(boolean fallback)