Atlas Stream Processing Monitoring and Alerting
Atlas Stream Processing provides monitoring and alerts so that users can leverage performance and status insights to refine their workflows.
Stream Processor Monitoring Methods
Atlas Stream Processing provides the following methods for on-demand reporting about your stream processors:
The sp.processor.sample() method allows you to see a small sample of
the documents output by a currently running stream processor of your
choosing. Users can compare the sampled results against their expected
results to diagnose any errors in their aggregation pipeline design.
The sp.processor.stats() method returns a variety of runtime
statistics about a stream processor of your choosing, including, but
not limited to:
Number of messages ingested
Number of messages successfully processed
Number of messages sent to your Dead Letter Queue
In-memory size of your pipeline state
Pipeline definition
Stream Processor Alerts
Atlas Stream Processing triggers alerts when processors change state, or a processor meets various ingestion or output thresholds. For a list of available Atlas Stream Processing alerts, see Atlas Stream Processing Alerts. To learn more about alert configuration, see Configure Alert Settings.
You can target Atlas Stream Processing alerts in the following ways:
All stream processors within a project
All stream processors within a stream processing instance matching the configured predicate
All stream processors the names of which match the configured predicate
For targets other than all stream processors, you can configure multiple targets for the same alert.