Stream Processor Logs
To help monitor the stream processor queues capacity, new WARN level logs are added for each stream processor. These warning logs will start to appear in Application Logs if any individual queue is at or more than 70% capacity. For value stream, datatable and stream processors, while the queue is full to its capacity, any attempt to add entry to the queues will be rejected. For Persistent Property Processor, it will retry (with Producer Timeout) to add entry to the queue and after the retry timeout, if the entry could not be added, the entry will be rejected. The rejected entries are counted against the new totalWritesRejected metrics mentioned above.
Below is an example of these logs statements.
PersistentPropertyProcessor
2022-02-09 02:56:15.465+0000 [L: WARN] [O: c.t.p.p.StreamEntryProcessor] [I: ] [U: ] [S: ] [P: ] [T: Timer-6] PersistentPropertyProcessor queue #9 is currently 82.0 percent full that is at or above 70.0 percent threshold
82022-02-09 02:56:15.465+0000 [L: WARN] [O: c.t.p.p.StreamEntryProcessor] [I: ] [U: ] [S: ] [P: ] [T: Timer-6] PersistentPropertyProcessor queue #11 is currently 82.0 percent full that is at or above 70.0 percent threshold
92022-02-09 02:56:15.465+0000 [L: WARN] [O: c.t.p.p.StreamEntryProcessor] [I: ] [U: ] [S: ] [P: ] [T: Timer-6] PersistentPropertyProcessor queue #13 is currently 81.0 percent full that is at or above 70.0 percent threshold
102022-02-09 02:56:15.465+0000 [L: WARN] [O: c.t.p.p.StreamEntryProcessor] [I: ] [U: ] [S: ] [P: ] [T: Timer-6] PersistentPropertyProcessor queue #14 is currently 82.0 percent full that is at or above 70.0 percent threshold
112022-02-09 02:56:15.465+0000 [L: WARN] [O: c.t.p.p.StreamEntryProcessor] [I: ] [U: ] [S: ] [P: ] [T: Timer-6] PersistentPropertyProcessor queue #16 is currently 79.0 percent full that is at or above 70.0 percent threshold
BatchValueStreamProcessor
platform1_1 | 2022-02-22 21:09:08.208+0000 [L: WARN] [O: c.t.p.p.StreamEntryProcessor] [I: ] [U: ] [S: ] [P: platform1] [T: Timer-9] BatchValueStreamProcessor queue #81 is currently 100.0 percent full that is at or above 70.0 percent threshold
platform1_1 | 2022-02-22 21:09:08.208+0000 [L: WARN] [O: c.t.p.p.StreamEntryProcessor] [I: ] [U: ] [S: ] [P: platform1] [T: Timer-9] BatchValueStreamProcessor queue #82 is currently 100.0 percent full that is at or above 70.0 percent threshold
platform1_1 | 2022-02-22 21:09:08.208+0000 [L: WARN] [O: c.t.p.p.StreamEntryProcessor] [I: ] [U: ] [S: ] [P: platform1] [T: Timer-9] BatchValueStreamProcessor queue #83 is currently 100.0 percent full that is at or above 70.0 percent threshold
platform1_1 | 2022-02-22 21:09:08.208+0000 [L: WARN] [O: c.t.p.p.StreamEntryProcessor] [I: ] [U: ] [S: ] [P: platform1] [T: Timer-9] BatchValueStreamProcessor queue #84 is currently 100.0 percent full that is at or above 70.0 percent threshold
platform1_1 | 2022-02-22 21:09:08.208+0000 [L: WARN] [O: c.t.p.p.StreamEntryProcessor] [I: ] [U: ] [S: ] [P: platform1] [T: Timer-9] BatchValueStreamProcessor queue #85 is currently 100.0 percent full that is at or above 70.0 percent threshold
The Application logs should be monitored for each stream entry processor to keep to the rejected entries to minimum.
Following is the template for this log statement:
{ProcessorName} queue #{queue number} is curently {percentage} percent full that is at or above 70.0 percent threshold
Following are the names of all the stream processors which can appear in above logs statements in place of {ProcessorName}:
• BatchStreamProcessor
• BatchValueStreamProcessor
• ValueStreamProcessor
• DataTableProcessor
• InfluxdbStreamProcessor
• InfluxdbValueStreamProcessor
• PersistentPropertyProcessor
• StreamProcessor