@outputai/core worker shutdown. Nothing in your code needs to change, but a deploy that lands on steps still running now exits 0 instead of 1, several shutdown log lines changed, and OUTPUT_TEMPORAL_LOG_LEVEL rejects invalid values.
If you alert on worker exit codes or match worker log lines, read Exit code on a forced shutdown and Shutdown logs.
What changed
- A signal shutdown that abandons in-flight work at
TEMPORAL_SHUTDOWN_FORCE_TIMEexits0instead of1. - A service that fails to stop is logged as a warning and never changes the exit code. This was already true for every service except the worker.
- The
... errorwarning of a failed service stop is now... failed, and carries the time the stop took inshutdownDurationMs. - Every successful service stop logs
... completewithshutdownDurationMs. - The worker logs
Shutting downwith its shutdown budget as soon as shutdown starts, andByecarriesshutdownDurationMs. - Temporal’s
Worker failedentry for an expired force time is logged as a warning, not an error. - Temporal log entries that carry an error now show its
name,messageandstackinstead of{}. - The worker warns at startup when
TEMPORAL_SHUTDOWN_FORCE_TIMEis not bigger thanTEMPORAL_SHUTDOWN_GRACE_TIME. OUTPUT_TEMPORAL_LOG_LEVELis validated when the worker starts.
Exit code on a forced shutdown
When a signal starts the shutdown and a step is still running atTEMPORAL_SHUTDOWN_FORCE_TIME, the worker abandons it and exits. v0.16.0 exited 1 for that, so a routine deploy looked like a crash. v0.17.0 exits 0.
The abandoned step is not lost. Temporal notices its worker is gone when the step’s heartbeat times out, and retries it. See Worker Shutdown.
If you alert on a non-zero worker exit, those alerts no longer fire for a deploy that abandoned work. To tell that case apart, match the logs below. A worker that fails for any other reason while draining is not covered by the exit code. Temporal logs it as an error-level
Worker failed, so alert on that line.
Shutdown logs
An abandoned drain now reads like this in the logs:
[level] namespace: message { fields }. Production logs are JSON with the same fields.
Update log-based alerts and dashboards:
- A matcher on the text
Stopping Worker error(or another... errorstop line) must use... failed. - A matcher on an error-level
Worker failedstops matching a forced drain. If you baselined it as noise, remove it. - Anything that parses a Temporal log’s
errorfield now receives an object withname,messageandstackinstead of{}.
Shutting down is logged before the drain starts, so the budget is in the log even if the platform kills the process mid-drain. shutdownDurationMs on Bye is the whole shutdown, hook flush included. Compare it with your platform’s shutdown window. See Reading the Shutdown Logs.
A force time that is not bigger than the grace time now logs Temporal shutdown force time is not bigger than its grace time, so activities have no window to react to cancellation, with both values. Raise TEMPORAL_SHUTDOWN_FORCE_TIME above TEMPORAL_SHUTDOWN_GRACE_TIME.
OUTPUT_TEMPORAL_LOG_LEVEL is validated
The variable takes TRACE, DEBUG, INFO, WARN or ERROR, in any letter case, and defaults to INFO when unset or empty. Any other value, such as verbose or a Winston level like silly, now fails worker startup with a message naming the allowed values.
ERROR, the Worker failed warning of a forced shutdown is not logged, since it is no longer an error.
Checklist
- Bump
@outputai/coreand@outputai/clito0.17.0. - Remove or adjust alerts that treat a non-zero worker exit on a deploy as a failure. A forced drain exits
0. - Change log matchers from
Stopping Worker error(and the other... errorstop lines) to... failed. - Remove an error-level
Worker failedentry from your noise baseline. A forced drain logs it as a warning. - Update consumers of Temporal log entries that read an empty
errorobject. - Check that
OUTPUT_TEMPORAL_LOG_LEVEL, if set, is one ofTRACE,DEBUG,INFO,WARNorERROR. - If your deploys abandon work often, size
TEMPORAL_SHUTDOWN_GRACE_TIME,TEMPORAL_SHUTDOWN_FORCE_TIMEand your platform’s shutdown window together. See Worker Shutdown.