Debugging lost output from a child process in Node.js
I recently encountered an issue in a Node.js v12 service where output streamed from a child process is intermittently missing from downstream processing. The pipeline streams the data into a ZIP archive and uploads it to S3. The service is deployed on AWS Lambda. In this post, I describe the problem, walk through the debugging process, and evaluate several solutions. Data pipeline At a high level, the data pipeline inside the Lambda function looks like: ...