Repository navigation
read ECONNRESET at TLSWrap.onStreamRead #35824
Description
Activity
Can you post your code, the default server doesn't do that. One of the causes can be blocking the event loop.
Can you post your code, the default server doesn't do that. One of the causes can be blocking the event loop.
In order to avoid the influence of the framework, http2 module is used for testing.
'use strict' const http2 = require('http2') const fs = require('fs') process.chdir(__dirname) let serv = http2.createSecureServer({ cert : fs.readFileSync('./rsa/localhost.cert'), key : fs.readFileSync('./rsa/localhost.key') }) serv.on('stream', (stream, headers) => { stream.on('frameError', (type,code,id)=>{}) stream.on('error', (err) => {stream.destroy()}) stream.on('aborted', () => { if (!stream.destroyed) { stream.destroy() } }) stream.on('data', chunk => { }) stream.on('end', () => { stream.end('ok') }) }) serv.on('tlsClientError', (err, tls) => { console.error(err) tls.destroy() }) serv.setTimeout(5000, (sess) => { sess.destroy() }) serv.on('connection', (sock) => { sock.setTimeout(2000, () => { if (!sock.destroyed) { sock.destroy() } }) }) serv.listen(1235)
test command:
ab -n 3500 -c 50 'https://localhost:1235/'I confirm that there is indeed a bug in the TLS code, both in 14.x and master
- added a commit that references this issue
on Oct 27, 2020 @master-genius I replaced the uncatchable error with a tlsClientError event
@master-genius, @lpinca is right, your error can (and should be) handled in the user code:
serv.on('secureConnection', (sock) => { sock.on('error', (err) => { console.log(err); }); });@mmomtchev ok, thanks.
I had the same issue with my mediasoup server. Earlier, it randomly crashed with the same exception:
throw er; // Unhandled 'error' event ^ Error: read ECONNRESET at TLSWrap.onStreamRead (internal/stream_base_commons.js:209:20) Emitted 'error' event on TLSSocket instance at: at emitErrorNT (internal/streams/destroy.js:106:8) at emitErrorCloseNT (internal/streams/destroy.js:74:3) at processTicksAndRejections (internal/process/task_queues.js:80:21) { errno: -104, code: 'ECONNRESET', syscall: 'read' }after applying the 'error' handler
server.on('request', (req, res) => { req.socket.on('error', () => {}); res.end('Unavailable'); }); server.on('secureConnection', (socket) => { socket.on('error', () => {}); });I've started receiving another crash:
events.js:292 throw er; // Unhandled 'error' event ^ Error: write ECONNRESET at WriteWrap.onWriteComplete [as oncomplete] (internal/stream_base_commons.js:94:16) at handleWriteReq (internal/stream_base_commons.js:65:21) at writeGeneric (internal/stream_base_commons.js:145:15) at TLSSocket.Socket._writeGeneric (net.js:785:11) at TLSSocket.Socket._write (net.js:797:8) at writeOrBuffer (_stream_writable.js:352:12) at TLSSocket.Writable.write (_stream_writable.js:303:10) at WebSocketRequest.accept (/home/newvid/node_modules/websocket/lib/WebSocketRequest.js:450:21) at /home/newvid/node_modules/protoo-server/lib/transports/WebSocketServer.js:110:33 at Object.task (/home/newvid/server.js:221:25) Emitted 'error' event on TLSSocket instance at: at emitErrorNT (internal/streams/destroy.js:106:8) at emitErrorCloseNT (internal/streams/destroy.js:74:3) at processTicksAndRejections (internal/process/task_queues.js:80:21) { errno: -104, code: 'ECONNRESET', syscall: 'write' }Considering that THERE IS an
errorhandler and that this bug is relatively fresh (25 days ago), it might indicate that there is one more related bug. That's why I'm reporting this right now.The
protoo-serverpackage is a simple wrapper aroundwebsocketserver, and it has the following (https://github.com/versatica/protoo/blob/master/server/lib/transports/WebSocketServer.js#L60)[line] to address the same issue, but it doesn't help either.I use node
v14.15.0on Ubuntu16.04 LTS. Going to install node v15 and see, whether the bug is still there.@Kukunin This is different than the recent issue #36180
The tricky part withECONNRESETis that it is usually reported by the other end, not the one that it is at fault
If your server saysECONNRESETthis means your client responded with aRSTmessage
What is your client? Is it Node? Can you capture a network trace of the session?My client is a browser. My server is an SFU for WebRTC using mediasoup, it handles a lot of clients simultaneously.
The problem is not ECONNRESET itself, I know what it is. The problem is that the whole server process crashes with
Unhandled 'error' eventdespite the handlers ARE there.I've upgraded to Node v15.2.1, it hasn't been crashing for 26 hours already. Maybe it's something that was fixed already.
I captured network traffic with
tcpdumpand I can see a lot of RST flags and only few of them (happens once-twice per day) actually crashes the app. I wasn't able to reproduce this locally with a dummy client (on Python), that sends RST flags, maybe because locally I have node v15.0.x.The TLS phase is also important: if you go through this discussion, my PR and @lpinca's comment, you will see that the TLS error handler is in Node during the handshake then once the user data starts flowing, it is expected that the user code handles it.
That part of the code has some key differences between 14.x and 15.x, so it is possible that this was fixed.
Try to identify if thisRSTcomes during the handshake or during the data transfer.
You can also try running withNODE_DEBUGset totls
Either there is an error handler that is missing (some of them like that one onsecureConnectionare not very intuitive), either someone forgot about an'error'in Node. Your error is on thewritecall, so it is definitely not something very common. This means that the remote end closed the socket before the normal end of the TLS connection.Reacted by Chad FaragherAnd instead of simply ignoring the
'error', try also to add aconsole.error()to your handler both inside the handler and in the initialization to make sure it is called.Thank you for your answers.
I can confirm, that upgrading to node
15.2.1has fixed the problem - it hasn't been happening for the last 3 days.Reacted by Afanasii Kurakin and Otávio BertuciniHi @mmomtchev,
Currently i am using the Node.js v17.6.0 and getting the same TLS -104 error and because of this hosted kubernets POD getting crashed.Client is, Another NodeJs server using "http" lib to make the POST call.
Error we got
`node:events:505
throw er; // Unhandled 'error' event
^Error: read ECONNRESET
at TLSWrap.onStreamRead (node:internal/stream_base_commons:217:20)
Emitted 'error' event on ClientRequest instance at:
at TLSSocket.socketErrorListener (node:_http_client:442:9)
at TLSSocket.emit (node:events:527:28)
at emitErrorNT (node:internal/streams/destroy:164:8)
at emitErrorCloseNT (node:internal/streams/destroy:129:3)
at processTicksAndRejections (node:internal/process/task_queues:83:21) {
errno: -104,
code: 'ECONNRESET',
syscall: 'read'
}Node.js v17.6.0`
and we had, all expected cases
`httpServer.createServer((request, response) => {
try{
//response.setHeader('Content-Type', 'text/plain');
response.setHeader('Content-Type', 'text/plain');
/* if boadcaster is not assigned then do needful *//* console.log("Start") */ response.setTimeout(parseInt(process.env.DEFAULT_API_REQUEST_TIME_OUT), function (_pConnectionTimeOut) { /* Send response */ sendResponse(response, responseObj.HTTP_STATUS.REQUESTTIMEOUT, responseObj.ERROR_CODES.SERVICE_TIME_OUT, responseObj.ERROR.TYPE.REQUESTTIMEOUT, responseObj.MESSAGES.DEFAULT.REQUESTTIMEOUT); request.ab }); let strQueryString = urlObj.parse(request.url, true).query; let strEndPoint = urlObj.parse(request.url, true).pathname; let body = ""; request.on('error', (err) => { console.log("Request error") console.log(err) request.destroy(); //response.write(err); response.end(); }).on('socket',function (socket) { socket.on('timeout', function() { request.destroy(); /* Send response */ sendResponse(response, responseObj.HTTP_STATUS.REQUESTTIMEOUT, responseObj.ERROR_CODES.SERVICE_TIME_OUT, responseObj.ERROR.TYPE.REQUESTTIMEOUT, responseObj.MESSAGES.DEFAULT.REQUESTTIMEOUT); }); }).on('data', (chunk) => { body += chunk.toString() }).on('end', () => { /* Validating and upstreating the requst */ processHTTPRequest(strEndPoint, request, response, body, (pAPIResponse) => { /* Writing the Resposne text */ //response.write(pAPIResponse); //response.end(); //echo(pAPIResponse); }); }); }catch(exception){ console.log("===================HTTP OBJECT ERROR===============") console.log(exception); console.log("===================HTTP OBJECT ERROR===============") }/* on client conention error /
}).on("clientError",function(err,socket){
console.log("Client Connection Error ");
console.log(err);
console.log("Client Connection Error");
/ On client connection error /
//socket.end(JSON.parse({"error": responseObj.HTTP_STATUS.BAD_REQUEST,"error_description": responseObj.MESSAGES.DEFAULT.BAD_REQUEST_BODY,"error_code": responseObj.HTTP_STATUS.BAD_REQUEST,"statuscode": responseObj.HTTP_STATUS.BAD_REQUEST}));
}).on("secureConnection",function(socket){
console.log("secure Connection Error ");
/ On client connection error */
//socket.end(JSON.parse({"error": responseObj.HTTP_STATUS.BAD_REQUEST,"error_description": responseObj.MESSAGES.DEFAULT.BAD_REQUEST_BODY,"error_code": responseObj.HTTP_STATUS.BAD_REQUEST,"statuscode": responseObj.HTTP_STATUS.BAD_REQUEST}));
}).listen(process.env.HTTP_ENCY_PORT);`this occurred, when, generally when traffic is reaching to the 4000+ TPS.
Kindly help.
Thanks & Regards
Jaiswar Vipin Kumar R.@jaiswarvipin, this is Sparta
Reacted by Tymoteusz Klocek and Dima ZudochkinReacted by Graham Ballantyne and JustinReacted by jirkadev and carlosencaladatrilogyReacted by regine-altaI am sorry, did understand, cloud you pls elaborate and help
- <script src="/sitelet?url=https%3A%2F%2Fcdnjs.cloudflare.com%2Fajax%2Flibs%2Fsocket.io%2F3.1.0%2Fsocket.io.js" integrity="sha512-+l9L4lMTFNy3dEglQpprf7jQBhQsQ3/WvOnjaN/+/L4i0jOstgScV0q2TjfvRF4V+ZePMDuZYIQtg5T4MKr+MQ==" crossorigin="anonymous"></script> in device
in server.js
setInterval(() => {
require('./app_node/routes/web')(app, sessionMap, startDEVICE)
}, 30000); Hi team,
Need help on this issue. My application was deployed in GKE (Google Cloud Project) and seeing the below errorread ECONNRESET
at TLSWrap.onStreamRead (internal/stream_base_commons.js:209:20)I am using the https call module and this happens only when there is connecting from external vendor api which is not internal network and I don't see this issue when there is internal applications with in the network.
Currently I am using the Node version in my docker file as
14.15.0. So based on the above comments I see this is an bug in Node V14. I just want to check is this fix on Node V16 LTS version@gopinash, I will explain you the problem with this issue.
The problem is that an ex-employer of mine organised a false rape accusation against me that he was able to cover up on not less than 6 different judicial procedures in the French system - and he has been paying people to post dicks and viagra ever since - for about 10 years now. So I did found the problem with that issue, but what bothers me is that every time I submit a PR, I get more dicks, viagra and whatever - because of his firm belief that I will take this shit to be able to work in Node. I won't. I have already contacted the OpenJS foundation once, I will probably do so again, this time getting all the companies involved in this project aboard, and if this continues, I will surely post my PRs somewhere else explaining the whole situation. Because some people simply refuse to accept that some things in life are off limits and that we live in a civilised society.Reacted by Anurag PandeyReacted by Mikita Kiryk, catcat, txwmemoria, Richard, Anurag Pandey, David and xtclovversimilar issue here, any hint help to solve the issue would be appreciated?
os: MacOS
node: v18.9.0
ts-node: v9.0.0AxiosError: read ECONNRESET at TCP.onStreamRead (node:internal/stream_base_commons:217:20) { syscall: 'read', code: 'ECONNRESET', errno: -54, config: { transitional: { silentJSONParsing: true, forcedJSONParsing: true, clarifyTimeoutError: false }, adapter: [Function: httpAdapter], transformRequest: [ [Function: transformRequest] ], transformResponse: [ [Function: transformResponse] ], timeout: 0, xsrfCookieName: 'XSRF-TOKEN', xsrfHeaderName: 'X-XSRF-TOKEN', maxContentLength: -1, maxBodyLength: -1, env: { FormData: [Function] }, validateStatus: [Function: validateStatus], headers: { Accept: 'application/json', 'User-Agent': 'axios/0.27.2' }, url: 'http://127.0.0.1:8080/v1/', method: 'get', withCredentials: true, cancelToken: CancelToken { promise: [Promise], _listeners: [] }, data: undefined },looks the issue is due to the concurrent issue, when i limit the count of promises to 10 in
await Promise.all(promises);, the issue fixedReacted by Dyarlen IberWhat steps will reproduce the bug?
[dev:start] Error: read ECONNRESET
[dev:start] at TLSWrap.onStreamRead (node:internal/stream_base_commons:217:20) {
[dev:start] errno: -4077,
[dev:start] code: 'ECONNRESET',
[dev:start] syscall: 'read',
[dev:start] response: undefined
[dev:start] }I'm also encountering errors with Next.js 14.
Reacted by David and Degubi
What steps will reproduce the bug?
I create http2 server and benchmark use ab:
after this, the server is crashed.
I had try to handle all 'error' event (socket, http2Stream, http2Session), but all are ineffective.