Skip to content
This repository has been archived by the owner on May 14, 2024. It is now read-only.

ECONNRESET errors when idling #318

Closed
mrapczynski opened this issue Nov 3, 2015 · 31 comments
Closed

ECONNRESET errors when idling #318

mrapczynski opened this issue Nov 3, 2015 · 31 comments
Labels

Comments

@mrapczynski
Copy link

I'm trying to figure out the source of the error below, or how to catch it properly and mitigate it. I am using LDAPJS as part of an identity management pipeline to move user accounts from our ERP into Active Directory (connected over TLS with a self-signed cert). There is often idle periods in the database where no events are being dispatched, and in turn LDAPJS raises these exceptions when I assume it has nothing to do.

12:20:54 bsis-0 Error: read ECONNRESET
    at exports._errnoException (util.js:812:11)
    at TLSWrap.onread (net.js:542:26)
2015-11-03 12:20:54: App name:bsis id:0 exited with code 1
12:20:54 PM2 App name:bsis id:0 exited with code 1
2015-11-03 12:21:54: Starting execution sequence in -fork mode- for app name:bsis id:0
12:21:54 PM2 Starting execution sequence in -fork mode- for app name:bsis id:0
2015-11-03 12:21:54: App name:bsis id:0 online
12:21:54 PM2 App name:bsis id:0 online
12:21:54 bsis-0 WARNING: NODE_APP_INSTANCE value of '0' did not match any instance config file names.
12:21:54 bsis-0 WARNING: See https://github.com/lorenwest/node-config/wiki/Strict-Mode
12:37:54 bsis-0 Error: read ECONNRESET
    at exports._errnoException (util.js:812:11)
    at TLSWrap.onread (net.js:542:26)
2015-11-03 12:37:54: App name:bsis id:0 exited with code 1
12:37:54 PM2 App name:bsis id:0 exited with code 1
2015-11-03 12:38:54: Starting execution sequence in -fork mode- for app name:bsis id:0
12:38:54 PM2 Starting execution sequence in -fork mode- for app name:bsis id:0
2015-11-03 12:38:54: App name:bsis id:0 online
12:38:54 PM2 App name:bsis id:0 online
12:38:54 bsis-0 WARNING: NODE_APP_INSTANCE value of '0' did not match any instance config file names.
12:38:54 bsis-0 WARNING: See https://github.com/lorenwest/node-config/wiki/Strict-Mode
12:54:55 bsis-0 Error: read ECONNRESET
    at exports._errnoException (util.js:812:11)
    at TLSWrap.onread (net.js:542:26)
2015-11-03 12:54:55: App name:bsis id:0 exited with code 1
12:54:55 PM2 App name:bsis id:0 exited with code 1

They only show up because of my process.on('uncaughtException') handler which in turns logs the event and then auto-restarts the broker.

Ideas?

@mrapczynski
Copy link
Author

One other detail - don't ask me why I thought to do this - but since I continue to see these reset errors in the log, I quickly compared the amount of time between each event and it looks to be approx. 17 minutes:

1:29PM - 1:12 = 17
12:54PM - 12:37 = 17
12:37PM - 12:21 = 16

Maybe this is a setting somewhere?

@pfmooney
Copy link
Contributor

pfmooney commented Nov 4, 2015

It's quite possible that the LDAP server you're connecting to will time out idle clients. Subscribing to the error event on the client will allow you to receive these events and take appropriate action.

@mrapczynski
Copy link
Author

@pfmooney OK I'll check into that. In the background, does LDAPJS pick up these errors and try to reconnect, or completely bails until I restart my Node process?

@pfmooney
Copy link
Contributor

pfmooney commented Nov 4, 2015

While ldapjs had client options to automatically reconnect in the face of socket errors, they have not yet been documented. Alternatively, you can perform such actions manually by instantiating a new connection when your client encounters an error.

@tapmodo
Copy link

tapmodo commented Nov 13, 2015

As noted above, the documentation for the Client API doesn't mention auto-reconnecting or how to handle an eventual ECONNRESET from the server. I can't imagine an LDAP server that never closes or resets the connection, or never goes down. Appears to be option reconnect: true?

The documentation also does not mention how to close a connection, which is necessary if you are writing a small script that should gracefully shut down after doing its work. I looked in the source and appears this can be done with client.destroy()

Both of these points would be useful to mention in the documentation. Thanks!

@randomsock
Copy link

👍

Thanks very much for doing the digging @tapmodo. reconnect: true does indeed auto-reconnect after this failure. This strikes me as a really important resilience feature so is a bit surprising it's not documented. You do still need the client.on('error', ...) handler to stop the disconnection from bringing the process down first, as @pfmooney pointed out.

So an over-simplified solution looks like:

var options = {
    url: 'ldap://myad.foo.net:3268',
    reconnect: true
};
var client = ldapjs.createClient(options);
client.on('error', function(err) {
    console.warn('LDAP connection failed, but fear not, it will reconnect OK', err);
});

Job done - thanks guys!

@dustinsmith1024
Copy link

This doesn't seem to reconnect properly for me. Anyone else having issues? The program doesn't exit but it just hangs on any attempt to re-authenticate.

@mjoyce91
Copy link

mjoyce91 commented Jan 4, 2016

@dustinsmith1024 - I've been having ECONNRESET errors as well. I've tried adding reconnect:true option, and using client.destroy() one minute after every query. This only started happening after I migrated from an AD on an Azure server to one on an AWS server, so I think it has some thing to do with what @pfmooney mentioned - the server is timing out idle connections. I just can't get the error to be handled any where.

@saigop
Copy link

saigop commented Jan 4, 2016

Where to add this entry reconnect:true and client.destroy()?

Thanks in advance

@saigop
Copy link

saigop commented Jan 4, 2016

I have changed the LDAP idle timeout in AD, post that it looks fine, will keep monitoring.

thanks.

@mjoyce91
Copy link

mjoyce91 commented Jan 5, 2016

@saigop - I believe you put reconnect : true in your options object that you pass into ldap.createClient. I moved my client.unbind() and client.destroy() into my resp.on handlers and I think that did the trick.

CKarper added a commit to CKarper/promised-ldap that referenced this issue Feb 3, 2016
As per ldapjs/node-ldapjs#318 , sometimes you will need to call destroy to completely close a client connection so you don't idle error out after ~17 minutes.
@the1mills
Copy link

@saigop "I have changed the LDAP idle timeout in AD, post that it looks fine, will keep monitoring. " what is AD??

@mjoyce91
Copy link

@the1mills Active Directory

@ma-zal
Copy link

ma-zal commented Feb 15, 2016

Without using TLS, same issue.

Error: read ECONNRESET
    at errnoException (net.js:905:11)
    at TCP.onread (net.js:559:19)

@jontowles
Copy link

When I add the reconnect I get a:

Err:OperationsError: 000004DC: LdapErr: DSID-0C090748, comment: In order to perform this operation a successful bind must be completed on the connection

I think some insight into how we need to gracefully handle this in NodeJS would be helpful. Realistically we need to ask do we close the connection, do we let is close on its own, should we be destroying the connection in intervals?

@jontowles
Copy link

I've been looking into this more deeply and I'd love to see some responses.

We added reconnect: true along with bind/unbind as we were seeing basically an LDAP break around 15m after starting the service.

We still continue to see everyone's favorite error: [2016-05-23 22:04:44.068] [ERROR] Server - Exception ->Error: read ECONNRESETerrStack: Error: read ECONNRESET
at exports._errnoException (util.js:870:11)
at TLSWrap.onread (net.js:552:26)

It's much more sporadic now with the exception that the re-connect and unbind ensure that we no longer see any issues because it automatically reconnects.

A packet capture shows us that even though we did an unbind, we see the RST from our LDAP server, which leads to an exchange of SYN and ACH packets and once our NodeJS server and LDAP exchange two sets of ACH packets another RST ensues basically saying "send me something! no you send me something! okay we'll close now"

At this point, this just looks like noise to me as everything works fine outside of that.

@mrapczynski
Copy link
Author

I recently experimented with setting up my own LDAP connection pool using the npm package pool2, and so far it's going very well. After weeks of testing, we rolled the update to production with no incidents.

Basically I have the pool give me connections and then automatically unbind them if they age out thus completely avoiding the idle timeout issue that I experience with Active Directory.

I have never found that reconnect: true worked, and so prior to this I would just process.exit(0) if I got an idle timeout forcing my app worker to re-generate and pick up a fresh new connection.

@jontowles
Copy link

The thing that I learned is that its not an idle session timeout. For some reason LDAPJS keeps trying to connect to LDAP even when no attempt to bind is occurring.

@Jackyjjc
Copy link

any update on this issue? we are using the ldapauth fork library which build on top of ldapjs. It is suffering from this same issue due to this bug in ldapjs and here are the discussion: vesse/node-ldapauth-fork#23

@PaulBernier
Copy link

+1 What's the proper way to close a client? Proper way to reconnect? Those sound like essential points that need to be documented please. Thanks.

@AshMartian
Copy link

+3

@Nepoxx
Copy link

Nepoxx commented Dec 20, 2016

This issue was opened a year ago, has this changed?

edit:

My current workaround is simply this:

const ldapClient = ldapjs.createClient({
  url: config.get('ldap.url'),
  reconnect: true
})

ldapClient.on('error', err => {
  logger.error(err.message)
})

Once in a while I'll get a read ECONNRESET error, but the client is still usable and seems to re-create connections as they are needed.

@banzy
Copy link

banzy commented May 26, 2017

I've solved the problem removing the client definition out of the main node loop. So I'm building the client just when it's necessary and after the call I destroy it.

  app.post('/item', function(req, res) { 
       const ldapClient = ldapjs.createClient({
           url: 'ldap://ldap.server.com:3880',
           reconnect: false
        });
    ... 
        res.on('end', function(result) {
            console.log('status: ' + result.status);
            ldapClient.destroy();
        });

@ORESoftware
Copy link

ORESoftware commented Jun 9, 2017

For some reason

client.destroy()

is not documented, as far as I can tell.

Should we call unbind() before destroy()?

client.unbind(function(){
   client.destroy();
});

@dpatte
Copy link

dpatte commented Jan 23, 2018

Seeing this:
Error: read ECONNRESET
at errnoException utiil.js:873:11)
at TCP.onread (net.js:557:26)

The timing seems random. I'm on 1.4

@ms007
Copy link

ms007 commented Jun 13, 2018

destroy calls unbind internally

@jsumners
Copy link
Member

I think this topic has been sufficiently covered in this thread. If someone would like to condense it into some better documentation and submit a PR that would be fantastic. I am closing this issue so I can continue triaging old issues.

@VikR0001
Copy link

I just figured this out, at least in my use case.

I was getting ECONNRESET. It turned out that the way my client was set up, it was hitting the server with an API call a ton of times really quickly -- and it only needed to hit the endpoint once.

When I fixed that, the error was gone.

@1nbuc
Copy link

1nbuc commented Jun 28, 2022

Since I still had to deal with some diffrent errors using the fixes above I finally found a solution working out for me.

const createClient = () => ldap.createClient({
    url: config.ldap.URL,
    reconnect: true,
});

let client = createClient();

const rebindClient = () => {
    client.destroy();
    client = createClient();
    client.bind(config.ldap.BIND_DN, config.ldap.BIND_CREDENTIALS, (err) => {
        if (err) {
            console.error(err);
        }
    });
};
rebindClient();

client.on('error', (err) => {
    if (err.errno === -4077) {
        rebindClient();
    } else {
        console.error(err);
    }
});

@longhibianca
Copy link

longhibianca commented Sep 27, 2022

I have an application that add and remove groups in AD, and that uses ldapjs. I'm getting ECONNRESET error many times. For exemple, when I get econnreset error 5 times, i can listen for the error and call a function in my code that restart the execution. But sometimes I received econnreset error 30 times or more (consecutivelly), and my application couldn't restart.
Here are some pieces of my code:

  • Create the client:
    var ldapOptions = { tlsOptions: tlsOptions, url: url, reconnect: true, } this.url = url; this.client = ldapjs.createClient(ldapOptions); this.client.on("error", (err) => { console.log(err), this.client.destroy() })

  • authenticate function:
    authenticate() { var promiseToAuthenticate = new Promise((resolved, rejected) => { this.client.bind(this.dn, this.password, (err) => { if (err) return rejected(err); else resolved(); }); this.client.on('timeout', (err) => console.log('Timeout.....', err)) this.client.on("error", (err) => console.log(err)) }); return promiseToAuthenticate; }

  • Close ldap connection
    async close() { this.client.unbind(); //this.client.destroy(); console.log('Conexão do Active Directory encerrada.') }

Do you know how to stop getting econnreset error or how is the corret way to catch this error or how to correctly close ldap connections?

@jsumners
Copy link
Member

⚠️ This issue has been locked due to age. If you have encountered a recent
problem that seems to be covered by this issue, please open a new issue.

Please include a minimal reproducible example
when opening a new issue.

@ldapjs ldapjs locked as resolved and limited conversation to collaborators Mar 10, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Projects
None yet
Development

No branches or pull requests