Network Discovery problem with 1.0.7

[piet]

Hi Remo,

thank you so much for the new release 

Unfortunetly I have problems during the discovery process.

I use a fresh system install using the NeDiO7 ISO image and I did not copy any data, except some settings in the nedi.conf and my seedfiles.

When starting the new discovery process (using nedi.pl -p -u seedfile) I get the following message for each discovered device

Code: [Select]

Use of uninitialized value in concatenation (.) or string at /var/nedi/inc/libcli-iopty.pm line 479.and at the end, the discovery process crashes with the message

Code: [Select]

Resource temporarily unavailable at /var/nedi/inc/libcli-iopty.pm line 919.

Do you have any idea what could cause that problem?

Piet

[rickli]

It's just a test, if anyone actually uses those features 

Most likely related to this: http://forum.nedi.ch/index.php?topic=1244.0

Are you using Cisco devices with SSH? Do you get the resource unavailable problem after about 128 devices? It's most likely because I haven't found a way to close SSH sessions problerly, thus they stay open until the discovery finishes. A quick fix might be to edit /etc/login.conf and make sure         :openfiles-cur=128:\ in the default section is changed to be bigger than the amound of devices you've got. Hope this will work, as I don't really have enough devices to test myself right now...

[piet]

Thank you very much for your answer.

Yes, we use Cisco devices with SSH!

I tested again, but the error happens immediately after a fresh install when run the discovery process the very first time.

I currently have just 40 devices in the database, but whenever a new device is discovered, the error happens. I increased the :openfiles-cur=128:\ to 1024 anyway, but without success.

Maybe it happens because I use the same userid with different passwords?

Here an extract of the output:

Prepare (CLI)  ----------------------------------------------------------------
SSH :user1;1:22 Tout:2s OS:IOS EN:(.+?)#\s?$
PTY :Forking ssh -o 'StrictHostKeyChecking no' -l user1 10.10.10.10
CLI2:Matched Password:, sending password
CLI3:Password sent
Use of uninitialized value in concatenation (.) or string at /var/nedi/inc/libcli-iopty.pm line 479.
CLI4:Matched DESEEB033WC001AS01> (or gen. prompt with enpass &  cmd), enabling
ERR :pattern match timed-out
SSH :user1;2:22 Tout:2s OS:IOS EN:(.+?)#\s?$
PTY :Forking ssh -o 'StrictHostKeyChecking no' -l user1 10.10.10.10
CLI2:Matched Password:, sending password
CLI3:Password sent
Use of uninitialized value in concatenation (.) or string at /var/nedi/inc/libcli-iopty.pm line 479.
CLI4:Matched DESEEB033WC001AS01> (or gen. prompt with enpass &  cmd), enabling
ERR :pattern match timed-out
SSH :user1;3:22 Tout:2s OS:IOS EN:(.+?)#\s?$
PTY :Forking ssh -o 'StrictHostKeyChecking no' -l user1 10.10.10.10
CLI2:Matched Password:, sending password
CLI3:Password sent
Use of uninitialized value in concatenation (.) or string at /var/nedi/inc/libcli-iopty.pm line 479.
CLI4:Matched DESEEB033WC001AS01> (or gen. prompt with enpass &  cmd), enabling
ERR :pattern match timed-out
SSH :user2;1:22 Tout:2s OS:IOS EN:(.+?)#\s?$
PTY :Forking ssh -o 'StrictHostKeyChecking no' -l user2 10.10.10.10
CLI2:Matched Password:, sending password
CLI3:Password sent
CLI3:Matched Password:, login failed
SSH :user2;2:22 Tout:2s OS:IOS EN:(.+?)#\s?$
PTY :Forking ssh -o 'StrictHostKeyChecking no' -l user2 10.10.10.10

That's the one issue I have, but the more important problem for me is, that the whole discovery preocess crashes suddenly - I could not find any regularity 

The message I receive from NeDi when the process crashes is

Resource temporarily unavailable at /var/nedi/inc/libcli-iopty.pm line 919.

What can I do to eliminate this problem?

Another question I have is, in which file(s) do I have to enter the fixes you posted here http://forum.nedi.ch/index.php?topic=1244.0 ?

Piet

[pc_sg]

About same user and different password (and/or enable password) is that the meaning of this parte of nedi.conf

Code: [Select]

# The users for telnet and ssh access:
# - Put most frequent ones first.
# - Leave enablepass empty, if the user is priviledged already.
# - Use a dummy pass (and proper enablepass) if no login is required to connect.
# - Append ;1 ;2 etc. to user, if different pw are used.
# - Nortel CLI capable devices may require to configure cmd-interface cli to avoid menus!
# user pass enablepass
;usr nedi pa55 enpa55
;usr admin enpa55
;usr edmin enterasys
;usr xmin extreme
usr admin;1 pwd1 enpwd1
usr admin;2 pwd2
usr admin;3 pwd3
usr admin;4 pwd4 enpwd1
usr admin;5 pwd1 enpwd2

the last fifth are other examples, where username is always admin (the default on our Cisco devices) and you can guess what you need to put instead of pwd# ed enpwd#.

If you try with -v verbose option and read logs, you should find your best configuration.

Paolo

[rickli]

It can be applied as patch. Copy to patch.txt then:
n7:~#patch /var/nedi/inc/libcli-iopty.pm patch.txt

If the patching doesn't help add this at line 468. I'm curios what you'll see:

print "PRE :$pre\nMTCH:$match\n" ;

[piet]

I applied the patch as described and the

Use of uninitialized value in concatenation (.) or string at /var/nedi/inc/libcli-iopty.pm line 479

problem disappeard 

Unfortunately, the

Resource temporarily unavailable

problem still remain 

The last nedi output just before the error occur is:

Prepare (CLI)  ----------------------------------------------------------------
SSH :user1;1:22 Tout:2s OS:IOS-wl EN:(.+?)#\s?$
PTY :Forking ssh -o 'StrictHostKeyChecking no' -l user1 10.10.10.22
CLI2:Matched password:, sending password
CLI3:Password sent
CLI3:Matched denied, login failed
SSH :user1;2:22 Tout:2s OS:IOS-wl EN:(.+?)#\s?$
PTY :Forking ssh -o 'StrictHostKeyChecking no' -l user1 10.10.10.22
Resource temporarily unavailable at /var/nedi/inc/libcli-iopty.pm line 926.

Before I started the discovery, I initialized the database again, so this happened with an empty DB - but the device where the error happened is not the very first one but no. 43 in the seedfile. The devices 1 - 42 are discoverd without any problems.
After starting the discorvery process again, it happened with seed59...

I am not sure, where to enter the additional print command - in line 468 of the unpatched or of the patched libcli-iopty.pm?
Could you please advice?

Thank you so much again for your support!

[rickli]

There should be a commented print line at 468 as well. You can just modify that one...

Did you see this?
CLI3:Matched denied, login failed

Is the pw set correctly? If you add -d you'll get input.log and output.log to trace what exactly gets sent and received...

I don't quite see, why you get the resource problem though. I do hope someone finds a fix to properly terminate the SSH pty!

[piet]

I added the print command in line 480 (after the commented print command line)

I use several different userids, passwords and different enable passwords for the devices in different locations - therefore it is not always the first userid - password combination, which works.

The users section of nedi.conf looks like this:

usr   user1;1   password1   enapw1
usr   user1;2   password1   enapw2
usr   user1;3   password1   enapw3
usr   user2;1   password2   enapw1
usr   user2;2   password3   enapw1
usr   user3   password4   enapw4
usr   user4   password5   enapw5

I run the discovery process again with a fresh empty database - the first time the error occurs with seed no. 43

I attached three parts from the total discovery output - one, where the first userid - password combination works, another one where the 4th combination works and the last one, where the error occurs (the 4th userid password combination would work with that device as well)

[rickli]

With this many accounts I can see how the resource problem comes up, when trying all of them. There's hope it will dissapear once all logins are found.

Can you try restarting the discovery and have an eye on the progress each time?

[piet]

I run several tests again - but without success at the end 

But the whole story...

The availble patch has been applied.
I have several different seed files for diferent locations but just one nedi.conf. I tried to optimize the sequence of the userid - password list in nedi.conf and I got better results during the discovery process.
I started again with one specific seed file and with an empty database and the error appears later than before, when the userlist in nedi.conf was not optimized.
Every time I started the discovery process, the error appears several devices later and at the end, all devices of that seedfile are discovered without error.

Than I took the next seedfile and performed the same steps until all devices have been discoverd without errors.

I was happy and I thought, everything would be OK now.
But in real life, I run many discovery processes with different seed files in parallel and then I get the error again (Resource temporarily unavailable at /var/nedi/inc/libcli-iopty.pm line 927.)
The time nedi requires for a discovery of all devices of all locations one after another would be more than 24 hrs - but I would like to discover each location at least once a day...
But because of this problem, some locations are not completely discovered at all...

In addition, I receive the following error during the discovery process:

Use of uninitialized value in concatenation (.) or string at /var/nedi/inc/libsnmp.pm line 292.

Do you have any additional hint?

[rickli]

line 292:
         $main::dev{$na}{cul}= "$misc::sysobj{$so}{cul};$misc::sysobj{$so}{mmu}";

So, can you have a look at the device's .def MemCPU entry (2nd argument is memory multiplier or mmu)?

I on the other will figure out why the heck I put it there 

Now to your other problem. Open filehandles is a global number, thus running several discoveries in parallel will use them all up, apparently. I really do hope to find a fix for this. Maybe forking will solve it, but I guess 1oh8 will tell. Are you using SSH to just backup configs or also to get the bridge forwarding entries? Because in the first case, you could backup configs during the night in a single run, while Skipping all other information (see crontab for an example).

[piet]

I did some manual optimization in the crontab to reduce parallel discovery as much as possible - is better now but sometimes, the problem still occurs.

I use SSH for getting bridge forwarding entries only (getfwd is set to dyn in nedi.conf). If I set that to SNMP only (getfwd snmp), do I loose information?

I wanted to get rid of that problem, before start backing up configs.

[rickli]

It'll be slower, if you have a lot of vlans on those switches. With getfwd sec you could also read port-security (static MAC) entries, which doesn't seem very popular these days...so I'd say you don't lose anything...

[quartz]

It can be applied as patch. Copy to patch.txt then:
n7:~#patch /var/nedi/inc/libcli-iopty.pm patch.txt

If the patching doesn't help add this at line 468. I'm curios what you'll see:

print "PRE :$pre\nMTCH:$match\n" ;

Could someone explain to an absolut windows nerd where to get and how to install that patch? It seems I have the same issue here on my site.

[quartz]

Found the Patch on the main site and got it installed with the description there.
Sorry for to quick questioning.