Welcome, Guest. Please login or register.

Author Topic: Network Discovery problem with 1.0.7  (Read 8026 times)

piet

  • Newbie
  • *
  • Posts: 31
    • View Profile
Network Discovery problem with 1.0.7
« on: February 13, 2012, 02:12:06 PM »
Hi Remo,

thank you so much for the new release  :)

Unfortunetly I have problems during the discovery process.

I use a fresh system install using the NeDiO7 ISO image and I did not copy any data, except some settings in the nedi.conf and my seedfiles.

When starting the new discovery process (using nedi.pl -p -u seedfile) I get the following message for each discovered device
Code: [Select]
Use of uninitialized value in concatenation (.) or string at /var/nedi/inc/libcli-iopty.pm line 479.and at the end, the discovery process crashes with the message
Code: [Select]
Resource temporarily unavailable at /var/nedi/inc/libcli-iopty.pm line 919. :(

Do you have any idea what could cause that problem?

Piet

rickli

  • Administrator
  • Hero Member
  • *****
  • Posts: 2697
    • View Profile
    • NeDi
Re: Network Discovery problem with 1.0.7
« Reply #1 on: February 13, 2012, 08:23:58 PM »
It's just a test, if anyone actually uses those features  ;)

Most likely related to this: http://forum.nedi.ch/index.php?topic=1244.0

Are you using Cisco devices with SSH? Do you get the resource unavailable problem after about 128 devices? It's most likely because I haven't found a way to close SSH sessions problerly, thus they stay open until the discovery finishes. A quick fix might be to edit /etc/login.conf and make sure         :openfiles-cur=128:\ in the default section is changed to be bigger than the amound of devices you've got. Hope this will work, as I don't really have enough devices to test myself right now...
Please consider Other-Invoices on your NeDi installation for an annual contribution, tx!
-Remo

piet

  • Newbie
  • *
  • Posts: 31
    • View Profile
Re: Network Discovery problem with 1.0.7
« Reply #2 on: February 15, 2012, 10:07:26 AM »
Thank you very much for your answer.

Yes, we use Cisco devices with SSH!

I tested again, but the error happens immediately after a fresh install when run the discovery process the very first time.

I currently have just 40 devices in the database, but whenever a new device is discovered, the error happens. I increased the :openfiles-cur=128:\ to 1024 anyway, but without success.

Maybe it happens because I use the same userid with different passwords?

Here an extract of the output:

Quote
Prepare (CLI)  ----------------------------------------------------------------
SSH :user1;1:22 Tout:2s OS:IOS EN:(.+?)#\s?$
PTY :Forking ssh -o 'StrictHostKeyChecking no' -l user1 10.10.10.10
CLI2:Matched Password:, sending password
CLI3:Password sent
Use of uninitialized value in concatenation (.) or string at /var/nedi/inc/libcli-iopty.pm line 479.
CLI4:Matched DESEEB033WC001AS01> (or gen. prompt with enpass &  cmd), enabling
ERR :pattern match timed-out
SSH :user1;2:22 Tout:2s OS:IOS EN:(.+?)#\s?$
PTY :Forking ssh -o 'StrictHostKeyChecking no' -l user1 10.10.10.10
CLI2:Matched Password:, sending password
CLI3:Password sent
Use of uninitialized value in concatenation (.) or string at /var/nedi/inc/libcli-iopty.pm line 479.
CLI4:Matched DESEEB033WC001AS01> (or gen. prompt with enpass &  cmd), enabling
ERR :pattern match timed-out
SSH :user1;3:22 Tout:2s OS:IOS EN:(.+?)#\s?$
PTY :Forking ssh -o 'StrictHostKeyChecking no' -l user1 10.10.10.10
CLI2:Matched Password:, sending password
CLI3:Password sent
Use of uninitialized value in concatenation (.) or string at /var/nedi/inc/libcli-iopty.pm line 479.
CLI4:Matched DESEEB033WC001AS01> (or gen. prompt with enpass &  cmd), enabling
ERR :pattern match timed-out
SSH :user2;1:22 Tout:2s OS:IOS EN:(.+?)#\s?$
PTY :Forking ssh -o 'StrictHostKeyChecking no' -l user2 10.10.10.10
CLI2:Matched Password:, sending password
CLI3:Password sent
CLI3:Matched Password:, login failed
SSH :user2;2:22 Tout:2s OS:IOS EN:(.+?)#\s?$
PTY :Forking ssh -o 'StrictHostKeyChecking no' -l user2 10.10.10.10

That's the one issue I have, but the more important problem for me is, that the whole discovery preocess crashes suddenly - I could not find any regularity  :(

The message I receive from NeDi when the process crashes is
Quote
Resource temporarily unavailable at /var/nedi/inc/libcli-iopty.pm line 919.

What can I do to eliminate this problem?

Another question I have is, in which file(s) do I have to enter the fixes you posted here http://forum.nedi.ch/index.php?topic=1244.0 ?

Piet



pc_sg

  • Sr. Member
  • ****
  • Posts: 265
    • View Profile
Re: Network Discovery problem with 1.0.7
« Reply #3 on: February 15, 2012, 01:17:08 PM »
About same user and different password (and/or enable password) is that the meaning of this parte of nedi.conf
Code: [Select]
# The users for telnet and ssh access:
# - Put most frequent ones first.
# - Leave enablepass empty, if the user is priviledged already.
# - Use a dummy pass (and proper enablepass) if no login is required to connect.
# - Append ;1 ;2 etc. to user, if different pw are used.
# - Nortel CLI capable devices may require to configure cmd-interface cli to avoid menus!
# user pass enablepass
;usr nedi pa55 enpa55
;usr admin enpa55
;usr edmin enterasys
;usr xmin extreme
usr admin;1 pwd1 enpwd1
usr admin;2 pwd2
usr admin;3 pwd3
usr admin;4 pwd4 enpwd1
usr admin;5 pwd1 enpwd2

the last fifth are other examples, where username is always admin (the default on our Cisco devices) and you can guess what you need to put instead of pwd# ed enpwd#.

If you try with -v verbose option and read logs, you should find your best configuration.

Paolo

rickli

  • Administrator
  • Hero Member
  • *****
  • Posts: 2697
    • View Profile
    • NeDi
Re: Network Discovery problem with 1.0.7
« Reply #4 on: February 15, 2012, 09:09:14 PM »
It can be applied as patch. Copy to patch.txt then:
n7:~#patch /var/nedi/inc/libcli-iopty.pm patch.txt

If the patching doesn't help add this at line 468. I'm curios what you'll see:

print "PRE :$pre\nMTCH:$match\n" ;
Please consider Other-Invoices on your NeDi installation for an annual contribution, tx!
-Remo

piet

  • Newbie
  • *
  • Posts: 31
    • View Profile
Re: Network Discovery problem with 1.0.7
« Reply #5 on: February 16, 2012, 06:33:45 PM »
I applied the patch as described and the
Quote
Use of uninitialized value in concatenation (.) or string at /var/nedi/inc/libcli-iopty.pm line 479
problem disappeard  :)

Unfortunately, the
Quote
Resource temporarily unavailable
problem still remain  :(

The last nedi output just before the error occur is:
Quote
Prepare (CLI)  ----------------------------------------------------------------
SSH :user1;1:22 Tout:2s OS:IOS-wl EN:(.+?)#\s?$
PTY :Forking ssh -o 'StrictHostKeyChecking no' -l user1 10.10.10.22
CLI2:Matched password:, sending password
CLI3:Password sent
CLI3:Matched denied, login failed
SSH :user1;2:22 Tout:2s OS:IOS-wl EN:(.+?)#\s?$
PTY :Forking ssh -o 'StrictHostKeyChecking no' -l user1 10.10.10.22
Resource temporarily unavailable at /var/nedi/inc/libcli-iopty.pm line 926.

Before I started the discovery, I initialized the database again, so this happened with an empty DB - but the device where the error happened is not the very first one but no. 43 in the seedfile. The devices 1 - 42 are discoverd without any problems.
After starting the discorvery process again, it happened with seed59...

I am not sure, where to enter the additional print command - in line 468 of the unpatched or of the patched libcli-iopty.pm?
Could you please advice?

Thank you so much again for your support!


rickli

  • Administrator
  • Hero Member
  • *****
  • Posts: 2697
    • View Profile
    • NeDi
Re: Network Discovery problem with 1.0.7
« Reply #6 on: February 16, 2012, 10:42:39 PM »
There should be a commented print line at 468 as well. You can just modify that one...

Did you see this?
CLI3:Matched denied, login failed

Is the pw set correctly? If you add -d you'll get input.log and output.log to trace what exactly gets sent and received...

I don't quite see, why you get the resource problem though. I do hope someone finds a fix to properly terminate the SSH pty!
Please consider Other-Invoices on your NeDi installation for an annual contribution, tx!
-Remo

piet

  • Newbie
  • *
  • Posts: 31
    • View Profile
Re: Network Discovery problem with 1.0.7
« Reply #7 on: February 18, 2012, 04:30:12 PM »
I added the print command in line 480 (after the commented print command line)

I use several different userids, passwords and different enable passwords for the devices in different locations - therefore it is not always the first userid - password combination, which works.

The users section of nedi.conf looks like this:
Quote
usr   user1;1   password1   enapw1
usr   user1;2   password1   enapw2
usr   user1;3   password1   enapw3
usr   user2;1   password2   enapw1
usr   user2;2   password3   enapw1
usr   user3   password4   enapw4
usr   user4   password5   enapw5

I run the discovery process again with a fresh empty database - the first time the error occurs with seed no. 43

I attached three parts from the total discovery output - one, where the first userid - password combination works, another one where the 4th combination works and the last one, where the error occurs (the 4th userid password combination would work with that device as well)


rickli

  • Administrator
  • Hero Member
  • *****
  • Posts: 2697
    • View Profile
    • NeDi
Re: Network Discovery problem with 1.0.7
« Reply #8 on: February 18, 2012, 07:32:06 PM »
With this many accounts I can see how the resource problem comes up, when trying all of them. There's hope it will dissapear once all logins are found.

Can you try restarting the discovery and have an eye on the progress each time?

Please consider Other-Invoices on your NeDi installation for an annual contribution, tx!
-Remo

piet

  • Newbie
  • *
  • Posts: 31
    • View Profile
Re: Network Discovery problem with 1.0.7
« Reply #9 on: February 24, 2012, 05:36:00 PM »
I run several tests again - but without success at the end  :(

But the whole story...

The availble patch has been applied.
I have several different seed files for diferent locations but just one nedi.conf. I tried to optimize the sequence of the userid - password list in nedi.conf and I got better results during the discovery process.
I started again with one specific seed file and with an empty database and the error appears later than before, when the userlist in nedi.conf was not optimized.
Every time I started the discovery process, the error appears several devices later and at the end, all devices of that seedfile are discovered without error.

Than I took the next seedfile and performed the same steps until all devices have been discoverd without errors.

I was happy and I thought, everything would be OK now.
But in real life, I run many discovery processes with different seed files in parallel and then I get the error again (Resource temporarily unavailable at /var/nedi/inc/libcli-iopty.pm line 927.)
The time nedi requires for a discovery of all devices of all locations one after another would be more than 24 hrs - but I would like to discover each location at least once a day...
But because of this problem, some locations are not completely discovered at all...

In addition, I receive the following error during the discovery process:
Quote
Use of uninitialized value in concatenation (.) or string at /var/nedi/inc/libsnmp.pm line 292.

Do you have any additional hint?



rickli

  • Administrator
  • Hero Member
  • *****
  • Posts: 2697
    • View Profile
    • NeDi
Re: Network Discovery problem with 1.0.7
« Reply #10 on: February 24, 2012, 07:18:59 PM »
line 292:
         $main::dev{$na}{cul}= "$misc::sysobj{$so}{cul};$misc::sysobj{$so}{mmu}";

So, can you have a look at the device's .def MemCPU entry (2nd argument is memory multiplier or mmu)?

I on the other will figure out why the heck I put it there  ;)

Now to your other problem. Open filehandles is a global number, thus running several discoveries in parallel will use them all up, apparently. I really do hope to find a fix for this. Maybe forking will solve it, but I guess 1oh8 will tell. Are you using SSH to just backup configs or also to get the bridge forwarding entries? Because in the first case, you could backup configs during the night in a single run, while Skipping all other information (see crontab for an example).
Please consider Other-Invoices on your NeDi installation for an annual contribution, tx!
-Remo

piet

  • Newbie
  • *
  • Posts: 31
    • View Profile
Re: Network Discovery problem with 1.0.7
« Reply #11 on: March 01, 2012, 04:35:03 PM »
I did some manual optimization in the crontab to reduce parallel discovery as much as possible - is better now but sometimes, the problem still occurs.

I use SSH for getting bridge forwarding entries only (getfwd is set to dyn in nedi.conf). If I set that to SNMP only (getfwd snmp), do I loose information?

I wanted to get rid of that problem, before start backing up configs.


rickli

  • Administrator
  • Hero Member
  • *****
  • Posts: 2697
    • View Profile
    • NeDi
Re: Network Discovery problem with 1.0.7
« Reply #12 on: March 01, 2012, 10:23:08 PM »
It'll be slower, if you have a lot of vlans on those switches. With getfwd sec you could also read port-security (static MAC) entries, which doesn't seem very popular these days...so I'd say you don't lose anything...
Please consider Other-Invoices on your NeDi installation for an annual contribution, tx!
-Remo

quartz

  • Guest
Re: Network Discovery problem with 1.0.7
« Reply #13 on: March 27, 2012, 12:49:18 PM »
It can be applied as patch. Copy to patch.txt then:
n7:~#patch /var/nedi/inc/libcli-iopty.pm patch.txt

If the patching doesn't help add this at line 468. I'm curios what you'll see:

print "PRE :$pre\nMTCH:$match\n" ;


Could someone explain to an absolut windows nerd where to get and how to install that patch? It seems I have the same issue here on my site.

quartz

  • Guest
Re: Network Discovery problem with 1.0.7
« Reply #14 on: March 27, 2012, 02:07:24 PM »
Found the Patch on the main site and got it installed with the description there.
Sorry for to quick questioning.