Welcome, Guest. Please login or register.

Author Topic: Parallel discovery on different networks  (Read 3366 times)

cmesser

  • Jr. Member
  • **
  • Posts: 80
    • View Profile
Parallel discovery on different networks
« on: November 03, 2009, 10:26:32 AM »
Hi,

We are glad to have such a great tool like nedi at hand and we use it in our LAN for a year now.

But I'm working in a global  company, and it takes too long to discover all LANs with a seed for every
location in the seed list.
Is it possible to run different nedi discoveries at the same time but for different LANs?
I wonder what happens to the database if I try this.

rickli

  • Administrator
  • Hero Member
  • *****
  • Posts: 2691
    • View Profile
    • NeDi
Re: Parallel discovery on different networks
« Reply #1 on: November 03, 2009, 06:21:04 PM »
As of 1.0.5 the links will be updated per device as well (I dropped MAC and OUI based "guessing" algorithms as that was never reliable anyways). This means the nodes are the only shared table within a discovery run.

You could run several nedi processes at the same time, as long as the locations don't "see" each other (use border in that case). Nodes shouldn't sacrifice accuracy as long as there's no Layer 2 connections between your locations. Make sure they don't finish at the same time or a node update might get lost (this is something I could address, if this approach works well!)

Create a seedlist for each instance you want to run. How many is up to you really, but I'd start with 2 and go from there. Remember the more parallel runs the higher the load on the machine and links (plus less time savings with higher number of runs) e.g.:

1 thread takes 6h
2 threads take 3h
3 threads take 2h
4 threads take 1.5h

...plus slowdown due to higher load.

How many devices, nodes and locations are we talking about, anyway?

I haven't tested this myself, so you're on your own, but it might definitely be worth a try...

Parallel discovery is an old story and it's a pity we're still not there, but it's tricky stuff ;-)
Please consider Other-Invoices on your NeDi installation for an annual contribution, tx!
-Remo

cmesser

  • Jr. Member
  • **
  • Posts: 80
    • View Profile
Re: Parallel discovery on different networks
« Reply #2 on: November 04, 2009, 11:03:49 AM »
Hi Remo,

what a quick reply! A little something from your wish list is on its way ;)

You are talking about 1.0.5. Is it near to be released? I run 1.0.4.
Should it be possible with this version too?

What we talking about is a company with roundabout 150 locations, from big to
small ones. I guess with about 5000 devices. But due to NeDi isn't an official tool for us (up to now :-)
I started at a location with 800 devices.
(And also, I'm not that crazy to try to discover all locations with one server)

By the way, our switches are all configured for SSH2 and I'm looking forward to a version that
will support that.


rickli

  • Administrator
  • Hero Member
  • *****
  • Posts: 2691
    • View Profile
    • NeDi
Re: Parallel discovery on different networks
« Reply #3 on: November 07, 2009, 07:03:46 PM »
Hey thanks, coincidentally I kept thinking about this and started implementing a locking mechanism for the nodes table, to make sure parallel discoveries don't interfere. I also think it's a better approach to start nedi threads manually as it can be controlled much better avoid risks of filling up a link etc...

Hmm, 5k devices? That's gotta be a new record!!! How long do those 800 devs take?

SSH2 and SNMPv3 is something I want to attack for 1.0.6...
Please consider Other-Invoices on your NeDi installation for an annual contribution, tx!
-Remo

cmesser

  • Jr. Member
  • **
  • Posts: 80
    • View Profile
Re: Parallel discovery on different networks
« Reply #4 on: November 09, 2009, 11:32:56 AM »
I'm a little bit puzzled about your answer  :-\
Should it be possible with 1.0.4 or do I have to wait for 1.0.5 ?

The other thing:
I excluded the LWAP APs so I end up with ca. 720 devices to discover.
Also (because of SSH2) I'm not pulling the configs.
This task takes about 75 minutes.

rickli

  • Administrator
  • Hero Member
  • *****
  • Posts: 2691
    • View Profile
    • NeDi
Re: Parallel discovery on different networks
« Reply #5 on: November 09, 2009, 06:08:04 PM »
Sorry for being not clear enough. I implemented the locking mechanism with 1.0.5 (which is not available yet). In the previous version (1.0.4), links are also calculated at the end of the discovery, just like the nodes. This has changed in 1.0.5 as well for the same reason.

I understand LWAP APs don't support SNMP anyways, so you're not going to loose much there. If you match them with nosnmpdev,  they still show up as devices (Just not actively being polled).

I reckon with 3 parallel discoveries, you should be well on your way to use an hourly schedule with plenty of slack...

Have you tried pulling configs? That part should actually work already...
Please consider Other-Invoices on your NeDi installation for an annual contribution, tx!
-Remo

cmesser

  • Jr. Member
  • **
  • Posts: 80
    • View Profile
Re: Parallel discovery on different networks
« Reply #6 on: November 11, 2009, 10:13:36 AM »
Hi Remo,

now I know what you are talking about. I tested with 3 parallel discovers.
It worked very well beside the links. These are empty at the devices that are discovered
in the two jobs that are finished first.
So I'm looking forward to install 1.0.5 soon  ;D

The other point: I'm fine with discovering the LWAPs as nodes, because we still have some
IOS APs left (for a bridge) that I want to discover. Unfortunately I can't distinguish between LWAP and IOS.

Pulling configs  should work? I thought it needs shell access? And it doesn't worked with 1.0 .
I'll give it another try  :)  ........
Doesn't work :( I guess because of the SSH2 issue.
« Last Edit: November 11, 2009, 11:06:09 AM by cmesser »

cmesser

  • Jr. Member
  • **
  • Posts: 80
    • View Profile
Re: Parallel discovery on different networks
« Reply #7 on: November 20, 2009, 11:01:32 AM »
I tested parallel discovery with the RC, and it works well  :)
Maybe 1oh5 is the staring point to use NeDi in more of our locations,
and possibly it will become a standard tool for us in the future.

Thanks a lot!

rickli

  • Administrator
  • Hero Member
  • *****
  • Posts: 2691
    • View Profile
    • NeDi
Re: Parallel discovery on different networks
« Reply #8 on: November 20, 2009, 05:24:57 PM »
Beautiful! Hope it stays that way...

Oh and thanks for the donation!
Please consider Other-Invoices on your NeDi installation for an annual contribution, tx!
-Remo