Welcome, Guest. Please login or register.

Author Topic: rrdstep and gaps in interface and network total graphs  (Read 2575 times)

jonesamu

  • Guest
rrdstep and gaps in interface and network total graphs
« on: October 16, 2013, 08:24:20 PM »
Rickli,

Having trouble with gaps in interface and network totals graphs.  I have been adjusting the rrdstep but I have not been too successful.  Can you explain how rrdstep affects the graphs?  Also is the rrdstep affected by the number of threads running?

When the rrdstep was set too 1800 (30mins) network total graphs seemed to be fine but the interface graphs had gaps especially on the interfaces in a down status.  When I changed to 1200 (20mins), or 2700 (45mins) the network total graphs do not update or they stagger in amount of devices.

We run a discovery of the network once a hour on the hour:
Code: [Select]
0 1-23 * * * /var/nedi/nedi.pl -vp -SmgafoWjitedbwAO > /tmp/nedi-`date +\%H`.run 2>&1

We then parallelize a gather of found devices every 20 mins 5 mins after the hour:
Code: [Select]
5,25,45 * * * * /var/nedi/nedi.pl -vA 'RIGHT(INET_NTOA(devip), 1) IN (0, 5)' > /tmp/nedi-`date +\%H\%M\A`.run 2>&1
5,25,45 * * * * /var/nedi/nedi.pl -vA 'RIGHT(INET_NTOA(devip), 1) IN (1, 6)' > /tmp/nedi-`date +\%H\%M\B`.run 2>&1
5,25,45 * * * * /var/nedi/nedi.pl -vA 'RIGHT(INET_NTOA(devip), 1) IN (2, 7)' > /tmp/nedi-`date +\%H\%M\C`.run 2>&1
5,25,45 * * * * /var/nedi/nedi.pl -vA 'RIGHT(INET_NTOA(devip), 1) IN (3, 8)' > /tmp/nedi-`date +\%H\%M\D`.run 2>&1
5,25,45 * * * * /var/nedi/nedi.pl -vA 'RIGHT(INET_NTOA(devip), 1) IN (4, 9)' > /tmp/nedi-`date +\%H\%M\E`.run 2>&1

nedi version 1.0.8-116

Thanks for a great product!
Sam Jones
« Last Edit: October 16, 2013, 08:29:48 PM by jonesamu »

rickli

  • Administrator
  • Hero Member
  • *****
  • Posts: 2780
    • View Profile
    • NeDi
Re: rrdstep and gaps in interface and network total graphs
« Reply #1 on: October 16, 2013, 10:14:24 PM »
First of all, rrdstep is set in the rrdfile upon creation. If you change discovery intervals/rrdstep after that, you'll need to recreate the rrds...

In general you need to make sure the previous discovery finishes, before the next one starts or you'll never see topgraphs (as they're only drawn, if 1 thread is detected).

When parallelizing, you should check how long each single thread takes (in your example hopefully less than 20 min). I'd even go for half the interval time, to make avoid problems when a lot of unreachable devices are encountered.

How many devices do you have (are they stacks, chassis)?
How long does a single discovery take?
Please consider Other-Invoices on your NeDi installation for an annual contribution, tx!
-Remo

jonesamu

  • Guest
Re: rrdstep and gaps in interface and network total graphs
« Reply #2 on: October 16, 2013, 10:28:34 PM »
Quote
How many devices do you have (are they stacks, chassis)?
How long does a single discovery take?

We discover just over 500 devices with less than 15 stacks or chassis.
The hourly discovery takes about 20-25 mins.

The longest gathering thread is 15 mins if we get no node locks.

Is there a limit on how many threads can run at one time?

rickli

  • Administrator
  • Hero Member
  • *****
  • Posts: 2780
    • View Profile
    • NeDi
Re: rrdstep and gaps in interface and network total graphs
« Reply #3 on: October 17, 2013, 06:40:41 PM »
I know of an installation running 80 threads in parallel, so no :)

You could skip the modules during the day (-Sm) and discover them with the configs. This should make it a bit faster and avoid false "module removed" messages on busy devices.

Are you using Cisco devices? Do you read forwarding tables via CLI, if so? That should speed up things considerably as opposed to SNMP.

I'd generally recommend using hourly discoveries in networks this size. Use Cacti or the likes to get 5min. graphs of more important interfaces.
Please consider Other-Invoices on your NeDi installation for an annual contribution, tx!
-Remo

jonesamu

  • Guest
Re: rrdstep and gaps in interface and network total graphs
« Reply #4 on: October 17, 2013, 07:11:07 PM »
Quote
Are you using Cisco devices?
Do you read forwarding tables via CLI, if so?

Most if not all our devices are Cisco.

I believe so but how would I check to make sure that the forwarding tables are checked via CLI and not SNMP?

rickli

  • Administrator
  • Hero Member
  • *****
  • Posts: 2780
    • View Profile
    • NeDi
Re: rrdstep and gaps in interface and network total graphs
« Reply #5 on: October 18, 2013, 09:41:58 AM »
With -v, if you see lines starting with FWDS: it's SNMP and if FWDC: it's CLI :)

Please consider Other-Invoices on your NeDi installation for an annual contribution, tx!
-Remo

jonesamu

  • Guest
Re: rrdstep and gaps in interface and network total graphs
« Reply #6 on: October 18, 2013, 04:33:31 PM »
Quote
With -v, if you see lines starting with FWDS: it's SNMP and if FWDC: it's CLI

How do I change from SNMP to CLI?

Also many of the switches I am checking I get this error:
Code: [Select]
DISC:Cli bridge fwd = no working user
which then nedi proceeds to check the forwarding table with SNMP.

What can I do to resolve this to speed up my discovery?
« Last Edit: October 18, 2013, 04:57:06 PM by jonesamu »

rickli

  • Administrator
  • Hero Member
  • *****
  • Posts: 2780
    • View Profile
    • NeDi
Re: rrdstep and gaps in interface and network total graphs
« Reply #7 on: October 18, 2013, 10:38:35 PM »
You'll need to add user and password in nedi.conf. Then in Devices-Status reset CLI access (click on key) and backup the config (2nd radar icon). If this works, you should see FWDC during the next discovery. I forgot to mention that the speedup depends on the number of vlans on the Cisco device is NeDi has to iterate through them with SNMP...
Please consider Other-Invoices on your NeDi installation for an annual contribution, tx!
-Remo