Welcome, Guest. Please login or register.

Author Topic: Play with NeDi  (Read 3477 times)

oxo

  • Guest
Play with NeDi
« on: January 11, 2010, 11:39:06 PM »
What started as a little fix to speed up a 5 min NeDi to collect RRD data has turned out to be a major rewrite.
I achieved 10000 rrd (int) per minute after a major rewrite (using SNMP for BridgeFwd), using RRDs instead of system calls to create/update RRD's and logic that only updated RRD's in the 5 min run  (using only the code that gave data for the RRD). Other speedups where:
- removal of hash copies when making snmp enquires
- single create dbh for a run and single snmp session for a device.
http://odp.bzr.sourceforge.net/bzr/odp/annotate/head%3A/others/nedi/branches/nedi-1.05rc04oxo/bin/nedi.pl

At the present moment, I am rewriting the db module to not use positional SQL returns but use hash so I that in the end, I have a data dictionary that shows the intermal perl names for the table entries used. This will give me a better chance when returning to the main and SNMP logic.
http://odp.bzr.sourceforge.net/bzr/odp/annotate/head%3A/others/nedi/branches/nedi-1.05rc04oxo/lib/NeDi/db.pm
http://odp.bzr.sourceforge.net/bzr/odp/annotate/head%3A/others/nedi/branches/nedi-1.05rc04oxo/var/share/neditables.sql

The code is still compatible with the html (which is only slightly modified) and has no support for CLI for BridgeFwd or backup as yet.

I will be using the finished code in 3 installations when I am finished.

It is my intention that code can be installed as a perl module, just like any other module. So the structure is different from a standard NeDi (lib instead of inc etc etc).
This means one can make a little perl script and
use NeDi;
http://odp.bzr.sourceforge.net/bzr/odp/annotate/head%3A/others/nedi/branches/nedi-1.05rc04oxo/bin/test.pl

« Last Edit: January 11, 2010, 11:47:14 PM by oxo »

rickli

  • Administrator
  • Hero Member
  • *****
  • Posts: 2777
    • View Profile
    • NeDi
Re: Play with NeDi
« Reply #1 on: January 12, 2010, 12:27:00 AM »
Hope we find a way of adding this as a discovery run just focusing on the RRDs. No forwarding or anything else. Plus it should be completely optional to my slooOow (and quiet) approach... Of course all my statistic stuff needs to be looked at  :o

What do you mean with this?

Quote
using SNMP for BridgeFwd), using RRDs instead of system calls


For the latter I guess you're using rrd.pm rather than my system call approach? Which is probably much faster?

Just wondering do you write an RRD in any case (e.g. delta is 0)?
Please consider Other-Invoices on your NeDi installation for an annual contribution, tx!
-Remo

oxo

  • Guest
Re: Play with NeDi
« Reply #2 on: January 12, 2010, 07:03:00 PM »
I forgot to write that a "standard" nedi run that does cam tables etc etc takes 6 min for 10000 interfaces
A -Q option just creates the rrd files (hourly run), -R updates (5 min run).
0 delta values are also stored.

RRDs is http://oss.oetiker.ch/rrdtool/prog/RRDs.en.html which gives a great speed increase withing the current NeDi 105 code, without altering anything else...
However, each of the changes I have made help speed up the code (even my rewrite of Identify).

I have only implemented collecting cam tables by SNMP (no cli for me).
TopRRD is also on hold ...

I am aiming for that all files (def, oui)  should be stored in MySQL tables
- that means when I get multiple NeDi machines running, I don't need to keep the files in sync.
(Oh, and if you look at the DB creating SQL, you will see a rrd definition: now when I get that working you will see something that is very strange ... can you guess what? )

How about a meeting in february, which should have given me time to round off some of the edges of my code (only been writing it for a month) ?
« Last Edit: January 12, 2010, 07:11:22 PM by oxo »