NeDi Community

NeDi General => Other => Topic started by: steffen1 on July 29, 2009, 11:22:43 AM

Title: SNMPv3 solved, Bugfixes, Nagios-Event-Propagation and more
Post by: steffen1 on July 29, 2009, 11:22:43 AM
Servus Community,

Did some bugfixes and extensions in the past to nedi, that was necessary for my last projects dealing with nedi. Thats are 3 libs for v1.0.4 and 2 libs for v1.0 (less features, than described below), that should replace the original libs located at ./inc - see attachments.

I hope and wish, that NeDi's father will take the fixes and features to the offical code  :)

Extended libdb-msq:
1. to can propagade NeDi messages into nagios,
2. sub WriteInt to avoid false positives in virtual and unpredictable bandwitch interface alerting
3. write config to file in case the file has been lost before
4. filename is none blanked now (strftime)
5. dont delete Links in case of adding a single device

Extended libsnmp:
1. sub Enterprise => Memory message + KB representation
2. restructuring of SNMP-session => New sub openSNMP
   + more fault tolerance and output in case of failed SNMP-session creation
3. get SNMPv3 working + sensing v2/v3, only 1 Rule if mixing v2/3:
   same v2-Community may not be a v3-Username!
4. more fault tolerance within .def file parsing: strict \t has been replaced
    by any whitespace characters. E.g. ASA-def (670) within v1.0.4 package will not work
    with the original strict \t parsing, because the version sensing is initiated by
    .def files.

Extended libmisc:
1. sub WriteCfg: filename is none blanked now (strftime)
2. sub InitSeed:
   - more fault tolerance, strict \t has been replaced by any whitespace characters
     and 3rd field will be read as initial SNMP version
   - -A => will set initial Community and initial SNMP version to avoid self generated
           AuthTraps in environments with plenty communities and mixed v2/v3


Q: What is the reason to have a strict \t instead of \s+ parsing for nedi.conf?


Steffen

P.S.: if you used the attached nedi-libs-netmon -2009-07.zip please update with the renewed one from today (2.9.2009)
nedi-libs-netmon -2009-09.zip. There were some issues with libsnmp.pl and moni.pl is v3 able as well now.

bugfix-1: avoid v1 fallback after failed v3
bugfix-2: more efficient and relyable probing
bugfix-3: v3-auth (not-priv) is now working as well
bugfix-4: All nodes has seen at the device that own the Default Gateway
             forgot one very special thing while restructuring of SNMP-session creation:
             Bridge Forwarding MIB opens VLAN based views only, if you extend the VLAN-number
             after the community or V3-user, e.g.: You will get the nodes in VLAN701 only,
             if you query the device using public@701 instead of public.
Title: Re: SNMPv3 solved, Bugfixes, Nagios-Event-Propagation and more
Post by: rickli on July 29, 2009, 06:35:31 PM
Thanks steffen, will certainly look at it when I find the time...darn busy job!
Title: Re: SNMPv3 solved, Bugfixes, Nagios-Event-Propagation and more
Post by: rickli on August 26, 2009, 12:30:21 AM
I started with strict \t because of the disclaimer (per request) and ported the format to the seedlist and .defs in order to allow spaces within values as well. It seem most of the users accepted it...
Title: Re: SNMPv3 solved, Bugfixes, Nagios-Event-Propagation and more
Post by: steffen1 on October 09, 2009, 10:57:43 AM
attached my 1st try to make the PHP frontend SNMPv3 able as well beside some cosmetic bugfixes in libsnmp.pl and moni.pl.
Title: Re: SNMPv3 solved, Bugfixes, Nagios-Event-Propagation and more
Post by: steffen1 on January 19, 2010, 07:36:00 PM
attached the version compatible to v1.0.5 and a update to v1.0.4. Cant upload a complete bundle compatible to nedi from v1.0 due to the upload limit of 128KB.
Title: Re: SNMPv3 solved, Bugfixes, Nagios-Event-Propagation and more
Post by: steffen1 on May 31, 2010, 03:10:33 PM
Nedi extensions 06/2010 - v1.0.5 based:
   - improved LLDP implementation
   - improved Juniper support (modules and changetracking works now), recalculation of that many virtual interfaces to it physical ones that Juniper (is completly different to other vendors at this planet) produces.
   - moved System Monitoring (CPU,Mem,Tmp,MemIO,Custom) from Discovering to Moni-Daemon. With this decoupling we do Discovering one time per day and Monitoring each 5 Minutes (better granularity for CPU utilisation). To avoid a new side effect: I started a device locking mechanism via DB, to skip CPU alarms during nedi discovering of that particular devices. Because you can get false posites at CPU-Utilisation due to the CPU overload caused by nedi discovering itself with some sensible network devices.
   - endstation tracking: extended Bridge-MIB-Implementation with an additional method, used for instance by juniper as well. Got idea from Cacti-MacTrack Juniper-Modul :)
   - did implementation of VLAN- context's from Bridge-MIB SNMPv3 konform .
   - modules and tracking of that now possible with devices using a multi dimensional module index, e.g. Juniper
   - additional Key-Words in nedi.conf:
     a) noalert-vips   172.31.*.15 => to can switch off duplicate IP-adresse checking for VIP's, e.g. Multicast-RendezVousPoints, VRRP, HSRP - as RegExp
     b) mac-logfile   macs.log => for troubleshooting of all nedi ARP and MAC activities
     c) rrd-ifs   no => possibility to can deactivate interface monitoring and storing to RRD's

p.s.: there are some bugfixes and extensions at the attached lib-image from 2010-06-08.
Title: Re: SNMPv3 solved, Bugfixes, Nagios-Event-Propagation and more
Post by: rickli on June 10, 2010, 10:11:35 PM
Thanks! I'll use this as a base for the new snmplib. But it's hard to keep up with all you guys contributing those great additions to the project  :-[

Vacation will help!!!
Title: Re: SNMPv3 solved, Bugfixes, Nagios-Event-Propagation and more
Post by: steffen1 on June 11, 2010, 11:16:43 AM
yes, I can imagine that it will take a lot of your time to check the ideas from contributrors beside your own ideas. But I'm glad that you take this time and it makes me also happy, that parts of my code could be a part of your great nedi.

But do me a favor: Please take the updated lib-image from 2010-06-08 (I recognized downloaded 0 times), because ther are some bugfixes compared to the first one from 2010-06-01.

Steffen
Title: Re: SNMPv3 solved, Bugfixes, Nagios-Event-Propagation and more
Post by: rickli on June 15, 2010, 06:04:16 PM
Ok, will do. You see I hadn't even time to download it yet. Was in Paris a couple weeks ago and I'm off to Geneva tomorrow...another week gone! Hopefully I can use summer vacation to wirelessly make some progress (meaning I sit by the pool, connecting to the lab in my basement, or so)  8)
Title: Re: SNMPv3 solved, Bugfixes, Nagios-Event-Propagation and more
Post by: tristanbob on June 19, 2010, 12:20:24 AM
Steffen1,

Thanks for your contribution to Nedi!  This is a very useful project and we appreciate all assistance and communication.  I think I have been using Nedi for over 5 years and we still use it on a daily basis!

Remo, any consideration into using a distributed versioning tool, something like Launchpad?  This way you can work on your code, and developers like Steffen can create branches, then you can consider merging the branches. (DISCLAIMER: I am not a programmer)

https://launchpad.net/

Tristan
Title: Re: SNMPv3 solved, Bugfixes, Nagios-Event-Propagation and more
Post by: steffen1 on June 23, 2010, 03:48:18 PM
Yes, Tristan, there are certainly some opportunities to can improve code sharing and the way of how to develop together between contributers and main developer. But its not only a question of tool or platform we use, more such things as how we agree each other. Each of us has different priorities and ideas what is important for nedi and how many time to can spend in developing. I think we are at the first step and at a good way organizing the work for nedi more centralized e.g. with any kind of SVN in remote future. But consider also we do this in our sparetime and to establish the SVN platform and organization will also consume some time. In the step now I'm glad to have this simple possibility in this forum to can contribute and to have a chance that provided features without we cant use nedi will get into the official release.

Steffen

------------------------------------ Part-2 -----------------------

The attached Image contains the  more advanced version of my Nedi-Netmon libs with the try to create MAC-based links. The libs are working at the same level as before, but the new feature is not ready - I think short before. You will see it in the summary statistic: "MAC Links: 489 mac adjacensies processed, 0 created\n". Most changes for this I did in libmisc.pl. I post it in the hope :o that any other contributer or Remo is interested in this feature as well and has ideas to can share to get this working.

We use NeDi in consulting situations and that feature will boost the opportunites for using nedi in several situations and networks where you are not allowed to make changes a lot. Because with this feature, NeDi will be independend from deployed Layer2 neighborship protocols as CDP or LLDP and its completeness in the networks. The method to create MAC-based links must work, because its the primary method for creating adjacencies between devices in commercial products in this sector (e.g. EMC Smarts, netXpose ...) and a big drawback in nedi currently compared to this tools.

Steffen

Title: Re: SNMPv3 solved, Bugfixes, Nagios-Event-Propagation and more
Post by: rickli on June 27, 2010, 11:42:20 AM
Thanks Tristan for your input and thanks Steffen for answering with one of my reasons  :) Indeed I change most of the contributed code to fit my needs. Assuming it still works for everyone (especially the contributers)...

Also many thanks Steffen, for your continued efforts! I actually dropped MAC based links in favor of parallelizing in 1.0.5 due to it's (un)reliability. I'll look at your ideas. There might be a way to get it back in along the nodes calculation...
Title: Re: SNMPv3 solved, Bugfixes, Nagios-Event-Propagation and more
Post by: steffen1 on June 29, 2010, 08:31:04 AM
Yes, Remo, I mixed the MAC based link calculation under the nodes calculation. ;)

I think I've got (suddenly) the clue after watching the node table of a network and recognizing that there are a lot of devices despite having not set option -N.

This (I use the negative of this in MAC based links) could better work, if you read the entire interface table (from previous discoveries) into the appropriate hashes before network discovery to complete this decision with out timed MAC entries as well:
Code: [Select]
if (!exists($ifmac{$mc}) and !exists($ifip{$arp{$mc}}) == 0 or $main::opt{N}) { # Don't use devices as nodes unless desired
...
if(exists $ifmac{$mc}){         # with none IP nodes

I'll try it after having more time for nedi again, Steffen
Title: Re: SNMPv3 solved, Bugfixes, Nagios-Event-Propagation and more
Post by: steffen1 on July 14, 2010, 11:28:11 PM
Attached some bugfixes (libsnmp & libmisc only). The improved behaviour for Juniper devices and the desire to make everything autosensing inside the code without touching existing defs, opened some bugs in endstation tracking (nodetable) for cisco devices compared to the original nedi libs.

Hope all bugs has been gone with attached version.
Title: Re: SNMPv3 solved, Bugfixes, Nagios-Event-Propagation and more
Post by: wilga on August 26, 2010, 09:28:59 AM
I tried the updates in nedi-libs-netmon-2010-07-14.v105, hoping to get it working for my Juniper devices.

it seems to have stopped the CDP-based discovery as only the items statically defined in the seedfile are still discovered.  How can I get back the CDP-based autodiscovery?

Thanks,

my discovery is normally running with
nedi.pl -op

My Juniper devices were discovered & worked for the first time, which is great
The only problem is CDP discovery seems to be no longer working

This is the output from a manual rediscovery with -v
Start link discovering by CDP:
 CDP: 0 neighbors detected.
IF:     154 interfaces written to nedi.interfaces
MOD:    4 modules written to nedi.modules
VLAN:   24 vlans written to nedi.vlans
NET:    24 networks written to nedi.networks
LINK:   0 (ignoring 0 static) links written to nedi.links
           0/1-3s



Title: Re: SNMPv3 solved, Bugfixes, Nagios-Event-Propagation and more
Post by: steffen1 on September 15, 2010, 04:25:33 PM
Hello Wilga,

To get Juniper devices working you will need the Juniper device packages as well, see:

http://forum.nedi.ch/index.php?topic=537.0

For Link discovery you will need to setup LLDP between Juniper devices. Because CDP is Cisco properitary, you cant get Juniper devices working with CDP.

Steffen
Title: Re: SNMPv3 solved, Bugfixes, Nagios-Event-Propagation and more
Post by: rickli on October 09, 2010, 01:07:41 PM
I just read Steffens post again about loading the dev macs from the DB. Coincidently I added this recently for other reasons 8) You're absolutely right, this will be of great value for the topology calculation....and even remove another hurdle for threaded discovery!
Title: Re: SNMPv3 solved, Bugfixes, Nagios-Event-Propagation and more
Post by: steffen1 on December 03, 2010, 04:55:05 PM
little update to netmon-nedi-libs:
sub WriteDev@libdb-msq: detect rename of devices and fix it, to avoid duplicate IP's.
see attached file
Title: Re: SNMPv3 solved, Bugfixes, Nagios-Event-Propagation and more
Post by: rickli on December 04, 2010, 06:10:51 PM
I've added mapna and maplo to cover renamed devices and fix those without location info. Probably, not as elegant as your solution...

BTW context really belongs to get calls not to creating the session. A very security minded "customer" showed me his approach. Just got v3 read access working now! I still have to find a suitable oid, for testing write access...
Title: Re: SNMPv3 solved, Bugfixes, Nagios-Event-Propagation and more
Post by: steffen1 on December 04, 2010, 10:24:14 PM
one idea for an OID: whats abaut the location string in system MIB? So you can also control the rollout of SNMP location from NeDi with SNMPv3.
Title: Re: SNMPv3 solved, Bugfixes, Nagios-Event-Propagation and more
Post by: rickli on December 05, 2010, 12:19:33 AM
I'm getting pretty good results now with changing snmpEnableAuthenTraps. I can also set ifadmin status in Devices-Status as promised 8) (Far from being finished, though)
Title: Re: SNMPv3 solved, Bugfixes, Nagios-Event-Propagation and more
Post by: steffen1 on December 21, 2010, 06:45:50 PM
Hello,

I did some new staff and fixed some older issues into the netmon-nedi-libs, see attached zip, most importand changes&news:
1. Management-IP-Selection is not dependend on a network loopback design, new Prios: 1. pingable, 2. SysName=DNS/etc-hostsname, 3. loopback
2. Netfilter could be combined with CIDR (new, dependend to Net::CIDR::Lite) and RegExp (backward compatible) ';' seperated, see nedi.conf in zip
3. MAC-based links are working now
4. Reports-Devices.php=>Unlinked: show all isolated devices and linked devices that are not discovered yet
5. new nedi options: -S (skip Bridge Forwarding), -P programmers opt to can quicker optimize some post processing, -O to make something only, e.g. Backup Configs
6. for large CDP/LLDP Link-tables: increased SNMP-max_msg_size to 64K
7. fixed SNMPv3 context issue, thx Remo for this hint
8. auto detection and cleanup of device renaming to avoid duplicate management IP's in nedi
---
9. started new idea (not finished yet): link isolated cloudes of devices with L3 links.

merry christmas, Steffen

Title: Re: SNMPv3 solved, Bugfixes, Nagios-Event-Propagation and more
Post by: paedi on October 24, 2011, 06:30:33 PM
Hi Steffen
I used Nedi in a Cisco environment at my former company and it worked very well. I now changed to a company that has a "Juniper-Only" environment. I'm therefore very much interested to use your files. I'm wondering whether the files you uploaded (mainly 1.0.5) are compatible with 1.0.6? I have to say, so far I am using Nedi without any of your files.
The switches in our environment are setup as several virtual chassis, I can imagine this being a nightmare for Nedi. I.e. the hosts all show as connected to a reth interface on the virtual chassi, rather than to what physical switch, this is not much of a help. ;-) Is there a readme.txt file coming with your files? I was wondering where I need to copy your files to and whether I need to rename them from something like Devices-Graph.v1-0-5.php to Devices-Graph.php. This probably depends on how you have included them in your nedi.conf file. Have you a already a "remedy" for the non existing Juniper Modules? With cisco devices, I had a very nice view of all the modules in the switches/routers.
Thanks for your contribution Steffen to support Juniper devices anyway. And of course thanks to Remo as well. ;-)
Patrick
Title: Re: SNMPv3 solved, Bugfixes, Nagios-Event-Propagation and more
Post by: steffen1 on November 03, 2011, 11:31:39 PM
Hi Patrick,

Attached current version of netmon_nedi-libs including readme, install script, device files, icons ...
You will need 7zip to can extract this 2 zip/7z images.

Steffen

P.S.: @Remo is it possible to allow 7z as file type having much higher compression ratio than zip?
Title: Re: SNMPv3 solved, Bugfixes, Nagios-Event-Propagation and more
Post by: rickli on November 18, 2011, 06:30:31 PM
Sure, if I find out how to configure this :)
Title: Re: SNMPv3 solved, Bugfixes, Nagios-Event-Propagation and more
Post by: steffen1 on December 18, 2012, 05:53:26 PM
Hello,

After some time I'll come back with a complete reworked version of nedi-netmon extensions compatible to v1.0.7 now and certainly new for you a config controlled complete propagation into Nagios/Icinga/OMD Monitoring, that Im called nedi2nagios.
The concept of this stage is to seperate 2 main competencies of Network Management into their leading OpenSource tools:
- NeDi          => Config&Inventory-management,reporting,discovering,Network-Knowledge-Base,Endstation-tracking
- Nagios&AddOns => Monitoring&Alerting

Main part of the nedi-netmon extensions rework was to concentrate all changes and extensions in the libnetmon.pm as far as possible.
To be OMD multisite capable it isnt possible 100% currently. So also some other nedi files need to be replaced (by the installer),
but with just very light changes.

The installation is handled by attached image containing nnedi_install.sh (-? for help) that is covering also the netmon extensions to previous nedi versions. But keep in mind only v1.0.7 will be compatible to OMD based Nagios/Icinga.
Nedi and Nagios need to be installed before.

Some hints for installation and usage could be found in the 2 readme's inside the attached image.

Steffen
Title: Re: SNMPv3 solved, Bugfixes, Nagios-Event-Propagation and more
Post by: 123ralle on February 19, 2015, 08:30:32 AM
Hallo Steffen,

very good work and i'am quite interested in the integration.
Is there a chance that the whole thing is adapted to the current version of Nedi? (1.4)

Lieb Gruesse,
Ralf