Welcome, Guest. Please login or register.

Author Topic: Idea for a more scaleable definition file system  (Read 5725 times)

steffen1

  • Guest
Idea for a more scaleable definition file system
« on: December 23, 2009, 07:02:02 PM »
Hallo,

For nearly one year ago I  got an idea how to structure device support in a more efficient way as many tools and nedi too does. It's a design in a tree structure using similarities between different models in one device family with the aim to break the linearity between plenty of definition files and devices to support.
I used this idea in a own system for excel/perl based asset scanning/deployment/config-compliance, we called NCA (Network Configuration Assessment). Because we describe CLI-commands and SNMP-OIDs in one file for the norming stage it will not be compatibel into NeDi as it is. But the idea could be helpful for quicker growing device support in NeDi, without the need to change the def-files, but only the way to interpret the def-files in the parsing code. Lets explain from an example:

We have some root def's to specify the device families, e.g.:
9.5.def Cisco-Cat
9.1.def Cisco-IOS
2272.def Nortel Passport
45.def Nortel S5-Agent or the older Bay staff
45.6.def Nortel Trapeze (its strange, completly different devices - WLAN-controller, that have completly nothing to do with S5-Agent, but are located at the same tree - so we need this tagged as a root-def)
...


And we have some none root def's (e.g. 9.1.113.def), to explain exceptions of special device models, that do not inherit all properties from its root-def. At the none root def's only the properties that are different from its device family root must be defined - all other properties will be inherit and unsupported properties will be tagged, e.g. by '-'.

All defs of models that completly behave as it's device family root or an other modified model will become redundant.

The result will be much less definition files and an autosupport for newer devices.
How you will find this idea for nedi?

Steffen


« Last Edit: December 23, 2009, 07:55:18 PM by steffen1 »

rickli

  • Administrator
  • Hero Member
  • *****
  • Posts: 2728
    • View Profile
    • NeDi
Re: Idea for a more scaleable definition file system
« Reply #1 on: December 24, 2009, 01:22:43 PM »
Ages ago NeDi was guessing, which MIB groups are supported on a particular device. With support for more models, this got rather inefficient, so I started the .def file concept. At the same time Max Baker came up with SNMP::Info, which is probably more what you want.

Of course the autosupport aspect would be nice (again), but from my experience with inconsistencies, this might also be a double edged sword...

The reason I kept the .defs flat and simple was, that everybody (without programming skills) is able to support his own devices. I realized there's a lot of redundant information in there, but it's a compromise I can accept.

With the new CLI library, the commands could be moved into .defs as well, to keep everything in the same place though...
Please consider Other-Invoices on your NeDi installation for an annual contribution, tx!
-Remo

steffen1

  • Guest
Re: Idea for a more scaleable definition file system
« Reply #2 on: December 25, 2009, 12:15:30 PM »
Thx for your answer on christmas :)

I know SNMP::Info and Netdisco too, but the model of SNMP::Info is not that what I want. One reason is that code is a part of the device package, so you can't provide an easy self helping service for device support as nedi with it's def-file concept does. So you will be dependend allways from the vendor of SNMP::Info. Another reason is, that you are dependend from the vendor MIB's bringing a lot of complexity into the device packages.

No, I prefere your def-file concept and to keep the def's simple as they are. I also dont want guessing of device properties - I want keeping the deterministic. But I think this simple concept could be improved for more scaleability in the future and that is the idea behind, what I mean in the 1st post.

I can imagine, that the lots of redundancy will make it not easier to develop new def's. The more def's you allready own, the more difficult will it be to keep the overview over that many def's.

Playing with guessed numbers, this could be one result:

Flat, linear, unused similarities, redundancies:
2007: 150 devices, 150def's
2008: 200 devices, 200def's
2009: 250 devices, 250def's
2010: 300 devices, 300def's
...

Tree-based, using similarities, no redundancies:
2010: 300 devices, 38def's
2011: 350 devices, 45def's
2012: 400 devices, 49def's
2013: 450 devices, 52def's

rickli

  • Administrator
  • Hero Member
  • *****
  • Posts: 2728
    • View Profile
    • NeDi
Re: Idea for a more scaleable definition file system
« Reply #3 on: December 25, 2009, 02:20:52 PM »
These days are where I can spend a lot of time with NeDi  ;) Got to be quick today though, family's coming soon...

I agree with your idea and we can work on a scheme, if you want. What I was trying to say, I expect problems due to inconsistency problems. I've come across devices supposedly providing MIB-A, Now this sometimes changes with a FW upgrade or is crippled to begin with. When you have many .defs, you can still put granular workarounds in place. Trust me when I say, it gets messy, when relying on dependencies...

Merry X-mas Steffen & thanks for your support! (This goes to all the others too, of course!)
Please consider Other-Invoices on your NeDi installation for an annual contribution, tx!
-Remo

steffen1

  • Guest
Re: Idea for a more scaleable definition file system
« Reply #4 on: December 25, 2009, 08:47:36 PM »
Merry X-mess for you and your family too, Remo and also many thanks for your support all the time I using Nedi and allways got an answer when I had problems with it.

Yes I will work with you on a scheme and I hope to convince you that you will not loose any flexibility with the reworked scheme. If you need a workarround for a special device, just create a new def-file for it to overwrite properties from the root of device family even as before.

I have an idea, how to can get it running:
1. MySQL table, that contain all properties of a def-file as fields, e.g. from other.def
2. a little sh-script that imports all current defs into this MySQL-table
3. Viewing this table in PHP-MySQLAdmin to see what sysobjid's are best for beeing the root of its device families, what redundancies and what devices with exception's we have
4. a little sh-script that can export a row to a def-file in a new dir, e.g. sysobj2 with 2 modes:
a) complete for a root-def
b) just difference properties to its root-def for leaf-defs
5. do the exports using this scriprt into sysobj2 (I expect much less than currently 250 defs)
6. extending code for def-file parsing, controlled by a new tag in nedi.conf, e.g. def-mode=flat|tree, so you can easely switch between the old and new def-file system.
7. tests

to clearify the scheme in more detail we can use private mail or / and phone / skype



« Last Edit: December 25, 2009, 08:49:32 PM by steffen1 »

oxo

  • Guest
Re: Idea for a more scaleable definition file system
« Reply #5 on: December 26, 2009, 09:54:27 PM »
I have a Table def for the sysObjectDef if you want:

Code: [Select]
$sql{ sysObjectDef } = "CREATE TABLE sysObjectDef (
id INT unsigned NOT NULL AUTO_INCREMENT,
sysObjectId VARCHAR(32) COMMENT 'sysObjectId',
SNMPv INT unsigned,
Type VARCHAR(32),
OS VARCHAR(32),
Icon VARCHAR(32),
Dispro VARCHAR(32),
Serial VARCHAR(32),
Bimage VARCHAR(32),
VLnams VARCHAR(32),
VTPdom VARCHAR(32),
VTPmod VARCHAR(32),
IFalia VARCHAR(32),
IFalix VARCHAR(32),
IFvlan VARCHAR(32),
IFvlix VARCHAR(32),
IFdupl VARCHAR(32),
IFduix VARCHAR(32),
Halfdp VARCHAR(32),
Modesc VARCHAR(32),
Moclas VARCHAR(32),
Movalu VARCHAR(32),
Moslot VARCHAR(32),
Modhw VARCHAR(32),
Modsw VARCHAR(32),
Modfw VARCHAR(32),
Modser VARCHAR(32),
Momodl VARCHAR(32),
CPUutl VARCHAR(32),
Temp VARCHAR(32),
MemCPU VARCHAR(32),
Custom VARCHAR(32),
index ( id ) )";

EDIT: added some more code to save into MySQL in "play with nedi"
« Last Edit: December 28, 2009, 08:10:23 PM by oxo »

steffen1

  • Guest
Re: Idea for a more scaleable definition file system
« Reply #6 on: December 29, 2009, 10:30:45 AM »
thx, oxo, would be very helpful. It seams to be a part from a perl script. Does this script makes something like im/export of def-files as well and is it possible to get the complete script in this case to save time for the def-investigation step?

steffen1

  • Guest
Re: Idea for a more scaleable definition file system
« Reply #7 on: January 03, 2010, 12:12:56 AM »
attached the script for the MySQL-import of all definition files and a first aggregation for Root-Def
analysis exported into Excel2000. The script will import all def-files into table sysObjDef and aggregate the first 2 digets after ENTERPRISE-OID of sysObjID to Table RootDefs.
« Last Edit: January 06, 2010, 01:12:06 AM by steffen1 »

steffen1

  • Guest
Re: Idea for a more scaleable definition file system
« Reply #8 on: January 06, 2010, 01:37:16 AM »
Now I finished the work for the def redesign. Attached is the zip containing the import script and the 3  tables in excel format for quick viewing the concept, that is as following:
  • free of reduncancies
  • RootDefs = complete definition per vendor or system main class, just 15
  • TypeDefs = device model specification, just a fiew fields - same count of defs than current
  • SpecDefs = device model specific properties as diff to its root. All device models that contains allmost no NeDi attributes are linked to other-def

It should be working at the same level as current with the advantages:
  • auto support of new devices from vendors or system main classes we've got RootDefs allready
  • better overview and easier to find mistakes in the defs
  • quicker development of new defs in future
  • new possibilities for the DefGen Utility. It could be improved to autoprobe a new device with attributes of its RootDef and the table OIDScore with the aim to reduce the amount of manual searching for attributes (OIDs)

I did'nt start the export script to new def-files, because I find to keep the def-file-system in MySQL has a lot of advantages.
« Last Edit: January 06, 2010, 02:12:03 AM by steffen1 »