Welcome, Guest. Please login or register.

Author Topic: more than 1 entry per mac  (Read 4207 times)

dobst

  • Full Member
  • ***
  • Posts: 144
    • View Profile
more than 1 entry per mac
« on: June 28, 2013, 05:49:25 PM »
I've got 6000 out of 45000 mac addresses which occure up to 6 times in the nodes table (53000 entries total). Is that a correct behavior?

rickli

  • Administrator
  • Hero Member
  • *****
  • Posts: 2526
    • View Profile
    • NeDi
Re: more than 1 entry per mac
« Reply #1 on: June 29, 2013, 01:22:28 AM »
No, not really. Are you running parallel threads? Or do you just have the same MACs in different vlans? Can you try to follow one of those duplicates in a -v output? I've seen some issues during development of 108, but I thought to have fixed them...
Please consider Other-Invoices on your NeDi installation for an annual contribution, tx!
-Remo

dobst

  • Full Member
  • ***
  • Posts: 144
    • View Profile
Re: more than 1 entry per mac
« Reply #2 on: July 01, 2013, 10:08:43 AM »
Yes, as described here I'm running parallel threads but each switch will be polled by exact one thread. Only during nightly discovery a 6th thread is running parallel with '-prn -SmgfoWjitedbwAO'

Code: [Select]
SELECT count(*) FROM nodes ;--> 57667

Code: [Select]
SELECT count(DISTINCT mac) FROM nodes;
--> 45359

Code: [Select]
SELECT
count(*)
FROM
(
SELECT
*,
count(*)
FROM
nodes
GROUP BY
mac
HAVING
count(*) > 1
ORDER BY
lastseen DESC,
count(*) DESC
) AS RESULT
;
--> 5896 (entries with more than one mac)

Example #1
Code: [Select]
name nodip mac oui firstseen lastseen device ifname vlanid ifmetric ifupdate ifchanges ipupdate ipchanges iplost arpval HEX(nodip6) tcpports udpports nodtype nodos osupdate noduser
- 0 009027cf421c INTEL CORPORATION 1371448801 1372663801 c37pebo1 Gi1/0/1 984 4358 1371448801 2 \N 0 0 0 '' 0
- 0 009027cf421c INTEL CORPORATION 1371448801 1371448801 c45seebc1 Gi1/1 984 4361 1371448801 1 \N 0 0 0 '' 0
- 0 009027cf421c INTEL CORPORATION 1371448801 1371448801 c45seebc1 Gi1/1 984 4361 1371448801 1 \N 0 0 0 '' 0
- 0 009027cf421c INTEL CORPORATION 1371448801 1371448801 c45seebc1 Gi1/1 984 4361 1371448801 1 \N 0 0 0 '' 0
- 0 009027cf421c INTEL CORPORATION 1371448801 1371448801 c37pebo1 Gi1/0/1 984 4358 1371448801 2 \N 0 0 0 '' 0

Example #2
Code: [Select]
name nodip mac oui firstseen lastseen device ifname vlanid ifmetric ifupdate ifchanges ipupdate ipchanges iplost arpval HEX(nodip6) tcpports udpports nodtype nodos osupdate noduser
3232268698 000d6152xxxx Giga-Byte Technology Co., Ltd. 1372278601 1372663801 c45bbz3og Gi3/6 982 256 1372278601 1 1372663801 1 0 1 '' 0
- 0 000d6152xxxx Giga-Byte Technology Co., Ltd. 1372278601 1372278601 switch2 Gi3/6 982 256 1372278601 1 \N 0 0 0 '' 0
- 0 000d6152xxxx Giga-Byte Technology Co., Ltd. 1372278601 1372278601 switch2 Gi3/6 982 256 1372278601 1 \N 0 0 0 '' 0
- 0 000d6152xxxx Giga-Byte Technology Co., Ltd. 1372278601 1372278601 switch2 Gi3/6 982 256 1372278601 1 \N 0 0 0 '' 0
- 0 000d6152xxxx Giga-Byte Technology Co., Ltd. 1372278601 1372278601 switch2 Gi3/6 982 256 1372278601 1 \N 0 0 0 '' 0

Following up the node für #2:

Code: [Select]
server:/opt/nedi # ./nedi.pl -vt switch2 | grep 000d6152xxxx
FWDS:000d6152xxxx on Gi3/6 Vl982
UPIP:000d615250a8 EXISTING no update 0.0.0.0 = -
server:/opt/nedi #

looks ok.

Interesstingly all nodes of #2 appear with exact the same attributes (same switch, same if, same vlan, ...). Maybe as a consequence of that I get events like

Code: [Select]
id level time source info class device
510120 100 1372430701 c45bbz3og New IP address 192.168.129.154 found for 000d615250a8 secj c45bbz3og
509418 100 1372429801 c45bbz3og New IP address 192.168.129.154 found for 000d615250a8 secj c45bbz3og
510771 100 1372431601 c45bbz3og New IP address 192.168.129.154 found for 000d615250a8 secj c45bbz3og

every scan... On friday I deleted all the entries with more than one mac address but immediatly it gets filled up again... :-( Can you check whether nedi updates existing entries instead of creating a new one? (mysql REPLACE INTO).

rickli

  • Administrator
  • Hero Member
  • *****
  • Posts: 2526
    • View Profile
    • NeDi
Re: more than 1 entry per mac
« Reply #3 on: July 01, 2013, 11:04:33 PM »
libdb's WriteNode has been optimized to cope with +100k nodes, but it's not using REPLACE INTO but rather uses firstseen and lastseen to decide whether to insert, update or delete...

I suspect your 6th thread causing this. Are you not skipping arp tables intenionally? What is it supposed to do?
Please consider Other-Invoices on your NeDi installation for an annual contribution, tx!
-Remo

dobst

  • Full Member
  • ***
  • Posts: 144
    • View Profile
Re: more than 1 entry per mac
« Reply #4 on: July 02, 2013, 05:32:01 PM »
What I did:
  • delete duplicates
  • --> no duplicates
  • single threaded update via
Code: [Select]
/opt/nedi/nedi.pl -A
--> no duplicates

second try:
  • check for duplicates (=0)
  • multi threaded update via
Code: [Select]
/opt/nedi/nedi.pl -A 'RIGHT(INET_NTOA(devip), 1) IN (0, 5)' > /dev/null 2>&1 &
/opt/nedi/nedi.pl -A 'RIGHT(INET_NTOA(devip), 1) IN (1, 6)' > /dev/null 2>&1 &
/opt/nedi/nedi.pl -A 'RIGHT(INET_NTOA(devip), 1) IN (2, 7)' > /dev/null 2>&1 &
/opt/nedi/nedi.pl -A 'RIGHT(INET_NTOA(devip), 1) IN (3, 8)' > /dev/null 2>&1 &
/opt/nedi/nedi.pl -A 'RIGHT(INET_NTOA(devip), 1) IN (4, 9)' > /dev/null 2>&1 &
    --> duplicates

    Ok, multiple entries are generated by multi threading. But how could that happen if each device is processed by just one thread? For example: These entries were inserted during the multi threaded update but they differ in nodip, ipupdate, ipchanges and iplost:
    Code: [Select]
    name nodip mac oui firstseen lastseen device ifname vlanid ifmetric ifupdate ifchanges ipupdate ipchanges iplost arpval HEX(nodip6) tcpports udpports nodtype nodos osupdate noduser
    - 0 e4d3f171xxxx Cisco 1372776102 1372776102 c29ubc1 Gi1/0/33 702 256 1372776102 0 \N 0 0 0 '' 0
    - 0 e4d3f171xxxx Cisco 1372776102 1372776102 c29ubc1 Gi1/0/33 702 256 1372776102 0 \N 0 0 0 '' 0
    168100660 e4d3f171xxxx Cisco 1372776102 1372776102 c29ubc1 Gi1/0/33 702 256 1372776102 0 1372776102 1 1 1 '' 0
    168100660 e4d3f171xxxx Cisco 1372776102 1372776102 c29ubc1 Gi1/0/33 702 256 1372776102 0 1372776102 1 0 1 '' 0
    168100660 e4d3f171xxxx Cisco 1372776102 1372776102 c29ubc1 Gi1/0/33 702 256 1372776102 0 1372776102 1 1 1 '' 0

    Other nodes appear on different devices:
    Code: [Select]
    name nodip mac oui firstseen lastseen device ifname vlanid ifmetric ifupdate ifchanges ipupdate ipchanges iplost arpval HEX(nodip6) tcpports udpports nodtype nodos osupdate noduser
    - 0 e4d3f1c8xxxx Cisco 1372776102 1372776102 c45gwzc22u Te5/1 702 4360 1372776102 0 \N 0 0 0 '' 0
    - 0 e4d3f1c8xxxx Cisco 1372776102 1372776102 c29ubc10w Gi0/2 702 4356 1372776102 1 \N 0 0 0 '' 0
    1681006xx e4d3f1c8xxxx Cisco 1372776102 1372776102 c29ubc2 Fa0/1 702 256 1372776102 3 1372776102 1 1 1 '' 0
    1681006xx e4d3f1c8xxxx Cisco 1372776102 1372776102 c65ub Gi1/40 702 4355 1372776102 2 1372776102 1 0 1 '' 0
    1681006xx e4d3f1c8xxxx Cisco 1372776102 1372776102 c65ub Gi1/40 702 4355 1372776102 2 1372776102 1 1 1 '' 0

    If I poll these 4 switches above, i get these 5 entries:

    Code: [Select]
    nedi@viewer:~> /opt/nedi/nedi.pl -vt c45gwzc22u | grep e4d3f1c8xxxx
    FWDS:e4d3f1c8xxxx on Te5/1 Vl702
    UPIP:e4d3f1c8xxxx EXISTING no update 10.5.3.62 =
    nedi@viewer:~> /opt/nedi/nedi.pl -vt c29ubc10w | grep e4d3f1c8xxxx
    FWDS:e4d3f1c8xxxx on Gi0/2 Vl702
    UPIP:e4d3f1c8xxxx EXISTING no update 10.5.3.62 =
    nedi@viewer:~> /opt/nedi/nedi.pl -vt c29ubc2 | grep e4d3f1c8xxxx
    FWDS:e4d3f1c8xxxx on Fa0/1 Vl702
    UPIP:e4d3f1c8xxxx EXISTING no update 10.5.3.62 =
    nedi@viewer:~> /opt/nedi/nedi.pl -vt c65ub | grep e4d3f1c8xxxx
    ARPS:e4d3f1c8xxxx 10.5.3.62 on Vl702 vl702
    FWDS:e4d3f1c8xxxx on Gi1/40 Vl702
    UPIP:e4d3f1c8xxxx EXISTING no update 10.5.3.62 =
    nedi@viewer:~>

    How does NeDi know which of these interfaces is the one the client is connected to directly?
    « Last Edit: July 02, 2013, 05:49:06 PM by dobst »

    rickli

    • Administrator
    • Hero Member
    • *****
    • Posts: 2526
      • View Profile
      • NeDi
    Re: more than 1 entry per mac
    « Reply #5 on: July 02, 2013, 07:56:15 PM »
    That's what ifmetric is for. In Node Reports - Node Distribution, hover over IF Metric to get more info...

    Sorry for asking, but you are using official 1.0.8-116? Some testers encountered this, when threads did not find any existing nodes (e.g. 1st run) and then just added everything they found. Nodelock should really prevent parallel threads from adding duplicates, as other threads wait for "unlocking" and then proceed with building nodes:


    nedi.pl line around 277...

                &db::Update('system',"value=\"$$\"",'name="nodlock"');         # Set node lock in system table...
                &misc::Prt("MAIN:Nodes table locked at ".localtime(time)." by PID $$\n","Building nodes\n");
                &db::ReadNod();
                &misc::BuildNod();
                &misc::Prt("\nMAIN:Nodes table unlocked at ".localtime(time)."\n"," done\n");
                &db::WriteNod();
                &db::Update('system','value="0"','name="nodlock"');         # ...unlock them again
    Please consider Other-Invoices on your NeDi installation for an annual contribution, tx!
    -Remo

    dobst

    • Full Member
    • ***
    • Posts: 144
      • View Profile
    Re: more than 1 entry per mac
    « Reply #6 on: July 02, 2013, 10:05:18 PM »
    Yes, I'm running the latest version:

    Code: [Select]
    viewer:/opt/nedi # ./nedi.pl -? | tail -n 2
    Unknown option: ?
    NeDi 1.0.8-116 (C) 2001-2013 Remo Rickli & contributors

    Which behavior is supposed to be expected during scan? I could see the number of duplicated nodes increasing after the first of five threads stopped. Each stopped thread creates more duplicate entries. Do you expect all threads waiting until the last one has finished scanning devices and writing nodes afterwards to the nodes table?
    « Last Edit: July 02, 2013, 10:17:04 PM by dobst »

    rickli

    • Administrator
    • Hero Member
    • *****
    • Posts: 2526
      • View Profile
      • NeDi
    Re: more than 1 entry per mac
    « Reply #7 on: July 02, 2013, 11:30:22 PM »
    Not quite. The code shows, that nodlock is set in the system table, before the nodes table is read and updated. Once done, the nodlock is set to 0. This mechanism should prevent any other thread to meddle....so I thought!

    Do you see any NeDi (as source) events? The only thing I can think of is, when the first thread reads no nodes, it'll insert all it's got. If the second doesn't wait for the 1st one to finish writing, it'll do the same. Question is, why does it not look at nodlock? Can you check whether the PIDs of the thread match the one in nodlock? add sleep 10 before unlocking, in case it's too fast...
    Please consider Other-Invoices on your NeDi installation for an annual contribution, tx!
    -Remo

    dobst

    • Full Member
    • ***
    • Posts: 144
      • View Profile
    Re: more than 1 entry per mac
    « Reply #8 on: July 03, 2013, 02:50:44 PM »
    Code: [Select]
    /opt/nedi/nedi.pl -A 'RIGHT(INET_NTOA(devip), 1) IN (0, 5)' > /dev/null 2>&1 &
    /opt/nedi/nedi.pl -A 'RIGHT(INET_NTOA(devip), 1) IN (1, 6)' > /dev/null 2>&1 &
    /opt/nedi/nedi.pl -A 'RIGHT(INET_NTOA(devip), 1) IN (2, 7)' > /dev/null 2>&1 &
    /opt/nedi/nedi.pl -A 'RIGHT(INET_NTOA(devip), 1) IN (3, 8)' > /dev/null 2>&1 &
    /opt/nedi/nedi.pl -A 'RIGHT(INET_NTOA(devip), 1) IN (4, 9)' > /dev/null 2>&1 &

    Behavior during scan:
    Code: [Select]
    Every 2.0s: ps -ef | grep nedi.pl                                                                                    Wed Jul  3 14:02:15 2013

    root       745 28879  0 13:32 pts/17   00:00:01 watch ps -ef | grep nedi.pl
    nedi      3718     1 23 14:00 ?        00:00:32 /usr/bin/perl /opt/nedi/nedi.pl -A RIGHT(INET_NTOA(devip), 1) IN (0, 5)
    nedi      3719     1 22 14:00 ?        00:00:30 /usr/bin/perl /opt/nedi/nedi.pl -A RIGHT(INET_NTOA(devip), 1) IN (1, 6)
    nedi      3722     1 19 14:00 ?        00:00:26 /usr/bin/perl /opt/nedi/nedi.pl -A RIGHT(INET_NTOA(devip), 1) IN (2, 7)
    nedi      3724     1 20 14:00 ?        00:00:26 /usr/bin/perl /opt/nedi/nedi.pl -A RIGHT(INET_NTOA(devip), 1) IN (3, 8)
    nedi      3725     1 16 14:00 ?        00:00:22 /usr/bin/perl /opt/nedi/nedi.pl -A RIGHT(INET_NTOA(devip), 1) IN (4, 9)
    root      3985   745  0 14:02 pts/17   00:00:00 sh -c ps -ef | grep nedi.pl
    root      3987  3985  0 14:02 pts/17   00:00:00 grep nedi.pl

    name value
    nodlock 0
    threads 4
    first 1372852801

    name value
    nodlock 3724
    threads 4
    first 1372852801

    name value
    nodlock 0
    threads 3
    first 1372852801

    Every 2.0s: ps -ef | grep nedi.pl                                                                                    Wed Jul  3 14:06:44 2013

    root       745 28879  0 13:32 pts/17   00:00:01 watch ps -ef | grep nedi.pl
    nedi      3718     1  9 14:00 ?        00:00:37 /usr/bin/perl /opt/nedi/nedi.pl -A RIGHT(INET_NTOA(devip), 1) IN (0, 5)
    nedi      3719     1 18 14:00 ?        00:01:12 /usr/bin/perl /opt/nedi/nedi.pl -A RIGHT(INET_NTOA(devip), 1) IN (1, 6)
    nedi      3722     1 18 14:00 ?        00:01:16 /usr/bin/perl /opt/nedi/nedi.pl -A RIGHT(INET_NTOA(devip), 1) IN (2, 7)
    nedi      3725     1 15 14:00 ?        00:01:04 /usr/bin/perl /opt/nedi/nedi.pl -A RIGHT(INET_NTOA(devip), 1) IN (4, 9)
    root      4475   745  0 14:06 pts/17   00:00:00 sh -c ps -ef | grep nedi.pl
    root      4477  4475  0 14:06 pts/17   00:00:00 grep nedi.pl

    name value
    nodlock 3719
    threads 3
    first 1372852801

    name value
    nodlock 0
    threads 2
    first 1372852801

    Every 2.0s: ps -ef | grep nedi.pl                                                                                    Wed Jul  3 14:08:07 2013

    root       745 28879  0 13:32 pts/17   00:00:01 watch ps -ef | grep nedi.pl
    nedi      3718     1 11 14:00 ?        00:00:53 /usr/bin/perl /opt/nedi/nedi.pl -A RIGHT(INET_NTOA(devip), 1) IN (0, 5)
    nedi      3722     1 15 14:00 ?        00:01:16 /usr/bin/perl /opt/nedi/nedi.pl -A RIGHT(INET_NTOA(devip), 1) IN (2, 7)
    nedi      3725     1 16 14:00 ?        00:01:19 /usr/bin/perl /opt/nedi/nedi.pl -A RIGHT(INET_NTOA(devip), 1) IN (4, 9)
    root      4645   745  0 14:08 pts/17   00:00:00 sh -c ps -ef | grep nedi.pl
    root      4647  4645  0 14:08 pts/17   00:00:00 grep nedi.pl

    name value
    nodlock 3725
    threads 2
    first 1372852801

    name value
    nodlock 0
    threads 1
    first 1372852801

    Every 2.0s: ps -ef | grep nedi.pl                                                                                    Wed Jul  3 14:08:31 2013

    root       745 28879  0 13:32 pts/17   00:00:01 watch ps -ef | grep nedi.pl
    nedi      3718     1 11 14:00 ?        00:00:58 /usr/bin/perl /opt/nedi/nedi.pl -A RIGHT(INET_NTOA(devip), 1) IN (0, 5)
    nedi      3722     1 15 14:00 ?        00:01:16 /usr/bin/perl /opt/nedi/nedi.pl -A RIGHT(INET_NTOA(devip), 1) IN (2, 7)
    root      4687   745  0 14:08 pts/17   00:00:00 sh -c ps -ef | grep nedi.pl
    root      4689  4687  0 14:08 pts/17   00:00:00 grep nedi.pl

    name value
    nodlock 3722
    threads 1
    first 1372852801

    name value
    nodlock 0
    threads 1
    first 1372852801

    Every 2.0s: ps -ef | grep nedi.pl                                                                                    Wed Jul  3 14:09:38 2013

    root       745 28879  0 13:32 pts/17   00:00:01 watch ps -ef | grep nedi.pl
    nedi      3718     1 12 14:00 ?        00:01:12 /usr/bin/perl /opt/nedi/nedi.pl -A RIGHT(INET_NTOA(devip), 1) IN (0, 5)
    root      4800   745  0 14:09 pts/17   00:00:00 sh -c ps -ef | grep nedi.pl
    root      4802  4800  0 14:09 pts/17   00:00:00 grep nedi.pl

    name value
    nodlock 3718
    threads 0
    first 1372852801

    name value
    nodlock 0
    threads -1
    first 1372852801

    Output of NeDi:
    Code: [Select]
    Id Stufe Zeit Quelle Klasse
    Info
    62650 3.Jul 13 14:15 NeDi -1 thread(s) error, 1st from Wed Jul 3 14:00:01 2013 make sure interval is longer than discovery takes!
    62246 3.Jul 13 14:00 NeDi -3 thread(s) error, 1st from Wed Jul 3 13:45:01 2013 make sure interval is longer than discovery takes!
    62245 3.Jul 13 14:00 NeDi -3 thread(s) error, 1st from Wed Jul 3 13:45:01 2013 make sure interval is longer than discovery takes!
    60361 3.Jul 13 13:00 NeDi -2 thread(s) error, 1st from Wed Jul 3 12:45:01 2013 make sure interval is longer than discovery takes!

    I repeated the procedure twice and first thread count differed from 2 to 4 and never was 5. So I started the threads a bit delayed and the thread count was correct:
    Code: [Select]
    /opt/nedi/nedi.pl -A 'RIGHT(INET_NTOA(devip), 1) IN (0, 5)' > /dev/null 2>&1 &
    sleep 1s
    /opt/nedi/nedi.pl -A 'RIGHT(INET_NTOA(devip), 1) IN (1, 6)' > /dev/null 2>&1 &
    sleep 1s
    /opt/nedi/nedi.pl -A 'RIGHT(INET_NTOA(devip), 1) IN (2, 7)' > /dev/null 2>&1 &
    sleep 1s
    /opt/nedi/nedi.pl -A 'RIGHT(INET_NTOA(devip), 1) IN (3, 8)' > /dev/null 2>&1 &
    sleep 1s
    /opt/nedi/nedi.pl -A 'RIGHT(INET_NTOA(devip), 1) IN (4, 9)' > /dev/null 2>&1 &

    So I deleted duplicates and started delayed parallel threads again and it seems to work. No duplicates so far....

    --> problem meight be the parallel access to the threads variable in the system table.

    rickli

    • Administrator
    • Hero Member
    • *****
    • Posts: 2526
      • View Profile
      • NeDi
    Re: more than 1 entry per mac
    « Reply #9 on: July 03, 2013, 08:50:59 PM »
    Wow, thanks for this thorough analysis! With 15min intervals, the hardcoded 5min wait for unlocking nodes in each thread is probably not optimal. Looks like they're kililing each other while busy with the nodes. How long does building the nodes for the first time (and subsequent times) take, do you know?

    Last but not least, have you tried running those threads staggered, so they won't even encounter any nodlocks?
    « Last Edit: July 03, 2013, 08:52:36 PM by rickli »
    Please consider Other-Invoices on your NeDi installation for an annual contribution, tx!
    -Remo

    dobst

    • Full Member
    • ***
    • Posts: 144
      • View Profile
    Re: more than 1 entry per mac
    « Reply #10 on: July 04, 2013, 08:43:25 AM »
    I dont't think these 5 threads block each other during writing nodes because each thread takes a different time to run - they don't finish at the same moment (in contrast to run 10 parallel threads). I also couln't see one thread killing an other. Building nodes takes less than 30 sec per thread.

    Do you mean by staggered starting a thread when the privious has finshed? What would be the advantage compared to running just one thread scanning all devices? It might be an good idea to sort threads by execution time to prevent blocking each other.

    rickli

    • Administrator
    • Hero Member
    • *****
    • Posts: 2526
      • View Profile
      • NeDi
    Re: more than 1 entry per mac
    « Reply #11 on: July 04, 2013, 09:39:29 PM »
    ok, just wanted to make sure. You absolutely understand how it should be :) Them not finishing at the same time and an individual time to build nodes of 30s should really work fine.

    You could doublecheck by looking for the following lines on -v output:
    MAIN:Nodes locked by PID xy, waiting for unlock

    Each line causes a thread to wait $pause (not 5, sorry). You might as well lower it to 30s, in case you see them...

    Let me know, so I can try to reproduce in my lab...
    « Last Edit: July 04, 2013, 09:41:25 PM by rickli »
    Please consider Other-Invoices on your NeDi installation for an annual contribution, tx!
    -Remo