Dear imonitor users, potential users, former users, and interested parties:

Summary for those of you new to this newsletter: 

I am working on a project on Internet [and local network] monitoring, and have a small device [raspberry pi] which I can deploy to perform this service. I have  23 "guinea pigs" deployed.  I initially targeted the service to users on the mountain in Jasper GA (windstream ISP), but it is applicable universally, and I have "customers" across the country on Windstream, ATT, Spectrum, Comcast, CenturyLink, and several other ISPs.  It spans all access technologies from ADSL, VDSL, WADSL, cable and fiber.

I have one spare gadget for rent [free]!   Let me know if you interested.  OR..... preferably, you can purchase an raspberry pi 3B or 3B+ and I can send you a microSD image!  Nerds might be interested in this alternative.  Currently, I can only offer an image [3B or 3B+, not 4B at this time].  But you can receive the image and then use the rpi for your own purposes - just don't mess with the scripts [guinea pigs are certainly encouraged to use existing rpis for their own -additional- purposes]. 

https://www.amazon.com/gp/product/B07BLRSKBV/ref=ppx_yo_dt_b_asin_title_o00_s00?ie=UTF8&psc=1

As usual, I am extremely grateful for the use of your ISP connection to develop this service across many ISPs.   It has been invaluable, and much fun. 

As always, you can refer to the main information page at https://johnloop.com/imonitor/imonitor.html  There are images of the web site, email, plots, etc. 

There is a new "quick intro" doc at https://johnloop.com/imonitor/QuickManual.pdf  [also attached to this email]

Newsletter 2-1-2020  previous newsletter https://johnloop.com/imonitor/newsletter12-1-2019.html

There is exciting "new" news in these areas:

Exploratory development of this package of features for the raspberry pi3B and 3B+ is nearing an end.  It is not foreseen to make major changes or additions going forward. 

1. There are now 3 different realtime performance measurements/plots, and an archive plot to record the historical average of the results.  Each of these performs a ping every minute and records reachability/delay.  The three measurements are:

b. "TCP" ping to top100 web sites, representing overall Internet web performance.  This tends to be to the nearest CDN. This is "AUTO" determined unless "LOCKed"

c. "Deep" ICMP ping, where each customer pings the ICMP ping target of the other customers.  This is "guaranteed" to be across the Internet.  The list is updated nightly. 

The combined plot shows all three graphs on one 24 hour plot [representing yesterday].  This plot [BOTHYpingresult.png attached to your email] is an excellent reveal of overall performance, especially when compared to historical plots [available on the rpi web page], and when compared to other customers [available on the server web page -10 below]. This plot is available on the rpi web page, and is attached to the daily email.  A representative BOTHYpingresult.png plot is attached [pi24BOTH...]. Notice the longer "deep" ICMP pings representing overall Internet sluggishness [from this pi24]. 

Comparisons between/among customers can be viewed here:  https://johnloop.com/imonitor/customerplots/customerplots.html This view will be upgraded in the next months.  This is a great comparison between technologies across the country.  

2. An archive plot is created which takes the average of each of the 3 daily measurements in 1 above and plots a cumulative graph.  This plot [ICMP-TCP-DelayArchive.png] is available on the rpi web page, and is attached to the daily email. A representative plot is attached [pi9archive.png].  Notice the consistency of the day to day averages, which represents good long term performance. 

3. An historical host table [called "arp table" on rpi web page].  This table [actually a "history" of the arp cache on the rpi] records the appearance of hosts on your local private network [wifi and eth as long as they are on the same network], and accumulates their appearance.  It monitors the arp cache, so it will even pick up hosts that do not respond to pings, including windows 10 which defaultly does not respond to ping!  This should be an invaluable record on your network "intruders" going forward!  This is only available on the rpi web page -tho the daily email does list NEW/GONE hosts.

The daily scan continues to report current IP hosts and changes, "gone" and "new" each day -reported in the daily email.  Beware that the "ghosts" may come and go as they wake up and are discovered -"new", or go to sleep and become "gone."  These tend to be phones, kindles, tablets, cameras, etc.  By referencing the arp table, you can ID the hosts that come and go. 

4. A "Daily news item" in the newsletter listing significant additions/changes/notes to the imonitor capabilities.

5. The speedtest archive plot has always been available to watch a history of your nightly and daily speedtests.  This is only available on the rpi web page. 

6. The top part of the rpi web page [which is linked in the daily email] links to all these most important plots.  I am including an image of the current rpi web page as an attachment 2 [jmonitorWEB.pdf]. -click on "data snapshots"- You should peruse it to gauge the large range of performance stats and data I am collecting about your Internet performance and your network stats! 

7. The "alerts" "seem" to be working properly [there are always collection problems if interruptions of less than 5 minutes occur and are difficult to track/alert].  Alerts are only available if you have not invoked "standalone."  The alert list is as follows:

a. "boot" alert:  this email is sent whenever the rpi loses power and is rebooted.  This is SCHEDULED on the first of the month, otherwise it is unintended.  It records the "last online time" and the "new online time" and the offline minutes.  For more detail, there is a record on the rpi web page.

b. "access alert: DNS pull target timeout"  This will happen occasionally and should not elicit concern unless many repeated occurrences. 

c. "access alert: OFFLINE to Internet/ONLINE to router"  This happens when the router loses Internet conectivity but is still connected to the router.

d. "access alert: IP address change.  Your Internet address changed from: to: Last and Now online Time are listed

e. "access alert: DNS server change.  Your DNS server [distributed by DHCP] has change from: to: or "reup'd"

8. The "QuickManual" -providing quick setup info, has been updated.  It is attached as 2nd attachment. 

9. I am adding a 4th attachment [Customer....pdf] which shows a spreadsheet listing the "guinea pigs" and a record of the [avg] TCP response times over the past month [collected weekly].  The consistency of the times is quite amazing...  Maybe explained by the fact that it is an AVERAGE over 100 top web sites which are reliably online 24/7/365 via a CDN.  The deep pings may give a better feel for "entire" Internet performance [...not possible -like measuring the temperature of the earth!]. 

10. Server side plots.  I collect and make available at my server daily customer performance graphs.  This will allow comparison across ISPs, access rates and technologies.  These will be available at https://johnloop.com/imonitor/customerplots/  -first and second links.  This is a simple directory listing for now.

11. There is a new speedtest script  -dated 10/2019- which I will be incorporating.  This uses the latest algorithms from ookla/speedtest.net.  This should vastly improve the speedtest capabilities!  Should be available by 2-5. 

12.  I would be remiss if I didn't mention a "competing" product which is very complementary to this software.  This is the "fing" software  -google it.  You can get a fing app for your phone and a fing install for your PC.  It does an especially nice job of detecting and listing objects on your network using a database it collects from its [registered] users.  Yes, you have to register....  and your info is collected into their cloud.  But it is more capable of ID'ing objects on your network because of its large MAC address database.  There is even a separate "gadget" you can order for your network which does more functions.  This is typical of "web/app" appearances these days - nice fancy graphics.  I don't bemoan this [I wish I were as good], but I can't see the detail for the fluff, as is the case with much of this art.  My development is all detail of course!!  I have attached a screenshot of the "device" page. 

This is what I will be working on in the next few months:

1. Improved Rpi web page and server web page.

2. Quantify the traffic injected by the rpi onto your network/Internet connection.

3. Transition to pi4.  Repackage scripts to make it a raspberry pi distribution.

4. Explore the possibility of making a phone app to link to the rpi.  Android first. [any advice???!!]

5. Updating the speedtest script

1. The Rpi does a scan of your local network at 6PM and reports "new" and "gone" hosts compared to 6PM day before.  This is done using the host IP address.  There is no universal way to "ID" IP hosts -getting their name e.g.- so this is useful only if you understand what is on your network.  I will be adding a scan of these active hosts to keep you alerted as to the services open on your network.  If you are running these on wifi [especially], these are potential intrustion avenues for lurking hackers.  Wifi traffic activity [unlike ethernet] is completely visible up to 200 feet from your router, and can be compromised by determined hackers.  [this scans ONLY the network that the rpi is on.  Some customers have added networks behind their routers].

2. The historical offline plot now works correctly, and will be of more value going forward.  The historical speedtest plot is working, so you can see an archive of your speedtest results.   -available via the email links or on the rpi web page.

3. The archive plots of your ICMP and TCP ping responses is a wonderful way to gauge longterm performance.  -available on the rpi web page.

4. The alerts now seem to work :)  you will get an email alert for the following:

- a reboot [which may represent power cycle in your house].  Ideally connect the rpi and your router to a UPS! 

- an Internet IP address change for your modem [which can be expected occasionally].  Useful if you have services running and need to get to them remotely.

- for "offline" events, where Internet access is interrupted. 

- for DNS interruptions, which may or may not be offline events.   DNS changes are also reported.

You can turn off all email of on the mgmt config on the web page, or just enable the Saturday status email.  The alerts are quantized in that they "may" not detect alert events lasting less than 5 minutes - such as DNS server or Internet timeouts.  This is probably best to "smooth" the alerts.  You can always see the offline events by looking at the ping plots or the actual alerts on the rpi web page.

5. The Rpis have been updated and upgraded to the latest code.  3B is "jessie."  3B+ is "stretch."

6. Temperature info has added a degF current reading on the rpi web page - as well as yesterday plot.  I can add a temp probe if desired!  This is used by several people to monitor rental properties. -the temp plots are archived on the rpi web page.

7. The emails are archived on the rpi as well.

I continue to fine tune the alert scripts.  I may decrease the quantization to 1 min [from 5 min] since the script is more optimized.  I will be adding a scan of the active hosts on the network [1 above] to try to ID services and summarize them.  This would list both ethernet and wifi services available [if the rpi is logged into your wifi].  Every day you would get a list of the "services you are offering!"

You receive a list of the wifi networks visible on your network [listing your neighbor's as well] - reported in the daily email.  These are the networks visible, and should not be accessible unless you have the wifi key.  The rpi can login to your wifi network as well as the ethernet by providing the SSID/key.  You can do this, or I can do it remotely.