Dear imonitor users, potential users, former users, and interested parties:
Summary for those of you new to this newsletter:
I am working on a project on Internet [and local network] monitoring, and have a small device [raspberry pi] which I can deploy to perform this service. I have 23 "guinea pigs" deployed. I initially targeted the service to users on the mountain in Jasper GA (windstream ISP), but it is applicable universally, and I have "customers" across the country on Windstream, ATT, Spectrum, Comcast, CenturyLink, and several other ISPs. It spans all access technologies from ADSL, VDSL, WADSL, cable and fiber.
I have one spare gadget for rent [free]! Let me know if you interested. OR..... preferably, you can purchase an raspberry pi 3B or 3B+ and I can send you a microSD image! Nerds might be interested in this alternative. Currently, I can only offer an image [3B or 3B+, not 4B at this time]. But you can receive the image and then use the rpi for your own purposes - just don't mess with the scripts [guinea pigs are certainly encouraged to use existing rpis for their own -additional- purposes].
https://www.amazon.com/gp/product/B07BLRSKBV/ref=ppx_yo_dt_b_asin_title_o00_s00?ie=UTF8&psc=1
As usual, I am extremely grateful for the use of your ISP connection to develop this service across many ISPs. It has been invaluable, and much fun.
As always, you can refer to the main information page at https://johnloop.com/imonitor/imonitor.html There are images of the web site, email, plots, etc.
There is a new "quick intro" doc at https://johnloop.com/imonitor/QuickManual.pdf [also attached to this email]
Use the site https://johnloop.com/imonitor/customerplots/customerplots.html to see representative plots, plus the "guinea pigs" and their technologies.
Newsletter 5-1-2020 previous newsletter https://johnloop.com/imonitor/newsletter2-1-2020.html
1. In addition to the three pings [discussed in the last newsletter], I have added an actual http browse to google.com as a "final" measurement, and I plot this on the same graph. Two browse's [actual linux curl] are performed per hour, signified by the circles on the plots. Only the google index page is loaded, not all the images. See the attached png as an example.
2. There is now a capability to generate a realtime plot, like the daily email plot [representing "yesterday"], representing 2AM->current time. Go to the rpi web page [linked in your email], OK the popup, then click on "data snapshots" and then click on the "PLOTIT." You will get a new webpage with a link to the realtime plot. It will take a minute to generate. There is a sample attached, showing snapshot taken at 10:50AM. Notice the date at the top, and the current time on the right of the plot. The attached shows a sustained offline time [my power and Internet were off]. Notice how nice a facility this is for tracking online/offline behavior. The orange squares/bar [top] is the actual offline [close to a 1 minute resolution]; the yellow squares/bar show occasional tcp failures as well as those during offline. You need to ignore the connecting lines [gnuplot is asked to plot "lines" and not "dots"]. Your rpi and your modem/router should be on a UPC to monitor all these interruptions, and the ethernet [or wifi] left connected.
3. I will be adapting the scripts for a pi4. So there will be images for a pi 3B/3B+ and pi4. These will run standalone without any connection to my server at all. All you need to do is burn an image to a microSD and plug it into the raspberry pi. You can then point your browser to the raspberry pi [find its IP in your router] and have available all the network stats. I hope to have these images available by 6-1. I can also burn a microSD image and just send it to you if you have a raspberry pi you can dedicate to this. With no connection to my server of course, there will be no daily emails or alerts, and no updates. These are "generic" pi images with my added scripts.
4. I will continue the program with my 23 guinea pigs for the immediate future. These will continue to receive the daily emails and alerts.
5. The daily 3 plots [combined plot, archive delay plot, and speedtest archive] from the guinea pigs will be available here https://johnloop.com/imonitor/customerplots/customerplots.html
6. I have decided NOT to develop an app for the Android [or the iphone]. There is really no advantage to this; you can always browse to the rpi page from the phone.
7. I will also NOT be developing a "pkg" distribution. There are hundreds of modifications to the raspian image, so an image distro is best. As mentioned, I will be working on these over the next month.
8. How to interpret the daily/on-call plots [look at attached png for reference].
There are 3 main LINE plots: The ping delays of a close router [pingresult], the TCP delays of top 100 web sites [Timetcpping], and the ping delays of a far-away router [CustPingSites]. These are the three colored lines. The close ping delays and the TCP delays are every minute for 23 hours [2AM till next 1AM], the far-away ping delays are about 20 minutes in the middle of the hour [there are straight lines connecting these]. Typically, the close ping delays should be less than the TCP delays and be fairly constant; the TCP delays will vary widely, normally between 20-100msec. The TCP delays will typically be to the nearest Content Distribution Network (CDN) which host most of the major web sites. The far ping delays may be the longest, since they are the ping targets of the other customers.
The fourth plot is the points that represent the twice hourly fetch of www.google.com. This will typically have the longest delay. DNS name resolution will have to be performed for this, and for the TCP delays. The ping delays do not need a DNS name resolution.
The fifth and sixth "plot" is actually an overlay on the plots which will represent offline times experienced, and TCP timeouts experienced, but still on-line. These will appear as bars when there are extended periods of offline state, or multiple TCP ping failures. These are plotted at the 250 and 220 msec delay times.
9. Assumptions used in developing these scripts.
Since the pi does not have a realtime clock, it is not possible to know the "correct" time with certainty -only when an Internet connection is established can the "correct" time be acquired via the ntp service. For this reason, the scripts only compute valid times once the rpi acquires an Internet connection, via an ethernet or a wifi connection to the local router. If this connection is "interrupted" the scripts can only "estimate" times, based on a clock that will gradually lose correct time. When booting, the clock may be totally incorrect [depending on length of time the pi was powered off]. The "alerts" sometimes report incorrect/repeated times due to these clock/power inconsistencies. Thus, it is assumed that the pi is powered at all times and has an active ethernet or wifi connection. For the best performance, the pi -and your modem/router- should be connected to a small UPS to bridge glitches and short power outages.
10. I will have an email announcing the general imonitor, called an "imonitorg." Just order a raspberry pi and ask for an image.
Previous 2-1-2020 newsletter information
1. There are now 3 different realtime performance measurements/plots, and an archive plot to record the historical average of the results. Each of these performs a ping every minute and records reachability/delay. The three measurements are:
a. "Near" ICMP ping, at most 5 hops into network, representing "near" ISP performance. This continues to be "AUTO" determined unless "LOCKed"
b. "TCP" ping to top100 web sites, representing overall Internet web performance. This tends to be to the nearest CDN. This is "AUTO" determined unless "LOCKed"
c. "Deep" ICMP ping, where each customer pings the ICMP ping target of the other customers. This is "guaranteed" to be across the Internet. The list is updated nightly.
The combined plot shows all three graphs on one 24 hour plot [representing yesterday]. This plot [BOTHYpingresult.png attached to your email] is an excellent reveal of overall performance, especially when compared to historical plots [available on the rpi web page], and when compared to other customers [available on the server web page -10 below]. This plot is available on the rpi web page, and is attached to the daily email. A representative BOTHYpingresult.png plot is attached [pi24BOTH...]. Notice the longer "deep" ICMP pings representing overall Internet sluggishness [from this pi24].
Comparisons between/among customers can be viewed here: https://johnloop.com/imonitor/customerplots/customerplots.html This view will be upgraded in the next months. This is a great comparison between technologies across the country.
2. An archive plot is created which takes the average of each of the 3 daily measurements in 1 above and plots a cumulative graph. This plot [ICMP-TCP-DelayArchive.png] is available on the rpi web page, and is attached to the daily email. A representative plot is attached [pi9archive.png]. Notice the consistency of the day to day averages, which represents good long term performance.
3. An historical host table [called "arp table" on rpi web page]. This table [actually a "history" of the arp cache on the rpi] records the appearance of hosts on your local private network [wifi and eth as long as they are on the same network], and accumulates their appearance. It monitors the arp cache, so it will even pick up hosts that do not respond to pings, including windows 10 which defaultly does not respond to ping! This should be an invaluable record on your network "intruders" going forward! This is only available on the rpi web page -tho the daily email does list NEW/GONE hosts.
The daily scan continues to report current IP hosts and changes, "gone" and "new" each day -reported in the daily email. Beware that the "ghosts" may come and go as they wake up and are discovered -"new", or go to sleep and become "gone." These tend to be phones, kindles, tablets, cameras, etc. By referencing the arp table, you can ID the hosts that come and go.
4. A "Daily news item" in the newsletter listing significant additions/changes/notes to the imonitor capabilities.
5. The speedtest archive plot has always been available to watch a history of your nightly and daily speedtests. This is only available on the rpi web page.
6. The top part of the rpi web page [which is linked in the daily email] links to all these most important plots. I am including an image of the current rpi web page as an attachment 2 [jmonitorWEB.pdf]. -click on "data snapshots"- You should peruse it to gauge the large range of performance stats and data I am collecting about your Internet performance and your network stats!
7. The "alerts" "seem" to be working properly [there are always collection problems if interruptions of less than 5 minutes occur and are difficult to track/alert]. Alerts are only available if you have not invoked "standalone." The alert list is as follows:
a. "boot" alert: this email is sent whenever the rpi loses power and is rebooted. This is SCHEDULED on the first of the month, otherwise it is unintended. It records the "last online time" and the "new online time" and the offline minutes. For more detail, there is a record on the rpi web page.
b. "access alert: DNS pull target timeout" This will happen occasionally and should not elicit concern unless many repeated occurrences.
c. "access alert: OFFLINE to Internet/ONLINE to router" This happens when the router loses Internet conectivity but is still connected to the router.
d. "access alert: IP address change. Your Internet address changed from: to: Last and Now online Time are listed
e. "access alert: DNS server change. Your DNS server [distributed by DHCP] has change from: to: or "reup'd"
8. The "QuickManual" -providing quick setup info, has been updated. It is attached as 2nd attachment.
9. I am adding a 4th attachment [Customer....pdf] which shows a spreadsheet listing the "guinea pigs" and a record of the [avg] TCP response times over the past month [collected weekly]. The consistency of the times is quite amazing... Maybe explained by the fact that it is an AVERAGE over 100 top web sites which are reliably online 24/7/365 via a CDN. The deep pings may give a better feel for "entire" Internet performance [...not possible -like measuring the temperature of the earth!].
10. Server side plots. I collect and make available at my server daily customer performance graphs. This will allow comparison across ISPs, access rates and technologies. These will be available at https://johnloop.com/imonitor/customerplots/ -first and second links. This is a simple directory listing for now.
11. There is a new speedtest script -dated 10/2019- which I will be incorporating. This uses the latest algorithms from ookla/speedtest.net. This should vastly improve the speedtest capabilities! Should be available by 2-5.
12. I would be remiss if I didn't mention a "competing" product which is very complementary to this software. This is the "fing" software -google it. You can get a fing app for your phone and a fing install for your PC. It does an especially nice job of detecting and listing objects on your network using a database it collects from its [registered] users. Yes, you have to register.... and your info is collected into their cloud. But it is more capable of ID'ing objects on your network because of its large MAC address database. There is even a separate "gadget" you can order for your network which does more functions. This is typical of "web/app" appearances these days - nice fancy graphics. I don't bemoan this [I wish I were as good], but I can't see the detail for the fluff, as is the case with much of this art. My development is all detail of course!! I have attached a screenshot of the "device" page.