Dear imonitor users, potential users, former users, and interested parties:
Summary for those of you new to this newsletter:
I am working on a project on Internet monitoring, and have a small device which I can deploy to help measure/monitor your Internet service. I have several "guinea pigs" deployed and am always interested in expanding my "customer" base. I initially targeted the service to users on the mountain in Jasper GA (windstream ISP), but it is applicable universally, and I have "customers" on Windstream, ATT, Spectrum, Hughes, Comcast, Qwest, CenturyLink, and several others. I am always interested in getting "customers - guinea pigs" on other ISPs. Please read on if you are interested. I am particularly interested in comments and suggestions from my more "nerdy" customers!!
Please let me know if you would like one of these devices. I still have a few available to deploy free to "guinea pigs"! I can give
login access for nerds! It is a wonderful way to learn linux. The temp monitoring can also be very revealing as to HVAC performance -putting temp probe in house.
As usual, I am extremely grateful for the use of your ISP connection to develop this service across many ISPs. It has been invaluable.
As always, you can refer to the main information page at https://johnloop.com/imonitor.html
That page will contain a lot of details and diagrams of the service.
There are additional detailed references linked on that page.
There is a new "quick intro" doc at https://johnloop.com/imonitor/QuickManual.pdf [also attached to this email]
Newsletter 7-1-2019 previous newsletter https://johnloop.com/imonitor/newsletter3-3-2019.html
There is news in these areas
1. The plots for ICMP and TCP ping response times are a great overall way to gauge your Internet connection. Yesterday's ICMP and TCP ping plots are attached to your daily email. You can link to the yesterday plots, and the running plot for today -to last hour- in your daily email. Most illuminating is to go to the rpi web page [linked in the email], click on "counts" at top and scroll down to the "current month plot archive." Here you can click on a single plot from any day [the date on the file indicates the plot creation, the plot applies to the day before]. Click the browser "back" arrow and go back to the archive and click on the next one. The "previous" link is a directory where the plots for the previous year will be archived.
I will continue work to make the plots more meaningful. Hoping to put actual time instead of "count" on the horizontal axis. Right now the ping counts start at 2AM and continue to 1AM the next morning [23 hours], representing a count of 1357. This can be seen on the plot. 2AM will be on the left, and each 60 count represents an hour. 1AM will be on the right at the 1357 count. The steadiness of the ping response times is most important. If there are bumps in the graph, it represents congestion is encounterd. Outright timeouts are declared when the ping reply is not returned after 1 sec -1000msec. These represent massive congestion, or offline times. If the Internet possessed unlimited bandwidth [and the ping targets were never busy...], these times would never vary. So it is an interesting indication just how open the Internet is for your connection. Some of you have excellent connections, others not so!
The TCP pings are generated for all 24 hours for a count of 1410 [unless there are periods of being offline, which will reduce this count]. Most importantly, the TCP ping is only performed when the rpi is declared "online." Look for correlation of ICMP congestion coincident with the TCP congestion. The TCP target is configurable, but right now it is suggested as either www.google.com or www.msftncsi.com. It represents a much deeper target in the Internet than the ICMP ping target, which is configured to be a few hops away.
2. The temperature probe plots are quite useful to monitor HVAC performance over long periods of time. An easy way to spot HVAC problems! They are listed in the same plot directory, and archived as well. Ask me if you want one of the probes. Pretty good reason for being a guinea pig is just for this capability! Pretty simple, but you will have to plug 3 cable jacks onto the rpi circuit board [I will have instructions] I will then have to enable the probe. You can put the temp probe anyplace of course - it has a 10 foot cable- but the rpi must be within ethernet or wifi of your router.
3. A lot of work has been completed on selecting more sensible ICMP ping targets. An "auto" routine at 1AM each nite determines -for the next day- the first pingable hop from your Internet presence starting at hop 5 and working backward. This is the ping target for the day -starting at 2AM. This will make much more sense in monitoring "near end" ISP local Internet performance.
4. A "ping lock" feature is added allowing you to "LOCK" the ping target [or return it to "AUTO"]. You have the option of entering your own ping target on the rpi web page, or using the results of the auto feature and then lock the result. Use the "route.txt" listing in the "counts" quick link and scroll down to "1AM IP route to IXC" and you can see the 5 hops into the Internet from your presence [FQDN on left, and raw IP on right]. Normally the ping target will be the fifth hop -if it is pingable-. Your situation may dictate a closer ping target to monitor. Notice the left panel to get some feeling for the actual "location" of the ping target. Overall, it is best to leave this in "auto" mode unless there are circumstances you understand.
5. The DNS server assigned by your DHCP is monitored, and a day over day change is flagged in the email. Often this is the router itself, so this will not [should not] change over time. At other times, the DHCP provided is an actual Internet site, probably the ISP DNS server.
6. A test is performed each night to determine if your DNS queries are being intercepted by your ISP. This is known as using a transparent DNS proxy, and may be used by some ISPs to inject their own DNS servers for adds or other reasons. It is not done very often and is considered "bad behavior." It will prevent you from specifying your own DNS server manually. It is often very advantageous to manually specify one of the current "universal" DNS servers such as 8.8.8.8, 9.9.9.9, 1.1.1.1 or 208.67.222.222 server in your PC/laptop or even in your router if possible. These DNS servers far exceed the services and capabilities of routine ISP DNS servers. They are even scattered around the Internet so they may be closer than your ISP-assigned DNS server. One of the worst Internet "atrocities" is DNS hijacking, where susceptible routers can be configured with a malicious DNS server, thus directing your name queries to malicious actors, and potentially directing you to web site imposters. The use of one of these servers goes a long way to protect you in DNS name resolution, esp the 9.9.9.9.
The optimum case would be to specify a DNS server such as 8.8.8.8 [google], 9.9.9.9 [ibm], 1.1.1.1 [cloudflare] or 208.67.222.222 [opendns] in your router. The router may pass itself off as the relay, or it might actually pass these servers off on DHCP. The Rpi can then monitor any DNS change on the DHCP.
There is no universal way to setup the DNS server in your router, but it is very worthwhile to make this change if you can figure it out!
7. I have added a javascript popup to the rpi web page directing you to a very short intro document to the use of this service -"QuickManual.pdf". I have attached this doc here as a "noteworthy" addition. The document is a very short into to setting up your imonitor gadget and a list of features.
8. If you have not enabled the internal scan of your network via the rpi internal web page, you should try this. The results of this scan will reveal hosts and open service ports inside your network, which might be very important if you live in a wifi maze! The results of this scan are not uploaded and do not leave the local rpi. There is a link to this scan in your email. I can help you interpret it if you want. It can get very complicated!
9. On the external scan of your connection from the Internet, thanks to Jerry for discovering the TCP/113 problem. This is ignored on most modern interfaces, but my fwall was returning a closed for it. So all 1000 first ports are normally shown as "filtered" now. Caution should be exercised if you have open ports on the Internet of course!!! Besides the basic scan of the first 1000/other important ports on your Internet connection that are performed each night, I am scanning the CWMP port 7547, which was used by the ISPs for many years to manage/update your router. This is less and less the case, but it is a noteworthy port that can be vulnerable on a router.
10. Going forward, there will be less emphasis on doing simultaneous wifi and ethernet testing. Some routers have a hard time living with one host appearing on two different interfaces on the same network. I am having difficulty working around/with this situation. The initial goal was to do the basic testing over which interface the rpi uses as default - we have both ethernet and wifi customers. If you are default wifi, both the basic and the wifi testing is performed over the wifi interface. If you are using primarily ethernet, but also supply the wifi password of your network, I can perform local -wifi- testing to the router, thus gauging the wifi local performance. This is a noble goal, and does actually work for the majority of cases, but there are situations, when the two interfaces are on the same network that the DHCP-router or linux complains.
11. Remember the email options. If you are weary of a daily email, turn daily email to NO -you will get a weekly email on Saturday. OR turn email to OFF and you will get nothing at all, and you will have to rely on the rpi web page for all your info. Turning email OFF will also eliminate the alerts that are generated.
12. Remember the management options. You can completely lock me out by turning on STANDALONE. More preferably, just disable management until the next 1AM. I will then have to ask you when I want to upgrade/access the rpi. If you are a guinea pig, please let me in occasionally to update.
I am eternally grateful for allowing me to use your Internet connection to develop this service. I am hoping it is a more valuable service as time goes on.
General description of rpi monitor tests:
Bash scripts running on a small computer on your network attempt a connect to a target every minute of every day. Statistics are reset at beginning of month. The target is configurable, default is google.com.
The scripts perform a speedtest at 3AM and 3PM. This is done from the Rpi of course!
The scripts perform a single ping every minute of every day to a stable Internet IP address. These are counted and reset at beginning of every day. There will be an entry for each ping failure. The ping target is configurable.
The wait time for both the connect and the ping is 15 seconds; no response and a failure is declared. There are 44640 minutes in a 31 day month. I don't do quite that many.... housekeeping and such.
Last minute ping delays
Yesterday ping delay average and plot
Today ping delay plot -from 2AM to when you click the link.
Last hour ping delay plot
Route to IXC
Yesterday temp average [if enabled]. Up/down limits can be set and you can be alerted.
Scan of wifi networks from your location listing all SSID and sig levels, other info
Scan of your Internet connection to check for vulnerability
Scan of your internal network -if enabled
Week's worth of stats available in piweeklytars directory
Internet IP address changes are recorded, and you can be alerted for a change
Boot times of the Rpi, which may represent power bounces in your house. You are alerted.
A browsable web page [safely on the raspberry pi on your local network] with all these stats available at any time. Link provided in the email.
Access to your router [gateway] via the web page.
Raspberry Pi can be behind a secondary router.
There is absolutely no monitoring, capturing of your Internet traffic. The raspberry pi generates its own to collect stats. The bandwidth demands on your connection average less than 1Kb/s.
I can give you access/login to the raspberry pi if you wish.
I can give you an SD micro with the software if you have your own raspberry pi 3B or 3B+.
John Loop pccitizen@gmail.com txt to 706 669 7164 or jdloop@johnloop.com