Developing lightweight computation at the DSG edge

Commit e6e94a56 authored by Roger Pueyo Centelles's avatar Roger Pueyo Centelles
Browse files

Update README


Signed-off-by: Roger Pueyo Centelles's avatarRoger Pueyo Centelles <rpueyo@ac.upc.edu>
parent 87cd582f
# uc-monitor-go-test [![Go Report Card](https://goreportcard.com/badge/lightkone.guifi.net/lightkone/uc-monitor-go-test)](https://goreportcard.com/report/lightkone.guifi.net/lightkone/uc-monitor-go-test)
A proof-of-concept application that leverages [AntidoteDB](https://syncfree.github.io/antidote/) to orchestrate the Guifi.net network nodes monitoring system for the [LightKone](https://www.lightkone.eu/) project.
An initial implementation, in the form of microservices, of a distributed, decentralised and resilient monitoring system for the Guifi.net network nodes. It uses [AntidoteDB](https://syncfree.github.io/antidote/) and is part of the [LightKone](https://www.lightkone.eu/) project.
## Description
The monitoring application is distributed in three main blocks:
- Network description fetching and feeding (functional)
- Nodes assignment among the different monitoring servers (functional, WiP)
- Actual nodes monitorisation (functional, WiP)
- Network description fetching and feeding: functional
- Automated nodes assignment among the different monitoring servers: functional
- Actual nodes monitorisation
- Uptime: functional
- SNMP: WiP
This proof of concept takes advantage of the [AntidoteDB Java tutorial](https://github.com/SyncFree/antidote-java-tutorial) by [Deepthi Akkoorath (@deepthidevaki)](https://github.com/deepthidevaki) and uses [Mathias Weber (@mweberUKL)](https://github.com/mweberUKL)'s Go client for AntidoteDB (previously, [João Neto (@joaomlneto)](https://github.com/joaomlneto)'s [HTTP/HTTPS REST API for AntidoteDB](https://github.com/LightKone/antidote-rest-server) was used).
This implementation uses [Mathias Weber (@mweberUKL)](https://github.com/mweberUKL)'s Go client for AntidoteDB.
This document briefly explains the applications, how to run them and how the data are structured.
It takes advantage of the [AntidoteDB Java tutorial](https://github.com/SyncFree/antidote-java-tutorial) by [Deepthi Akkoorath (@deepthidevaki)](https://github.com/deepthidevaki) and [João Neto (@joaomlneto)](https://github.com/joaomlneto)'s [HTTP/HTTPS REST API for AntidoteDB](https://github.com/LightKone/antidote-rest-server).
## Installation
......@@ -36,7 +41,7 @@ Get the tutorial's source code [here](https://github.com/SyncFree/antidote-java-
### HTTP/HTTPS REST API
The AntidoteDB REST API server is no longer needed, but you can follow the [instructions here](https://github.com/LightKone/antidote-rest-server) to install it, as it comes handy to perform quick tests.
The AntidoteDB REST API server is no longer used by the monitoring applications, but it comes handy to perform quick tests like the ones in this document. You can follow the [instructions here](https://github.com/LightKone/antidote-rest-server) to install it..
## Running the application
......@@ -60,7 +65,7 @@ $ go run monitor-fetch.go
```
Other small sub-networks' descriptions are available in the `assets/cnml` folder, some of them overlapping and some not.
Beyond the development and testing phase, the whole Guifi.net CNML description can be loaded (warning, it's HUGE):
Beyond the development and testing phase, the whole Guifi.net CNML description can be loaded (warning, it's HUGE, and it's GROWING):
```bash
$ go run monitor-fetch.go -cnml_file ../assets/cnml/guifi.xml
......@@ -86,7 +91,7 @@ Different options can be specified, like the monitor's `ID`, the maximum number
<details><summary><b>$ go run monitor-assign.go -id 12345 -maxDevs 5 -minMons 3</b> (click here to see the whole content)</summary>
<p>
`
```
Initializing...
Using ID 12345
Setting timestamp to 1560419260
......@@ -252,7 +257,6 @@ Reassignation of devices
5 devices currently assigned to this monitor (maximum: 5 devices)
Updating the current cnml...
Not assigning any new device
```
</p>
</details>
......@@ -262,7 +266,7 @@ Not assigning any new device
`monitor-assign` not only cares about assigning itself the devices to monitor, but also pushes this information to AntidoteDB, so that other monitors can indirectly coordinate and pick the right devices to monitor.
Additionally, it periodically sanitizes the global assignation in AntidoteDB, by pruning assignations from monitors that are not in the system anymore (crashed, unresponsive, in another network partition...)
### Monitor the nodes' liveliness (ping RTT and TTL) [WiP]
### Monitor the nodes' liveliness (ping RTT and TTL)
The `monitor-ping` pings the different devices the monitor is assigned periodically, and writes the information to AntidoteDB.
To have enough resolution for graphing, each device must be pinged *at least* every five minutes (and, ideally, this will happen simultaneously in other monitors, achieving the required redundancy).
The `monitor-ping` instance **must** use the same `ID` parameter as the local `monitor-assign` instance (in the future they will be merged into a single process):
......@@ -270,7 +274,7 @@ The `monitor-ping` instance **must** use the same `ID` parameter as the local `m
<details><summary><b>$ go run monitor-ping.go -id 12345</b> (click here to see the whole content)</summary>
<p>
`
```
Initializing...
Using ID 12345
Initialization done. Entering infinite loop...
......@@ -319,7 +323,7 @@ $ sudo sysctl -w net.ipv4.ping_group_range="0 2147483647"
as per this note on Linux support:
```this library attempts to send an "unprivileged" ping via UDP. On Linux, this must be enabled by setting```
```[...] this library attempts to send an "unprivileged" ping via UDP. On Linux, this must be enabled by setting [...]```
### Monitor the nodes' network traffic (SNMP) [WiP]
The `monitor-snmp` periodically asks the different devices the monitor is assigned periodically for their [SNMP information](https://en.wikipedia.org/wiki/Simple_Network_Management_Protocol) to get details about their interfaces and the inbound/outbound traffic.
......@@ -379,7 +383,7 @@ $ curl localhost:3000/set/read/device-26932/ipv4s
The `monitors` *set* in the `device-i` *bucket* is an `array` of `strings`, each `string` containing the ID of a monitor the device is assigned to (i.e. the `ID` of a monitor that is in charge of monitoring the device). For example:
```bash
$ curl localhost:3000/set/read/device-26932/monitors
["a45632","a47363", "21435"]
["a45632","a47363","21435"]
```
##### device-i (bucket) => graphserver (LWW register)
......@@ -403,85 +407,27 @@ The `rawping` *map* in the `device-i` *bucket* is a collection of nested maps wh
The following call will not work<sup>2</sup>, but gives an example of the data structure:
```bash
$ curl localhost:3000/map/list/device-26932/rawping/
{["2018", "2019"]}
{["2018-12-31","2019-01-01","2019-01-02","2019-01-03","2019-01-04","2019-01-05","2019-01-06", ... ,"2019-03-13","2019-03-14"]}
```
##### device-i (bucket) => rawping (map) => year (map)
The `year` *map* in the `rawping` *map* in the `device-i` *bucket* is a collection of nested maps where the raw ping data are stored.
##### device-i (bucket) => rawping (map) => year-month-day (map)
The `year-month-day` *map* in the `rawping` *map* in the `device-i` *bucket* contains the raw ping data stored during a given day.
The following call will not work<sup>2</sup>, but gives an example of the data structure:
```bash
$ curl localhost:3000/map/list/device-26932/rawping/2019/
{["01","02","03","04","05","06"]}
$ curl localhost:3000/map/list/device-26932/rawping/2019-03-14/
{["00-00-00-21435","00-00-00-a45632","00-00-00-a47363","00-05-00-21435","00-05-00-a45632","00-05-00-a47363","00-10-00-21435","00-10-00-a45632","00-10-00-a47363","23-50-00-21435","23-50-00-a45632","23-50-00-a47363","23-55-00-21435","23-55-00-a45632","23-55-00-a47363"]}
```
##### device-i (bucket) => rawping (map) => year (map) => month (map)
The `month` *map* in the `year` *map* in the `rawping` *map* in the `device-i` *bucket* is a collection of nested maps where the raw ping data are stored.
##### device-i (bucket) => rawping (map) => year-month-day (map) => hour-minute-second-monitorID (set)
The `year-month-day` *map* in the `rawping` *map* in the `device-i` *bucket* contains the raw ping data captured at a given moment by a specific monitor.
The following call will not work<sup>2</sup>, but gives an example of the data structure:
```bash
$ curl localhost:3000/map/list/device-26932/rawping/2019/01/
{["01","02","03","04","05","06","07","08","09","10","11","12","13","14","15",
"16","17","18","19","20","21","22","23","24","25","26","27","28","29","30","31"]}
```
##### device-i (bucket) => rawping (map) => year (map) => month (map) => day (map)
The `day` *map* in the `month` *map* in the `year` *map* in the `rawping` *map* in the `device-i` *bucket* is a collection of nested maps where the raw ping data are stored.
The following call will not work<sup>2</sup>, but gives an example of the data structure:
```bash
$ curl localhost:3000/map/list/device-26932/rawping/2019/01/03/
{["rtt","ttl"]}
```
##### device-i (bucket) => rawping (map) => year (map) => month (map) => day (map) => rtt (map)
The `rtt` *map* in the `day` *map* in the `month` *map* in the `year` *map* in the `rawping` *map* in the `device-i` *bucket* is a collection of `sets` named with a HHmmss-monitorID format where the raw RTT (round-trip time) are (times are UTC).
The following call will not work<sup>2</sup>, but gives an example of the data structure:
```bash
$ curl localhost:3000/map/list/device-26932/rawping/2019/01/03/rtt/
{["000000-1234"],["000000-3456"],["000000-5678"],
["000230-1234"],["000230-3456"],["000230-5678"],
["000500-1234"],["000500-3456"],["000500-5678"],
["000730-1234"],["000730-3456"],["000730-5678"],
...
["235730-1234"],["235730-3456"],["235730-5678"],
```
##### device-i (bucket) => rawping (map) => year (map) => month (map) => day (map) => rtt (map) => HHmmss-monitorID (set)
The `HHmmss-monitorID` *set* in the `rtt` *map* in the `day` *map* in the `month` *map* in the `year` *map* in the `rawping` *map* in the `device-i` *bucket* is a `set` where the raw RTT (round-trip time) values from the ping probe performed at time *HHmmss* (UTC) by monitor *monitorID* are stored. Set values are in microseconds (µs).
The following call will not work<sup>2</sup>, but gives an example of the data structure:
```bash
$ curl localhost:3000/set/read/device-26932/rawping/2019/01/03/rtt/235730-3456/
{["12452","9873","10347","12906","96991"]}
```
##### device-i (bucket) => rawping (map) => year (map) => month (map) => day (map) => ttl (map)
The `ttl` *map* in the `day` *map* in the `month` *map* in the `year` *map* in the `rawping` *map* in the `device-i` *bucket* is a collection of `sets` named with a HHmmss-monitorID format where the raw TTL (time-to-live) are.
The following call will not work<sup>2</sup>, but gives an example of the data structure:
```bash
$ curl localhost:3000/map/list/device-26932/rawping/2019/01/03/ttl/
{["000000-1234"],["000000-3456"],["000000-5678"],
["000230-1234"],["000230-3456"],["000230-5678"],
["000500-1234"],["000500-3456"],["000500-5678"],
["000730-1234"],["000730-3456"],["000730-5678"],
...
["235730-1234"],["235730-3456"],["235730-5678"],
```
##### device-i (bucket) => rawping (map) => year (map) => month (map) => day (map) => ttl (map) => HHmmss-monitorID (set)
The `HHmmss-monitorID` *set* in the `ttl` *map* in the `day` *map* in the `month` *map* in the `year` *map* in the `rawping` *map* in the `device-i` *bucket* is a `set` where the raw TTL (time to live) values from the ping probe performed at time *HHmmss* (UTC) by monitor *monitorID* are stored. Set values are integers ≤ 64.
The following call will not work<sup>2</sup>, but gives an example of the data structure:
```bash
$ curl localhost:3000/set/read/device-26932/rawping/2019/01/03/ttl/235730-3456/
{["59","59","58","59","59"]}
$ curl localhost:3000/map/list/device-26932/rawping/2019-03-14/23-55-00-a45632
{["12452","9873","10347","12906","96991","62"]}
```
where the 5 first numbers correspond to the ping RTT, in nanoseconds, and the last one to the TTL count.
---
<sup>1</sup> LWW: last writer wins
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment