Developing lightweight computation at the DSG edge

README.md 6.49 KB
Newer Older
1
2
# uc-monitor-go-test [![Go Report Card](https://goreportcard.com/badge/lightkone.guifi.net/lightkone/uc-monitor-go-test)](https://goreportcard.com/report/lightkone.guifi.net/lightkone/uc-monitor-go-test)

Roger Pueyo Centelles's avatar
Roger Pueyo Centelles committed
3
A proof-of-concept application that leverages [AntidoteDB](https://syncfree.github.io/antidote/) to orchestrate the Guifi.net network nodes monitoring system for the [LightKone](https://www.lightkone.eu/) project.
felix's avatar
felix committed
4

Roger Pueyo Centelles's avatar
Roger Pueyo Centelles committed
5
6
7
8
9
## Description
The monitoring application is distributed in three main blocks:
 - Network description fetching and feeding (functional)
 - Nodes assignment among the different monitoring servers [WiP]
 - Actual nodes monitorisation [WiP]
felix's avatar
felix committed
10

Roger Pueyo Centelles's avatar
Roger Pueyo Centelles committed
11
This proof of concept is built on top of the [AntidoteDB Java tutorial](https://github.com/SyncFree/antidote-java-tutorial) by [Deepthi Akkoorath (@deepthidevaki)](https://github.com/deepthidevaki) and uses [João Neto (@joaomlneto)](https://github.com/joaomlneto)'s [HTTP/HTTPS REST API for AntidoteDB](https://github.com/LightKone/antidote-rest-server).
felix's avatar
felix committed
12

Roger Pueyo Centelles's avatar
Roger Pueyo Centelles committed
13
## Installation
felix's avatar
felix committed
14

Roger Pueyo Centelles's avatar
Roger Pueyo Centelles committed
15
16
### Docker
Install `docker-ce` using your
felix's avatar
felix committed
17

Roger Pueyo Centelles's avatar
Roger Pueyo Centelles committed
18
19
### Go and required libraries
Install Go using your operating system's package manager or follow the [instructions here](https://golang.org/doc/install).
felix's avatar
felix committed
20

Roger Pueyo Centelles's avatar
Roger Pueyo Centelles committed
21
22
After installing Go, download and install the folowing external libraries needed:
 - golang/glog:
felix's avatar
felix committed
23

Roger Pueyo Centelles's avatar
Roger Pueyo Centelles committed
24
 `go get github.com/golang/glog`
felix's avatar
felix committed
25

Roger Pueyo Centelles's avatar
Roger Pueyo Centelles committed
26
 - sparrc/go-ping:
felix's avatar
felix committed
27

Roger Pueyo Centelles's avatar
Roger Pueyo Centelles committed
28
 `go get github.com/sparrc/go-ping`
felix's avatar
felix committed
29

Roger Pueyo Centelles's avatar
Roger Pueyo Centelles committed
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
### AntidoteDB Java tutorial
Get the tutorial's source code [here](https://github.com/SyncFree/antidote-java-tutorial) to download the code and [start the two AntidoteDB nodes](https://github.com/SyncFree/antidote-java-tutorial#starting-antidote-nodes).

### HTTP/HTTPS REST API

Install the AntidoteDB REST API server following the [instructions here](https://github.com/LightKone/antidote-rest-server). Once everything is installed, start the server with the `antidote-rest-server` command. The server connects to one of the AntidoteDB instances (the one running on port 8087) from the Java tutorial.

## Building and running the application

### Fetch the network description and feed it to AntidoteDB
The whole Guifi.net network description is included in the `cnml.xml` file, which can be downloaded from the Guifi.net website, and is used by default:
```
$ cd src/monitor/fetch/
$ go run monitor-fetch.go
57642 nodes read from cnml.xml
53461 devices read from cnml.xml
39988 devices exported to /tmp/gmonitor2/devs.json
0 devices removed from AntidoteDB (0 success, 0 fail)
0 IPv4 addresses removed from AntidoteDB (0 success, 0 fail)
10634 devices added to AntidoteDB (10634 success, 0 fail) ...
```
Roger Pueyo Centelles's avatar
Roger Pueyo Centelles committed
51
etc. Smaller subnetworks can be used instead, like the Guifi-UPC subnetwork, for testing purposes:
Roger Pueyo Centelles's avatar
Roger Pueyo Centelles committed
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
```
$ go run monitor-fetch.go -cnml_file upc.xml
19 nodes read from upc.xml
0 devices removed from AntidoteDB (0 success, 0 fail)
0 IPv4 addresses removed from AntidoteDB (0 success, 0 fail)
49 devices added to AntidoteDB (49 success, 0 fail)
67 IPv4 addresses added or updated to AntidoteDB (67 success, 0 fail)
```

### Assign network nodes to monitoring servers [WiP]
```
go run monitor-assign.go
```

### Monitor the nodes [WiP]
```
go run monitor-ping.go  <host>
```
### Note
Previously as of this note on Linux support: _this library attempts to send an "unprivileged" ping via UDP. On Linux, this must be enabled by setting_:

```sudo sysctl -w net.ipv4.ping_group_range="0 2147483647"```
Roger Pueyo Centelles's avatar
Roger Pueyo Centelles committed
74
75
76
77
78
79
80
81
82

## Data structures in AntidoteDB

### Devices/Monitors assignation
The primary data source for this application is the CNML file. The `monitor-fetch` application parses the specified CNML file and pushes its contents to AntidoteDB. There is a single `monitor-fetch` instance, and its writes/updates are __authoritative__.

In AntidoteDB, the data are structured as follows:

#### guifi (bucket)
Roger Pueyo Centelles's avatar
Roger Pueyo Centelles committed
83
The `guifi` bucket contains the lists of monitors and network devices to be monitored:
Roger Pueyo Centelles's avatar
Roger Pueyo Centelles committed
84
85
86
87
88
89
90
91
92
93
94
##### guifi (bucket) => devices (set)
The `devices` *set* in the `guifi` *bucket* is an `array` of `strings`, each `string` containing the `ID` of a Guifi.net device. For example:
```bash
$ curl localhost:3000/set/read/guifi/devices
  ["22110","26932","38720","40605","40962","41175","42331","42626","42627","42628",
   "46654","46656","47103","48030","51580","57728","59001","60415","64962","64963",
   "64965","64966","65291","65720","72843","73952","79715","81297","82096","82097",
   "82098","82099","82103","82104","82105","82111","83865","85877","87503","90228",
   "92032","92802","92803","92804","94210","94965","96225","96676","96684"]
```

Roger Pueyo Centelles's avatar
Roger Pueyo Centelles committed
95
96
97
98
##### guifi (bucket) => monitors (set)
The `monitors` *set* in the `guifi` *bucket* is an `array` of `strings`, each `string` containing the `ID` of a Guifi.net monitor. For example:
```bash
$ curl localhost:3000/set/read/guifi/monitors
Roger Pueyo Centelles's avatar
Roger Pueyo Centelles committed
99
  ["a45632","a47363", "21435"]
Roger Pueyo Centelles's avatar
Roger Pueyo Centelles committed
100
```
Roger Pueyo Centelles's avatar
Roger Pueyo Centelles committed
101
When started, each of the monitoring instances register to the system by adding their `ID` to this *set*.
Roger Pueyo Centelles's avatar
Roger Pueyo Centelles committed
102

Roger Pueyo Centelles's avatar
Roger Pueyo Centelles committed
103
104
105
106
107
108
##### guifi (bucket) => checksum (LWW<sup>1</sup> register)
The `checksum` *LWW register* in the `guifi` *bucket* is a `strings` containing the *SHA256 checksum* of the CNML data fetched from the Guifi.net website and pushed to the database. For example:
```bash
$ curl localhost:3000/register/read/guifi/checksum
  35aaa826b841ed412897691bb1f50278d742ef9a76da9750a8ae509d3b01f8ee
```
Roger Pueyo Centelles's avatar
Roger Pueyo Centelles committed
109

Roger Pueyo Centelles's avatar
Roger Pueyo Centelles committed
110
111
112
113
114
115
116
117
118
119
120
121
122
123
#### device-i (bucket)
The `device-i` bucket, where `i` is the numeric `ID` of a device in the `guifi/devices` set, contains the information about a Guifi.net device:

##### device-i (bucket) => ipv4s (set)
The `ipv4s` *set* in the `device-i` *bucket* is an `array` of `strings`, each `string` containing an IPv4 address of the device. For example:
```bash
$ curl localhost:3000/set/read/device-26932/ipv4s
  ["10.139.37.226","172.25.40.188","172.25.40.189"]
```

##### device-i (bucket) => monitors (set)
The `monitors` *set* in the `device-i` *bucket* is an `array` of `strings`, each `string` containing the ID of a monitor the device is assigned to (i.e. the `ID` of a monitor that is in charge of monitoring the device). For example:
```bash
$ curl localhost:3000/set/read/device-26932/monitors
Roger Pueyo Centelles's avatar
Roger Pueyo Centelles committed
124
  ["a45632","a47363", "21435"]
Roger Pueyo Centelles's avatar
Roger Pueyo Centelles committed
125
```
Roger Pueyo Centelles's avatar
Roger Pueyo Centelles committed
126
127

##### device-i (bucket) => graphserver (LWW register)
Roger Pueyo Centelles's avatar
Roger Pueyo Centelles committed
128
129
130
131
132
133
134
135
136
The `graphserver` *LWW register* in the `device-i` *bucket* is a `string` containing the ID of a monitor the device is assigned to **in the Guifi.net** website (i.e., not automatically assigned by the monitoring application, but done manually on the Guifi.net website, and included in the CNML). For example:

```bash
$ curl localhost:3000/register/read/device-26932/graphserver
  71808
```

---
<sup>1</sup> LWW: last writer wins