Developing lightweight computation at the DSG edge

Commit a41ab420 authored by Roger Pueyo Centelles's avatar Roger Pueyo Centelles
Browse files

Import qmp-cooker repository

qmp-cooker repository is contributed by p4u and is based on LibreMesh's lime-sdk cooker (https://github.com/libremesh/lime-sdk)
parent 208aba02
17.01.0
tmp
dl
feeds
feeds.conf
output
files
*.local
communities/
lede/
This diff is collapsed.
## qmp-cooker
\ No newline at end of file
# lime-sdk cooker
LibreMesh **software development kit** uses the [LEDE](https://lede-project.org/docs/start) SDK and ImageBuilder to generate (**cook**) LibreMesh packages and firmware. If you want to create your own LibreMesh flavor because you need some specific configuration or you just want to have control over your binaries, the cooker is your friend!
Basic usage example for cooking a firmware for TpLink 4300:
`./cooker -c ar71xx/generic --flavor=lime_default --profile=tl-wdr4300-v1`
## Preparing the local environment
Before using lime-sdk, make sure your Linux system has the required dependencies installed. You might follow these instructions (look for _Examples of Package Installations_) https://lede-project.org/docs/guide-developer/install-buildsystem
## Targets, profiles and flavors
LibreMesh can be used on many different devices (target and profile) and can be packed in many different ways (flavors), depending on your needs. To this end, it is important to choose the right options for building your firmware.
To generate a firmware, the _-c_ option must be used (**c**ook). But it requires to specify at least the target and subtarget of your router and optionally (recommended) the profile and flavor.
`./cooker -c <target/subtarget> --profile=<profile name> --flavor=<flavor name>`
For instance, this will work for a TpLink WDR4300:
`./cooker -c ar71xx/generic --profile=tl-wdr4300-v1 --flavor=lime_default`
##### Target
Target references to the router architecture, usually depends on the manufactor and the set of chips used for building the hardware. Therefore, you must know the target and subtarget before using cooker. As we use LEDE, this information can be found here https://lede-project.org/toh/start. The most common targets are currently _ar71xx/generic_ (Atheros) and _ramips/mt7620_ (Ramips). Once we know the target, we must find the specific profile.
To see the list of available targets execute:
`./cooker --targets`
##### Profile
The profile is the specific brand/model of the router. Each target has a list of hardware profiles than can be choosed. Cooker build all profiles from a target by default, but it is better if your find and choose the specific profile.
To see the list of available profiles for a specific target execute:
`./cooker --profiles=<target/subtarget>`
For instance:
`./cooker --profiles=ar71xx/generic`
##### Flavor
LibreMesh is a modular system, so it can be cooked on many different ways. There are some predefined that we call _flavor_, however anyone can create its own set of packets and options (for instance the default flavors include _bmx6_ and _batman-adv_ as routing protocols, but you might create other kinds of setup).
One of the most important things regarding the flavor is the internal flash size of your router. This must be taken into account when choosing a flavor.
Currently there are three main flavors:
* **lime_default:** the recommended for routers with more than 4MB of flash. It includes all required and optional software.
* **lime_mini:** the recommended for routers of 4MB, made for end-users, includes a minimal web interface, but new software cannot be installed (opkg is not available).
* **lime_zero:** for advanced users, it does not include web interface, just the basic software to mesh the network but it does include opkg, so new software can be installed.
## Building and cooking
These are two different steps. **Building** means to compile and prepare all the required packages for LibreMesh. To **cook** means taking the packages (depending on the flavor) and generating the firmware ready to install on your device.
The standard steps to generate a firmware would be: firstly build and secondly cook, like this:
`./cooker -b ar71xx/generic`
`./cooker -c ar71xx/generic --profile=tl-wdr4300-v1 --flavor=lime_default`
However, cooker is smart enough to detect the missing steps and transparently execute them. If we choose to cook before building, it will automatically build before cooking. Therefore, for debugging purposes it is better to execute the steps separately.
On the other hand, if you do not want to build locally (since it requires some special software installed on your Linux machine), you can just cook using the online precompiled binaries.
##### Building locally or fetch remote?
Cooker can locally build the LibreMesh packages or fetch the remote precompiled ones. For most users there is no real need for building, since using the remote ones might be a better (and fast) option. To remotelly fetch the packages the special option _--remote_ must be used when cooking, like this:
`./cooker -c ar71xx/generic --profile=tl-wdr4300-v1 --flavor=lime_default --remote`
## Communities
It is not required to create and/or use a community profile. But if you are constantly cooking LibreMesh for your network and you need a specific configuration, you might want to spend some time creating a community profile. This will make things easier and better coordinated for your community.
A community is mainly a set of files you want to include in the output firmware. For instance, if you want to pre-configure the WiFi SSID, mode or channels, you might want to include a specific _/etc/config/lime-defaults_ file as shown in this article http://libremesh.org/docs/en_config.html.
Also _/etc/shadow_ for setting an initial root password or _/etc/uci-defaults/_ one-time executed scripts might be useful for your setup.
The default way to create or use a community is to use this Git repository https://github.com/libremesh/network-profiles (ask for writing access in the users mailing list). The directory structure of the Git repository is:
`/<community name>/<device profile name>/<files and directories>`
Both community and device profile names can be any of your choice (must exist!) , since they are only used for identifying it. When executing a cook order, you can specify the community profile like this:
`./cooker -c ar71xx/generic --profile=tl-wdr4300-v1 --flavor=lime_default --community=CommunityName/ProfileName`
## Using development branch
If you want to get the last LEDE source because it includes some new feature or it supports some new hardware, you can use the lime-sdk branch named _develop_. However as LEDE source is changing daily, we cannot assure the correct working of the firmware.
It is recommended to start with a new Git clone instead of reuse an existing one. Once the lime-sdk source is cloned, change the branch: `git checkout develop`
## Add your own feed repository
If you want to compile and/or cook your own feed package repository, you can follow one of the following methods.
##### For a permanent build environment
If it is a permanent change on your cooker setup, better add your repository (or modify the existing ones) to the feeds file
cp feeds.conf.default feeds.conf.default.local
vim feeds.conf.default.local
Edit and save the new created file _feeds.conf.default.local_ and force the reinstall of the feeds
./cooker -f --force
Crete and add to the SDK config file the new packages you want to include (if any)
cp libremesh.sdk.config libremesh.sdk.config.local
echo "CONFIG_PACKAGE_myNewPackage=m" >> libremesh.sdk.config.local
Add your new flavor (or modify the existing ones)
cp flavors.conf flavors.conf.local
vim flavors.conf.local
Finally build and cook as usual but adding also your new packages
./cooker -b ar71xx/generic
./cooker -c ar71xx/generic --profile=tl-wdr4300-v1 --flavor=lime_new_flavor
##### For a casual cooking on a existing feed repository
Download the standard feeds (if not previously downloaded)
./cooker -f
Modify the source code of the existing feed
cd feeds/libremesh
git checkout feature/somethingToTest
Build the code and cook as usual
./cooker -b ar71xx/generic
./cooker -c ar71xx/generic --profile=tl-wdr4300-v1 --flavor=lime_default --extra-pkg="someExtraPackage?"
## Forking lime-sdk for your community
If you like to manage your own set of flavors, options and/or repositories, you might fork the lime-sdk code to your own Git repository. To preserve the compatibility with the official source (so merges can be easily done), none of the original files must be modified.
To this end, _cooker_ will look first for the files named _.local_ and will use them instead. Therefore make a copy of _options_ and _flavors_.
cp options.conf options.conf.local
cp flavors.conf flavors.conf.local
Modify them as your own wish and add them to the Git repository.
git add *.local
git commit -m 'Add local options and flavors'
git push
Time to time, if you want to update the code with the official one you might add a new remote and perform a merge.
git remote add official https://github.com/libremesh/lime-sdk.git
git fetch official
git merge official/master
git push origin/master
## Advanced help
Usage: ./cooker [-f [--force]] [-d <target> [--sdk|ib|force]] [-i <target> [--sdk-file=<file>|ib-file=<file>]]
[--download-all|build-all|update-feeds] [--targets|flavors|communities|profiles=<target>]
[-b <target> [--no-update|no-link-ib|remote|clean|package=<pkg>]]
[-c <target> [--profile=<profile>|no-update|remote|flavor=<flavor>|community=<path>|extra-pkg=<list>]]
[--help]
--help : show full help with examples
--download-all : download all SDK and ImageBuilders
--build-all : build SDK for all available tagets
--cook-all : cook firmwares for all available targets and profiles
--targets : list all officialy supported targets
--profiles=<target> : list available hardware profiles for a specific target
--flavors : list available LibreMesh flavors for cooking
--communities : list available community profiles
--update-feeds : update previously downloaded feeds (only works for Git feeds)
-f : download feeds based on feeds.conf.default file. Feeds will be shared among all targets
--force : force reinstall of feeds (remove old if exist)
-d <target> : download SDK and ImageBuilder for specific target
--sdk : download only SDK
--ib : download only ImageBuilder
--force : force reinstall of SDK and/or ImageBuilder (remove old if exist)
-i <target> : install local/custom SDK or ImageBuilder
--sdk-file=<file> : specify SDK file to unpack
--ib-file=<file> : specify ImageBuilder file to unpack
-b <target> : build SDK for specific target and link it to the ImageBuilder
--no-link-ib : do not download and link ImageBuilder when building the SDK
--no-update : do not update feeds when building SDK
--clean : clean sources before compiling
--package=<pkg_name> : build only a package (and its dependencies)
-c <target> : cook the firmware for specific target. Can be used with next options
--profile=<profile> : use <profile> when cooking firmware (default is all available target profiles)
--flavor=<flavor> : use <flavor> when cooking firmware (default lime_default)
--extra-pkg=<pkg_list> : specify extra packages (separated by spaces) for including into the output firmware
--remote : instead of building local SDK packages. Use only remote repositories for cooking
--community=<name/prof> : specify which network community and profile device to use (if any)
--no-update : do not update package list when cooking (requires patch_ib_no_update.sh snippet)
Examples:
- Build packages using the SDK and cook the firmware for target tl-wdr3500-v1 and flavor generic (all in one command)
./cooker -c ar71xx/generic --flavor=lime_default --profile=tl-wdr3500-v1
- Cook the firmware without compiling the SDK but using only remote precompiled binaries
./cooker -c ar71xx/generic --remote --flavor=lime_mini --profile=tl-wdr3500-v1
- Build SDK and cook ar71xx target with all available profiles (step by step)
./cooker -d ar71xx/generic # download SDK and IB
./cooker -f # download and prepare feeds
./cooker -b ar71xx/generic # build the SDK and link it to IB
./cooker -c ar71xx/generic --flavor=lime_default # cook all firmwares for target ar71xx/generic
- If you want to use an existing community network profile, specify it when cooking (in addition to the device profile)
./cooker -c ar71xx/generic --flavor=lime_default --community=quintanalibre.org.ar/comun --profile=tl-wdr3500-v1
- If not profile defined, cook all profiles of target. Also --extra-pkg option can be used to add extra packages when cooking
./cooker -c ar71xx/generic --flavor=lime_zero --extra-pkg="luci-proto-3g iperf"
- To see/debug build errors use J (number of threads) and V (verbose) system vars
J=1 V=s ./cooker -b ar71xx/generic --profile=tl-wdr3500-v1
This diff is collapsed.
src-git base https://github.com/lede-project/source.git;v17.01.2
src-git packages https://git.lede-project.org/feed/packages.git;lede-17.01
src-git luci https://git.lede-project.org/project/luci.git;lede-17.01
src-git routing https://git.lede-project.org/feed/routing.git;master
src-git telephony https://git.lede-project.org/feed/telephony.git;lede-17.01
src-git libremesh https://github.com/libremesh/lime-packages.git;17.06
src-git libremap https://github.com/libremap/libremap-agent-openwrt.git;master
src-git limeui https://github.com/libremesh/lime-ui-ng.git;master
# Flavors
lede_vanilla="luci-ssl"
lime_default="lime-full -dnsmasq"
lime_mini="lime-basic -opkg -wpad-mini hostapd-mini -kmod-usb-core -kmod-usb-ledtrig-usbport -kmod-usb2 -ppp -dnsmasq -ppp-mod-pppoe -6relayd -odhcp6c -odhcpd -iptables -ip6tables"
lime_zero="lime-basic-no-ui -wpad-mini hostapd-mini -ppp -dnsmasq -ppp-mod-pppoe -6relayd -odhcp6c -odhcpd -iptables -ip6tables"
lime_newui_test="lime-full lime-webui-ng-luci -dnsmasq"
src/gz libremesh http://repo.libremesh.org/current/packages/{ARCH}/libremesh
src/gz libremap http://repo.libremesh.org/current/packages/{ARCH}/libremap
src/gz limeui http://repo.libremesh.org/current/packages/{ARCH}/limeui
src/gz lm_routing http://repo.libremesh.org/current/packages/{ARCH}/routing
# CONFIG_ALL_NONSHARED is not set
# CONFIG_ALL_KMODS is not set
# CONFIG_ALL is not set
CONFIG_SIGNED_PACKAGES=n
CONFIG_DEFAULT_dnsmasq=n
CONFIG_PACKAGE_dnsmasq=n
CONFIG_PACKAGE_dnsmasq-dhcpv6=y
CONFIG_PACKAGE_ALFRED_BATHOSTS=y
CONFIG_PACKAGE_lime-full=y
CONFIG_PACKAGE_lime-basic=m
CONFIG_PACKAGE_lime-basic-no-ui=m
CONFIG_PACKAGE_lime-basic-uing=m
CONFIG_PACKAGE_lime-freifunk=m
CONFIG_PACKAGE_lime-full=m
CONFIG_PACKAGE_lime-ap-watchping=m
CONFIG_PACKAGE_lime-debug=m
CONFIG_PACKAGE_lime-hwd-ground-routing=m
CONFIG_PACKAGE_lime-hwd-openwrt-wan=m
CONFIG_PACKAGE_lime-hwd-usbradio=m
CONFIG_PACKAGE_lime-map-agent=m
CONFIG_PACKAGE_lime-proto-anygw=m
CONFIG_PACKAGE_lime-proto-batadv=m
CONFIG_PACKAGE_lime-proto-bgp=m
CONFIG_PACKAGE_lime-proto-bmx6=m
CONFIG_PACKAGE_lime-proto-bmx7=m
CONFIG_PACKAGE_lime-proto-eigennet=m
CONFIG_PACKAGE_lime-proto-olsr2=m
CONFIG_PACKAGE_lime-proto-qmp=m
CONFIG_PACKAGE_lime-proto-wan=m
CONFIG_PACKAGE_lime-system=m
CONFIG_PACKAGE_lime-webui=m
CONFIG_LIMEWEBUI_ES=y
CONFIG_PACKAGE_lime-webui-ng=m
CONFIG_PACKAGE_lime-webui-ng-luci=m
CONFIG_PACKAGE_luci-app-lime-location=m
CONFIG_PACKAGE_luci-lib-libremap=m
CONFIG_PACKAGE_luci-lib-libremap-babel=m
CONFIG_PACKAGE_luci-lib-libremap-bmx6=m
CONFIG_PACKAGE_luci-lib-libremap-contact=m
CONFIG_PACKAGE_luci-lib-libremap-freifunk=m
CONFIG_PACKAGE_luci-lib-libremap-location=m
CONFIG_PACKAGE_luci-lib-libremap-olsr=m
CONFIG_PACKAGE_luci-lib-libremap-qmp=m
CONFIG_PACKAGE_luci-lib-libremap-system=m
CONFIG_PACKAGE_luci-lib-libremap-wireless=m
CONFIG_LUCI_SRCDIET=y
CONFIG_PACKAGE_libremap-agent=m
CONFIG_PACKAGE_lime-api-orange-rpc=m
release=17.01.2
base_url=https://downloads.lede-project.org/releases/$release/targets/
communities_git=https://github.com/libremesh/network-profiles.git
communities_dir=communities
tmp_dir=tmp
feeds_default_file=feeds.conf.default
feeds_file=feeds.conf
feeds_dir=feeds
files_dir=files
flavors_file=flavors.conf
sdk_config=libremesh.sdk.config
sdk_install_packages="libustream-openssl firewall"
sdk_install_repos="libremesh libremap limeui"
remote_pkg_repos="libremesh.repositories.conf"
default_flavor="lime_default"
targets_list=targets.list
downloads_dir=dl
make_j=1
bin_output=output
src-git base https://github.com/lede-project/source.git;v17.01.2
src-git packages https://git.lede-project.org/feed/packages.git;lede-17.01
src-git luci https://git.lede-project.org/project/luci.git;lede-17.01
src-git routing https://git.lede-project.org/feed/routing.git;master
src-git telephony https://git.lede-project.org/feed/telephony.git;lede-17.01
src-git qmp https://dev.qmp.cat/qmp.git;master
src-git libremap https://github.com/libremap/libremap-agent-openwrt.git;master
# Flavors
lede_vanilla="luci-ssl"
qmp_big="qmp-big-node"
qmp_small="qmp-small-node"
qmp_tiny="qmp-tiny-node -opkg -wpad-mini hostapd-mini -kmod-usb-core -kmod-usb-ledtrig-usbport -kmod-usb2 -ppp -ppp-mod-pppoe -6relayd -odhcp6c -odhcpd -ip6tables -firewall"
# CONFIG_ALL_NONSHARED is not set
# CONFIG_ALL_KMODS is not set
# CONFIG_ALL is not set
CONFIG_SIGNED_PACKAGES=n
CONFIG_PACKAGE_dnsmasq-dhcpv6=y
CONFIG_PACKAGE_qmp-big-node=m
CONFIG_PACKAGE_qmp-small-node=m
CONFIG_PACKAGE_qmp-tiny-node=m
CONFIG_PACKAGE_luci-app-lime-location=m
CONFIG_PACKAGE_luci-lib-libremap=m
CONFIG_PACKAGE_luci-lib-libremap-babel=m
CONFIG_PACKAGE_luci-lib-libremap-bmx6=m
CONFIG_PACKAGE_luci-lib-libremap-contact=m
CONFIG_PACKAGE_luci-lib-libremap-freifunk=m
CONFIG_PACKAGE_luci-lib-libremap-location=m
CONFIG_PACKAGE_luci-lib-libremap-olsr=m
CONFIG_PACKAGE_luci-lib-libremap-qmp=m
CONFIG_PACKAGE_luci-lib-libremap-system=m
CONFIG_PACKAGE_luci-lib-libremap-wireless=m
CONFIG_LUCI_SRCDIET=y
CONFIG_PACKAGE_libremap-agent=m
FLAVORS="lime_default lime_zero lime_mini"
COMM=""
export J=1
./cooker --download-all
bash snippets/copy_signing_keys.sh
./cooker --update-feeds
./cooker --build-all # --clean
for F in $FLAVORS; do
./cooker --cook-all --flavor=$F
for C in $COMM; do
./cooker --cook-all --flavor=$F --community=$C
done
done
#!/bin/bash
. options.conf
RELEASE="$release"
[ ! -d $RELEASE ] && echo "Dir $RELEASE not found" && exit 1
[ ! -d keys ] && mkdir keys
[ ! -f keys/key-build ] || [ ! -f keys/key-build.pub ] && {
echo "Signing keys not available. For signing packages you must create and copy them into keys/ directory."
echo "File names must be keys/key-build (private) and keys/key-build.pub (public)"
exit 0
}
find $RELEASE/ -maxdepth 3 -type d -name 'sdk' | while read d; do
echo "Installing keys on $d..."
cp -f keys/key-build* $d/
done
cat $sdk_config | sed s/CONFIG_SIGNED_PACKAGES=n/CONFIG_SIGNED_PACKAGES=y/g > ${sdk_config}.local
#!/bin/bash
. options.conf
OUTDIR="${1:-repository}"
[ ! -d $OUTDIR ] && mkdir -p $OUTDIR
echo "-> Output directory: $OUTDIR"
for link in $OUTDIR/*; do
unlink $link 2>/dev/null
done
for target in $(cat $targets_list); do
for arch in $release/$target/sdk/bin/packages/*; do
[ -d "$arch" ] && {
echo "-> Creating symlink for $arch"
ln -s $PWD/$arch $OUTDIR/ 2>/dev/null
}
done
done
#!/bin/bash
. options.conf
echo "Patches all installed ImageBuilders to not opkg update if NO_UPDATE=1"
RELEASE="$release"
[ ! -d $RELEASE ] && echo "Dir $RELEASE not found" && exit 1
find $RELEASE/ -maxdepth 3 -type d -name 'ib' | while read d; do
echo "Patching $d..."
sed -i s/'\($(OPKG) update || true;\)'/'[ -z \"$(NO_UPDATE)\" ] \&\& \1'/g $d/Makefile
done
#!/bin/bash
. options.conf
RELEASE="$release"
echo '--- Makefile2 2017-03-30 07:44:01.402299190 +0300
+++ Makefile 2017-03-30 07:43:19.065015108 +0300
@@ -123,6 +123,14 @@
ifneq ($(USER_FILES),)
$(MAKE) copy_files
endif
+ifneq ($(FILES_REMOVE),)
+ @echo
+ @echo Remove useless files
+
+ while read filename; do \
+ rm -rfv "$(TARGET_DIR)$$filename"; \
+ done < $(FILES_REMOVE);
+endif
$(MAKE) package_postinst
$(MAKE) build_image
$(MAKE) checksum
' > tmp/ib_remove_files.patch
[ ! -d $RELEASE ] && echo "Dir $RELEASE not found" && exit 1
find $RELEASE/ -maxdepth 3 -type d -name 'ib' | while read d; do
echo "Patching $d..."
patch -p1 $d/Makefile < tmp/ib_remove_files.patch
done
#!/bin/bash
. options.conf
base_feed="$feeds_dir/base"
echo "Patch will be applied to $base_feed"
[ ! -d $base_feed ] && {
echo "$base_feed does not exist. Before applying this snippet you must clone the feeds (option -f of cooker)"
exit 1
}
patch_file="$PWD/$tmp_dir/regdb.patch"
cat > $patch_file << EOF
diff --git a/package/kernel/mac80211/files/regdb.txt b/package/kernel/mac80211/files/regdb.txt
index c4a9b2d15f..ec2c21543a 100644
--- a/package/kernel/mac80211/files/regdb.txt
+++ b/package/kernel/mac80211/files/regdb.txt
@@ -1180,8 +1180,8 @@ country TW: DFS-FCC
(5725 - 5850 @ 80), (30)
country TZ:
- (2402 - 2482 @ 40), (20)
- (5735 - 5835 @ 80), (30)
+ (2402 - 2484 @ 40), (30)
+ (5150 - 5835 @ 80), (30)
# Source:
# #914 / 06 Sep 2007: http://www.ucrf.gov.ua/uk/doc/nkrz/1196068874
EOF
( cd $base_feed && git apply $patch_file && {
echo "Patch applied, now you can use the special channel TZ when you deploy a mesh network on International Waters"
} || echo "Patch does not apply, maybe it is already applied orLEDE source has changed" )
ar71xx/generic
ramips/rt305x
ramips/rt3883
ramips/mt7620
ramips/mt7621
ramips/mt7628
ramips/mt7688
lantiq/xrx200
lantiq/xway
x86/generic
x86/64
x86/geode
mvebu/generic
ipq806x/generic
brcm2708/bcm2708
brcm2708/bcm2709
brcm2708/bcm2710
octeon/generic
sunxi/generic
mediatek/generic
arm64/generic
armvirt/generic
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment