CityLab Testbed - User contributions [en]

Usage Policy

2025-07-28T09:51:31Z

Dvdakker: /* Firewall Policy */

Experimenters must comply with the [https://www.fed4fire.eu/terms/ Fed4Fire Umbrella Terms and Conditions] when running experiments on the CityLab Testbed.
We're currently also working on the ''Local Terms and Conditions'' for the CityLab testbed. Once these have been finalized, experimenters will be required to comply with these terms and conditions as well. In the meantime, we ask all experimenters to effectively ''behave nicely'' and use ''caution'' and ''common sense'' when running experiments.

The main thing to keep in mind is that in contrast to, for example [http://doc.ilabt.iminds.be/ilabt-documentation/wilabfacility.html w.iLab-2], the CityLab testbed is ''not'' deployed in a fully-isolated ''playground''-environment. Instead, the testbed is deployed in an ''outdoor'' city-environment surrounded by wireless devices operating in the same frequency bands using the same wireless technologies. This basically means that when you run experiments on the CityLab testbed you are running them in an ''uncontrolled'' and possibly ''hostile'' environment. People for instance love connecting to unsecured WiFi access points and there are plenty of amateur-radio enthusiasts around that love tinkering with [https://www.gnuradio.org/ GNU Radio] and a [https://www.rtl-sdr.com/about-rtl-sdr/ hacked DVB-T dongle]. You should therefore take the necessary precautions (e.g.: secure your access points) to prevent people from gaining unauthorized access to the CityLab testbed infrastructure.

To help you along the remainder of this page outlines some of the things to take into account when doing your experiments. If things are not 100% clear or you have a question about this, please don't hesitate to contact us at {{SafeMailTo|admin|lab.cityofthings.eu}}.

== Regulatory Requirements ==

Since the CityLab testbed is deployed ''outdoors'', you are required to adhere to rules of the [http://www.bipt.be BIPT] (Belgian Institute for Postal services and Telecommunication) regarding wireless communications.
To help you along we've listed some of the regulations below and provided links to others. These are however only intended to guide you along. For a definitive reference of what ''is'' and ''is not'' allowed you should consult the [http://www.bipt.be/nl/operatoren/radio/frequentiebeheer/frequentieplan/tabel Frequency Plan] of the BIPT. As usual, contacting the testbed administrators at {{SafeMailTo|admin|lab.cityofthings.eu}} in case of uncertainty is always a good idea.

* '''WiFi @2.4GHz & 5GHz''': Consult [https://www.wirelessinfo.be/wifi-frequentie-tabel/ this website] for a nice summary (in Dutch) of the WiFi-channels & TX Powers you are allowed to use. If you no not speak Dutch, you may try the [https://translate.google.com/translate?sl=nl&tl=en&u=https%3A%2F%2Fwww.wirelessinfo.be%2Fwifi-frequentie-tabel Google Translated] version instead, or contact us at {{SafeMailTo|admin|lab.cityofthings.eu}} if you have any questions.

* '''IEEE 802.15.4 @2.4GHz''': The same limits as for WiFi @2.4GHz apply. Since IEEE 802.15.4 operates in the same frequencies as WiFi and has a much lower TX-Power (0dBm instead of 20dBm), this should not be a problem.

For 868MHz and 433MHz technologies (this includes IEEE 802.15.4, LoRa and Dash7) the following limits apply:
{| class="wikitable" border="1"
|-
! Frequency Band
! Allowed Frequencies
! Max TX-Power
! Max. Duty Cycle
! Additional Requirements
|-
| 433 MHz
| 433.05 MHz - 434.04 MHz
| 10mW (10dBm)
| 10%
|-
| rowspan="5" | 868MHz
| 863 MHz - 865 MHz
| 25mW (14dBm)
| 0.1%
| Must use CCA (Clear Channel Assessments)
|-
| 865 MHz - 868.6 MHz
| 25mW (14dBm)
| 1%
| Duty Cycle requirement is waived if CCA is used
|-
| 868.7 MHz - 869.2 MHz
| 25mW (14dBm)
| 0.1%
| Must use CCA
|-
| 869.4 MHz - 869.64 MHz
| 25mW (14dBm)
| 0.1%
| Must use CCA
|-
| 869.7 MHz - 870 MHz
| 25mW (14dBm)
| 0.1%
| Must use CCA
|}

== WiFi Testing ==

The following rules apply when doing WiFi tests on the CityLab testbed:

* '''No (Active) WiFi tests in the city campus during the day'''. Most of the CityLab testbed nodes are deployed at the City Campus of the University of Antwerp. Since this campus also houses around 1000 employees and 10000 students who all rely on a stable WiFi-connection to do their work, ''active'' WiFi tests are '''not''' allowed during the [https://www.uantwerpen.be/en/library/services/locations-and-opening-hours/stadscampus/ opening hours] of the City Campus' Library. ''Passive tests'' (such as background RSSI measurements), that do not require packets to be transmitted ''are'' allowed during the day. For active tests, you can use the nodes at the [[Nodes#Middelheim_Campus|Middelheim Campus]] to prepare your tests during normal office hours and then schedule your tests to run after-hours on the nodes in the City Campus. Exemptions to this rule may be granted if needed and your experiments won't have too-large an impact on WiFi performance (e.g.: PING-ing is probably fine, iperf's probably aren't.). In either case this is something to be discussed with the CityLab System Administrators '''before''' you run your experiments.

* '''Secure your WiFi connections'''. As described above, the CityLab nodes are deployed in an area where everyone with a smartphone is able to connect to unsecured WiFi Access Points. To prevent unauthorized access to the CityLab infrastructure you are '''required''' to secure '''all''' your wireless connections with WPA2 and a ''secure'' password. If, for whatever reason, you cannot use encryption for your experiments, contact the CityLab System Administrators({{SafeMailTo|admin|lab.cityofthings.eu}}), so we can try to work out an alternative solution.

* '''Don't provide Internet Access to external devices'''. If required for your experiments, you ''are'' allowed to connect external devices to the CityLab testbed infrastructure on two conditions:
#You must have ''full control'' over the devices you connect to theCityLab infrastructure (e.g: your own smartphone/laptop)
#You may '''not''' under any circumstances allow these devices to access the internet ''through'' the testbed infrastructure.

* '''No network penetration testing'''. As discussed in the [https://www.fed4fire.eu/terms/ General Fed4Fire Terms] you are not allowed to use the infrastructure for any illegal activity. Just so we're clear on this: that includes 'testing' how secure other WiFi networks in the area are.

== Firewall Policy ==

To prevent any problems resulting from improper use of the CityLab infrastructure on the rest of the University Network, we apply strict filtering on both incoming and outgoing network connections. At this point only the following connections are allowed:

* '''Incoming''':
** Ping
** SSH access only via the jFed SSH proxy. Direct SSH connections from the internet are not allowed.

* '''Outgoing''': Ping, SSH, HTTP, HTTPS, NTP, DNS, FTP

This is by no means a definitive list and more protocols may be added in the future. We also don't mind punching additional holes in the firewall to accommodate your experiments (just so long as you don't ask us to allow for example Bittorrent traffic). If you do require additional ports to be opened, please send an e-mail to {{SafeMailTo|admin|lab.cityofthings.eu}} and we'll see what we can do.

Getting Started

2025-07-28T09:49:48Z

Dvdakker: /* Configuration of Access Point */

== Prerequisites ==

=== JFed Account ===
To access the CityLab testbed you need:
* A [https://portal.fed4fire.eu JFed account]. If you do not have one already, you will have to request one. How this is done is explained [https://doc.fed4fire.eu/getanaccount.html here].
* You must be a member of a JFed-project that is authorised to run experiments on the Citylab testbed.

If you join a JFed-project that has already been authorised, you will automatically also be granted access to the Citylab testbed. If you create a new JFed-project for your experiments, you can request for it to be authorised by sending an e-mail to {{SafeMailTo|admin|lab.cityofthings.eu}}, explaining why your projects need access to the testbed. Once your project has been authorised, any new members you add to your project will automatically also be granted access.

Please note that if you create a new JFed-project during signup both the application for a jFed account as well as the authorisation of your JFed-project must be approved. The Citylab administrators are '''not''' automatically notified of the fact that you created a new project. You have to ask for it to be authorised yourself by sending an e-mail to {{SafeMailTo|admin|lab.cityofthings.eu}}. As long as you did not receive confirmation of both approvals, you are not ready to start using the testbed.

=== JFed Software ===
You also need to have the JFed software installed on your computer. You can do this before or after you create your JFed account. Go to [https://jfed.ilabt.imec.be/downloads/ https://jfed.ilabt.imec.be/downloads/] and download the appropriate installer for your operating system. On that website, you can also find some documentation on how to use this software (e.g. regarding certificates).

== Using the testbed for the first time ==
=== Create and activate your experiment using jFed ===
Follow these steps to activate your nodes using jFed:
* Fire up jFed. If all goes well, you should be prompted to provide your User certificate and Password. Browse to the location where you stored the .pem-file (see jFed documentation regarding certificates), provide your password and click Login.
[[File:jfed_login.png|Log in|480px]]
* Click on New
* Drag in 3 wireless nodes.
[[File:jfed_experiment.png|Drag in 3 wireless nodes|480px]]
* Configure which nodes you want to use:
** Click the node, and select ''imec CityLab Antwerp''. By default, WiLab2 is selected.
** Click on ''specific node''and select a node there. '''Make sure you check on the map to select nodes that are in reach of each other!''' The CityLab nodes can be far away from each other, so if you don't pay attention, you will end up with an experiment that does not support wireless communication between the selected nodes! See [[#Using the node map|using the node map]] for more information.
[[File:jfed_select_nodes.png|Select the appropriate nodes|480px]]
* Right click, Configure Node to change the properties (name/testbed/disk image/specific node).
* Name the three fixed nodes: ap and sta1 and sta2.
* Click Run to start your slice (Green arrow on the top left) and fill in a unique name for your slice.
[[File: jfed_run.png|Start the experiment|480px]]
* When all nodes turn green, your experiment is succesfully activated.
[[File:jfed_started.png|The experiment is running|480px]]

=== Configuration of Access Point ===
SSH to your AP node (double click it in jFed). '''Important''': For security purposes, the Citylab nodes are '''only''' accessible via the SSH proxy. Make sure it is enabled before you try to login to them !!

Become root:

<code>sudo su</code>

Create a config file for the hostapd program:

<code>nano /root/hostapd.conf</code>

Add the following content to the config file:

<pre>interface=wlp1s0
driver=nl80211
country_code=BE
ssid=demoX
hw_mode=Z
channel=Y

auth_algs=1
wpa=2
wpa_passphrase=
wpa_key_mgmt=WPA-PSK
wpa_pairwise=TKIP
rsn_pairwise=CCMP</pre>

Replace X with a random number. Replace Y with your channel(1-11 for g, 36/40/44 for a), Z with the WiFi mode (a or g) and PWD with a ''secure'' password. Start hostapd. The above config will setup an AP on wlan0 using 802.11a or g, channel Y, with SSID demoX:

<code>hostapd /root/hostapd.conf</code>

Open a second ssh terminal and give an IP address to the wlp1s0 interface so we can test the connection to the clients (in the next steps). Be sure to replace X with your number:

<code>sudo su; ifconfig wlp1s0 192.168.X.1/24</code>

=== Configuration of stations ===
For each of the two stations, open a ssh terminal and become root:

<code>sudo su</code>

Create a config file for the wpa_supplicant program:

<code>wpa_passphrase demoX PWD >/root/wpa_supplicant.conf</code>

In the above command replace demoX and PWD with the SSID and WPA2 password you configured in the hostapd.conf file on the access point.

Put the wireless interface into managed mode and start the wpa_supplicant tool to connect to the AP.

<code>iwconfig wlp1s0 mode managed</code>

<code>wpa_supplicant -iwlp1s0 -c/root/wpa_supplicant.conf</code>

Open a second ssh terminal and specify an IP address for the wlp1s0 interface:

<code>sudo su; ifconfig wlan0 192.168.X.Z/24</code>

make sure Z is not 1 and is different on both stations. Now, check if you can ping from one station to the other.

== Using the node map ==
Note: A full-screen version of the node map can be found [https://doc.lab.cityofthings.eu/nodemap/ here]
The node map is a city map of Antwerp where all CityLab nodes in the City Center are marked (Zoom out for the nodes at Campus CMI). Every node is shown on the map by by a clickable pin. When you click on a pin, a small text balloon opens that contains a link that brings you to a web page with some more information on that node.

You can use the node map to search for nodes that are in each others vicinity in order to set up an experiment with nodes that can actually communicate with the technology you will use. Due to the scale of the CityLab testbed, not all nodes can directly communicate with each other.

Please not that, while on the map, nodes might appear to be in close proximity, they do not necessarily have wireless connectivity (e.g. because there are obstacles blocking the line-of-sight between them). This is also highly dependent on technology. To help you select which nodes might be suitable for your experiment, we created an overlay on the map that shows which nodes have a direct connection with each other. To view this overlay, first select a type of wireless link from the drop-down menu. Then, you can optionally set filters based on RSSI values or on reliability. When you click the ''update'' button, the overlay will appear.

The overlay consists of colour-coded lines that give a visual clue of how good or bad a certain wireless link is. The lines themselves are also clickable, showing you the details of the link conditions. Do note that these figures come from a specially designed experiment to measure the link conditions. As these link conditions are also affected by external factors, they might change. You may also notice that some nodes appear not to have any links to any other nodes. This simply means that the link conditions have not been measured since this node was added. We do occasionally re-measure the link conditions but since we have to reserve ''all'' the nodes in the testbed for an entire weekend to do so, we don't want to do this too often. In essence: You can use this node map as a guide, but do not consider it to be absolute reality...
<iframe key="docs" path="nodemap/?embed=1" width="100%" height="800" />

== Embedded Radios ==

Once you've managed to start your first experiment, check out the [[Embedded Radios Quick Start Guide]] for more information on how to start using those.

== Troubleshooting ==
=== Working around connectivity issues ===

If you get a ''connection error'' while configuring the nodes or starting a test, your host network is probably blocking outgoing TCP connections to port 12369 (the Geni AM port). You can work around this by enabling the ''jFed Proxy''. To do so, click on Preferences, select the ''Proxy'' tab and click Run Proxy Test. Select the ''Always'' option for both Proxy for jFed and Proxy for SSH connections.

[[File:Jfed_proxy_settings.png|480px]]

Embedded Radios Quick Start Guide

2023-04-05T13:30:16Z

Dvdakker: /* Experiment Setup */

This quick start guide will explain how to get started with the "LoRa" (EFM32GG+RFM95W) an the OpenUSB modules of the Citylab testbed nodes.
== Prerequisites ==

This quick start guide assumes you already have access to the Citylab Testbed and that you are familiar with the steps required to start experiments. If that is not the case then please follow the [[Getting Started]] guide first.

== Experiment Setup ==

* Open the JFed GUI, create a new experiment and drag in two wireless nodes.
[[File:jfed_embedded_setup_01.png|Experiment Setup|480px]]
* Configure the nodes. Make sure that you:
** Set the correct testbed for the node (defaults to w-iLab.2, needs to be set to Citylab)
** Set Disk image to <code>Ubuntu 20.04 LTS 64-bit (CoT) with ARM gcc compiler</code>. This is important as we'll need the ARM gcc toolchain to compile the images for the devices.
** Explicitly set the specific node to use. '''Make sure that the nodes you select are within range of each other!'''. You can use the [https://doc.lab.cityofthings.eu/nodemap/ Node Map] to help you with this.
** name one of the nodes <code>nodeTX</code> and the other one <nodeRX>. You can use other node names but those are the node names that will be used throughout this guide.
[[File:jfed_embedded_setup_02.png|Experiment Setup|600px]]

* Click "Run" to start the experiment and wait the the nodes to finish initialising. ''Important'': It'll take a bit longer than usual to start the experiment as the <code>Ubuntu 20.04 LTS 64-bit (CoT) with ARM gcc compiler</code> image we're using in this experiment is quite large so it'll take some time to load it onto the nodes.

== Getting started with the EFM32GG+RFM95W (LoRa) module ==

To help you get started with the "EFM32GG+RFM95W" module, we've ported the [https://github.com/Lora-net/LoRaMac-node LoraWAN network stack] to the "EFM32GG+RFM95W". This port can be found in the following Github repository:
* https://github.com/imec-idlab/LoRaMac-node

In this guide, we'll use that repository to set up a basic broadcast tx/rx between two nodes but the stack has many other capabilities as well.

=== Preparing the receiver node ===

1) SSH into the <code>nodeRX</code> and clone the <code>LoRaMac-node</code> repository
<pre>
git clone https://github.com/imec-idlab/LoRaMac-node
</pre>
2) Change to the root dir of the repository
<pre>
cd LoraMac-node
</pre>
3) Edit the file <code>Makefile.local</code> and make sure it has the following contents:
<pre>
TARGET = beacon_rx
CFLAGS += -DUSE_BAND_868=1
CFLAGS += -DUSE_MODEM_LORA=1
</pre>
The <code>TARGET</code> line tells the compiler which application to build. The other lines tell the application to use the LoRa radio in the 868MHz band
4) Compile the binary using the <code>make</code> command:
<pre>
make -j4
</pre>
If all went well you should now have a file named <code>beacon_rx.bin</code> in the local directory. If you get a complaint that no compiler could be found then, most likely you are not using the correct disk image.
5) Flash the binary onto the EFM32GG device using the <code>flash_lora</code> script:
<pre>
flash_lora ./beacon_rx.bin
</pre>

6) Use <code>picocom</code> to listen to the serial port of the 'Lora' device
<pre>
sudo picocom -b 115200 /dev/ttyLORA
</pre>
In the above command:
* <code>115200</code> is the baud rate to use
* <code>/dev/ttyLORA </code> is the path to the serial port of the Lora device. (This is a symlink to the actual 'ttyUSB' device that auto-generated via udev-rules present in all 'CoT' disk images)

At this point you won't see anything yet, but once the transmitter node has also been prepared you'll see text appearing here

=== Preparing the transmitter node ===

The steps needed to prepare the transmitter node are almost identical to the steps needed to prepare the receiver node.

1) Open a new terminal, SSH into the <code>nodeTX</code> and clone the <code>LoRaMac-node</code> repository

2) Change to the root dir of the repository

3) Edit the file <code>Makefile.local</code> and make sure it has the following contents:
<pre>
TARGET = beacon_tx
CFLAGS += -DUSE_BAND_868=1
CFLAGS += -DUSE_MODEM_LORA=1
</pre>
The only difference between the <code>Makefile.local</code> used here and the one used for the receiver is that we are now building the <code>beacon_tx</code> rather than the <code>beacon_rx</code> app.
The <code>TARGET</code> line tells the compiler which application to build. The other lines tell the application to use the LoRa radio in the 868MHz band

4) Compile the binary using the <code>make</code> command.

5) Flash the binary onto the EFM32GG device using the <code>flash_lora</code> script:
<pre>
flash_lora ./beacon_tx.bin
</pre>

6) Use <code>picocom</code> to listen to the serial port of the 'Lora' device
<pre>
sudo picocom -b 115200 /dev/ttyLORA
</pre>
You should now see lines similar to the ones below appearing:
<pre>
Sent Packet using LORA. size=16, counter=3 TimeOnAir(us)=52
Sent Packet using LORA. size=16, counter=4 TimeOnAir(us)=52
Sent Packet using LORA. size=16, counter=5 TimeOnAir(us)=52
</pre>

7) Switch back to the terminal of the <code>nodeRX</code> testbed node. You should see lines similar to the ones below appearing
<pre>
0x520B4614248AB601,10,-33,25
0x520B4614248AB601,11,-35,25
0x520B4614248AB601,12,-34,26
0x520B4614248AB601,13,-33,24
</pre>
In the reported lines, the first field is the UUID of the node sending the beacons, the second field is the sequence number of the packet and the 3rd & 4th fields are the RSSI & LQI of the received beacon packet.

=== Experiment Cleanup ===

In both the <code>nodeTX</code> and <code>nodeTX</code> terminals, perform the following actions to clean up the experiment:

1) Press the 'Control' key and then press the 'A' and 'X' one after the other while holding the 'Control' key pressed down. This control-sequence stops the running <code>picocom</code> program

2) Run the <code>erase_lora</code> program to wipe the LoRa devices.
<pre>
erase_lora
</pre>
It is important you do this since otherwise the transmitter node will keep sending beacons.

== Running Contiki-NG on the OpenUSB Radios ==

For this guide we re-use the same JFed experiment setup as for the 'LoRa device' (discussed above). In this section we'll flash two OpenUSB devices and ping from one to the other. The guide in this section is based on the [https://docs.contiki-ng.org/en/develop/doc/tutorials/Hello%2C-World%21.html Hello World], [https://docs.contiki-ng.org/en/develop/doc/tutorials/Shell.html Shell] and [https://docs.contiki-ng.org/en/develop/doc/tutorials/IPv6-ping.html IPv6 Ping] tutorials on the Contiki-NG documentation website, but focusses specifically on the OpenMote platform.

For more information on how to use Contiki-NG consult the Contiki-NG documentation at:
* https://docs.contiki-ng.org/en/develop/doc/tutorials/
* https://github.com/contiki-ng/contiki-ng/wiki

=== Cloning, Compiling & Flashing the nodes ===

1) SSH into both the <code>nodeTX</code> and <code>nodeRX</code> experiment nodes in two separate terminals. Then perform the steps below '''on both experiment nodes'''

2) Clone the contiki-ng git repository
<pre>
git clone https://github.com/contiki-ng/contiki-ng/
</pre>

3) Change to the root of the source tree
<pre>
cd contiki-ng
</pre>

4) Pull in all submodules (please note: this will take a while)
<pre>
git submodule update --init --recursive
</pre>

4) move to the 'app' directory of the <code>hello-world</code> example application
<pre>
cd examples/hello-world
</pre>

5) Edit the file <code>hello-world.c</code> and change the following line:
<pre>
etimer_set(&timer, CLOCK_SECOND * 10);
</pre>
Into the following one:
<pre>
etimer_set(&timer, CLOCK_SECOND * 120);
</pre>
The reason we do this is because we'll be opening a 'shell' to the device later on and having the "''Hello World''" prompt interrupting the shell session every 10 seconds is annoying.

6) Edit the <code>Makefile</code> and add the line <code>MODULES += os/services/shell</code> just below the <code>all: $(CONTIKI_PROJECT)</code> line. The end result should look like this:
<pre>
CONTIKI_PROJECT = hello-world
all: $(CONTIKI_PROJECT)

MODULES += os/services/shell

CONTIKI = ../..
include $(CONTIKI)/Makefile.include
</pre>

7) Run <code>make distclean</code> to ensure the build environment is clean

8) Compile the hello-world application for the Openmote platform using the following command
<pre>
make TARGET=openmote BOARD=openmote-cc2538
</pre>

9) Flash the hello-world application on the device using the following command:
<pre>
sudo make TARGET=openmote BOARD=openmote-cc2538 PORT=/dev/ttyOPENUSB hello-world.upload
</pre>

10) Connect with <code>picocom</code> to the serial port of the openmote:
<pre>
sudo picocom -c -b 115200 --imap lfcrlf /dev/ttyOPENUSB
</pre>
In the above command:
* <code>115200</code> is the baudrate
* <code>-c</code> turns on local echo (so you can see what you type)
* <code>--imap lfcrlf</code> automaticall converts the "\n" printed by the Openmote into "\r\n" so the terminal behaves as you expect it to

11) Type <code>help</code> and press enter to see a list of all commands available

=== Pinging from one node to another ===

If you've followed the steps above you should now have two 'terminal' windows open. One where you are logged in to the serial console of the OpenUSB devices of <code>nodeTX</code> and one where you are logged in to the console of the OpenUSB device connected to <code>nodeRX</code>.

12) In the <code>nodeRX</code> terminal, enter <code>ip-addr</code> and press enter to retrieve the link-local address of the Openmote. The result should be similar to this:
<pre>
#0012.4b00.0YYY.XXXX> ip-addr
Node IPv6 addresses:
-- fe80::212:4b00:YYY:XXXX
#0012.4b00.0YYY.XXXX>
</pre>

13) In the <code>nodeTX</code> terminal, enter the following command:
<pre>ping <ip>
</pre>
Where <code><ip></code> is the ''ip-address'' of the <code>nodeRX</code>. The result should be similar to this:
<pre>
#0012.4b00.0YYY.ZZZZ> ping fe80::212:4b00:YYY:XXXX
Pinging fe80::212:4b00:613:1506
Received ping reply from fe80::212:4b00:YYY:XXXX, len 4, ttl 64, delay 78 ms
</pre>

=== Experiment Cleanup ===

In both the <code>nodeTX</code> and <code>nodeTX</code> terminals, perform the following actions to clean up the experiment:

1) Press the 'Control' key and then press the 'A' and 'X' one after the other while holding the 'Control' key pressed down. This control-sequence stops the running <code>picocom</code> program

2) From the <code>examples/hello-world</code> directory in the contiki-ng source tree, run the following command to wipe the flash on the OpenUSB device:
<pre>
../../tools/cc2538-bsl/cc2538-bsl.py -e -p /dev/ttyOPENUSB --bootloader-invert-lines
</pre>
It is important you do this since otherwise the OpenUSB devices will remain accessible over the wireless interface.

File:Jfed embedded setup 02.png

2023-04-05T13:28:20Z

Dvdakker:

2023-04-05T12:33:10Z

Dvdakker: /* Running Contiki-NG on the OpenUSB Radios */

This quick start guide will explain how to get started with the "LoRa" (EFM32GG+RFM95W) an the OpenUSB modules of the Citylab testbed nodes.
== Prerequisites ==

This quick start guide assumes you already have access to the Citylab Testbed and that you are familiar with the steps required to start experiments. If that is not the case then please follow the [[Getting Started]] guide first.

== Experiment Setup ==

* Open the JFed GUI, create a new experiment and drag in two wireless nodes.

* Configure the nodes. Make sure that you:
** Set the correct testbed for the node (defaults to w-iLab.2, needs to be set to Citylab)
** Set Disk image to <code>Ubuntu 20.04 LTS 64-bit (CoT) with ARM gcc compiler</code>. This is important as we'll need the ARM gcc toolchain to compile the images for the devices.
** Explicitly set the specific node to use. '''Make sure that the nodes you select are within range of each other!'''. You can use the [https://doc.lab.cityofthings.eu/nodemap/ Node Map] to help you with this.
** name one of the nodes <code>node_tx</code> and the other one <node_rx>. You can use other node names but those are the node names that will be used throughout this guide.

* Click "Run" to start the experiment and wait the the nodes to finish initialising. ''Important'': It'll take a bit longer than usual to start the experiment as the <code>Ubuntu 20.04 LTS 64-bit (CoT) with ARM gcc compiler</code> image we're using in this experiment is quite large so it'll take some time to load it onto the nodes.

== Getting started with the EFM32GG+RFM95W (LoRa) module ==

To help you get started with the "EFM32GG+RFM95W" module, we've ported the [https://github.com/Lora-net/LoRaMac-node LoraWAN network stack] to the "EFM32GG+RFM95W". This port can be found in the following Github repository:
* https://github.com/imec-idlab/LoRaMac-node

In this guide, we'll use that repository to set up a basic broadcast tx/rx between two nodes but the stack has many other capabilities as well.

=== Preparing the receiver node ===

1) SSH into the <code>node_rx</code> and clone the <code>LoRaMac-node</code> repository
<pre>
git clone https://github.com/imec-idlab/LoRaMac-node
</pre>
2) Change to the root dir of the repository
<pre>
cd LoraMac-node
</pre>
3) Edit the file <code>Makefile.local</code> and make sure it has the following contents:
<pre>
TARGET = beacon_rx
CFLAGS += -DUSE_BAND_868=1
CFLAGS += -DUSE_MODEM_LORA=1
</pre>
The <code>TARGET</code> line tells the compiler which application to build. The other lines tell the application to use the LoRa radio in the 868MHz band
4) Compile the binary using the <code>make</code> command:
<pre>
make -j4
</pre>
If all went well you should now have a file named <code>beacon_rx.bin</code> in the local directory. If you get a complaint that no compiler could be found then, most likely you are not using the correct disk image.
5) Flash the binary onto the EFM32GG device using the <code>flash_lora</code> script:
<pre>
flash_lora ./beacon_rx.bin
</pre>

6) Use <code>picocom</code> to listen to the serial port of the 'Lora' device
<pre>
sudo picocom -b 115200 /dev/ttyLORA
</pre>
In the above command:
* <code>115200</code> is the baud rate to use
* <code>/dev/ttyLORA </code> is the path to the serial port of the Lora device. (This is a symlink to the actual 'ttyUSB' device that auto-generated via udev-rules present in all 'CoT' disk images)

At this point you won't see anything yet, but once the transmitter node has also been prepared you'll see text appearing here

=== Preparing the transmitter node ===

The steps needed to prepare the transmitter node are almost identical to the steps needed to prepare the receiver node.

1) Open a new terminal, SSH into the <code>node_tx</code> and clone the <code>LoRaMac-node</code> repository

2) Change to the root dir of the repository

3) Edit the file <code>Makefile.local</code> and make sure it has the following contents:
<pre>
TARGET = beacon_tx
CFLAGS += -DUSE_BAND_868=1
CFLAGS += -DUSE_MODEM_LORA=1
</pre>
The only difference between the <code>Makefile.local</code> used here and the one used for the receiver is that we are now building the <code>beacon_tx</code> rather than the <code>beacon_rx</code> app.
The <code>TARGET</code> line tells the compiler which application to build. The other lines tell the application to use the LoRa radio in the 868MHz band

4) Compile the binary using the <code>make</code> command.

5) Flash the binary onto the EFM32GG device using the <code>flash_lora</code> script:
<pre>
flash_lora ./beacon_tx.bin
</pre>

6) Use <code>picocom</code> to listen to the serial port of the 'Lora' device
<pre>
sudo picocom -b 115200 /dev/ttyLORA
</pre>
You should now see lines similar to the ones below appearing:
<pre>
Sent Packet using LORA. size=16, counter=3 TimeOnAir(us)=52
Sent Packet using LORA. size=16, counter=4 TimeOnAir(us)=52
Sent Packet using LORA. size=16, counter=5 TimeOnAir(us)=52
</pre>

7) Switch back to the terminal of the <code>node_rx</code> testbed node. You should see lines similar to the ones below appearing
<pre>
0x520B4614248AB601,10,-33,25
0x520B4614248AB601,11,-35,25
0x520B4614248AB601,12,-34,26
0x520B4614248AB601,13,-33,24
</pre>
In the reported lines, the first field is the UUID of the node sending the beacons, the second field is the sequence number of the packet and the 3rd & 4th fields are the RSSI & LQI of the received beacon packet.

=== Experiment Cleanup ===

In both the <code>node_tx</code> and <code>node_tx</code>, perform the following actions to clean up the experiment:

1) Press the 'Control' key and then press the 'A' and 'X' one after the other while holding the 'Control' key pressed down. This control-sequence stops the running <code>picocom</code> program

2) Run the <code>erase_lora</code> program to wipe the LoRa devices.
<pre>
erase_lora
</pre>
It is important you do this since otherwise the transmitter node will keep sending beacons.

== Running Contiki-NG on the OpenUSB Radios ==

For this guide we re-use the same JFed experiment setup as for the 'LoRa device' (discussed above). In this section we'll flash two OpenUSB devices and ping from one to the other. The guide in this section is based on the [https://docs.contiki-ng.org/en/develop/doc/tutorials/Hello%2C-World%21.html Hello World], [https://docs.contiki-ng.org/en/develop/doc/tutorials/Shell.html Shell] and [https://docs.contiki-ng.org/en/develop/doc/tutorials/IPv6-ping.html IPv6 Ping] tutorials on the Contiki-NG documentation website, but focusses specifically on the OpenMote platform.

For more information on how to use Contiki-NG consult the Contiki-NG documentation at:
* https://docs.contiki-ng.org/en/develop/doc/tutorials/
* https://github.com/contiki-ng/contiki-ng/wiki

=== Cloning, Compiling & Flashing the nodes ===

1) SSH into both the <code>node_tx</code> and <code>node_rx</code> experiment nodes in two separate terminals. Then perform the steps below '''on both experiment nodes'''

2) Clone the contiki-ng git repository
<pre>
git clone https://github.com/contiki-ng/contiki-ng/
</pre>

3) Change to the root of the source tree
<pre>
cd contiki-ng
</pre>

4) Pull in all submodules (please note: this will take a while)
<pre>
git submodule update --init --recursive
</pre>

4) move to the 'app' directory of the <code>hello-world</code> example application
<pre>
cd examples/hello-world
</pre>

5) Edit the file <code>hello-world.c</code> and change the following line:
<pre>
etimer_set(&timer, CLOCK_SECOND * 10);
</pre>
Into the following one:
<pre>
etimer_set(&timer, CLOCK_SECOND * 120);
</pre>
The reason we do this is because we'll be opening a 'shell' to the device later on and having the "''Hello World''" prompt interrupting the shell session every 10 seconds is annoying.

6) Edit the <code>Makefile</code> and add the line <code>MODULES += os/services/shell</code> just below the <code>all: $(CONTIKI_PROJECT)</code> line. The end result should look like this:
<pre>
CONTIKI_PROJECT = hello-world
all: $(CONTIKI_PROJECT)

MODULES += os/services/shell

CONTIKI = ../..
include $(CONTIKI)/Makefile.include
</pre>

7) Run <code>make distclean</code> to ensure the build environment is clean

8) Compile the hello-world application for the Openmote platform using the following command
<pre>
make TARGET=openmote BOARD=openmote-cc2538
</pre>

9) Flash the hello-world application on the device using the following command:
<pre>
sudo make TARGET=openmote BOARD=openmote-cc2538 PORT=/dev/ttyOPENUSB hello-world.upload
</pre>

10) Connect with <code>picocom</code> to the serial port of the openmote:
<pre>
sudo picocom -c -b 115200 --imap lfcrlf /dev/ttyOPENUSB
</pre>
In the above command:
* <code>115200</code> is the baudrate
* <code>-c</code> turns on local echo (so you can see what you type)
* <code>--imap lfcrlf</code> automaticall converts the "\n" printed by the Openmote into "\r\n" so the terminal behaves as you expect it to

11) Type <code>help</code> and press enter to see a list of all commands available

=== Pinging from one node to another ===

If you've followed the steps above you should now have two 'terminal' windows open. One where you are logged in to the serial console of the OpenUSB devices of <code>node_tx</code> and one where you are logged in to the console of the OpenUSB device connected to <code>node_rx</code>.

12) In the <code>node_rx</code> terminal, enter <code>ip-addr</code> and press enter to retrieve the link-local address of the Openmote. The result should be similar to this:
<pre>
#0012.4b00.0YYY.XXXX> ip-addr
Node IPv6 addresses:
-- fe80::212:4b00:YYY:XXXX
#0012.4b00.0YYY.XXXX>
</pre>

Embedded Radios Quick Start Guide

2023-04-05T11:35:44Z

Dvdakker: /* Running Contiki on the OpenUSB Radios */

This quick start guide will explain how to get started with the "LoRa" (EFM32GG+RFM95W) an the OpenUSB modules of the Citylab testbed nodes.
== Prerequisites ==

This quick start guide assumes you already have access to the Citylab Testbed and that you are familiar with the steps required to start experiments. If that is not the case then please follow the [[Getting Started]] guide first.

== Experiment Setup ==

* Open the JFed GUI, create a new experiment and drag in two wireless nodes.

* Configure the nodes. Make sure that you:
** Set the correct testbed for the node (defaults to w-iLab.2, needs to be set to Citylab)
** Set Disk image to <code>Ubuntu 20.04 LTS 64-bit (CoT) with ARM gcc compiler</code>. This is important as we'll need the ARM gcc toolchain to compile the images for the devices.
** Explicitly set the specific node to use. '''Make sure that the nodes you select are within range of each other!'''. You can use the [https://doc.lab.cityofthings.eu/nodemap/ Node Map] to help you with this.
** name one of the nodes <code>node_tx</code> and the other one <node_rx>. You can use other node names but those are the node names that will be used throughout this guide.

* Click "Run" to start the experiment and wait the the nodes to finish initialising. ''Important'': It'll take a bit longer than usual to start the experiment as the <code>Ubuntu 20.04 LTS 64-bit (CoT) with ARM gcc compiler</code> image we're using in this experiment is quite large so it'll take some time to load it onto the nodes.

== Getting started with the EFM32GG+RFM95W (LoRa) module ==

To help you get started with the "EFM32GG+RFM95W" module, we've ported the [https://github.com/Lora-net/LoRaMac-node LoraWAN network stack] to the "EFM32GG+RFM95W". This port can be found in the following Github repository:
* https://github.com/imec-idlab/LoRaMac-node

In this guide, we'll use that repository to set up a basic broadcast tx/rx between two nodes but the stack has many other capabilities as well.

=== Preparing the receiver node ===

1) SSH into the <code>node_rx</code> and clone the <code>LoRaMac-node</code> repository
<pre>
git clone https://github.com/imec-idlab/LoRaMac-node
</pre>
2) Change to the root dir of the repository
<pre>
cd LoraMac-node
</pre>
3) Edit the file <code>Makefile.local</code> and make sure it has the following contents:
<pre>
TARGET = beacon_rx
CFLAGS += -DUSE_BAND_868=1
CFLAGS += -DUSE_MODEM_LORA=1
</pre>
The <code>TARGET</code> line tells the compiler which application to build. The other lines tell the application to use the LoRa radio in the 868MHz band
4) Compile the binary using the <code>make</code> command:
<pre>
make -j4
</pre>
If all went well you should now have a file named <code>beacon_rx.bin</code> in the local directory. If you get a complaint that no compiler could be found then, most likely you are not using the correct disk image.
5) Flash the binary onto the EFM32GG device using the <code>flash_lora</code> script:
<pre>
flash_lora ./beacon_rx.bin
</pre>

6) Use <code>picocom</code> to listen to the serial port of the 'Lora' device
<pre>
sudo picocom -b 115200 /dev/ttyLORA
</pre>
In the above command:
* <code>115200</code> is the baud rate to use
* <code>/dev/ttyLORA </code> is the path to the serial port of the Lora device. (This is a symlink to the actual 'ttyUSB' device that auto-generated via udev-rules present in all 'CoT' disk images)

At this point you won't see anything yet, but once the transmitter node has also been prepared you'll see text appearing here

=== Preparing the transmitter node ===

The steps needed to prepare the transmitter node are almost identical to the steps needed to prepare the receiver node.

1) Open a new terminal, SSH into the <code>node_tx</code> and clone the <code>LoRaMac-node</code> repository

2) Change to the root dir of the repository

3) Edit the file <code>Makefile.local</code> and make sure it has the following contents:
<pre>
TARGET = beacon_tx
CFLAGS += -DUSE_BAND_868=1
CFLAGS += -DUSE_MODEM_LORA=1
</pre>
The only difference between the <code>Makefile.local</code> used here and the one used for the receiver is that we are now building the <code>beacon_tx</code> rather than the <code>beacon_rx</code> app.
The <code>TARGET</code> line tells the compiler which application to build. The other lines tell the application to use the LoRa radio in the 868MHz band

4) Compile the binary using the <code>make</code> command.

5) Flash the binary onto the EFM32GG device using the <code>flash_lora</code> script:
<pre>
flash_lora ./beacon_tx.bin
</pre>

6) Use <code>picocom</code> to listen to the serial port of the 'Lora' device
<pre>
sudo picocom -b 115200 /dev/ttyLORA
</pre>
You should now see lines similar to the ones below appearing:
<pre>
Sent Packet using LORA. size=16, counter=3 TimeOnAir(us)=52
Sent Packet using LORA. size=16, counter=4 TimeOnAir(us)=52
Sent Packet using LORA. size=16, counter=5 TimeOnAir(us)=52
</pre>

7) Switch back to the terminal of the <code>node_rx</code> testbed node. You should see lines similar to the ones below appearing
<pre>
0x520B4614248AB601,10,-33,25
0x520B4614248AB601,11,-35,25
0x520B4614248AB601,12,-34,26
0x520B4614248AB601,13,-33,24
</pre>
In the reported lines, the first field is the UUID of the node sending the beacons, the second field is the sequence number of the packet and the 3rd & 4th fields are the RSSI & LQI of the received beacon packet.

=== Experiment Cleanup ===

In both the <code>node_tx</code> and <code>node_tx</code>, perform the following actions to clean up the experiment:

1) Press the 'Control' key and then press the 'A' and 'X' one after the other while holding the 'Control' key pressed down. This control-sequence stops the running <code>picocom</code> program

2) Run the <code>erase_lora</code> program to wipe the LoRa devices.
<pre>
erase_lora
</pre>
It is important you do this since otherwise the transmitter node will keep sending beacons.

== Running Contiki-NG on the OpenUSB Radios ==

For this guide we re-use the same JFed experiment setup as for the 'LoRa device' (discussed above). In this section we'll flash two OpenUSB devices and ping from one to the other. The guide in this section is based on the [https://docs.contiki-ng.org/en/develop/doc/tutorials/Hello%2C-World%21.html Hello World], [https://docs.contiki-ng.org/en/develop/doc/tutorials/Shell.html Shell] and [https://docs.contiki-ng.org/en/develop/doc/tutorials/IPv6-ping.html IPv6 Ping] tutorials on the Contiki-NG documentation website, but focusses specifically on the OpenMote platform.

For more information on how to use Contiki-NG consult the Contiki-NG documentation at:
* https://docs.contiki-ng.org/en/develop/doc/tutorials/
* https://github.com/contiki-ng/contiki-ng/wiki

=== Cloning, Compiling & Flashing the nodes ===

1) SSH into both the <code>node_tx</code> and <code>node_rx</code> experiment nodes in two separate terminals. Then perform the steps below '''on both experiment nodes'''

2) Clone the contiki-ng git repository
<pre>
git clone https://github.com/contiki-ng/contiki-ng/
</pre>

3) Change to the root of the source tree
<pre>
cd contiki-ng
</pre>

4) Pull in all submodules (please note: this will take a while)
<pre>
git submodule update --init --recursive
</pre>

4) move to the 'app' directory of the <code>hello-world</code> example application
<pre>
cd examples/hello-world
</pre>

5) Edit the file <code>hello-world.c</code> and change the following line:
<pre>
etimer_set(&timer, CLOCK_SECOND * 10);
</pre>
Into the following one:
<pre>
etimer_set(&timer, CLOCK_SECOND * 120);
</pre>
The reason we do this is because we'll be using the serial port to issue commands to the m

Embedded Radios Quick Start Guide

2023-04-05T11:22:03Z

Dvdakker: /* Running Contiki on the OpenUSB Module */

Embedded Radios Quick Start Guide

2023-04-05T11:08:04Z

Dvdakker: /* Experiment Setup */

Embedded Radios Quick Start Guide

2023-04-05T11:06:54Z

Dvdakker:

Embedded Radios Quick Start Guide

2023-04-05T08:40:25Z

Dvdakker: /* OpenMote / OpenUSB */

Embedded Radios Quick Start Guide

2023-04-05T08:39:41Z

Dvdakker: /* Experiment Cleanup */

Embedded Radios Quick Start Guide

2023-04-05T08:39:34Z

Dvdakker: /* Preparing the transmitter node */

Embedded Radios Quick Start Guide

2023-04-05T08:38:39Z

Dvdakker: /* Getting started with the EFM32GG+RFM95W (LoRa) module */

Embedded Radios Quick Start Guide

2023-04-05T07:51:18Z

Dvdakker: /* Getting started with the EFM32GG+RFM95W (LoRa) module */

Embedded Radios Quick Start Guide

2023-04-05T07:41:51Z

Dvdakker:

Embedded Radios Quick Start Guide

2023-04-05T07:14:00Z

Dvdakker: Dvdakker moved page Lora Quick Start Guide to Embedded Radios Quick Start Guide without leaving a redirect

== Prerequisites ==

This quick start guide assumes you already have access to the Citylab Testbed and that you are familiar with the steps required to start experiments. If that is not the case then please follow the [[Getting Started]] guide first.

== Experiment Setup ==

Open

Embedded Radios Quick Start Guide

2023-04-05T07:13:20Z

Dvdakker:

2021-08-11T10:28:19Z

Dvdakker: /* Creating a link between two nodes */

At this point, the CityLab testbed only has limited support for ''automatically'' creating links between nodes using the JFed interface. Standard IP-in-IP GRE Tunnels and Ethernet-in-IP GRE Tunnels (''gre-tunnel'' and ''egre-tunnel'' links in JFed) are supported. Adding support ''native'' Ethernet links (lan links in JFed) is on our TODO list. Since both 'gre' and 'egre' tunnels are point-to-point tunnels it is currently not (yet) possible to create a JFed-link with more than two connected nodes. As a workaround it ''is'' possible to bridge multiple 'egre'-tunnels together into a single (tunneled) L2-network once the test has started.

This page provides a quick How-To guide on how to create standard 'gre' or 'egre-tunnels' in JFed and then explains how to manually set up and bridge egre-tunnels manually (after the test has started).

== Creating Links between nodes through the JFed interface ==

=== Setting up links ===

Creating Links between nodes through the JFed interface is very straightforward and can be done as follows:

* Start JFed, click on 'New' and drag in two wireless nodes. For this tutorial we'll assume that these nodes are called ''apu1'' and ''apu2''
* Edit the node properties so both nodes are sourced from the ''City of Things Antwerp'' testbed. (By default Wilab2 is used).[[File:Jfed_experiment_2_nodes.png|Drag in two nodes from the ''City of Things Antwerp'' testbed and name them ''apu1'' and ''apu2''|720px]]
* Drag a line from ''apu1'' to ''apu2''. A ''link'' element connecting the two nodes is automatically created.
* If you are using JFed GUI 5.9 or above, link type of the link will already be set to 'gre'. (This is shown between braces behind the name of the link). [[File:Jfed_gre_link_2_nodes.png||300px]] If this is not the case, or you would like to create an 'egre'-link instead, the link-type will need to be manually configured:
*# Right-click on the ''link''-element, Click on ''Configure Link'' and select the ''Link Type''-tab in the options window.
*# Change the link-type to 'gre-tunnel' or 'egre-tunnel' in the drop-down menu. If you select 'gre-tunnel' an IP-in-IP tunnel will be created, if you select 'egre-tunnel' an Ehternet-in-IP tunnel will be created. [[File:JFed_link_options_type.png|Set the link-type to 'gre-tunnel'|480px]]
* JFed will automatically configure an IP-address on the tunnel interface for both nodes. optionally you can modify these IP-addresses in the ''General'' tab of the Link options-window [[File:JFed_link_options_general.png|Changes the IP-addresses for the link if needed|480px]]

* Click 'Run' to start your slice. When all nodes are green, your experiment has started and the GRE-tunnel between the two nodes has been configured. [[File:JFed_gre_link_2_nodes_running.png|The running experiment|720px]]

*SSH into the ''apu1'' node.
*Run <code>ifconfig link02</code> to verify that the GRE-tunnel has been created
<pre>
user@apu1:~$ ifconfig link02
link02 Link encap:UNSPEC HWaddr 8F-81-55-10-00-00-80-3F-00-00-00-00-00-00-00-00
inet addr:192.168.0.1 P-t-P:192.168.0.1 Mask:255.255.255.0
inet6 addr: fe80::200:5efe:8f81:5510/64 Scope:Link
UP POINTOPOINT RUNNING NOARP MTU:1434 Metric:1
RX packets:2 errors:0 dropped:0 overruns:0 frame:0
TX packets:3 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1
RX bytes:112 (112.0 B) TX bytes:168 (168.0 B)
</pre>
*Ping the ''apu2''-tunnel enpoint by running <code>ping -c 4 <ip></code> where <code><ip></code> is the IP-address configured for the tunnel endpoint on ''apu2''. If you have not changed this, this is <code>192.168.0.2</code>
<pre>
user@apu1:~$ ping -c 4 192.168.0.2
PING 192.168.0.2 (192.168.0.2) 56(84) bytes of data.
64 bytes from 192.168.0.2: icmp_seq=1 ttl=64 time=2.21 ms
64 bytes from 192.168.0.2: icmp_seq=2 ttl=64 time=1.60 ms
64 bytes from 192.168.0.2: icmp_seq=3 ttl=64 time=1.46 ms
64 bytes from 192.168.0.2: icmp_seq=4 ttl=64 time=1.51 ms

--- 192.168.0.2 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3004ms
rtt min/avg/max/mdev = 1.464/1.699/2.218/0.303 ms
</pre>

=== Limitations ===

==== No more than 2 nodes per link ====

It is not possible to create (E)GRE-links with more than two nodes. The JFed GUI does allow additional nodes to be added to the link, but when the test is started the link will only exist on two nodes. This limitation is caused by the fact that the GRE-protocol only supports point-to-point tunnels. It is possible to [[#Manually setting up EGRE-tunnels|set up]] and [[#Creating a link between three or more nodes|bridge]] multiple egre-tunnels, but this cannot be done through the JFed interface.

==== EGRE-tunnels only work with 'CoT' disk images ====

To enable support for 'egre-tunnel' links, a few tweaks had to be made to the 'client-side' emulab-tools installed in the testbed disk images. Since these tweaks are not (yet) incorporated in the upstream emulab-code base, they are also not present in the many emulab-images used around the world.

We have added these tweaks to the 'CoT' disk-images of the CityLab testbed (see [[Disk Images]]), so if you want to use egre-links you must use a 'CoT' disk image.

==== (E)GRE-tunnels between nodes of different testbeds cannot be created via JFed ====

Even at the best of times, creating links between nodes of different testbeds is an 'experimental feature' at best. While it is perfectly possible to set up GRE and EGRE tunnels between nodes of different testbeds, it is currently not possible to do so through the JFed interface.

==== Link impairment on GRE Tunnels cannot be configured through the JFed GUI ====

The way JFed normally adds impairment to a link is by adding a so-called 'impairment node' in between the nodes of a link to change the properties of the traffic flowing over the link. This mechanism however only works with ethernet interfaces and not with GRE Tunnels. That being said: it is possible to add link impairment to GRE tunnels manually. How to do so is [[#Setting_up_Link_impairment_on_GRE_Tunnels|explained below]].

== Manually setting up EGRE-tunnels ==

'''Important:''' although it is possible to manually set up GRE-tunnels it is much easier to do so through the JFed-interface. The only reason for creating them manually is to create a 'link' between more than two nodes or to create links with nodes of another testbed (both of which are currently not supported through JFed)

A number of [[:File:Gre-utils.tar.gz | scripts]] are provided to make it as easy as possible to create and manage EGRE-tunnels.
Before any EGRE-tunnels can be created, these scripts first need to be installed on every node.
This can be done by running the following one-liner:

<code>wget -O- https://doc.lab.cityofthings.eu/w/images/9/93/Gre-utils.tar.gz | sudo tar -C /usr/local/ -zxvf -</code>

This command will
* Download the [[:File:Gre-utils.tar.gz | Gre-utils.tar.gz]] archive
* Install the scripts in <code>/usr/local</code>

=== Creating a link between two nodes ===

[[File:Jfed-link-2-nodes.png|thumb|right|A simple link between two nodes.]]

A simple GRE tunnel can be created by running the <code>gre_add_tunnel</code> script on both nodes.

The setup shown in the example on the right can be created as follows:

* On '''apu1''' run: <code>sudo gre_add_tunnel -c 192.168.0.1/24 link0 apu2</code>
* On '''apu2''' run: <code>sudo gre_add_tunnel -c 192.168.0.2/24 link0 apu1</code>

In the above commands <code>link0</code> is the name of the interface to create and <code>apuX</code> is the host to create a tunnel to. The optional <code>-c</code> flag can be used to automatically configure an IPv4-address on the newly created interface.

To allow the node names configured in JFed to be used to configure GRE tunnels, the <code>gre_add_tunnel</code> script tries to behave somewhat intelligently when resolving hostnames:
* If the specified host is a Fully Qualified Domain Name (i.e: it contains a '.'), a regular DNS query is performed
* If the specified host is a simple hostname (no '.' in the hostname), the <code>gre_add_tunnel</code> script ''first'' tries to resolve the specified host to a node name specified in JFed. If this fails, the IP-address of the host is resolved using a normal DNS query.
* If the specified host is a valid IPv4 address, no DNS-resolution is performed.

Once the GRE tunnel is created, it can be used just like any other Ethernet interface, except that the MTU is a bit lower than normal.

Removing the tunnel is done using the <code>gre_del_tunnel</code> script.
To remove the tunnels created above for example, the following command would have to be run on each node:

<code>sudo gre_del_tunnel link0</code>

=== Creating a link between three or more nodes ===
[[File:Jfed-link-3-nodes.png|thumb|right|A slightly more complicated link between more than two nodes.]]

Creating a link between more than two nodes is (slightly) more complicated since GRE only allows Point-to-Point tunnels to be created. To work around this, GRE Tunnels are created from one 'central' node to all other nodes on the link. These GRE Tunnels are then 'bridged' together on the central node to form a shared subnet. Such a 'GRE Bridge' can easily be created using the <code>gre_add_bridge</code> script.

The 3-node setup shown in the example on the right can be created as follows:

# On '''apu1''' run: <code>sudo gre_add_bridge -c 192.168.0.1/24 link0 apu2 apu3</code> As for the <code>gre_add_tunnel</code> script, <code>link0</code> is the name of the interface to create and the <code>-c</code> flag can optionally be used to configure an IPv4 address on the interface. The remaining parameters, in this case <code>apu2</code> and <code>apu3</code> are the hosts to connect to. The resolution of these hostnames to IP-addresses is done in exactly the same way as for the <code>gre_add_tunnel</code> script.

#* On '''apu2''' run: <code>sudo gre_add_tunnel -c 192.168.0.2/24 link0 apu1</code>
#* On '''apu3''' run: <code>sudo gre_add_tunnel -c 192.168.0.3/24 link0 apu1</code>

In this case ''apu1'' is used as the central node and ''apu2'' and ''apu3'' connect to it in exactly the same way as in the [[#Creating a link between two nodes| two node]] scenario.

Once the 'GRE Bridge' has been created it can be used just like any other linux bridge. Additional interfaces can be added/removed using the <code>brctl</code> command. The bridge's stastus can be queried using the <code>brctl show</code> command:
user@apu1:~$ brctl show link0
bridge name bridge id STP enabled interfaces
link0 8000.a2b74a105a76 no link0_gre_0
link0_gre_1
In the above output, the <code>link0_gre_x</code> interfaces are the GRE tunnels to the individual nodes.

Removing the 'GRE Bridge' is done using the <code>gre_del_bridge_script</code>. It is highly recommended to use this script to remove the bridge (rather than the ''brctl'' command) as this script not only removes the bridge itself, but also cleans up the GRE-tunnels to the individual nodes.

To remove the bridge created in the above example the following command would have to be run

* On '''apu1''': <code>sudo gre_del_bridge link0</code>

Once the 'GRE Bridge' has been removed, the GRE-Tunnels on the other nodes also need to be cleaned up. This is done in exactly the same way as in the [[#Creating a link between two nodes| two node]] scenario:

<code>sudo gre_del_tunnel link0</code>

===Limitations===

==== Single tunnel between two nodes ====

GRE Tunnels are identified ''solely'' on the source and destination IP-address of the encapsulating packet. That means that it is ''not'' possible to establish two independent tunnels between the same two nodes at the same time.

For the [[#Creating a link between two nodes| two node]] scenario, this means that only a ''single'' link can be created between two nodes.

For the [[#Creating a link between three or more nodes| mutliple nodes]] scenario, a node can participate in multiple links as long as:
* each link uses a different 'central node'
* each 'central node' is only part of ''one'' link at the same time

It should be noted however, that GRE Tunnels can be used in combination with VLAN-tagging, so if you ''really'' need to create multiple links between two nodes, you can do so by defining multiple VLANs on the GRE tunnel interface. This however is not something that can be done using the scripts provided here.

==== Reduced MTU ====

Using a GRE Tunnel incurs an overhead of 38 bytes. This is because the original Ethernet packet needs to be prefixed with an additional IP- and Ethernet-header in order to send it to the other tunnel endpoint. To prevent fragmentation, the linux kernel automatically reduces the MTU of the GRE-interface. As a result, the MTU ''inside'' the tunnel is 38 bytes lower than the MTU ''outside'' the tunnel.

Except for a small decrease in performance this usually is not that big of a problem, with one notable exception. When, for instance, a WiFi AP-interface is bridged to a GRE Tunnel, the MTU is set to the smallest MTU of the 'slave' interfaces (in this case the GRE-tunnel). This lower MTU however is '''not''' automagically communicated to the connected WiFi clients and as a result these clients may still send packets that are larger than the MTU of the bridge. Since these packets are subsequently dropped by the linux bridge, this can lead to all sorts of nasty problems. The most common of which are dropped UDP packets and TCP sessions seem to work fine and then suddenly hang when more than a few bytes at a time are sent over them.

To resolve this issue the MTU of the other WiFi clients should be reduced to the MTU of the linux bridge. When static IP-addresses are used, this can easily be done using the <code>ip link</code> command:
ip link set mtu <mtu> dev <device>

If DHCP is used, the lower MTU should be advertised by the DHCP server.

== Setting up (E)GRE-Tunnels Citylab and the Virtual Wall ==

At this point it is not possible to set up 'multi-site' (e)GRE tunnels through the JFed interface, but it is possible to do so manually after the test has started. The process of doing so is very similar to manually creating EGRE-Tunnels within the citylab testbed itself, but the following caveats need to be taken into account:

* Setting up GRE Tunnels between nodes of different testbeds requires both endpoints to have a ''routable'' (i.e. public) IPv4 or IPv6 address and for GRE-traffic to be allowed by the firewall/router through which the nodes are connected to the internet. While all nodes of the citylab testbed meet the above requirement, this may not necessarily be the case for nodes of other testbeds.

* The ''MTU'' of the created link needs to be set manually. Otherwise you will most likely encounter MTU-mismatch issues given that, by default, the MTU of the created tunnel is calculated from the MTU of the uplink interface, and that not all testbeds use the same MTU on their uplink interface.

* GRE tunnels are '''not encrypted'''. Any data you send over them is sent plain-text over the internet so make sure not to send any sensitive data over them without adding encryption in some other way. If you need a tunnel which encrypts the traffic as well, OpenVPN is most likely a better solution.

The remainder of this section explains how to set up 'multi-site' (e)gre tunnels. We specifically focus on setting up GRE tunnels between Citylab and the [https://doc.ilabt.imec.be/ilabt/virtualwall/|Virtual Wall], but setting up GRE tunnels to other testbeds should be very similar.

=== Prerequisites ===

* Make sure that all nodes involved are running ''at least'' Ubuntu 16.04 (EGRE tunnels are not supported in Ubuntu 14.04 and before). For citylab nodes this is the case by default, but for the nodes on the virtual wall you will have to explicitly specify a different disk image when you set up the experiment as that testbed uses 'Ubuntu 14.04' by default.

* Setting up multi-site (e)gre tunnels is done with the same 'helper scripts' as [[#Manually_setting_up_EGRE-tunnels|used before]]. Before setting up the tunnels themselves, these scripts need to be installed on every node. This can be done using the following one-liner:
<code>wget -O- https://doc.lab.cityofthings.eu/w/images/9/93/Gre-utils.tar.gz | sudo tar -C /usr/local/ -zxvf -</code>

=== Creating a link between two nodes ===
[[File:Jfed-multisite-link-2-nodes.png|thumb|right|A 'simple' multi-site link with two nodes. 'citynode1' is a node of the Citylab testbed. 'vwallnode1' is a node of the Virtual Wall testbed.]]
A point-to-point (E)GRE tunnel can be created using the <code>gre_add_tunnel</code> script on both nodes.

The setup in the example on the right can be created as follows:

# Run the <code>hostname</code> command on both hosts to learn their fully qualified domain names (FQDNs). E.g:
#:<pre>root@citynode1:~# hostname
citynode1.gretest.wall2-ilabt-iminds-be.lab.cityofthings.eu</pre>
#:<pre>root@vwallnode1:~# hostname
vwallnode1.gretest.wall2-ilabt-iminds-be.wall2.ilabt.iminds.be</pre>
#: As shown in the examples above the FQDNs of the nodes depends not only on the hostname configured in JFed but also on the name of the experiment, project and the testbed the node belongs to. Therefore yuo will almost certainly need to use different FQDNs than the ones listed above
# Create the EGRE tunnel itself:
#* On '''citynode1''' run: <code> sudo gre_add_tunnel -c 192.168.0.1/24 -m 1400 -6 link0 <fqdn_of_vwallnode1></code>
#* On '''vwallnode1''' run: <code> sudo gre_add_tunnel -c 192.168.0.1/24 -m 1400 -6 link0 <fqdn_of_citynode1></code>

The <code>gre_add_tunnel</code> commands shown above are very similar to the ones used to set up an EGRE-tunnel between citylab nodes, but there are a few important differences:
* Rather than only specifying the hostname of the other endpoint, the full FQDN needs to be specified
* The <code>-m</code> flag is added to manually set the MTU on the link. As explained before, this is needed to prevent MTU-mismatch issues. In this specific example an MTU of <code>1400</code> bytes is specified since it provides enough headroom for multiple encapsulation headers and is 'good enough' for an initial test. For optimal performance however, the MTU should be set to <code>min(uplink MTU of all nodes involved) - <GRE Overhead>(38 bytes)</code>.
* The <code>-6</code> flag is added to instruct the <code>gre_add_tunnel</code> to create a 'GRE-over-IPv6' rather than a 'GRE-over-IPv4' tunnel. The reason for using IPv6 in this case is that the nodes of the virtual wall testbed don't have a public IPv4 address.

Instead of specifying the FQDNs of the nodes, it would also have been possible to specify the public IP address of the node directly but this is more cumbersome and error prone (especially when using IPv6). Once the tunnel has been created it can be used just like any regular Ethernet interface except that, once again, the MTU is somewhat lower.

As before, removing the tunnels is done using the <code>gre_del_tunnel</code> script:

<code>sudo gre_del_tunnel link0</code>

=== Creating a link between three or more nodes ===
[[File:Jfed-multisite-link-3-nodes.png|right|thumb|A multi-site link with three nodes. 'citynode1' and 'citynode2' are Citylab nodes. 'vwallnode1' is a node on the Virtual Wall testbed.]]

As with [[#Creating_a_link_between_three_or_more_nodes|GRE links between citylab nodes]], setting up a GRE-link with more than two nodes is done by:
#Creating GRE tunnels from all nodes to a 'central' node
#Bridging all of these tunnels together to create a single shared subnet.

When such an 'overlay subnet' spans multiple testbeds however, it is important to select the 'central node' very carefully. Consider for example the topology shown on the right. In this case a link is created to which two nodes of the citylab testbed and a single node of the virtual wall are connected. If, in this case, <code>vwallnode1</code> were to be selected as the 'central' node, then all traffic between <code>citynode1</code> and <code>citynode2</code> would be sent over the virtual wall testbed unnecessarily. If on the other hand <code>citynode1</code> or <code>citynode2</code> is used as a 'central' node, then this problem is avoided.

== Setting up Link impairment on GRE Tunnels ==

While it is not possible to configure link-impairment on GRE tunnels via the JFed interface, it is possible to do so manually once the test has started. This not only works for links created by JFed but also for links created manually.

Adding impairment to a link is done by running the following command on each node connected to the GRE-tunnel:

<code>sudo tc qdisc add dev <link> root netem loss random <loss_percentage>% rate <data_rate> delay <delay_in_micro_seconds></code>

In this command:
* <code><link></code> is the name of the gre tunnel interface
* <code><loss_percentage></code> is the percent of packets to be randomly dropped
* <code><data_rate></code> is the data rate limit to impose on the link (e.g 10mbit, 20kbit, ...)
* <code><delay_in_micro_seconds></code> is the delay (in microseconds) to add to packets sent from the host. (So: to add a 5ms delay you need to specify 5000)

As an example, the following command adds a 20% loss, a maximum data rate of 35mbit and a 7.5ms delay to link0

<code>sudo tc qdisc add dev link0 root netem loss random 20% rate 35mbit delay 7500</code>

'''Important''': The link impairment configured with the above command only affects packet that are ''sent'' from the host. Received packets are unaffected. Therefore it is imperative that you configure the same link impairment settings on both nodes connected to the GRE tunnel.

Removing link impairment on a link is done by running the following command

<code>sudo tc qdisc add dev <link> root netem</code>

To change the impairment settings on a link, you first need to remove the link impairment all together and then add it again with the new settings.

More information on the 'netem' traffic control queueing discipline can be found here: https://www.systutorials.com/docs/linux/man/8-tc-netem/

GRE Tunnels

2021-08-11T10:27:33Z

Dvdakker: /* Creating a link between three or more nodes */

At this point, the CityLab testbed only has limited support for ''automatically'' creating links between nodes using the JFed interface. Standard IP-in-IP GRE Tunnels and Ethernet-in-IP GRE Tunnels (''gre-tunnel'' and ''egre-tunnel'' links in JFed) are supported. Adding support ''native'' Ethernet links (lan links in JFed) is on our TODO list. Since both 'gre' and 'egre' tunnels are point-to-point tunnels it is currently not (yet) possible to create a JFed-link with more than two connected nodes. As a workaround it ''is'' possible to bridge multiple 'egre'-tunnels together into a single (tunneled) L2-network once the test has started.

This page provides a quick How-To guide on how to create standard 'gre' or 'egre-tunnels' in JFed and then explains how to manually set up and bridge egre-tunnels manually (after the test has started).

== Creating Links between nodes through the JFed interface ==

=== Setting up links ===

Creating Links between nodes through the JFed interface is very straightforward and can be done as follows:

* Start JFed, click on 'New' and drag in two wireless nodes. For this tutorial we'll assume that these nodes are called ''apu1'' and ''apu2''
* Edit the node properties so both nodes are sourced from the ''City of Things Antwerp'' testbed. (By default Wilab2 is used).[[File:Jfed_experiment_2_nodes.png|Drag in two nodes from the ''City of Things Antwerp'' testbed and name them ''apu1'' and ''apu2''|720px]]
* Drag a line from ''apu1'' to ''apu2''. A ''link'' element connecting the two nodes is automatically created.
* If you are using JFed GUI 5.9 or above, link type of the link will already be set to 'gre'. (This is shown between braces behind the name of the link). [[File:Jfed_gre_link_2_nodes.png||300px]] If this is not the case, or you would like to create an 'egre'-link instead, the link-type will need to be manually configured:
*# Right-click on the ''link''-element, Click on ''Configure Link'' and select the ''Link Type''-tab in the options window.
*# Change the link-type to 'gre-tunnel' or 'egre-tunnel' in the drop-down menu. If you select 'gre-tunnel' an IP-in-IP tunnel will be created, if you select 'egre-tunnel' an Ehternet-in-IP tunnel will be created. [[File:JFed_link_options_type.png|Set the link-type to 'gre-tunnel'|480px]]
* JFed will automatically configure an IP-address on the tunnel interface for both nodes. optionally you can modify these IP-addresses in the ''General'' tab of the Link options-window [[File:JFed_link_options_general.png|Changes the IP-addresses for the link if needed|480px]]

* Click 'Run' to start your slice. When all nodes are green, your experiment has started and the GRE-tunnel between the two nodes has been configured. [[File:JFed_gre_link_2_nodes_running.png|The running experiment|720px]]

*SSH into the ''apu1'' node.
*Run <code>ifconfig link02</code> to verify that the GRE-tunnel has been created
<pre>
user@apu1:~$ ifconfig link02
link02 Link encap:UNSPEC HWaddr 8F-81-55-10-00-00-80-3F-00-00-00-00-00-00-00-00
inet addr:192.168.0.1 P-t-P:192.168.0.1 Mask:255.255.255.0
inet6 addr: fe80::200:5efe:8f81:5510/64 Scope:Link
UP POINTOPOINT RUNNING NOARP MTU:1434 Metric:1
RX packets:2 errors:0 dropped:0 overruns:0 frame:0
TX packets:3 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1
RX bytes:112 (112.0 B) TX bytes:168 (168.0 B)
</pre>
*Ping the ''apu2''-tunnel enpoint by running <code>ping -c 4 <ip></code> where <code><ip></code> is the IP-address configured for the tunnel endpoint on ''apu2''. If you have not changed this, this is <code>192.168.0.2</code>
<pre>
user@apu1:~$ ping -c 4 192.168.0.2
PING 192.168.0.2 (192.168.0.2) 56(84) bytes of data.
64 bytes from 192.168.0.2: icmp_seq=1 ttl=64 time=2.21 ms
64 bytes from 192.168.0.2: icmp_seq=2 ttl=64 time=1.60 ms
64 bytes from 192.168.0.2: icmp_seq=3 ttl=64 time=1.46 ms
64 bytes from 192.168.0.2: icmp_seq=4 ttl=64 time=1.51 ms

--- 192.168.0.2 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3004ms
rtt min/avg/max/mdev = 1.464/1.699/2.218/0.303 ms
</pre>

=== Limitations ===

==== No more than 2 nodes per link ====

It is not possible to create (E)GRE-links with more than two nodes. The JFed GUI does allow additional nodes to be added to the link, but when the test is started the link will only exist on two nodes. This limitation is caused by the fact that the GRE-protocol only supports point-to-point tunnels. It is possible to [[#Manually setting up EGRE-tunnels|set up]] and [[#Creating a link between three or more nodes|bridge]] multiple egre-tunnels, but this cannot be done through the JFed interface.

==== EGRE-tunnels only work with 'CoT' disk images ====

To enable support for 'egre-tunnel' links, a few tweaks had to be made to the 'client-side' emulab-tools installed in the testbed disk images. Since these tweaks are not (yet) incorporated in the upstream emulab-code base, they are also not present in the many emulab-images used around the world.

We have added these tweaks to the 'CoT' disk-images of the CityLab testbed (see [[Disk Images]]), so if you want to use egre-links you must use a 'CoT' disk image.

==== (E)GRE-tunnels between nodes of different testbeds cannot be created via JFed ====

Even at the best of times, creating links between nodes of different testbeds is an 'experimental feature' at best. While it is perfectly possible to set up GRE and EGRE tunnels between nodes of different testbeds, it is currently not possible to do so through the JFed interface.

==== Link impairment on GRE Tunnels cannot be configured through the JFed GUI ====

The way JFed normally adds impairment to a link is by adding a so-called 'impairment node' in between the nodes of a link to change the properties of the traffic flowing over the link. This mechanism however only works with ethernet interfaces and not with GRE Tunnels. That being said: it is possible to add link impairment to GRE tunnels manually. How to do so is [[#Setting_up_Link_impairment_on_GRE_Tunnels|explained below]].

== Manually setting up EGRE-tunnels ==

'''Important:''' although it is possible to manually set up GRE-tunnels it is much easier to do so through the JFed-interface. The only reason for creating them manually is to create a 'link' between more than two nodes or to create links with nodes of another testbed (both of which are currently not supported through JFed)

A number of [[:File:Gre-utils.tar.gz | scripts]] are provided to make it as easy as possible to create and manage EGRE-tunnels.
Before any EGRE-tunnels can be created, these scripts first need to be installed on every node.
This can be done by running the following one-liner:

<code>wget -O- https://doc.lab.cityofthings.eu/w/images/9/93/Gre-utils.tar.gz | sudo tar -C /usr/local/ -zxvf -</code>

This command will
* Download the [[:File:Gre-utils.tar.gz | Gre-utils.tar.gz]] archive
* Install the scripts in <code>/usr/local</code>

=== Creating a link between two nodes ===

[[File:Jfed-link-2-nodes.png|thumb|right|A simple link between two nodes.]]

A simple GRE tunnel can be created by running the <code>gre_add_tunnel</code> script on both nodes.

The setup shown in the example on the right can be created as follows:

* On '''apu1''' run: <code>sudo gre_add_tunnel -c 192.168.0.1/24 link0 apu2</code>
* On '''apu2''' run: <code>sudo gre_add_tunnel -c 192.168.0.2/24 link0 apu1</code>

In the above commands <code>link0</code> is the name of the interface to create and <code>apuX</code> is the host to create a tunnel to. The optional <code>-c</code> flag can be used to automatically configure an IPv4-address on the newly created interface.

To allow the node names configured in JFed to be used to configure GRE tunnels, the <code>gre_add_tunnel</code> script tries to behave somewhat intelligently when resolving hostnames:
* If the specified host is a Fully Qualified Domain Name (i.e: it contains a '.'), a regular DNS query is performed
* If the specified host is a simple hostname (no '.' in the hostname), the <code>gre_add_tunnel</code> script ''first'' tries to resolve the specified host to a node name specified in JFed. If this fails, the IP-address of the host is resolved using a normal DNS query.
* If the specified host is a valid IPv4 address, no DNS-resolution is performed.

Once the GRE tunnel is created, it can be used just like any other Ethernet interface, except that the MTU is a bit lower than normal.

Removing the tunnel is done using the <code>gre_del_tunnel</code> script.
To remove the tunnels created above for example, the following command would have to be run on each node:

<code>sudo gre_del_tunnel link0</code>

=== Creating a link between three or more nodes ===
[[File:Jfed-link-3-nodes.png|thumb|right|A slightly more complicated link between more than two nodes.]]

Creating a link between more than two nodes is (slightly) more complicated since GRE only allows Point-to-Point tunnels to be created. To work around this, GRE Tunnels are created from one 'central' node to all other nodes on the link. These GRE Tunnels are then 'bridged' together on the central node to form a shared subnet. Such a 'GRE Bridge' can easily be created using the <code>gre_add_bridge</code> script.

The 3-node setup shown in the example on the right can be created as follows:

# On '''apu1''' run: <code>sudo gre_add_bridge -c 192.168.0.1/24 link0 apu2 apu3</code> As for the <code>gre_add_tunnel</code> script, <code>link0</code> is the name of the interface to create and the <code>-c</code> flag can optionally be used to configure an IPv4 address on the interface. The remaining parameters, in this case <code>apu2</code> and <code>apu3</code> are the hosts to connect to. The resolution of these hostnames to IP-addresses is done in exactly the same way as for the <code>gre_add_tunnel</code> script.

#* On '''apu2''' run: <code>sudo gre_add_tunnel -c 192.168.0.2/24 link0 apu1</code>
#* On '''apu3''' run: <code>sudo gre_add_tunnel -c 192.168.0.3/24 link0 apu1</code>

In this case ''apu1'' is used as the central node and ''apu2'' and ''apu3'' connect to it in exactly the same way as in the [[#Creating a link between two nodes| two node]] scenario.

Once the 'GRE Bridge' has been created it can be used just like any other linux bridge. Additional interfaces can be added/removed using the <code>brctl</code> command. The bridge's stastus can be queried using the <code>brctl show</code> command:
user@apu1:~$ brctl show link0
bridge name bridge id STP enabled interfaces
link0 8000.a2b74a105a76 no link0_gre_0
link0_gre_1
In the above output, the <code>link0_gre_x</code> interfaces are the GRE tunnels to the individual nodes.

Removing the 'GRE Bridge' is done using the <code>gre_del_bridge_script</code>. It is highly recommended to use this script to remove the bridge (rather than the ''brctl'' command) as this script not only removes the bridge itself, but also cleans up the GRE-tunnels to the individual nodes.

To remove the bridge created in the above example the following command would have to be run

* On '''apu1''': <code>sudo gre_del_bridge link0</code>

Once the 'GRE Bridge' has been removed, the GRE-Tunnels on the other nodes also need to be cleaned up. This is done in exactly the same way as in the [[#Creating a link between two nodes| two node]] scenario:

<code>sudo gre_del_tunnel link0</code>

===Limitations===

==== Single tunnel between two nodes ====

GRE Tunnels are identified ''solely'' on the source and destination IP-address of the encapsulating packet. That means that it is ''not'' possible to establish two independent tunnels between the same two nodes at the same time.

For the [[#Creating a link between two nodes| two node]] scenario, this means that only a ''single'' link can be created between two nodes.

For the [[#Creating a link between three or more nodes| mutliple nodes]] scenario, a node can participate in multiple links as long as:
* each link uses a different 'central node'
* each 'central node' is only part of ''one'' link at the same time

It should be noted however, that GRE Tunnels can be used in combination with VLAN-tagging, so if you ''really'' need to create multiple links between two nodes, you can do so by defining multiple VLANs on the GRE tunnel interface. This however is not something that can be done using the scripts provided here.

==== Reduced MTU ====

Using a GRE Tunnel incurs an overhead of 38 bytes. This is because the original Ethernet packet needs to be prefixed with an additional IP- and Ethernet-header in order to send it to the other tunnel endpoint. To prevent fragmentation, the linux kernel automatically reduces the MTU of the GRE-interface. As a result, the MTU ''inside'' the tunnel is 38 bytes lower than the MTU ''outside'' the tunnel.

Except for a small decrease in performance this usually is not that big of a problem, with one notable exception. When, for instance, a WiFi AP-interface is bridged to a GRE Tunnel, the MTU is set to the smallest MTU of the 'slave' interfaces (in this case the GRE-tunnel). This lower MTU however is '''not''' automagically communicated to the connected WiFi clients and as a result these clients may still send packets that are larger than the MTU of the bridge. Since these packets are subsequently dropped by the linux bridge, this can lead to all sorts of nasty problems. The most common of which are dropped UDP packets and TCP sessions seem to work fine and then suddenly hang when more than a few bytes at a time are sent over them.

To resolve this issue the MTU of the other WiFi clients should be reduced to the MTU of the linux bridge. When static IP-addresses are used, this can easily be done using the <code>ip link</code> command:
ip link set mtu <mtu> dev <device>

If DHCP is used, the lower MTU should be advertised by the DHCP server.

== Setting up (E)GRE-Tunnels Citylab and the Virtual Wall ==

At this point it is not possible to set up 'multi-site' (e)GRE tunnels through the JFed interface, but it is possible to do so manually after the test has started. The process of doing so is very similar to manually creating EGRE-Tunnels within the citylab testbed itself, but the following caveats need to be taken into account:

* Setting up GRE Tunnels between nodes of different testbeds requires both endpoints to have a ''routable'' (i.e. public) IPv4 or IPv6 address and for GRE-traffic to be allowed by the firewall/router through which the nodes are connected to the internet. While all nodes of the citylab testbed meet the above requirement, this may not necessarily be the case for nodes of other testbeds.

* The ''MTU'' of the created link needs to be set manually. Otherwise you will most likely encounter MTU-mismatch issues given that, by default, the MTU of the created tunnel is calculated from the MTU of the uplink interface, and that not all testbeds use the same MTU on their uplink interface.

* GRE tunnels are '''not encrypted'''. Any data you send over them is sent plain-text over the internet so make sure not to send any sensitive data over them without adding encryption in some other way. If you need a tunnel which encrypts the traffic as well, OpenVPN is most likely a better solution.

The remainder of this section explains how to set up 'multi-site' (e)gre tunnels. We specifically focus on setting up GRE tunnels between Citylab and the [https://doc.ilabt.imec.be/ilabt/virtualwall/|Virtual Wall], but setting up GRE tunnels to other testbeds should be very similar.

=== Prerequisites ===

* Make sure that all nodes involved are running ''at least'' Ubuntu 16.04 (EGRE tunnels are not supported in Ubuntu 14.04 and before). For citylab nodes this is the case by default, but for the nodes on the virtual wall you will have to explicitly specify a different disk image when you set up the experiment as that testbed uses 'Ubuntu 14.04' by default.

* Setting up multi-site (e)gre tunnels is done with the same 'helper scripts' as [[#Manually_setting_up_EGRE-tunnels|used before]]. Before setting up the tunnels themselves, these scripts need to be installed on every node. This can be done using the following one-liner:
<code>wget -O- https://doc.lab.cityofthings.eu/w/images/9/93/Gre-utils.tar.gz | sudo tar -C /usr/local/ -zxvf -</code>

=== Creating a link between two nodes ===
[[File:Jfed-multisite-link-2-nodes.png|thumb|right|A 'simple' multi-site link with two nodes.]]
A point-to-point (E)GRE tunnel can be created using the <code>gre_add_tunnel</code> script on both nodes.

The setup in the example on the right can be created as follows:

# Run the <code>hostname</code> command on both hosts to learn their fully qualified domain names (FQDNs). E.g:
#:<pre>root@citynode1:~# hostname
citynode1.gretest.wall2-ilabt-iminds-be.lab.cityofthings.eu</pre>
#:<pre>root@vwallnode1:~# hostname
vwallnode1.gretest.wall2-ilabt-iminds-be.wall2.ilabt.iminds.be</pre>
#: As shown in the examples above the FQDNs of the nodes depends not only on the hostname configured in JFed but also on the name of the experiment, project and the testbed the node belongs to. Therefore yuo will almost certainly need to use different FQDNs than the ones listed above
# Create the EGRE tunnel itself:
#* On '''citynode1''' run: <code> sudo gre_add_tunnel -c 192.168.0.1/24 -m 1400 -6 link0 <fqdn_of_vwallnode1></code>
#* On '''vwallnode1''' run: <code> sudo gre_add_tunnel -c 192.168.0.1/24 -m 1400 -6 link0 <fqdn_of_citynode1></code>

The <code>gre_add_tunnel</code> commands shown above are very similar to the ones used to set up an EGRE-tunnel between citylab nodes, but there are a few important differences:
* Rather than only specifying the hostname of the other endpoint, the full FQDN needs to be specified
* The <code>-m</code> flag is added to manually set the MTU on the link. As explained before, this is needed to prevent MTU-mismatch issues. In this specific example an MTU of <code>1400</code> bytes is specified since it provides enough headroom for multiple encapsulation headers and is 'good enough' for an initial test. For optimal performance however, the MTU should be set to <code>min(uplink MTU of all nodes involved) - <GRE Overhead>(38 bytes)</code>.
* The <code>-6</code> flag is added to instruct the <code>gre_add_tunnel</code> to create a 'GRE-over-IPv6' rather than a 'GRE-over-IPv4' tunnel. The reason for using IPv6 in this case is that the nodes of the virtual wall testbed don't have a public IPv4 address.

Instead of specifying the FQDNs of the nodes, it would also have been possible to specify the public IP address of the node directly but this is more cumbersome and error prone (especially when using IPv6). Once the tunnel has been created it can be used just like any regular Ethernet interface except that, once again, the MTU is somewhat lower.

As before, removing the tunnels is done using the <code>gre_del_tunnel</code> script:

<code>sudo gre_del_tunnel link0</code>

=== Creating a link between three or more nodes ===
[[File:Jfed-multisite-link-3-nodes.png|right|thumb|A multi-site link with three nodes. 'citynode1' and 'citynode2' are Citylab nodes. 'vwallnode1' is a node on the Virtual Wall testbed.]]

As with [[#Creating_a_link_between_three_or_more_nodes|GRE links between citylab nodes]], setting up a GRE-link with more than two nodes is done by:
#Creating GRE tunnels from all nodes to a 'central' node
#Bridging all of these tunnels together to create a single shared subnet.

When such an 'overlay subnet' spans multiple testbeds however, it is important to select the 'central node' very carefully. Consider for example the topology shown on the right. In this case a link is created to which two nodes of the citylab testbed and a single node of the virtual wall are connected. If, in this case, <code>vwallnode1</code> were to be selected as the 'central' node, then all traffic between <code>citynode1</code> and <code>citynode2</code> would be sent over the virtual wall testbed unnecessarily. If on the other hand <code>citynode1</code> or <code>citynode2</code> is used as a 'central' node, then this problem is avoided.

== Setting up Link impairment on GRE Tunnels ==

While it is not possible to configure link-impairment on GRE tunnels via the JFed interface, it is possible to do so manually once the test has started. This not only works for links created by JFed but also for links created manually.

Adding impairment to a link is done by running the following command on each node connected to the GRE-tunnel:

<code>sudo tc qdisc add dev <link> root netem loss random <loss_percentage>% rate <data_rate> delay <delay_in_micro_seconds></code>

In this command:
* <code><link></code> is the name of the gre tunnel interface
* <code><loss_percentage></code> is the percent of packets to be randomly dropped
* <code><data_rate></code> is the data rate limit to impose on the link (e.g 10mbit, 20kbit, ...)
* <code><delay_in_micro_seconds></code> is the delay (in microseconds) to add to packets sent from the host. (So: to add a 5ms delay you need to specify 5000)

As an example, the following command adds a 20% loss, a maximum data rate of 35mbit and a 7.5ms delay to link0

<code>sudo tc qdisc add dev link0 root netem loss random 20% rate 35mbit delay 7500</code>

'''Important''': The link impairment configured with the above command only affects packet that are ''sent'' from the host. Received packets are unaffected. Therefore it is imperative that you configure the same link impairment settings on both nodes connected to the GRE tunnel.

Removing link impairment on a link is done by running the following command

<code>sudo tc qdisc add dev <link> root netem</code>

To change the impairment settings on a link, you first need to remove the link impairment all together and then add it again with the new settings.

More information on the 'netem' traffic control queueing discipline can be found here: https://www.systutorials.com/docs/linux/man/8-tc-netem/

File:Jfed-multisite-link-3-nodes.png

2021-08-11T10:26:02Z

Dvdakker: