In a recent post, I told you about the NAPCORE Mobility Data Days. I was there as both, a supporter of the European Parking Association (EPA) as well as with my APDS hat on (APDS: Alliance for Parking Data Standards). At the conference, EPA signed a cooperation agreement with the EV Roaming Foundation.

Moving forward, we will work out ways to create smart and deep links between the two standards (APDS and OCPI), because often parking and charging happen at the same time. I will keep you posted.

Last week, I attended the NAPCORE Mobility Data Days in Budapest. Three days packed with interesting presentations, workshop sessions, and networking amongst mobility data enthusiasts. For those who don't know yet: NAPCORE is a cooperation to coordinate and harmonise more than 30 mobility data platforms across Europe. Sharing mobility-related data creates new opportunities...and it is more and more also becoming the law in the EU member states. So, if you work in the mobility sector, it is time to learn more. The linked website is a good starting point.

For mobility enthusiasts amongst you who are living and/or working in Germany, there is something you should definitely check out: the Mobilithek, an open mobility data exchange platform hosted and maintained by the German government (Federal Ministry for Digital and Transport, BMDV). You can find it here: https://www.mobilithek.info. It is a B2B platform where data providers and data consumers can find matching connections.

While it is in an early stage, it is certainly worth a visit, and you will see more and more data being provided there down the road.

Folks, please meet SteVe. SteVe is a software to manage charge points for electric vehicles. It was developed at the renowned RWTH Aachen university, and it is free to use (open sourced under the GPL license).

You can use SteVe to test and evaluate your ideas in the realm of e-mobility. SteVe comes with support for a variety of related protocols:

• OCPP1.2S and OCPP1.2.J
• OCPP1.5S and OCPP1.5J
• OCPP1.6S and OCPP1.6J

A group of SteVe enthusiasts maintains a curated list of charge point models that have been tested with SteVe. You can find it here: https://github.com/RWTH-i5-IDSG/steve/wiki/Charging-Station-Compatibility .

The SteVe team maintains a GitHub repository (https://github.com/steve-community/steve). There, you can find the source code as well as instructions on how to spin up your own SteVe instance.

For the German-speaking amongst you: the name "SteVe" is a combination of Ste (short for "Steckdose", German for "socket") and Ve (short for "Verwaltung", German for management).

Have fun with SteVe!

A while ago, I joined the APDS team (Alliance for Parking Data Standards), a group of people from the parking industry driving interoperability in the realm of mobility. Born as an initiative by major industry associations, it now has become the blueprint for an internal standard (ISO TS 5206-1) and inspired European standards like e.g. DATEX II (published as CEN/TS 16157-6).

It all started out with a parking data platform project in the UK, and here I am now: head of the APDS change control team. The change control team is your point of contact to suggest adjustments and incremental extensions to the standard. You think something's missing? Just let us know, and we'll work with you.

Some random thoughts

Why?

When it comes to open sourcing a project, they key question should be: “Why do I want to do this?”

Typical reasons are:

• growing a user community to find additional contributors or at least be notified about bugs (drive further project development, get support)
• recruiting (find good developers)
• marketing (offer your services based on a reference project)

A not-so-good reason is: “I just want to throw it over the fence so people can do with it whatever they like.”

How?

Another question to answer is: “How exactly do you want to handle/maintain this?”

Once you open source a project that is potentially interesting for others, there is a risk of this act generating overheads for you:

• People might have questions: Who will answer them? Will you allow GitHub issues to be opened?
• People might think they found a bug and expect you to do your own research and potentially fix it.
• There might even be people who would like to actively contribute to the project. Will you want to allow GitHub pull requests? Who will regularly review and assess
• Users/followers of the project might expect to see it continuously maintained, e.g. in terms of updating external dependencies. Who would be the one to do this?

My kubernetes students are moving towards making their pods interact with each other. Along those lines, they need to find a good approach to service discovery, i.e. how to find a healthy instance of a service that another service wants to consume. A good way to do (server-side) discovery is to make use of k8s'es DNS subsystem. All a  calling service has to do is using the target service's FQDN (fully qualified domain name). The structure of a FQDN in k8s is this:

http://{service name}.{namespace}.svc.{cluster}.local:{service port}

Example: if the service you want to reach out to

• is named "phonebook"
• is running in the "default" namespace
• in cluster "explorer"
• on port 8080

...then the fully qualified domain name to use in your http request would be

http://phonebook.default.svc.explorer.local:8080

I am tutoring some people starting with kubernetes. On their local machines, they're using minikube, and it lets them explore a lot of k8s features without the immediate need of a full-fledged cluster. My students do a lot of research themselves, but occasionally they get back to me with questions they can't seem to find an answer for. In order to not loose those questions, I am writing them down here. I will keep updating this post as they go.

LoadBalancer Services

One team member  has started to experiment with exposing services running inside her minikube to the outside world. She defined an object of kind Service, and in the spec part, she configured it to be of type LoadBalancer. She was expecting this to eventually assign an external ip. However, the EXTERNAL-IP element remained in state <pending>:

Well, in this case minikube tunnel is your friend. Open another terminal window, execute the "minikube tunnel" command, and voilà:

You now have an external ip (it will go away as soon as you close the tunnel window). Please note that this is specific to the use of minikube. When you're using the managed k8s environment of a public cloud provider such as AWS, Google, Microsoft or DigitalOcean, they will automatically handle everything required for a  "real" LoadBalancer Service type.

Minikube DNS

Minikube comes with DNS activated by default. So, most people wont' have to do anything, and it will work out of the box. A team member approached me because he wasn't that lucky. He is using a machine running Mac OS, and is using VirtualBox for virtualisation. In this combination, you need to patch a setting for your virtual machine.

Let's assume your VM is named "minikube" (this is the default, check it via minikube status). Open a terminal window and run the following commands:

minikube stop
VBoxManage modifyvm minikube --natdnshostresolver1 on
minikube start

This will do the trick and make DNS available to all your objects. For those who are curious about the details: you can find them here and here.

Exposing a Minikube Ingress

Next up was the attempt to expose the (training) Minikube so it was - temporarily - accessible from other machines on the local network. As this was not within the scope of the training and was only supposed to be temporary in nature, we used the kubectl port-forward command to accomplish this:

kubectl port-forward \
    --address=0.0.0.0 \
    --namespace=kube-system \
    deployment/ingress-nginx-controller 80:80

You should see a response similar to this one here:

Forwarding from 0.0.0.0:80 -> 80

As the response indicates, this will forward all traffic coming in via the Minikube host's network adapter on port 80 to the ingress controller listening on port 80 of our Minikube. Whenever an incoming request is handled, you'll see a corresponding log line:

Handling connection for 80

Logging

In the pre-k8s era, the team already had been using Elasticsearch and Kibana for centralized logging. The Docker containers (at the time) would then use the fluentd (td-agent) log driver to contact a locally-installed log shipping agent which in turn would send all received messages to the Elasticsearch instance. The team wanted to keep using Elasticsearch/Kibana, so the question was simple how to accomplish this. One of the team members did some digging and came across this article: https://mherman.org/blog/logging-in-kubernetes-with-elasticsearch-Kibana-fluentd/ . Another (similar) post can be found here: https://medium.com/trendyol-tech/forwarding-kubernetes-containers-logs-to-elasticsearch-with-fluent-bit-and-showing-logs-with-411587e54e22

Whenever a team is tasked with the development of a mobile application, one topic often is greatly underrated or simply not dealt with for other reasons like laziness or a deep dislike of related activities: application security.

Of course, somehow everyone knows that appropriate application security is a must-have, and sometimes a team just can't find a good approach on how to deal with it. Let me give you some snippets of inspiration in this regard here:

Regardless of the application technology chosen (web, hybrid, cross-platform), a mobile device must always be considered insecure. While not prominently visible to the end user, the communication between the (mobile) frontend and the backend needs to be secured against common and application-specific vulnerabilities.

Threats catalogue

Before a team can take measures in terms of application security, potential threats first need to be identified. This is typically done by means of compiling a threats catalogue listing potential threats from different categories like e.g.

• Organizational shortcomings
• Human failure
• Technical failure
• Force majeur (if applicable)

Keep in mind that security risks can be induced both, externally and internally. Also, they can be deliberate or accidental in nature. All those factors need to be considered when putting together the list of potential threats.

Security mechanisms/features

Once an initial version of the threats catalogue has been created, the team can go over it and see what can be done to address which risk. It helps to prioritize corresponding tasks by factors like “severity of consequences” and “likeliness to happen”.

Security is not a one-time task

It goes without saying that the topic of security must accompany the entire life cycle of an application. It is not something you get done once and forever.

• Every iteration shall have a task to answer the question “are the planned changes/enhancements likely to induce new security risks?”
• As a team, make sure to define a recurring task “threats catalogue update/revision”.

Obviously, those are just some examples of what should be taken into consideration. It is always a good idea to talk to other teams and find out how they have organized their work in this regard.

Stay secure! 🔐

Jez Humble (Twitter: @jezhumble) teaches a class on software product management at UC Berkeley's. For 2020 he recorded all his lectures, and you can find everything available for free (licensed CC-BY-SA) here (it is a Google document with links to YouTube video clips).

I personally like especially this one here which deals with OKRs and helps to focus on the right stuff...

In case you are using Docker on Linux systems (pretty much the standard), it is important to be aware of the fact that Docker is is using iptables rules to provide network isolation. This is something that usually happens automatically under the hood, so many people - like me - don't dive into the details of what's going on there. The other day however, I came across the fact that I probably should have given some more attention to how this actually works.

If you're using one of the major cloud providers (AWS, Azure, Google, IBM, ...), they will provide you with a "real" firewall one way or the other, and that's a good thing. If however you are self-hosting a web-facing machine for test purposes, you might be using ufw, the "uncomplicated firewall". This is an easy-to-use tool that lets you set up policies concerning which protocols from which sources are allowed to reach out to which port on particular network interfaces. When it comes to Docker, you should be aware of one important fact:

Docker does not honor ufw rules

Let's say e.g. you have a Docker container on your web-facing test server that is running NGINX, and you used the  -p 8080:8080 parameter to make this http server available outside of the container listing on port 8080. Now, the web - i.e. everyone - will be able to connect to this NGINX instance via http://yourdomain:8080/. Let's assume for a minute that you decide to make this endpoint unavailable to the public. Easy, he? You ssh to your machine, and a one-liner will do the job for you:

sudo ufw deny 8080

Mission accomplished. Are you sure? Go ahead, open the browser on your desktop machine, and navigate to http://yourdomain:8080...and you will see that NXGINX is still serving contents to the public. How is this even possible 😵??  You did close port #8080. Well, as mentioned earlier: Docker is manipulating iptables rules, but it doesn't give a sh.. about ufw, at least not out of the box.

Docker typically sets up two custom iptables chains. They're named DOCKER and DOCKER-USER. The DOCKER chain is used by Docker itself, and you don't want to mess with it. If you have rules/policies that you want to see applied, you should add them to the DOCKER-USER chain. The default behaviour is that all external source IP addresses are allowed to connect to your Docker host.

So, if you want to only allow external traffic from a particular IP address, you will have to add a corresponding rule to the DOCKER-USER chain:

iptables -I DOCKER-USER -i eth0 ! -s 192.168.1.11 -j DROP

(this is assuming that your host's external network interface is named eth0, and the only source IP you want to allow is 192.168.1.11).

One final note: some posts in various blogs and support sites suggest you go and turn off iptables manipulation by Docker (there is a key named iptables in the Docker configuration that defaults to true and can be set to false). This is not a good idea, as it will also turn off a lot of the wanted behaviour.

Hope this helps.

I have been using keycloak as my identity management solution for a couple of years now, and I have yet to see a different OSS solution that might make me consider a change.

In integration testing, staging and production systems, I am using a keycloak docker container with a postgresql companion container holding the data. While the keycloak admin console does offer export/import functionality to a certain extent, it is limited: users cannot be exported, neither can secrets (passwords etc.). There is however a keycloak-provided way of accomplishing this.

When starting up, keycloak checks for a bunch of system properties that control migration actions, export and import in particular. The following settings will make keycloak export all realms, client and user settings inclusive of their passwords:

-Dkeycloak.migration.action=export
-Dkeycloak.migration.provider=singleFile
-Dkeycloak.migration.file=/export/kcdump.json

Upon the next start-up, keycloak will write all details out to a file named /export/kcdump.json (in this example). There is also an alternative where you can have keycloak write a separate file per realm and another separate file for the users in a realm. If that's what you want (my favorite), you will have to go with the dir provider instead of the singleFile option, and you'll have to configure a directory name rather than a file name. This is how it looks like:

-Dkeycloak.migration.action=export
-Dkeycloak.migration.provider=dir
-Dkeycloak.migration.dir=/export

That way, you will end up with files master-realm.json, master-users-0.json, someother-realm.json, someother-users-0.json (if e.g. you have two realms named master and someother).

The process of importing from above json files into a fresh keycloak database will most likely not surprise you:

-Dkeycloak.migration.action=import
-Dkeycloak.migration.provider=singleFile
-Dkeycloak.migration.file=/export/kcdump.json
-Dkeycloak.migration.strategy=OVERWRITE_EXISTING

I think this an elegant way to "clone" a setup. Furthermore: json files are good candidates to be customized using an automated deployment tool such as e.g. Ansible. That's what I ususally do: I create jinja2 templates from the json realm definitions, customize them by injecting the variable values I need and then spin up a keycloak container that imports the customized files into the database. The official keycloak documentation for this feature can be found here: https://www.keycloak.org/docs/6.0/server_admin/#_export_import.

I am a big fan of Kafka, and I am using it a lot to connect my domains and services with scalable and performant asynchronous communication. Over the time, I sometimes run into a situation where I need to get rid of a topic that has become obsolete. In some (rare) cases, topics are "sticky", and even though I ordered a deletion, they somehow remain.

This is where zookeeper-shell comes into play. Let's assume for this example that I

• have a zookeeper instance running on a server named zookeeper-1, listening on port 2181.
• let's further assume that I have a topic named NOTNEEDEDANYMORE that I want to get rid of.
• let's finally assume that the normal way of deleting the topic didn't work (for whatever reason)

In this special situation, the following two commands will do the job:

$ zookeeper-shell zookeeper-1:2181
# now I am connected to the shell
[] rmr /brokers/topics/NOTNEEDEDANYMORE
[] rmr /admin/delete_topics/NOTNEEDEDANYMORE
# by now, the topic is gone
# quit the shell
[] quit
$

I don't know why exactly, but when it comes to high-performance no-SQL databases with clustering capabilities, many people seem to exclude Apache's Cassandra from their list of candidates. This might be a mistake. Cassandra has been around for quite a while now, so it is safe to say that it is a mature product fit for production. One good proof for this is the fact that Apple is running 25,000+ Cassandra nodes.

So, time to at least take a look. The following script expects to be run in a Linux environment with pre-installed Docker. It sets up a cluster with three nodes (granted, on the same machine, but hey, we're just getting started here).

# cassandra/run.sh
#
# setup script for a docker based cassandra cluster
#

# settings
CASSANDRA_VERSION="3.11.6"
NAME_TEMPLATE="cassandra_node"
NUMBER_OF_NODES=3
MEMORY_PER_CONTAINER="4g"
CLUSTER_NAME="plexx"
VOLUME_BASE="/opt/cassandra/node"
STARTUP_DELAY=60

# network: create own network if it doesn't exist already
CASSANDRA_NETWORK=`docker network ls --format="{{ .Name }}" |grep cassandra`
if [ -z "${CASSANDRA_NETWORK}" ]
then
	echo "cassandra network doesn't exist yet, so create it"
	docker network create cassandra
else
	echo "cassandra network already present"
fi

# stop and remove any existing containers (data will NOT be deleted)"
for node in $(seq $NUMBER_OF_NODES)
do
	CONTAINER_NAME="${NAME_TEMPLATE}${node}"
	echo "checking container $CONTAINER_NAME ..."
	NODE_IS_RUNNING=$(docker ps --format="{{ .Names }}" |grep "$CONTAINER_NAME")
	if [ -n "$NODE_IS_RUNNING" ]
	then
		echo "container ${CONTAINER_NAME} is running, stop it"
		docker stop "$CONTAINER_NAME"
	else
		echo "container $CONTAINER_NAME is not running"
	fi
	NODE_EXISTS=$(docker ps -a --format="{{ .Names }}" |grep "$CONTAINER_NAME")
	if [ -n "$NODE_EXISTS" ]
	then
		echo "remove pre-existing container $CONTAINER_NAME"
		docker rm "$CONTAINER_NAME"
	fi
done

# make sure host volumes exists
for node in $(seq $NUMBER_OF_NODES)
do
	MOUNT_POINT="${VOLUME_BASE}${node}"
	if [ -d "$MOUNT_POINT" ]
	then
		echo "mount point at $MOUNT_POINT exists"
	else
		echo "mount point at $MOUNT_POINT does not exist yet, so create it"
		sudo mkdir -p "$MOUNT_POINT"
	fi
done

# create and run the MASTER container
echo "create and start master"
docker run -d \
--name "${NAME_TEMPLATE}1" \
--network=cassandra \
--memory $MEMORY_PER_CONTAINER \
-e CASSANDRA_CLUSTER_NAME="$CLUSTER_NAME" \
-v "${VOLUME_BASE}1:/var/lib/cassandra" \
cassandra:"$CASSANDRA_VERSION"

# give the master some time to start up
echo "waiting $STARTUP_DELAY seconds..."
sleep "$STARTUP_DELAY"

# determine IP address of master
MASTER_IP="$(docker inspect --format='{{ .NetworkSettings.Networks.cassandra.IPAddress }}' ${NAME_TEMPLATE}1)"
echo "master node IP address is $MASTER_IP (will be used as seed)"

# all the other nodes 
for node in $(seq 2 $NUMBER_OF_NODES)
do
	echo "create and start node #$node"
	docker run -d \
	--name "${NAME_TEMPLATE}${node}" \
	--network=cassandra \
	--memory $MEMORY_PER_CONTAINER \
	-e CASSANDRA_CLUSTER_NAME="$CLUSTER_NAME" \
	-e CASSANDRA_SEEDS="$MASTER_IP" \
	-v "${VOLUME_BASE}${node}:/var/lib/cassandra" \
	cassandra:"$CASSANDRA_VERSION"
done

echo "done."

To verify that everything went well, issue the following command from your terminal command line: docker exec -it cassandra_node1 bash -c 'nodetool status'.  This should generate some output similar to this:

Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address     Load     Tokens  Owns (effective)  Host ID                               Rack
UN  172.18.0.2  326.58 KiB  256  100.0%            668b272b9fea  rack1
UN  172.18.0.3  319.04 KiB  256  100.0%            1bb5e1c153b7  rack1

A status of "UN" stands for Up and Normal. This is what you want. Note: Cassandra needs memory. Anyhting below the 4g that I configured in my script most likely will make it refuse to start up. Swap is fine, though.

Finally, here are the ports used by / to be configured for Cassandra:

I will follow up with a second post showing how to connect and talk to your new cluster using Go.

A robust database engine is the foundation of many applications, no matter where they are running. There is a vast variety of options you have to choose from. But when it comes to horizontal scaling and resilience, things quickly get complicated. Clustered databases with read and write access typically are no low-hanging fruits and often are reserved for the enterprise versions of commercial database products. Well, I have good news for you: there is a company named Cockroach Labs that is offering a database claiming to do exactly this (i.e. scale and distribute). And while there are commercial plans to run a managed cluster on AWS and/or Google, the self-hosted variant is free to use, event for commercial purposes. CockroachDB comes with the BSL (Business Source License), a model that is mostly generous unless you are planning to offer the product in a database-as-a-service mode. This is excluded, and the reason is that the company wants to protect themselves from being taken over by the big guys (Azure, AWS, Google) and losing this type of business to them. This has happened before with other popular open source database products.

While this obviously doesn't make a lot of sense for a production environment, you can easily spin up a CockroachDB cluster on one machine using Docker. This is what I will show you in this post.

Prerequisites

I won't go over the details of installing docker and docker-compose. There are many good tutorials out there on how to do this. So, I am assuming that you have a test machine with some flavour of Linux and docker-compose installed.

Starting the Cluster

Here's the docker-compose.yml file I used for my little test:

version: "3"
services:
  cockroach1:
    image: cockroachdb/cockroach
    command: start --insecure
    ports:
      - "8080:8080"
    volumes:
      - ./data/cockroach1:/cockroach/cockroach-data
    networks:
      cockroachnetwork:
        aliases:
          - cockroach1

  cockroach2:
    image: cockroachdb/cockroach
    command: start --insecure --join=cockroach1
    volumes:
      - ./data/cockroach2:/cockroach/cockroach-data
    depends_on:
      - cockroach1
    networks:
      cockroachnetwork:
        aliases:
          - cockroach2

  cockroach3:
    image: cockroachdb/cockroach
    command: start --insecure --join=cockroach1
    volumes:
      - ./data/cockroach3:/cockroach/cockroach-data
    depends_on:
      - cockroach1
    networks:
      cockroachnetwork:
        aliases:
          - cockroach3

networks:
  cockroachnetwork:
    driver: bridge

Fire it up by running docker-compose up, and there you have your clustered database. You can now attach to any of your three nodes and execute SQL statements. The command docker-compose exec cockroach1 ./cockroach sql --insecure will take you to a prompt where you can enter and run standard SQL commands. You can connect to any of the nodes, and your modifications will be replicated to all others.

Nice Specialties

I am relatively new to CockroachDB, too. So, I don't know how exactly they are doing it, but there are some specialties in this product that come in very handy.

No proprietary Drivers

CockroachDB does not come with its own set of drivers. Instead, its makers decided to establish compatibility with PostgreSQL. So, in your projects, you can just pretend to be using a Postgres database while in reality using CockroachDB.

No Master

There is no real master. All nodes are equally important...or replaceable. Try stopping one of the nodes (e.g. via  docker stop cockroach1), make some changes and restart the node. You will find all modifications to be replicated in the resurrected node, too.

I haven't done extensive error scenario testing yet, but I certainly will and will keep you posted.