Ernest Neijenhuis on Qstars IT Blog - Cloud, Kubernetes & Software Engineering

Docker about to announce Docker Module Runner

Sun, 30 Mar 2025 12:00:00 +0200

Docker is about to launch Docker Model Runner (DMR). This allows users to run AI large language models on their local system. Although not officially announced yet, multiple preview users have come up with their first impressions, like Nigel Poulton (author of The Kubernetes Book) did on LinkedIn. Meanwhile a DMR landing page has become available on the Docker website.

During KubeCon / CloudNativeCon Europe 2025 (this week in London) Docker will introduce DMR during one of the side events.

Big change

This is definitely a big change for Docker. Being the oldest runtime for Docker and OCI containers, once published in 2013 it exploded in popularity and containerization became the next big thing in compute. It was the only runtime available in the early versions of Kubernetes. However, in Kubernetes version 1.24 native Docker support was dropped from the codebase and the community moved merely to Containerd as their new favourite runtime for Kubernetes.

Meanwhile Docker was moving more into the Developer workstations, by means of Docker Desktop. This product has become very popular to run code locally in containers, as a local test environment for apps being developed, or running containerized tools.

In February Docker got a new CEO, named Don Johnson (no, not that guy from Miami Vice) who came over from Oracle. Now DMR is seeing the light soon, probably in Docker Desktop for Apple Silicon. Yes, Docker is on the move again.

Not a container

DMR will probably being introduced in Docker Desktop 4.40 for Mac. This would be a cool choice, since competitor Ollama showed some issues supporting their Silicon GPU’s. Docker promises to start with support for Silicon and NVidia GPU’s, which could be a game changer, since developers love to use MacBooks.

If you think those models run like any other container in Docker, you’re wrong. Docker introduces a new runtime, allowing models to run natively on the system, so not from a container! So far, a small set of supported models has been mentioned, including Llama, Gemma and Mistral. Although not running as a container, models are still stored as OCI artifacts, allowing you to use your existing container registry to store/cache models locally.

New subcommand and API

To use DMR from the command line, a new subcommand wil be introduced: docker model. This will allow you to pull, run and manage large language models. It is comparable to handling docker images, so it is easy to adapt. Using docker model pull <model> you can download an LLM to your system. Once downloaded, you can start it using docker model run <model>. There is also docker model list and docker model rm to keep things organized.

Docker will provide an API to access models programmatically. The API is compatible with the OpenAI API. This allows developers to easily migrate their code by simply changing the endpoint to localhost. This is another difference compared to Ollama, which implemented its own API.

What about Ollama?

So what does this mean for Ollama? One of the good things of free and open source software is that you always can choose which tool or technology best fits your purpose. So there will be a place for both solutions. For now DMR will still be in development, while Ollama is already very mature. Also Ollama has built an impressive library of LLM’s, while DMR is still working on this. Personally I would stick to Ollama for now, but keep an eye on how DMR evolves. I expect Docker Model Runner will take another 6 to 12 months to become a serious competitor.

More information

Qstars IT will attend KubeCon / CloudNativeCon Europe 2025, in April we will publish the highlights on this blog. Meanwhile you can always contact us to help you on your journey into containerization, to prepare your code for containerization, build CI pipelines for automation, to design/build your hosting platform or develop your next cloud-ready Python application.

Herding Cattle in your Data Center?

Wed, 04 Dec 2024 10:00:00 +0100

In most of our on-premise data centers, compute is traditionally realized by running a few large virtualization clusters (e.g. VMware, Nutanix), backed by one or more shared (file+block) storage solutions and a solid network. Just add power, climate control and security, and your data center is basically ready to go.

However, compute is on the move. Where virtual servers were the only/major type of workload, nowadays we also need to support hosting for Docker containers, serverless/functions, event driven solutions, webAssembly, etc… Meanwhile we learned that running containerized applications is better on smaller clusters than on one single cluster: the Pets versus Cattle paradigm.

Pets versus Cattle

For those who are not familiar with this concept, here is a short explanation. Once we ran servers for multiple years. We updated them periodically and applied changes when new applications/functionally was added to the server. We gave them cute names, noticed some ECC memory warning in the syslog of a server and remembered that we already saw the same error on this server about two months ago. After 5+ years the server got replaced and with tears in our eyes we removed the obsolete server from the rack and moved it to the museum rack. We handled those servers like pets, with all our love and care.

A few years ago we started containerizing, and started orchestrating these workloads in clusters like Docker Swarm or Kubernetes. Containers were recreated faster and faster, from months to days to minutes. Although containers still have host names and IP addresses, we don’t care about it. Containers are handled like cattle. Hosting became more application-centric, sysadmins moved from managing servers to managing clusters and related tools. The clusters became the new pets.

Clusters as cattle

While the number of containers increased rapidly, clusters grew exponentially and became more difficult to maintain. Thanks to DevOps we learned that it is easier to have multiple, smaller clusters: each team or application (group) gets its own cluster, so cluster maintenance could be delegated to the owning teams and aligned with the application release cycles. Also, when a cluster fails, only a limited set of functionality is lost, while other clusters/apps remain available. Cluster deployments become automated and available on-demand via API-call, web form and/or Infrastructure-as-Code. Our clusters now become cattle.

Compute as cattle

Not only container workloads are disposable, so become virtual servers. Containerization is still not a solution for all kinds of applications. For example, large database clusters and ERP systems still prefer to scale vertically and are not suitable to containerize. However it would be great to manage these servers like containers, to limit our tool set and create a universal way of managing workloads. This means that compute in our data center needs to change: virtual machines (and basically any type of workload) should be available on-demand, as cattle!

“Whatever your compute demand is, it should be available as cattle”

Kubernetes as the new compute layer

Nowadays Kubernetes is so much more than just a container hosting platform and orchestrator. Lots of effort has been put into hosting additional compute types, for example:

The KubeVirt project allows us to create and manage virtual machines as they where containers. KubeVirt requires KVM enabled nodes, since it uses this hypervisor.
SpinKube provides hosting WebAssembly containers, enabling workloads to start within milliseconds.
KEDA enables event driven solutions to Kubernetes (comparable to AWS Lambda, Azure Functions, etc). Whenever a call comes in, the required workload is started to handle it. Combined with WebAssembly function apps this can be very powerful.

Using Kubernetes as the new compute layer for your data center is quite different from the more traditional way (large virtualization clusters). It requires both a different skill set and mind set to build and manage. Infrastructure-as-Code should be in your genes to automate everything. You also need a portal to expose the available compute options and allow teams to deploy and maintain their own clusters.

Example solutions

Here are some example setups to create your new compute-as-cattle. These are commercially supported solutions, you can still build your own compute using the related open source solutions, although I don’t think this is suitable for most enterprise organizations.

All examples follow a basic pattern: there is a Management Cluster that runs software to deploy Workload Clusters. The Management Cluster also runs the Portal and API. One of the workload clusters will probably be a Services Cluster, running shared services you offer to the application teams; think of your Git repositories, container registry, helm repository, CI tools, etc…

SUSE

SUSE provides a solution based on Harvester and Rancher. Harvester enables bare metal provisioning of your physical servers and can initialize Kubernetes clusters (RKE2). Once the basic cluster is up, Rancher Prime imports the cluster and will manage it. It will also enable server virtualization (based on KubeVirt). Rancher provides a nice web interface to manage your clusters. It centralizes authentication/authorization and provides a web store for your own helm charts. Rancher also allows you to deploy clusters on popular clouds for a hybrid approach.

Red Hat

The Red Hat solution uses Advanced Cluster Management for Kubernetes (ACM), combined with OpenShift (Red Hat’s Kubernetes based hosting platform). ACM allows you to provision OpenShift clusters on bare metal. OpenShift is Red Hat’s implementation of Kubernetes and includes OpenShift Virtualization to deploy and manage virtual machines. ACM can also deploy clusters on popular clouds.

Summary

Kubernetes is a proven foundation for all popular types of compute and can be provisioned as cattle to populate your on-premise data center, optionally combined with cloud for a hybrid solution. Clusters are available on-demand thanks to a portal, API and Infrastructure-as-Code. The platform is maintained by a platform team, that applies Infrastructure-as-Code and CI-pipelines to deliver the platform. The team can work in sprints, delivering a new potential version of the platform in every sprint. There is no more single/large production cluster, instead workloads are spread over numerous clusters. Cluster maintenance ahs less impact and is repeatable, more predictable and eventually automated. Outages will be smaller and will have less impact.

“Running Cattle rather than Pets is a computing infrastructure best practice no matter what layer of the stack” (source)

Crossplane: Manage your cloud like you manage Kubernetes

Tue, 18 Jun 2024 12:00:00 +0200

Kubernetes has proven to be a very solid orchestrator for itself. So… why would you not orchestrate your complete infrastructure with Kubernetes? Crossplane provides a great way to do this, so you can manage your storage, databases, even other Kubernetes clusters and all other resources available outside your cluster (but within your (hybrid) cloud). Whenever you’re in a Kubernetes minded environment or not, Crossplane is a great alternative to tools like Terraform and worth evaluating on your IaC product selection.

What is Crossplane?

Crossplane provides self-service to your developers without needing to become an infrastructure expert. For example, if you need an S3 bucket to store data, you simply create a yaml object within Kubernetes to do so, like you create a deployment or service. Here is an example manifest for it:

apiVersion: s3.aws.upbound.io/v1beta1
kind: Bucket
metadata:
  name: your-web-bucket-a3df9
spec:
  forProvider:
    region: eu-west-3
  providerConfigRef:
    name: default

Once deployed in your Kubernetes cluster, Crossplane will pick it up and create your S3 bucket in AWS. Isn’t that neat? Within a few seconds the bucket is available:

$ kubectl get buckets
NAME                    READY   SYNCED   EXTERNAL-NAME           AGE
your-web-bucket-a3df9   True    True     your-web-bucket-a3df9   95s

$ aws s3 ls
2024-05-31 07:48:26 your-web-bucket-a3df9

This is just a kind of hello world example. For more complex solutions you can create a composition, combining multiple resources. A developer can now request an instance of such a composition, like it would be a single object. You can introduce variables (patches) to these compositions or add logic by adding composition functions.

You can create packages of compositions, composition resource definitions, providers, functions, etc. This enables easy deployment and versioning. To help you with this, a CLI was recently introduced. This CLI allows you to create and update packages. You can also use the CLI for testing and troubleshooting.

At the time of writing, Crossplane is six years old now. It is a CNCF incubating project and recently applied for graduation, which means there should be enough proof of maturity and should be ready to be used in production environments.

GitOps workflow

So Crossplane provides a Kubernetes way for doing Infrastructure-as-Code. But things get even more powerful when combining Crossplane with a GitOps agent, like Argo CD or Flux CD. Now you just create objects in your Git repository, get them reviewed and merge. Next the GitOps agent will pickup your changes and create/update/delete your resources, maybe along with your application deployment.

Installation

Crossplane installs as a controller in your Kubernetes cluster, accompanied with a bunch of CRD’s (custom resource definitions). After installing the main controller using their helm chart, you need to add providers for your cloud platform. You also create a secret allowing Crossplane to access your cloud. Installing Crossplane is very easy and the documentation comes with some quick start guides for AWS, Azure and GCP.

Besides the popular public cloud providers, there are also providers available for VMware vCloud, Terraform/Ansible, GitHub/GitLab/Bitbucket, Keycloak, Argo CD, OpenStack, Ceph and many others. Check the Crossplane Marketplace for a full list.

Crossplane at large

If you have a large environment, the best way to manage it is by creating a dedicated Kubernetes management cluster which only runs Crossplane (and probably Argo CD or Flux CD). Next tell Crossplane to deploy Kubernetes workload clusters (e.g. EKS, AKS, GCE), these clusters will host your applications.

If you still want to provide self-service to your developers, there are multiple ways to do so, for example:

Install Crossplane in the workload clusters and apply RBAC roles to selectively permit developers to create resources outside the cluster.
Use RBAC on the management cluster to selectively allow developers to deploy their resources.
Introduce an IdP (like Backstage) to allow developers to deploy resources in their own clusters/namespaces.

If you want to have more control on what your developers are allowed to do, Crossplane provides management policies. Or you can add a policy agent to your cluster, like Kyverno.

AWS Summit Amsterdam 2024 highlights and takeaways

Thu, 11 Apr 2024 10:09:06 +0200

This week the AWS Summit was in Amsterdam, and off course Qstars was there too! We attended lots of talks, strolled along the booths and had lots of nice conversations. The main topics where AI (like any IT conference nowadays) and serverless.

Artificial Intelligence

There are lots of AI resources available in AWS, as demonstrated in lots of talks. We think these are worth mentioning to start your AI adventures:

Amazon Q was introduced a few months ago and heavily promoted during the conference. This is a generative AI-powered assistent which can help you both on coding your infrastructure, as well as providing you support on all those AWS services. You can integrate it with editors like VScode (using the AWS Toolkit for Visual Studio Code).
Bedrock allows you to run lots of AI models, including Mistral AI, Cohere and Llama 2. This allows you to use such models within your applications. This allows you to apply AI functionallity in your own applications. E.g. you can use this to make summaries of your documents, generate images or create a virtual assistant.
SageMaker provides you a great way to build, train and deploy machine learning models. This is a great way to get more insights from your data. Your data analysts will love it. To get you started there is a nice tool called SageMaker Studio.

Serverless

AWS provides serverless compute by providing Lambda. Serverless means you don’t have to bother about the underlying infrastructure. You can just focus on running your application, AWS will do all the other stuff for you. It scales automatically and you pay per request. Many languages are supported, including Java, Python and Node.js.

Lambda really shines in distributed applications. This means that you build a workflow based on existing cloud resources, where Lambda is the resource to insert your custom code. To demonstrate this here is a simple example workflow: a user can upload documents to an S3 bucket. Whenever a document is uploaded, a lambda function is triggered, which extracts the document title and put it in a DynamoDB table. Meanwhile also Bedrock is called to add a summary. To allow scaling up, you may decide to add queues to improve/shape the flow.

There are multiple tools available to build such workflows, we think these are the most interesting ones:

Workflow Studio is a low-code IDE to build a distributed application with AWS Step Functions.
Application Composer allows you to generate infrastructure-as-code using the Serverless Application Model.

Takeaways

Master at least one cloud provider

Cloud has become mature, every company should at least explore the opportunities to benefit from the cloud. Whether you’re into infrastructure, software development of data analytics, you should know how to use at least one cloud. Initially it doesn’t matter what cloud provider you choose; once you know the possibilities of a single cloud provider, you can better select the most appropriate cloud provider for your needs. Also learn how to deploy your cloud resources using an Infrastructure-as-Code, for example Terraform.

Become familiar with AI

Artificial Intelligence (AI) is evolving and already provides lots of intesting features for you. Whether you include AI functions within the apps you develop, or use generative AI to assist coding your infrastructure, embrace the possibilities. Coding has become so much easier. And if you don’t trust publicly available services, run an LLM on your own laptop (e.g. with Ollama) or in your own datacenter or private cloud.

Forget server virtualization, embrace the real cloud

AWS provides EC2 (Elastic Compute) so you can run your applications on virtual machines. But this is basically just outsourcing your server hosting. Explore container hosting (like EKS) and serverless for your ever demanding compute needs.

KubeCon 2024 takeaways

Fri, 05 Apr 2024 10:09:06 +0200

Another KubeCon / CloudNativeCon Europe 2024, this time in Paris. And of course Qstars attended this great conference! We joined with five people to experience the latest developments on the cloud native landscape.

AI, AI, AI

In the first two minutes it became clear that Artificial Intelligence (AI) is the absolute main topic for this conference. AI is everywhere nowadays, and we (as platform engineers) have to deal with that. How can we prepare our infrastructure to handle these AI workloads? What are the best practices in terms of security, performance and scalability?

No surprise GPU producent NVIDIA is one of the main sponsors of this conference. They provide the hardware and drives that are needed for running AI models. GPU’s are getting more optimized for AI workloads. One of the changes is that you can now share a GPU to run multiple workloads in parallel, making it possible to share it over multiple pods of your Kubernetes cluster. Lots of effort has been put into resource management, including NVIDIA DRA (Dynamic Reource Allocation) and DRA functions within Kubernetes.

Running AI models has become easier thanks to some intesting tools. One of them is Ollama, a simple docker-look-a-like tool to download and run models on a local machine (with a GPU). Be sure to give it a try on your own laptop, combining it with the Continue plugin on Visual Studio Code you get a local Copilot experience without sending code out to a remote party; all the magic happens on your own laptop!

Forget the operating system, Kubernetes is your new base

Kubernetes is becoming the new base layer for your compute. Yes, there will be some linux down there, but that’s merely firmware nowadays. Kubernetes not only runs containers, but also WASM stacks, virtual machines and even serverless, so all your favourite cloud compute options are now available on one single platform, even on-prem. Let’s take a quick look on a few of these.

KubeVirt is already out there for a while and has even been adopted into Red Hat OpenShift. With KubeVirt you can run virtual machines on your Kubernetes cluster, for those workloads that you can’t or don’t want to containerize yet. It uses libvirt to run the virtual machines on your cluster nodes.

Web Assembly or (WASM) is a rather new technology that allows you to create very small stacks of code. It was originally created to run code in web browsers (client side), however it appears to be very powerfull for server side applications as well. One of the great features of a Web Assembly stack is the boot time, typically less then a millisecond. This is great for hosting serverless applications. One of the emerging tools for this is SpinKube.

KEDA (Kubernetes Event-Driven Autoscaling) is another way to run your apps in a serverless way. Based on triggers, KEDA can scale the number of pods up or down based on the load. It also comes with a whole bunch of built-in scalers for databases, messaging, pipelines, etc. This makes your landscape very flexible and cost effective.

Karpenter is a great tool to automatically scale your whole Kubernetes cluster. Depending on the requested load it scales your workers nodes up and down, again providing a flexible and cost efficient way of running your workloads.

Observability

So, observability is everything related to collecting, filtering, storing and visualising metrics, logs and traces. The more you automate, the more you need to keep an eye on, so observability is hotter than ever before.

To retreive all metrics from the bottom of your applications, eBPF is the way to go. The kernel provides a set of hooks for tracing and instrumentation. You can dynamically insert some code into the kernel to call these hooks and report back the results to a pod that runs a collector. This collector forwards all metrics to your favourite monitoring stack. A nice tool to explore the possibilities of eBPF is Bumblebee, as demonstrated at one of the workshops during the conference.

Maybe the best way to collect, filter and forward all monitoring data is OpenTelemetry. There are different ways to collect, including running a collector side-car with your app or use the eBPF-collector to collect directly from the kernel.

Prometheus is still a proven tool to store all your metrics in time. Meanwhile much effort has been put into the integration between Prometheus and OpenTelemetry. Combined with tools like Grafana you can build state-of-the-art dashboards for your applications and infrastructure.

Summary

The cloud native community is 10 years old now and still growing rapidly. It has become a great ecosystem which can truly help organisations to run their applications faster, more secure and cost effective. Whenever you need to select a new tool for a solution, always explore the CNCF landscape first!

Qstars to attend KubeCon 2024

Wed, 27 Mar 2024 13:28:26 +0100

Another KubeCon/CloudNativeCon is upcoming in Europe, this time Paris is the city hosting this big event. Alphons, Ernest, Luuk, RobV and Tom will travel to the city of light to bring back the latest trends and developments. And hopefully we will meet you all there as well! We’re eager to go. And once back we will report our findings on this blog again, and maybe even focus on some subjects in detail.

Looking forward to the conference, there are some important themes to mention:

Platform engineering
Observability and eBPF
AI

Platform engineering

Last year was platform engineering year, this subject hit every IT conference and we saw customers discussing, using and evaluating this. Still there are lots of developments going on, so it’s good to check out some related talks. Kubernetes seems to be one of the best platforms to adopt this best practice, however this merely depends on the IDP (Internal Developer Platform) you’re using. Is Backstage really to best IDP available? If so, are there any improvements on deploying and managing this? Where is the community heading to? And what alternatives are there? Lots to consider, so be sure to attend BackstageCon of de Platform Engineering Day on Tuesday, or join a few talks of the Platform Engineering track during the rest of the conference.

Observability and eBPF

eBPF provides a neat way to expose all kinds of data and metrics from your Kubernetes cluster, on a very detailed level, up to processes within your pods. In the world of observability, eBPF is currently one of the hottest themes. Last year we saw lots of possibilities and experimental tools on this. Meanwhile some tools became more mature and are ready to apply on your production workloads, taking away most complexity of eBPF. So what are these tools? Which do best fit to your clusters? And what value do they add to your ecosystem? Join Observability day on Tuesday or one of the observability talks during the conference.

AI

So… Artificial intelligence remains popular and increases your workloads. More applications are using AI models to perform all kinds of calculations and data analysis. So your clusters need GPU nodes to run these workloads. However, they might have special requirements and you may need to think about centralizing models. How do you handle this? What is the best way to manage this? What tools can help you? And what to expect in the upcoming years on these vast growing workloads? Be sure to attend some ML/AI + Data Processing talks.