EC1 Capital

ClusterHQ Announce Flocker

Migration of data within Docker containers
Julian Julian Thu 14th Aug 2014 19:35
https://www.flickr.com/photos/sergemelki/

Introducing Flocker 0.1, a lightweight volume & container manager for Docker

Today I’m very excited to announce the first release of our data-focused Docker management framework. We’ve worked hard all summer to bring our unique ZFS replication technology to Linux and Docker containers and we’re pleased to now get an opportunity to demonstrate what our technology can do.

So without further ado I’d like to announce Flocker 0.1, a lightweight data volume and container manager for Docker.

Making Docker data volumes portable

At ClusterHQ we know all too well about the problems with managing data in a distributed system, so we sought to make the first Docker management framework which has data at its heart. Flocker includes:

  • Straightforward Docker container management – deploy what you want, where you want
  • Easy data volume migration – if your node is about to fail, we’ve got you covered
  • Global routing – service a request for any container on any node

flocker-architecture-diagram

Using Flocker you’ll be able to deploy stateful services like databases, key-value stores and queues using Docker – with all the benefits that Docker deployment brings – without worrying about having your data locked up on local disks or a SAN.

How does Flocker work?

Our first Flocker release allows you to define your application configuration anddeployment configuration separately using small YAML files. Your applicationconfiguration defines which Docker images make up your application. The deploymentconfiguration describes where your containers should be deployed on your cluster, whether that is in a public or private cloud, on bare metal servers or a mix of all three. You can also specify which volumes your containers are using, and if you decide to move those containers, their volumes will be copied over as well.

Here is a short example (for more you can follow a full tutorial for installing and using Flocker):

This example describes a very simple single-instance MongoDB installation, and how to move it between two nodes. We’re using our own MongoDB images but they do not contain any special functionality – you can use whatever you wish.

Flocker requires two files to deploy an application. The first is the application configuration file, called application.yml in this example:

application.yml:

1
2
3
4
5
6
7
8
9
10
11
"version": 1
"applications":
  "mongodb-volume-example":
    "image": "clusterhq/mongodb"
    "ports":
    - "internal": 27017
      "external": 27017
    "volume":
      # The location within the container where the data volume will be
      # mounted:
      "mountpoint": "/data/db"

The second is the deployment configuration file, called deployment.yml in this example:

deployment.yml:

1
2
3
4
"version": 1
"nodes":
  "172.16.255.250": ["mongodb-example"]
  "172.16.255.251": []

We can easily deploy this application to our cluster using the flocker-deploy command available in the Flocker command-line tool (see installing flocker for details on installing the CLI):

1
2
3
4
$ flocker-deploy deployment.yml application.yml
ssh root@172.16.255.250 docker ps
CONTAINER ID    IMAGE                       COMMAND    CREATED         STATUS         PORTS                  NAMES
4d117c7e653e    clusterhq/mongodb:latest   mongod     2 seconds ago   Up 1 seconds   27017/tcp, 28017/tcp   mongodb-example

We can also insert some data into it:

1
2
3
4
5
6
7
8
$ mongo 172.16.255.250
MongoDB shell version: 2.4.9
connecting to: 172.16.255.250/test
> use example;
switched to db example
> db.records.insert({"the data": "it moves"})
> db.records.find({})
{ "_id" : ObjectId("53d80b08a3ad4df94a2a72d6"), "the data" : "it moves" }

Next we change our deployment.yml so that our container is located on a different node:

1
2
3
4
"version": 1
"nodes":
  "172.16.255.250": []
  "172.16.255.251": ["mongodb-example"]

Run the flocker-deploy command again and our container (and volume) has moved to the new node:

1
2
3
4
5
$ flocker-deploy deployment.yml application.yml
ssh root@172.16.255.250 docker ps
ssh root@172.16.255.251 docker ps
CONTAINER ID    IMAGE                       COMMAND    CREATED         STATUS         PORTS                  NAMES
4d117c7e653e    clusterhq/mongodb:latest   mongod     3 seconds ago   Up 2 seconds   27017/tcp, 28017/tcp   mongodb-example

Not only has the container moved, but the data is still available:

1
2
3
4
5
6
7
$ mongo 172.16.255.251
MongoDB shell version: 2.4.9
connecting to: 172.16.255.251/test
> use example;
switched to db example
> db.records.find({})
{ "_id" : ObjectId("53d80b08a3ad4df94a2a72d6"), "the data" : "it moves" }

This is a very simple example of using Flocker. You can also use Flocker to configure more complex multi-node applications and configure network ports so your containers can communicate. Flocker will make sure an application is reachable on all nodes in the cluster, no matter where it is currently located.

With this easy-to-use functionality, devops engineers can finally deploy Docker containers with data volumes – without worrying about how to relocate those volumes later.

This is just the beginning for Flocker. Using our technology we can also do a lot more, including continuous replication, branching and versioning of data volumes. We’ll be announcing exciting new features and functionality over the coming months as we work to make our data-driven Docker vision a reality.

Get Involved

Flocker is 100% free and open-source, available on GitHub at clusterhq/flocker. If you want to give it a try, head over to docs.clusterhq.com and follow along with a tutorial for migrating MongoDB or PostgreSQL. We have Vagrant images you can download and experiment with on your laptop.

We also want your feedback to make Flocker awesome. Let us know what you think by joining the Flocker Google Group or come hang out with us at #clusterhq on Freenode IRC. If you want to contribute directly to Flocker, we’d love your help.

Enjoy!

 

Great! You are now subscribed to blog.


You might also like


comments powered by Disqus