Alexander Fedulov

4 posts

Terraform: deploying containers on AWS Fargate

Recently AWS introduced a service called Fargate, which alows you to run containers without having to manage servers or clusters. It can currently be used on top of AWS Elastic Container Service (ECS) with support for Kubernetes (EKS) coming later this year. Please notice, that at the moment of writing, Fargate is only available in N.Virginia (us-east) region.

Terraform added support for a new Fargate launch type in their ECS module, but documentation is very scarse and there are a lot of things that need to be configured differently compared to a classic ECS task.

In this post I would like to share a minimal working example of deploying an image hosted on AWS Elastic Container Repository (ECR) to ECS Fargate.

First part of the tutorial includes basic details about how to use terraform and ECR. If you are just looking for specifics of configuration of aws_ecs_task_definition and aws_ecs_service resources, scroll down to the last section.

Setup

This tutorial is accompanied by an example of setting up bare-bone web server running on AWS Fargate and publicly accessible via an Application Load Balancer (ALB) DNS name: https://github.com/afedulov/terraform-fargate.git

git clone https://github.com/afedulov/terraform-fargate.git  
cd terraform-fargate/  

Easiest way to give terraform access to your AWS account is to export AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY into the shell that you are using to run terraform. This is fine for the purposes of this tutorial, but there are approaches with better security available, such as using aws-vault tool. If you use aws-vault, add --no-session parameter to avoid errors with tokens:

aws-vault exec <profile name> --no-session -- terraform plan  

If you do not want to make use of aws_vault, proceed as follows:

set +o history # disable history  
export AWS_ACCESS_KEY_ID=<insert key_id>  
export AWS_SECRET_ACCESS_KEY=<insert secret_access_key>  
set -o history # enable history  
export AWS_DEFAULT_REGION=us-east-1  

First you need to create a backend for Terraform state. Easiest option is to use S3. You can either create it manually via AWS UI, or use aws cli:

aws s3 mb s3://<terraform-fargate>  

Bucket names in s3 have to be globally unique. I will use terraform-fargate, but make sure you change your backend configuration in vars.tf to whatever bucket you've created. Enabling versioning on this bucket might be a good idea.

Next step is to run

terraform-fargate/terraform$ terraform init  

In case something went wrong during the initialization (for instance bucket created in a wrong region), rm -rf .terraform/ and run init again.

I will be using AWS Elastic Container registry (ECR) to store docker images. In order to apply the whole terraform plan, including container deployment, Docker image has to be first made available. Therefore first create a repository by using partial configuration (--target option)

terraform-fargate/terraform$ terraform apply --target aws_ecr_repository.myapp  

Notice repository URL in the output:

myapp-repo  = <generated_ecr_repo_url>  

For simplicity let's just export it to the environment:

export REPO_URL=$(terraform output myapp-repo)  

Docker => ECR

Now that the ECR repository has been created we can build and push Docker image to AWS.

You'll need AWS CLI tools to push images to ECR. Refer to this page for installation details. Execute the following (including $ and brackets) to login docker to ECR:

terraform-fargate/terraform$ $(aws ecr get-login --region us-east-1 --no-include-email)  

Now go to your Docker image folder, build it, label and upload it to ECR (do it in the same shell or manually substitute ${REPO_URL} by <generated_ecr_repo_url> from before).

terraform_fargate/docker/myapp $ docker build -t myapp .  
terraform_fargate/docker/myapp $ docker tag myapp ${REPO_URL}:latest  
terraform_fargate/docker/myapp $ docker push ${REPO_URL}:latest  

You shall now see upload progress.

If you get errors related to missing credentials, make sure that you've used the same region in 'ecr get-login' as the one where your ECR repos was created.

Deploying

terraform-fargate/terraform$ terraform apply  

After all changes are applied, you should see a DNS name of an ALB. Just open it in your Web browser to check a greeting message from your container deployed on AWS Fargate.

Outputs:

alb_dns_name = myapp-*********.us-east-1.elb.amazonaws.com  

You will probably want to play around with the configuration and maybe apply terraform destroy at some point. Before you do that, I recommend to prevent Terraform from deleting your ECR repositories. This will save you a lot of time needed to reinitialize ECR and re-upload your Docker images. To do this, go to ECS console -> Repositories -> -> Permissions -> Add; Tick Principal [✔] , ecr:BatchDeleteImage [✔] and ecr:DeleteRepository [✔].

Following sections will contain some details and peculiarities (as compared to classic ECS) of applied configuration.

Fargate access to ECR

ECR images are being pulled into the cluster via public Internet. There are two options to allow this: either assign your task a public IP directly (only available starting Terraform AWS Provider 1.9.0: https://github.com/terraform-providers/terraform-provider-aws/issues/3098#issuecomment-359552995) or configure networking using NAT and internet gateways (IGW). Here are some details: https://github.com/aws/amazon-ecs-agent/issues/1204. If you are just starting, it might take you a while to figure out all the missing pieces. Important part is the following: you'll have to configure 2 subnets: one private, with traffic to 0.0.0.0/0 being forwarded to a NAT gateway and a public one, with traffic to 0.0.0.0/0 being forwarded to an IGW. You'll then place your Fargate tasks into a private subnet. NAT + IGW is the configuration that I am using in the accompanying source code https://github.com/afedulov/terraform-fargate.git .

Peculiarities of Fargate configuration

Let's now take a look at configuration and emphasize what is specific to Fargate.

resource "aws_ecs_task_definition" "myapp" {  
  family                = "myapp"
  requires_compatibilities = ["FARGATE"]
  network_mode = "awsvpc"
  cpu = 256
  memory = 512
  container_definitions = "${data.template_file.myapp.rendered}"
  execution_role_arn = "${aws_iam_role.ecs_task_assume.arn}"
}

resource "aws_ecs_service" "myapp" {  
  name            = "myapp"
  cluster         = "${aws_ecs_cluster.fargate.id}"
  launch_type     = "FARGATE"
  task_definition = "${aws_ecs_task_definition.myapp.arn}"
  desired_count   = 1

  network_configuration = {
    subnets = ["${module.base_vpc.private_subnets[0]}"]
    security_groups = ["${aws_security_group.ecs.id}"]
  }

  load_balancer {
   target_group_arn = "${aws_alb_target_group.myapp.arn}"
   container_name = "myapp"
   container_port = 3000
  }

  depends_on = [
    "aws_alb_listener.myapp"
  ]
}

resource "aws_alb_target_group" "myapp" {  
  name = "myapp"
  protocol = "HTTP"
  port = "3000"
  vpc_id = "${module.base_vpc.vpc_id}"
  target_type = "ip"

  health_check {
    path = "/"
  }
}
  1. aws_ecs_task_definition:
    • network_mode must be set to "awsvpc"
    • cpu and memory have to be specified in the aws_ecs_task_definition block
    • cpu and memory have to be picked from a list of allowed combinations (see here)
    • requires_compatibilities must contain "FARGATE"
  2. aws_ecs_service:
    • launch_type must be set to "FARGATE"
    • Fargate service won't deploy without explicit network_configuration block
  3. aws_alb_target_group:
    • ALB target_type is by default "instance" and it has to be set explicitly to "ip" (there are no "instances", at least from the user perspective)

I hope this tutorial will help you save time making first steps in using AWS Fargate + ECR managed by Terraform.

Swapping Keys on Linux (Who needs CapsLock?)

I am a big fan of dropdown terminals, like Guake on Linux of ITerm2 on Mac. And I really like having it accessible just by pressing one button. My favorite placement for the Guake hotkey has been the backtick key ( ` - grave assent). The problem is obviously that you loose the backtick, which for many is not a big deal, but it is often used to highlight code snippets in various message boards with markdown and is also used in shell, so it might be inconvenient to have to copy/paste it every time. On the other hand there is arguably the most redundant, useless and frustration-inducing key on the keyboard - the CapsLock. I personally do not need, so it becomes a great candidate for a backtip impersonator.

Goals:

  • have backtick [ ` ] key open a dropdown terminal
  • make CapsLock output the backtick

I have only tested it on Ubuntu, but XKB is used in other distributions too, so give it a try.

Solution:

  • Modify /usr/share/X11/xkb/symbols/pc , section xkb_symbols "pc105":
   //key <CAPS> {  [ Caps_Lock     ]   };
   key <CAPS> {    [ grave, asciitilde ]   };
  • Modify /usr/share/X11/xkb/symbols/us (or whatever default layout you are using), section xkb_symbols "basic":
   //key <TLDE> {    [     grave, asciitilde  ]   };
   key <TLDE> {    [     XF86HomePage, asciitilde  ]   };

The idea is to make CapsLock behave entirely like a backtick/tilde button and to remap backtick to a virtual key that you do not use. I personally never use "HomePage" media key, so this is what my backtick became. You'll then just need to go to Guacke preferences and assign XF86HomePage as it's hotkey.
Those changes are not applied automatically, so you might need to logout/login, rm -rf /var/lib/xkb/* or dpkg-reconfigure xkb-data. That said, there seem to be some caches involved and for me personally rebooting worked most reliably to apply those changes.

Apache Zeppelin Notebooks Export

Apache Zeppelin is a web-based notebook for interactive data analysis. It has functionality similar to Jypyter (former IPython) but what is cool about it is that it allows using Scala instead of Python to utilize Spark interactively. This makes it a very handy tool if you want to quickly test code but do not want to go through the pain of using sbt assembly + ./bin/spark-submit development cycle (btw. check out a very interesting post by my colleague Artur Mkrtchyan about his findings on a hidden Spark REST API as an alternative to spark-submit).

What regards Apache Zeppelin, as of now, unfortunately, it does not support export of notebooks. Chances are you would need this feature to maybe share your notebooks with your colleagues, migrate from one machine to another or even to put your work under source version control.

Luckily under the hood saving and importing new notebooks is quite transparent, you can do this with just a few commands. So, without further ado, let's see how exactly can we do a manual export.

Notebooks are located in folders with random names in the Zeppelin notebook directory.

ls /opt/zeppelin/notebook/  
2A94M5J1Y  2A94M5J1Z  2AZU1YEZE  2B3D826UD  

The whole notebooks' definition is stored in a single file called note.json (including the source code).

ls /opt/zeppelin/notebook/2A94M5J1Y/  
note.json  

In order to export it to another machine, just copy the folder into another Zeppelin installation directory.

There is just one more thing you need to do though. Open the interpreter configuration file:

vim /opt/zeppelin/conf/interpreter.json  

you will see a section at the end called interpreterBindings. Add a new section with the ID of your imported notebook (should be the same as the folder name, otherwise check note.json file) and associate it with existing interpreters IDs.

"interpreterBindings": {
    "2B3D826UD": [
      "2AZN2E1JE",
      ...
      "2B219R99U",
      "2B46SWGGN"
    ],
    "2A94M5J1Y": [   <---- imported notebook
      "2AZN2E1JE",
      ...
      "2B219R99U",
      "2B46SWGGN"
    ]

Now restart zeppelin:

/opt/zeppelin/bin/zeppelin-daemon.sh stop
/opt/zeppelin/bin/zeppelin-daemon.sh start

That's it. You should now be able to access and run your imported notebook!

Dynamic DataSource Routing with Spring @Transactional

It is often desirable to distribute requests between multiple physical instances of SQL databases depending on the semantics of the executed query. In the simplest and most typical scenario, we would want to ensure unhindered write process to the master DB (INSERT/UPDATE) by offloading heavy SELECT queries to the replicas.

In this blog post I will show how to conveniently achieve this by using custom annotations on top of Spring transactional layer.

Spring provides a variation of DataSource, called AbstractRoutingDatasource. It can be used in place of standard DataSource implementations and enables a mechanism to determine which concrete DataSource to use for each operation at runtime. All you need to do is to extend it and to provide an implementation of an abstract determineCurrentLookupKey method. This is the place to implement your custom logic to determine the concrete DataSource. Returned Object serves as a lookup key. It is typically a String or en Enum, used as a qualifier in Spring configuration (details will follow).

package website.fedulov.routing.RoutingDataSource

import org.springframework.jdbc.datasource.lookup.AbstractRoutingDataSource;

public class RoutingDataSource extends AbstractRoutingDataSource {  
    @Override
    protected Object determineCurrentLookupKey() {
        return DbContextHolder.getDbType();
    }
}

You might be wondering what is that DbContextHolder object and how does it know which DataSource identifier to return? Keep in mind that determineCurrentLookupKey method will be called whenever TransactionsManager requests a connection. It is important to remember that each transaction is "associated" with a separate thread. More precisely, TransactionsManager binds Connection to the current thread. Therefore in order to dispatch different transactions to different target DataSources we have to make sure that every thread can reliably identify which DataSource is destined for it to be used. This makes it natural to utilize ThreadLocal variables for binding specific DataSource to a Thread and hence to a Transaction. This is how it is done:

public enum DbType {  
   MASTER,
   REPLICA1,
}

public class DbContextHolder {

   private static final ThreadLocal<DbType> contextHolder = new ThreadLocal<DbType>();

   public static void setDbType(DbType dbType) {
       if(dbType == null){
           throw new NullPointerException();
       }
      contextHolder.set(dbType);
   }

   public static DbType getDbType() {
      return (DbType) contextHolder.get();
   }

   public static void clearDbType() {
      contextHolder.remove();
   }
}

As you see, you can also use an enum as the key and Spring will take care of resolving it correctly based on the name. Associated DataSource configuration and keys might look like this:

   ....
<bean id="dataSource" class="website.fedulov.routing.RoutingDataSource">  
 <property name="targetDataSources">
   <map key-type="com.sabienzia.routing.DbType">
     <entry key="MASTER" value-ref="dataSourceMaster"/>
     <entry key="REPLICA1" value-ref="dataSourceReplica"/>
   </map>
 </property>
 <property name="defaultTargetDataSource" ref="dataSourceMaster"/>
</bean>

<bean id="dataSourceMaster" class="org.apache.commons.dbcp.BasicDataSource">  
  <property name="driverClassName" value="com.mysql.jdbc.Driver"/>
  <property name="url" value="${db.master.url}"/>
  <property name="username" value="${db.username}"/>
  <property name="password" value="${db.password}"/>
</bean>  
<bean id="dataSourceReplica" class="org.apache.commons.dbcp.BasicDataSource">  
  <property name="driverClassName" value="com.mysql.jdbc.Driver"/>
  <property name="url" value="${db.replica.url}"/>
  <property name="username" value="${db.username}"/>
  <property name="password" value="${db.password}"/>
</bean>  

We are almost there. At this point you might find yourself doing something like this:

@Service
public class BookService {

  private final BookRepository bookRepository;
  private final Mapper               mapper;

  @Inject
  public BookService(BookRepository bookRepository, Mapper mapper) {
    this.bookRepository = bookRepository;
    this.mapper = mapper;
  }

  @Transactional(readOnly = true)
  public Page<BookDTO> getBooks(Pageable p) {
    DbContextHolder.setDbType(DbType.REPLICA1);   // <----- set ThreadLocal DataSource lookup key
                                                  // all connection from here will go to REPLICA1
    Page<Book> booksPage = callActionRepo.findAll(p);
    List<BookDTO> pContent = CollectionMapper.map(mapper, callActionsPage.getContent(), BookDTO.class);
    DbContextHolder.clearDbType();               // <----- clear ThreadLocal setting
    return new PageImpl<BookDTO>(pContent, p, callActionsPage.getTotalElements());
  }

  ...//other methods

Now we can control which DataSource will be used and forward requests as we please. Looks good!

...Or does it? First of all, those static method calls to a magical DbContextHolder really stick out. They look like they do not belong the business logic. And they don't. Not only do they not communicate the purpose, but they seem fragile and error-prone (how about forgetting to clean the dbType). And what if an exception is thrown between the setDbType and cleanDbType? We cannot just ignore it. We need to be absolutely sure that we reset the dbType, otherwise Thread returned to the ThreadPool might be in a "broken" state, trying to write to a replica in the next call. So we need this:

  @Transactional(readOnly = true)
  public Page<BookDTO> getBooks(Pageable p) {
    try{
      DbContextHolder.setDbType(DbType.REPLICA1);   // <----- set ThreadLocal DataSource lookup key
                                                    // all connection from here will go to REPLICA1
      Page<Book> booksPage = callActionRepo.findAll(p);
      List<BookDTO> pContent = CollectionMapper.map(mapper, callActionsPage.getContent(), BookDTO.class);
       DbContextHolder.clearDbType();               // <----- clear ThreadLocal setting
    } catch (Exception e){
      throw new RuntimeException(e);
    } finally {
       DbContextHolder.clearDbType();               // <----- make sure ThreadLocal setting is cleared         
    }
    return new PageImpl<BookDTO>(pContent, p, callActionsPage.getTotalElements());
  }

Yikes >_< ! This definitely does not look like something I would like to put into every read only method. Can we do better? Of course! This pattern of "do something at the beginning of a method, then do something at the end" should ring a bell. Aspects to the rescue!

Let's define a neat little annotation with a meaningful name

import java.lang.annotation.ElementType;  
import java.lang.annotation.Retention;  
import java.lang.annotation.RetentionPolicy;  
import java.lang.annotation.Target;

@Target({ElementType.METHOD, ElementType.TYPE})
@Retention(RetentionPolicy.RUNTIME)
public @interface ReadOnlyConnection {

}

And an interceptor for it:

import org.aspectj.lang.ProceedingJoinPoint;  
import org.aspectj.lang.annotation.Around;  
import org.aspectj.lang.annotation.Aspect;  
import org.aspectj.lang.annotation.Pointcut;  
import org.springframework.beans.factory.annotation.Value;  
import org.springframework.core.Ordered;  
import org.springframework.stereotype.Component;

@Aspect
@Component
public class ReadOnlyConnectionInterceptor implements Ordered {

    private int order;

    @Value("20")
    public void setOrder(int order) {
        this.order = order;
    }

    @Override
    public int getOrder() {
       return order;
    }

    @Pointcut(value="execution(public * *(..))")
    public void anyPublicMethod() { }

    @Around("@annotation(readOnlyConnection)")
    public Object proceed(ProceedingJoinPoint pjp, ReadOnlyConnection readOnlyConnection) throws Throwable {
        try {
            DbContextHolder.setDbType(DbType.REPLICA1);
            Object result = pjp.proceed();
            DbContextHolder.clearDbType();
            return result;
        } finally {
            // restore state
            DbContextHolder.clearDbType();
        }
    }
}

This pointcut will make sure that our public methods annotated with @ReadOnlyConnection will get correct DbType.REPLICA1 DataSource set under the hood and cleared on the method exit (even if exception is thrown).

  @ReadOnlyConnection
  @Transactional(readOnly = true)
  public Page<BookDTO> getBooks(Pageable p) {
    Page<Book> booksPage = callActionRepo.findAll(p);
    List<BookDTO> pContent = CollectionMapper.map(mapper, callActionsPage.getContent(),                             BookDTO.class);
    return new PageImpl<BookDTO>(pContent, p, callActionsPage.getTotalElements());
  }

This looks much better. Business logic does not get cluttered with unnecessary "perpendicular" concepts and annotation name directly communicates that this method is designed to use our read-only data source. The last bit to clarify is the magical @Value("20"). It is used to set the order parameter of our interceptor. The thing is, we need to make sure that the DataSource type is set before the @Transactional annotation kicks in. Otherwise connection will already be bound to the thread at the time our @ReadOnlyConnection gets processed. So basically we need set the order below the order of transactions annotation (20 < 100). You can use this configuration:

<tx:annotation-driven order="100"/>