See all posts

The EC2 Spot Instance Life Cycle

Managing Spot Instances

Genestack Platform runs on various commodity compute clusters, from large-scale cloud providers like Amazon, to custom-built systems like IBM’s GPFS + LSF combination.

When we use Amazon’s EC2 for internal development, we use numerous “worker instances” to perform bioinformatic computations. The load, actual number of jobs needed to be run, is highly erratic. Keeping fixed number of workers is not a good solution: it might be too small in the afternoon when active testing is going on, and too big and wasteful at night. We built an internal system to manage worker instances, allocating and deallocating them based on current and expected load.

A cornerstone for successful management of EC2 instances is a good understanding of their life cycle. There are some resources on the topic, but they do not concern Spot Instances, are partly outdated and do not have as much technical detail as we would like. Spot instances are also a bit different from regular instances, so a bit of investigation was in order. So, if you need to monitor what’s going on in the cloud, read on.

From Request to Instance

Spot Instances are a way to buy Amazon’s spare capacity at a lower price. Check out AWS documentationto learn more about how they differ from regular ones. We will concentrate only on certain aspects here, assuming that reader already knows basic stuff.

So, how do you get a Spot Instance? You file a Spot (Instance) Request, and wait for its fulfilment. If you watch the AWS web interface closely, you will notice that at first your new Spot Request is in ‘open’ state, then it becomes ‘active’. It then begets an associated Instance. For a bit of time that Instance is ‘pending’, then becomes ‘running’, but still with status ‘initializing’, and only then it’s fully ready (‘running’/’ok’).

Now a bit about termination: if you cancel a Spot Request before it’s fulfilled and becomes ‘active’, its state changes to ‘cancelled’. If you terminate an instance associated with a Spot Request, its state becomes‘closed’ right away.

An important point is that Amazon starts charging you money when your instance starts ‘running’. While your instance is ‘initializing’  that’s when the OS boots and various startup procedures take place, including any services that you may need specifically. It is likely that your instance may be available to do work while still in the ‘initializing’ state.

Actual and Full Life Cycle

Here’s a cool and colourful flowchart illustrating Spot Request and Instance state sequence. ec2_cycle


Bold text in boxes denotes request or instance ‘state’, italics denotes spot request ‘status code’ or instance‘system status’.

And here are a few lines in Python (using Boto library, naturally) showing how to list everything already launched, but not ready yet.

# spot requests awaiting fulfilment
spot_reqs = ec2_connection.get_all_spot_instance_requests(
    filters={'state': 'open'}

# instances in 'pending' state
pending_instances = ec2_connection.get_only_instances(
    filters={'instance-state-name': 'pending'}

# get running instances with 'initializing' status
initting = ec2_connection.get_all_instance_status(
    filters={'system-status.status': 'initializing'}


See all posts

Get started now:

Sign Up for free




Thank you for subscribing

Your subscription has been confirmed.You've been added to our list and will get a message with our news soon!

Let your friends know about Genestack.


Thank you for signing up.

Check your inbox for a confirmation link and some tips on getting started.

Let your friends know about Genestack.


Oops, something went wrong!

Please try again, or register via the sign up page.