Briefly on the purpose of Functors, Applicatives and Monads

In a recent thread on /r/haskell about how to motivate the AMP proposal in teaching, I read a comment that finally helped me understand the purpose of Functors, Applicatives and Monads.

Wait, what is AMP?

For the reader that hasn’t followed this debacle, the AMP proposal is basically about making Applicative a superclass of Monad. That this hasn’t been so is often considered one of the historical errors Haskell carries around, since a Monad is always an Applicative.

The hierachy of Functor, Applicative and Monad is thus changed to,

Typeclass hierachy
class Functor f where
    -- ...

class Functor f => Applicative f where
    -- ...

class Applicative m => Monad m where
    -- ...

which makes more sense than the arbitrary split before (i.e. Monad was just “on its own”).

The concerns surrounding this change has mostly been about beginner-friendliness, since a Monad now needs an instance of both Applicative and Functor.

There are various other considerations, which the interested reader can find out more about on the GHC 7.10 migration page or the Haskell wiki entry about AMP.

Back to Functor, Applicative and Monad

I’m mostly going to paraphrase /u/ForTheFunctionGod’s comment (which you can find here), while trying to expand a bit on it with my own understanding.

One can naturally extend the intuition of a Functor to an Applicative and from there to a Monad.

An explanation of the first two might go as follows:

Let’s say I have a list of numbers and want to add 2 to each of these numbers. You can write fmap (+2) [1..10] (using Functor).

But what if I had two lists of numbers and wanted to add each number from the first list to each number from the second list? With Applicative you can do just that - it’s like fmap only it can take multiple arguments. You can write (+) <$> [1,5,10] <*> [11..13].

From here, one can motivate Monad by asking:

What if I wanted to abort midway through - say, if I encountered a problem? You can then write:
add xs ys = do
    x <- xs
    if x < 0 then []
    else do
        y <- ys
        if y < 0 then []
        else return (x+y)

Thus we have the natural hierachy:

  1. Functor: apply a function to a container.
  2. Applicative: apply a multi-argument function to multiple containers.
  3. Monad: like Applicative but I can decide what to do next after each step.

Examples of Functor, Applicative and Monad

While the above may help a bit in the “why?”, there is still the question of how to use them. I won’t go into much detail since there exists a wealth of information on this topic already (you can quickly google them), but just give some brief examples of what happens when using them.

Functor is the simplest, and can be thought of as a much more general map. Wherever you use map you can always replace it with fmap, but not the other way around.

> fmap (*2) [1..10]
[2,4,6,8,10,12,14,16,18,20]

fmap simply takes every element of the list and applies the function to it.

Applicative on the other hand is a bit more tricky to understand. The best starting point is probably to show where fmap is not enough.

Imagine we want to apply the function * (multiplication) to a list, we could try fmap (*) [1..3] but that would give us a bunch of partially applied functions back, like [(*1), (*2), (*3)].

Now, what can we do with this? We can for example map a value onto the list of partially applied functions, which would look something like this using fmap,

> let a = fmap (*) [1..3]
> fmap (\f -> f 9) a
[9,18,27]

which can be seen as applying a function that takes a function as argument an applies 9 to that function, to every function in the list a. The real problem comes when we want to apply a Functor function to Functor values, but I won’t get into much detail on that (you can read more here).

Luckily, instead of writing the above, we can use <*> which is a part of Applicative. We can instead write,

> fmap (*) [1..3] <*> [9]
[9,18,27]

Note that the reason we put 9 inside a context (here a list), is because Applicative expects everything to be a Functor. Since the first part is so common, Control.Applicative actually exports <$>, so we can do the following instead, replacing fmap,

> (*) <$> [1..3] <*> [9]
[9,18,27]

Admittedly, it’s a bit more interesting when we want to apply the multiple functions to multiple arguments, as such,

> (*) <$> [1,5,10] <*> [11..13]
[11,12,13,55,60,65,110,120,130]
> (*) <$> Just 3 <*> Just 5

or perhaps inside contexts, such as Maybe,

> (*) <$> Just 3 <*> Just 5
Just 15

but I won’t go into more detail about that, since that isn’t the purpose of this post.

Monad is the last, but perhaps most tricky. I’ll try to be as brief as possible though. Too see the “what happens next, based on what happened before” part, we will look into the Maybe monad used with do notation. It will not explain Monad usage in general, but should be enough to get the gist that “something” happens between the steps.

For example (note that I use ; instead of linebreaks because we are executing in the GHCi REPL),

> do Just 6; Nothing; Just 9
Nothing

You may be aware that a do block returns the last line executed in it - so why did it here return Nothing when the last line was Just 9? That is because on each step the bind function >>= (or =<< for the other direction) is applied. The Maybe Monad defines that if it encounters a Nothing then every subsequent actions also return a Nothing.

To understand this better, let’s take a look at the Maybe instance,

Maybe Monad typeclass instance
instance Monad Maybe where  
    return x = Just x  
    Nothing >>= f = Nothing  
    Just x >>= f  = f x  
    fail _ = Nothing  

Notably here is the line that says Nothing >>= f = Nothing. It throws away whatever future action it gets and just returns Nothing. This is in contrast to Just x >>= f = f x which can be seen as unpacking x from Just x and then applying the future action, f, to that x.

Wrap up

Hopefully this should have helped understand the difference of Functor, Applicative and Monad a bit, and how they work together. To understand these concepts more in depth, I recommend reading a bit up on them, and especially for Monad, try reading about the Writer and State Monads, how they are used and how they are implemented. One resource for that is the chapter on a few more monads in LYAH.

If you are confused about “What exactly is a Functor, is it a class or interface or whatever?”, then I gave my take on it here. Other than that, I encourage that you read a bit on Type Class’.

Finally, I can deeply recommend the book Haskell Programming - from first principle. It is still in development as of this date, but already features 700 pages of quality content. Simply the best Haskell book I have ever read (or, I’m still reading it as of writing).

Setting up UnrealIRCd and Anope IRC Services on EC2

Versions used:

  • UnrealIRCd 4.0.0-rc3
  • Anope 2.0.2

Having recently discovered sameroom.io I wanted to update the codetalk IRC server to be compliant with their authentication method. This basically just meant enabling SASL support, but while I was tinkering with stuff anyways, I thought I might as well streamline the setup process for the IRC server. In short, everything is fully automated and set up on AWS using EC2 and S3.

This will go through the process of doing exactly that, by talking about:

If you just want to throw yourself in the deep end, you can fork/clone the github repository, alter the configuration and script variables to fit your need and quickly be on with it. I recommend though that you skim through most of this to get an overview of what is happening.

SASL support

SASL (Simple Authentication and Security Layer) is a framework for authentication and data security. For all intents and purposes we won’t bother with how it works specifically, since we are only interested in getting it running.

UnrealIRCd supports authenticating with SASL using a remote service that supports SASL. Here enters Anope which is commonly used for handling nickname registration and other IRC services. Since 1.9.x Anope has supported SASL. To enable it, it needs to be compiled with the sasl modules (which it should be by default).

The configuration for SASL is two part, a simple block for UnrealIRCd and a block for Anope.

UnrealIRCd: The following links the Anope services to the IRC server and sets the SASL server to the services server.

services.conf
link services.myircserver.org {
    incoming {
        mask *;
    };
    outgoing {
        bind-ip  *;
        hostname 127.0.0.1;
        port     6697;
        options { ssl; autoconnect; };
    };
    password "MySecretPassword!";
    class    servers;
};

ulines {
	services.myircserver.org;
};
/* Enable SASL support */
set {
    sasl-server "services.myircserver.org";
};

Anope: Enabling SASL is fairly simple, and just requires that the module is loaded. The rest is handled by Anope.

modules.conf
/* Load the SASL module m_sasl */
module { name = "m_sasl" }

If you are using the sample config in the github repo, then SASL is already included.

Installing UnrealIRCd

Since we want to automate the setup and installation, we need to install UnrealIRCd in a way that requires absolutely no user input. First off though, UnrealIRCd depends on some packages, and these will need to be installed.`

Since we are building UnrealIRCd from source, we will need some build tools. These can be found in the Development Tools group. Other than that, there are some curl, ssl and other libraries that needs to be installed.

install-unrealircd.sh
yum -yq groupinstall "Development Tools" \
&& yum -yq install \
        curl \
        libcurl-devel \
        openssl-devel \
        openssl zlib \
        zlib-devel \
        ntp \
        c-ares \

Now we are able to compile UnrealIRCd from source. We will do all of this in one giant step:

install-unrealircd.sh
# UnrealIRCd version
UNREAL_VERSION="unrealircd-4.0.0-rc3"

curl https://www.unrealircd.org/downloads/$UNREAL_VERSION.tar.gz | tar xz \
&& cd $UNREAL_VERSION \
&& ./configure \
    --enable-ssl \
    --with-showlistmodes \
    --with-shunnotices \
    --with-confdir=/etc/unrealircd/config \
    --with-cachedir=/etc/unrealircd/cache \
    --with-scriptdir=/etc/unrealircd/unreal \
    --with-tmpdir=/etc/unrealircd/tmp \
    --with-modulesdir=/etc/unrealircd/modules \
    --with-logdir=/etc/unrealircd/log \
    --with-docdir=/etc/unrealircd/doc \
    --with-datadir=/etc/unrealircd/data \
    --with-pidfile=/etc/unrealircd/pid \
    --with-bindir=/usr/bin/unrealircd \
    --with-permissions=0600 \
    --enable-dynamic-linking \
&& make \
&& make install

This will install UnrealIRCd into /etc/unrealircd with SSL enabled. Since we have SSL enabled, we will also need an SSL certificate! Luckily, this can also be done through without any user input. It does require some information though, so you should substitute the variables with your information.

install-unrealircd.sh
# SSL certificate information
# The two-letter ISO abbreviation for your country
SSL_CERTIFICATE_COUNTRY="DK"
# The state or province where your organization is legally located
SSL_CERTIFICATE_STATE="Copenhagen"
# The city where your organization is legally located
SSL_CERTIFICATE_LOCATION="Copenhagen"
# The exact legal name of your organization
SSL_CERTIFICATE_ORGANIZATION="MyOrganization"
# Section of the organization
SSL_CERTIFICATE_ORGANIZATION_UNIT="IT"
# The fully qualified domain name for your web server
SSL_CERTIFICATE_COMMON_NAME="irc.myserver.org"
# Days the certificate is valid for
SSL_CERTIFICATE_DAYS=20000

mkdir -p /etc/unrealircd \
&& openssl req \
    -x509 \
    -newkey rsa:2048 \
    -keyout server.key.pem \
    -out server.cert.pem \
    -days $SSL_CERTIFICATE_DAYS \
    -nodes \
    -subj "/C=$SSL_CERTIFICATE_COUNTRY/ST=$SSL_CERTIFICATE_STATE/L=$SSL_CERTIFICATE_LOCATION/O=$SSL_CERTIFICATE_ORGANIZATION/OU=$SSL_CERTIFICATE_ORGANIZATION_UNIT/CN=$SSL_CERTIFICATE_COMMON_NAME" \
&& mv server.cert.pem /etc/unrealircd/config/ssl/ \
&& mv server.key.pem /etc/unrealircd/config/ssl/

Now you can put your configuration files into /etc/unrealircd/config. You can read more in the automating section about automatically including the configs.

Installing Anope IRC services

Again the goal is to install without any human interaction needed. This step will assume that UnrealIRCd has been installed first, since it needs some of the tools (namely the Development Tools packages).

Anope uses (or at least we use it here) cmake to build. This means we have to install cmake before doing anything else.

install-anope.sh
yum -y install cmake

Now we can compile Anope IRC services from source. We will fetch it and compile it in one step:

install-anope.sh
# Anope version
ANOPE_VERSION="2.0.2"

curl -L https://github.com/anope/anope/releases/download/$ANOPE_VERSION/anope-$ANOPE_VRSION-source.tar.gz | tar xz \
&& cd anope-$ANOPE_VERSION-source \
&& mv modules/extra/m_ssl_openssl.cpp modules/ \
&& mv modules/extra/m_sasl_dh-aes.cpp modules/ \
&& mkdir build \
&& cd build \
&& cmake \
    -DINSTDIR:STRING=/etc/anope \
    -DDEFUMASK:STRING=077  \
    -DCMAKE_BUILD_TYPE:STRING=RELEASE \
    -DUSE_RUN_CC_PL:BOOLEAN=ON \
    -DUSE_PCH:BOOLEAN=ON .. \
&& make \
&& make install

Depending on your SSL config you might also need to generate some certificates here. These are automatically generated in the full script included in the github repo, along with a sample configuration setup.

It is probably worth noting that two extra modules were included in the Anope build, namely m_ssl_openssl which enables SSL and m_sasl_dh-aes which enables AES on SASL. You can take a look at the extra modules in the modules/extra folder in the Anope source files.

Automating launch of EC2 and install

There are a couple of things that need to be set up first. For starters, we need to create a security group for the EC2 instance. You can do this in AWS Console -> EC2 -> Security Groups -> Create Security Group. If you don’t know what you want to open here, open the two TCP ports 6667 and 6697 to anywhere, and the SSH port 22 to anywhere also. The last one is optional, but it is quite nice to be able to check out the logs if anything goes wrong.

After this we need to create an IAM role for the instance, so that it can fetch configuration files and install scripts from S3. You can do this in AWS Console -> Identify & Access Management -> Roles -> Create New Role and then name the role. You then need to attach a policy to it. This can either be full S3 access, or a more limited policy. For more on the latter see the post about generating S3 bucket specific policies.

Now to the fun part! This will assume that you are using the install scripts found in the github repo, and that you have uploaded them to S3. In the repo there is a script to quickly upload the install/install-anope.sh and install/install-unrealircd.sh scripts, along with tar/gzipping the config files and uploading them. The script is aptly named upload-to-s3.sh.

First off, we will create an init script, which the instance will run on the first launch. This will take care of installing everything and moving the files into place, using the install scripts mentioned earlier. By having a IAM role with S3 read access attached to it, we can download objects from the S3 bucket directly.

init-ec2.sh
#!/bin/bash
# Bucket location
export AWS_S3_BUCKET="YourBucket/install"
export AWS_DEFAULT_REGION=eu-central-1

# Download the files from S3
echo "Downloading install files" >> /home/ec2-user/log.txt
aws s3 cp --region $AWS_DEFAULT_REGION s3://$AWS_S3_BUCKET/install/install-unrealircd.sh /home/ec2-user/install-unrealircd.sh >> /home/ec2-user/log.txt
aws s3 cp --region $AWS_DEFAULT_REGION s3://$AWS_S3_BUCKET/install/install-anope.sh /home/ec2-user/install-anope.sh >> /home/ec2-user/log.txt

# Make the scripts executable
echo "Making scripts executable" >> /home/ec2-user/log.txt
chmod +x /home/ec2-user/install-unrealircd.sh >> /home/ec2-user/log.txt
chmod +x /home/ec2-user/install-anope.sh >> /home/ec2-user/log.txt

# Installing UnrealIRCd
echo "Starting install of UnrealIRCd (check log-unrealircd.txt)" >> /home/ec2-user/log.txt
touch /home/ec2-user/log-unrealircd.txt
/home/ec2-user/install-unrealircd.sh >> /home/ec2-user/log-unrealircd.txt

# Installing Anope
echo "Starting install of Anope (check log-anope.txt)" >> /home/ec2-user/log.txt
touch /home/ec2-user/log-anope.txt
/home/ec2-user/install-anope.sh >> /home/ec2-user/log-anope.txt

The above fetches the scripts down, executes them which in turn installs UnrealIRCd and Anope.

While launching stuff from the console is indeed very fun…the first couple of times, it quickly gets tedious. Therefore we will utilize the AWS API, to create an EC2 instance, tag it and associate an elastic IP to it.

launch-ec2-instance.sh
#!/bin/bash

# AWS user credentials
export AWS_ACCESS_KEY_ID=MyAccessKey
export AWS_SECRET_ACCESS_KEY=MySecretKey
export AWS_DEFAULT_REGION=eu-central-1

# EC2 instance details
NAME_TAG="irc.myserver.org"
IMAGE_ID="ami-bc5b48d0" # Amazon Linux AMI 2015.09.1 (HVM), SSD Volume Type
SNAPSHOT_ID="snap-f1a95375" # That snapshot depends on the AMI above
INSTANCE_TYPE="t2.micro"
KEY_NAME="irc-server"
SECURITY_GROUP="IRC"
IAM_ROLE="irc.codetalk.io"
ELASTIC_IP=168.1.1.1

# Launch an EC2 instance
echo "> Launching the EC2 instance..."
INSTANCE_ID=$( aws ec2 run-instances \
    --image-id $IMAGE_ID \
    --instance-type $INSTANCE_TYPE \
    --key-name $KEY_NAME \
    --security-groups $SECURITY_GROUP \
    --iam-instance-profile Name=$IAM_ROLE \
    --block-device-mapping DeviceName=/dev/xvda,Ebs="{SnapshotId=$SNAPSHOT_ID,VolumeSize=30,DeleteOnTermination=true,VolumeType=gp2}" \
    --user-data file://initec2.sh \
    | jq --raw-output '.Instances[0].InstanceId' )
echo "> Instance $INSTANCE_ID is launching"

# Add name, environment and company tags to the instance
echo "> Adding tags to the instance $INSTANCE_ID"
aws ec2 create-tags \
    --resources $INSTANCE_ID \
    --tags "[
        {\"Key\": \"Name\", \"Value\": \"$NAME_TAG\"}
    ]"

# Waiting for the instance to be running
echo "> Waiting until $INSTANCE_ID is running..."
aws ec2 wait instance-running --instance-ids $INSTANCE_ID
echo "> Instance is up"

# Associate an elastic IP with the instance
echo "> Associating the IP $ELASTIC_IP with instance $INSTANCE_ID"
ASSOC_ID=$( aws ec2 associate-address --instance-id $INSTANCE_ID --public-ip $ELASTIC_IP )
echo "> Done!"

The script should mostly be self-explanatory. The important parts are under the # EC2 instance details comment. Here are the values that you should configure to match what you need. The AMI ID can be found in the AMI store (you can start launching an instance and stop after the first screen).

Github repo, ready to fork!

All of the scripts and configuration files to set it all up can be found in this github repo. You’ll want to change the configuration files in the config folder to fit your server details.

Furthermore you need to fit the credentials and server details to your own, in the init-ec2.sh, launch-ec2-instance.sh and upload-to-s3.sh scripts. Hopefully it should be evident from the naming of the variables, what it is they expect.

S3 bucket specific policy

I have recently started caring a bit more about security on my AWS applications, and to this end Identity & Access Management (IAM) users are a great way to limit access to a need-to-use-only basis.

Recently I set up my IRC server to download its configuration and install files from an S3 bucket. This meant that it needed to have read access to a specific bucket, and for this an IAM role was created.

There are two ways to generate policies:

I will generally advise to either use the generator completely or at least use it for the basis of the policy you want to create.

Using the policy generator

To access the generator go into IAM -> Policies -> Create Policy -> Policy Generator.

S3 Create Policy S3 Select Policy Generator

You will then be able to create the policy. To do this, first set the AWS service to Amazon S3 and then choose the actions that you want to support. Here I’ve chosen GetObject as I just want to allow the read of an object in the bucket, and nothing more.

Finally, you need to set the ARN of your bucket. This will be something like arn:aws:s3:::YourBucketName/* to allow access to all elements in your S3 bucket named YourBucketName. It’ll look a little something like in the left figure below.

S3 Edit Permissions S3 Review Policy

Finally add the statement, and you can click next which should present you with something like the right image above.

Manually creating a policy

You can also skip the generator and create your own policy directly. This will get you into a window looking much like the ‘Review Policy’ image in the earlier section.

Here you just need to input a name, description and the policy document itself. You can either use the generator as a basis for this document or craft your own. The following is the one generated from the policy generator which can be directly used here.

S3 bucket policy
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Stmt1447546160000",
            "Effect": "Allow",
            "Action": [
                "s3:GetObject"
            ],
            "Resource": [
                "arn:aws:s3:::YourBucketName/*"
            ]
        }
    ]
}

Hopefully this will get you going and buff up your security on your AWS setup.