slamii

Window functions

2019-06-13T00:00:00+00:00

Window functions in SQL, their syntax, benefits and coverage in MySQL and PostgreSQL.

Outline with an example

Window functions are functions applied to sets of rows. They can be used in SELECT or ORDER BY clauses. Think of an SQL SELECT query that selects $S$ rows and that for each row $r$ makes some calculation involving other rows, for example assigns rank to customer orders according to their value.

What a window function does is for each processed row $r$ in $S$ it defines a set of rows called a window, let’s denote it $W_r$, and this window provides the context for the window function to operate within. For example, if $r$ is a row describing an order, $W_r$ could be a set of all the orders from which to determine the rank of $r$. Windowing means applying calculations to a set of rows and returning a single value. Let’s be quick to jump to an example:

The query above could yield results like the following:

id      order_date  value     rank
------- ----------- --------- ----
4       2019-03-23  56        1
2       2019-02-20  40        2
3       2019-03-23  33        3

In this example the selected rows $S$ are all the rows from Sample.Table. For each row $r$, the OVER clause creates its own, independent window $W_r$. Here each window is the same and consists, again, of all the rows in the table, so $W_r = S$. This is because OVER does impose any restrictions on the windows, only ordering. Subsequently, both $r$ and $W_r$ are passed to the window function RANK(). It computes the rank of a given record $r$ as one more than the number of rows in $W_r$ that have a greater ordering value than $r$.

Syntax

The general form of a window function clause is function(<args>) OVER([<partition clause>][<order clause>][<frame clause>]]). Subsequent sections cover each part in more detail.

Partitioning

Specified by the PARTITION BY clause, partitioning restricts the scope of $W_r$ to only those rows from $S$ that have the same values in the partitioning columns as the current row $r$.

For example, the following query would print each sales value next to its associated customer id together with its percentage share among all sales values for this customer. Here SUM() is the window function and PARTITION BY restricts $W_r$ to only those rows that have the same value in customer_id as $r$ does.

Ordering

Ordering is defined by the ORDER BY clause which defines the order for the calculation within $W_r$. With ranking functions like RANK() or ROW_NUMBER() this is pretty straightforward. Compare the example in the outline section; we obviously have to specify by which column we want to rank the records and this is the role of the ORDER BY value DESC part.

However, for aggregate window functions like SUM() or AVG() ordering of the rows does not matter. In this case, it is only used to give meaning to the framing options which are covered in the next section.

Framing

Framing, like partitioning, is a filter that restricts the rows in $W_r$. It is applicable to aggregate window functions and to three of the offset functions: FIRST_VALUE(), LAST_VALUE(), and NTH_VALUE() and requires an ordering imposed on the window. Based on this ordering $W_r$ is framed to contain only a selection of rows.

The window frame clause can include three parts and takes the following form <window frame units> <window frame extent> [<window frame exclusion>]. We shall briefly describe each part before presenting a couple of short examples.

Frame units specify the way in which the frame is determined. It can be either ROWS, RANGE or GROUPS, the last one not supported by MySQL and only supported by PostgreSQL of version at least 11.

The frame extent part describes the boundaries quantitatively - once we know how we specify the frame, what units we use, we have to describe how big it is going to be. For this we use the statements UNBOUNDED PRECEDING/FOLLOWING, <n> PRECEDING/FOLLOWING or CURRENT ROW.

The window frame-exclusion option specifies what to do with the current row and its peers in case of ties. Has the form EXCLUDE CURRENT ROW/GROUP/TIES/NO OTHERS (default), not supported by MySQL, the last three supported by Postgres from version 11.

Framing examples

Suppose our dataset is contained within the table Example.Data and contains the following records:

id         value
-------    -------
        15
        20
        5
        15
        9

Framing with ROWS

The ROWS option allows to indicate the bounds of the frame as an offset from $r$ in the number of rows. Let’s take the example of the following query:

It would yield the following result:

 id        sum 
-------    --------
        35
        40
        55
        64
        64

For each $r$ the query defines $W_r$ to be $r$ together with all preceding and one following row (in the order given by id). Thus for id = 3 the sum is equal to 55 as a sum of value for id = 1, 2, 3, 4. One could use EXCLUDE CURRENT ROW in the query to exclude id = 3 here.

Framing with RANGE

The RANGE option defines the bounds in terms of a difference with respect to $r$’s value. Take a look at the following:

id        sum 
-------   --------
       5
       14
       30
       30
       20

Notice the change in ordering in both ORDER BY’s. Now we defined the sum to be over all the rows that have value at least 4 smaller than $r$ and at most equal to the value of $r$. Hence for example id = 5 has sum = 14 as the only rows within this range are id = 5, 3.

Framing with GROUPS

Finally, the GROUPS option frames the window to a number of groups of rows, a ‘group’ meaning the rows with the same value.

id        sum 
-------   --------
       14
       39
       50
       50
       20

Consequently, sum for id = 1 is 50 as a sum of id = 1, 4 (the value = 15 group) and id = 2 (the next group, value = 20). Had we used EXCLUDE GROUP here, id = 1, 4 would be neglected in this case, while EXCLUDE TIES would count only one of them.

Comparison to other approaches

An alternative to using window functions could be grouped queries. However, these give us insight into aggregates, but we lose the details. For example, let’s say we have a database of books, and each book has its weight and an owner. For each book, we want to calculate the percentage of the current weight of the bookworm’s total, and maybe a difference from the average. If you aggregate the data by owner, you don’t have access to individual weights.

A common way to handle this is to have a query that groups the data, define a table expression based on this query, and then join it with the main table to match the detail with the aggregate. Here’s a query that uses this approach:

But what if you also need to include percentages and differences from the global aggregates (including all the owners)? You would have to create another table expression and another join. It quickly gets complicated.

Another approach is to use subqueries:

It suffers from the same problem: complication. What’s more, most (if not all) SQL optimizers are not clever enough to use the fact that the subquery uses the same rows, and therefore perform multiple accesses.

Look how simple it becomes with window functions:

Bibliography

Microsoft SQL Server 2012 High-Performance T-SQL Using Window Functions, Itzik Ben-Gan
PostgreSQL 11’s Support for SQL Standard GROUPS and EXCLUDE Window Function Clauses, blog.jooq.org
Window Function Frame Specification, dev.mysql.com

Docker symlink security hole

2019-06-04T00:00:00+00:00

On 28th May a senior software engineer Aleksa Sarai discovered a security hole within Docker which can grant an attacker read and write access to any file on the host machine regardless of its permissions.

TOCTOU vulnerability

TOCTOU stands for time-of-check to time-of-use and is a race condition vulnerability resulting in inconsistent state of a system, usually a file system. When two or more processes operate on one resource, some of them may perform modifications based on an incorrect past state of that resource.

Consider a database of email recipients. One possible email would be sent to users who haven’t logged in to the app for at least 30 days. Let’s assume a job performs this task once a day, first selecting a set of users who satisfy the criterion (last login more than 30 days before), then resolving an email template, maybe calling an external service for some additional data, and then sending the emails to the selected users. While the server was resolving the template, Bob, who hasn’t been using the app for a while, logs in. He would still receive the email.

In general, despite their apparent simplicity, TOCTOU problems are usually very hard to solve when both consistency and performance are needed.

Bug location

The Docker vulnerability affects the FollowSymlinkInScope function used by many docker tools, notably docker cp that copies files between containers and the host machine. The purpose of FollowSymlinkInScope is to take a given symlink pointing to some path and safely resolve it as though the process was inside a container.

Take for example docker cp /path1 container1:/path2/sym1 executed on the host. It would copy a file from /path1 to the location within container1 to which sym1 points, say /loc1. /loc1 only makes sense within the container; it does not refer to /loc1 on the host machine. Hence sym1 has to be resolved to something else; to /loc1 at some hidden location $\ell$ on the host that Docker process uses internally as container1’s insides.

The problem with FollowSymlinkInScope is that after the symlink is resolved, it is passed around a little more before it is actually used - in the case of docker cp to do the copying. The usage phase assumes the path is safe to use since the resolution has been done in a safely manner before.

But it can happen that an attacker adds a symlink component to the path after the resolution but before the actual use, hitting the TOCTOU window. Then the symlink will not be treated specially and can therefore point to a file on the host machine.

Continuing the above example, container1:/path2/sym1 has been resolved to /loc1 within $\ell$. But suppose at attacker managed to swap /sym1 for another symlink, now pointing to /loc2, before docker cp started the actual copying. Since we are after the resolving phase, docker cp will not realise that it should search for /loc2 within $\ell$. It will take it as given, referring to a file on the host machine.

If used maliciously, this can give the attacker read and write permissions to any file on the host machine, even without appropriate permissions.

Exploitation with docker cp

Aleksa provided exemplary tools to showcase the security hole on the example of docker cp command. Let’s see what he did.

Attack setup

Assume we are a regular user on the host machine with docker permissions but without root access. We want to retrieve a root-only file from the system, say /w00t_w00t_im_a_flag. So the first thing that Aleksa does is he creates this file, prints some ‘secret’ message there and chmod’s it to 000 so that only root should have access to it.

Then he builds an runs a malicious container and subsequently starts an attack.

Container contents

The container used for the attack also contains /w00t_w00t_im_a_flag file with a different, ‘ordinary’ message. The container executes a single C program which creates a symlink /totally_safe_path pointing to /. It also creates an empty directory. The program then atomically swaps the two in a loop until the end of time.

The attack

docker cp container1:/totally_safe_path/w00t_w00t_im_a_flag is invoked in a never-ending loop, copying each time to a different directory. The underlying container keeps swapping around the symlink at /totally_safe_path. If it manages to insert the symlink to / after the resolution by FollowSymlinkInScope, then / will resolve to the root of the host machine and the file that should be inaccessible will be copied and compromised. If the symlink stays there throughout the operation of docker cp, / will resolve to the container’s root and we will simply copy the ‘ordinary’ message.

Aleksa commented that there is a <1% chance of hitting the TOCTOU window with this script. This means that it only takes roughly a 10s attack to get hold of the ‘secret’ file.

Fallout

Aleksa and Docker security team agreed that the public disclosure of the issue is legitimate even without a fix available.

The most comprehensive solution to the problem would require changes to the core pieces of Docker which is not possible. Some partial solutions are underway and as for now Docker officials said that possible attack scenario is “rare/unlikely” so the flaw, while serious, is not necessarily an emergency.

Bibliography

Monads for the inquisitive

2019-03-18T00:00:00+00:00

Why we need monads, what they are and where we can see them.

Motivational introduction

Outlines what purpose monads serve in computer science.
The shaded paragraphs are supposed to be more in-depth and give a mathematical background; they can be omitted and it should not affect the overall understanding.

Functional languages are founded on the idea of function composition; having $f: A \longrightarrow B$ and $g: B \longrightarrow C$ we can create a composition $gf: A \longrightarrow C$. Suppose now those functions have some additional, common behaviour. Maybe they sometimes don’t return a result (e.g. because something is undefined) or maybe they return a varying number of results.

In a strongly typed language we’d represent this additional behaviour by a different type of the return value. Let’s stick to the example of possibly not returning a value. We’d introduce a type constructor Maybe, $M$ for short, and our functions would have signatures $f:A \longrightarrow MB$ and $g:B \longrightarrow MC$.

We still do want to compose them. The additional behaviour shouldn’t be a problem; both functions have it. But now the types are incompatible. We want to apply $g$ after $f$ but $g$ accepts type $B$, not $MB$. We are stuck. Monads are a device to break this impasse; let’s keep this in mind for the rest of the section.

Taking a step back, another thing besides composition of functions that we can employ is the identity function $id: A \longrightarrow A$, a trivial computation for any type $A$. Now, it turns out that there is a mathematical structure built on top of these two properties - a category.

It belongs to category theory, a mathematical area that tries to describe things in the most general way possible. Thus the definition of a category is very non-specific. Intuitively, it's a thing with some objects and with some things between these objects (like functions between sets). Formally:

A category $\mathcal{C}$ is a collection of:

objects $A, B, \ldots$

arrows $f:A\longrightarrow B, \ldots$

where the arrows have the objects of $\mathcal{C}$ as domain and codomain. The arrows of corresponding domains and codomains have to be composable in the way you'd expect and each object has its identity arrow $id_A:A \longrightarrow A$ such that composing an identity with any function gives back the function, e.g. $fid_A = f = id_Bf$.

For more details refer to chapter 1 in [1]. So the types of a programming language together with its functions form a category.

Further into category theory, there is a notion of a monad. It is different from the computer science monad we are discussing and we are not going to give its definition (the reader can refer to 5.1 in [1]). But it is closely related to the one we are interested in.

A monad $M$ on a category $\mathcal{C}$ gives us another category, the Kleisli category $\mathcal{C}_\mathbb{M}$. The only thing we need to know about it is that its arrows are the arrows in $\mathcal{C}$ of the form $f:A\longrightarrow MB$. Because the Kleisli category is ineed a category, we can compose those arrows - which is precisely what we wanted.

The upshot is as follows: if we have functions with some additional behaviour represented by $f:A\longrightarrow MB$ and if the type constructor $M$ has some nice properties, then we can compose those functions despite an apparent type inconsistency. This is what monads are all about.

What is a monad?

Presents two definitions of a monad in computer science.
The shaded paragraphs serve as their comparison and discussion. They are more mathematically involved and can be safely skipped if they bring more harm than good.

Much in the spirit of the above reasoning, the definition of a computer science monad can be given as follows ([5]):

A monad is a map $M$:

sending every type $X$ of a given programming language to a new type $MX$.

equipped with a rule for composing two functions of the form $f: X \longrightarrow MY$ and $g: Y \longrightarrow MZ$ to $gf: X\longrightarrow MZ$. The functions are called Kleisli functions and the composition the Kleisli composition.

together with a map $\mathrm{pure}_X: X\longrightarrow MX$ for every type $X$ that acts trivially in the composition. The Kleisli composition is required to be associative.

The Kleisli functions are the functions with additional behaviour discussed in the previous section.

A more practical definition offered by Haskell [1] states:

A monad is:

a type constructor $m$

a function $a \longrightarrow ma$ building values of type $m$, called return

a combinator function called bind of the form $(ma, a \longrightarrow mb)\longrightarrow mb$ for any types $a, b$. Typically denoted $x\ {\scriptstyle \gg=}\ f$.

So given a monadic value $x$ of type $ma$ and Kleisli function $f: a\longrightarrow mb$ the bind function gives us a way to apply $f$ to $x$. The monad above has to also satisfy the laws equivalent to the third point in the previous definition.

A natural question to ask would be about the meaning of the $\mathrm{pure}$ function (and its counterpart from the second definition - bind). If all we want is a composition of Kleisli functions and this is guaranteed by the second point in the definition, why bother with the third?

$\mathrm{pure}$ and bind are in some sense trivial computations; ones such that composing them with others does not make any difference. They provide an "obvious" or "natural" way to get from a value in $X$ to a value in $MX$.

The fact that such trivial computations exist ensures that $M$ is not too weird, that it has some nice structure. Mathematically speaking, this means that the computations of the form $A\longrightarrow MB$ form a category (the Kleisli category we mentioned before).

The requirement of being a category is a really basic one. If we don't insist on it and don't require the existence of $\mathrm{pure}$ function, we would allow some very irregular and strange structures to be monands. Ones that would be irregular enough to not have any practical value anyway. It's better to simply disregard them.

For now we will leave it as an open question: what are those irregular structures? What "monads" would we get if we didn't require the $\mathrm{pure}$ and bind functions and their associated laws in the definition?

Another question we might want to ask concerns the relation between the two definitions. Are they equivalent? First points are clearly the same. $\mathrm{pure}$ for a given type is the same as bind and the associated laws are already stated to be equivalent. The only question is whether having Kleisli composition is the same as having the bind combinator.

Indeed if we have a bind function then we get a Kleisli composition by defining $gf(x) := f(x)\ {\scriptstyle \gg=}\ g$. Before looking into whether a Kleisli composition gives us bind let's look at the Haskell definition more closely; the findings below are not immediately obvious but the reader will be spared the detailed proofs.

The three points of the second definition together with its laws ensure that the return function is injective for any type $a$ unless $ma$ is trivial. In both cases $m$ is a functor and $(m, \mathrm{return}, \left(\mathrm{return}_m)^{-1}\right)$ is a monad in the category theory sense.

If $\mathrm{pure}_X$ from the first definition happens to be injective as well then indeed we get a bind from the second. In general however, it looks as though it need not be.

Again, we leave these as open questions: can we construct a monad as in the first definition without $\mathrm{pure}_X$ being injective? Can $M$ be non-monadic in category theory sense? Can it be non-functorial?

So in general the first definition seems broader than the second. However, in all practical and useful cases in which computer science is actually interested the two definitions coincide and one should forget the potential discrepancy.

Examples of monads in programming languages

Three examples: language-agnostic list constructor, Optionals from Java as semimonad and random number generators from Haskell.

A list constructor

A list constructor $L$ can be given a structure of a monad. Kleisli functions in this case are functions returning a varying number of results.

Let’s take an example of a function $f$ that takes a chess position and returns a list of possible chess positions after one move. We want compose two such functions $f: \mathrm{ChPos}\longrightarrow L\mathrm{ChPos}$ to get a function computing possible chess possitions after two moves.

How would we do that? $f(x)$ is a list of chess positions $[x_1, x_2, \ldots]$; we’d apply another $f$ to every such position getting a list of lists of positions $[[x_1’, \ldots], [x_2’, \ldots], \ldots]$. Then we’d flatten this list to a simple list of positions $[x_1’, \ldots, x_2’, \ldots]$.

The $\mathrm{ChPos}$ is just an example, but it’s clear that we could do the same with all functions returning a list of elements. The bind is the operator applying a function to every element of a list and flattening the result. The return for an element $x$ returns a list containing this single element $[x]$. These two together with $L$ form a monad

Optionals in Java

Optionals are Java-implementation of the $\mathrm{Maybe}$ we took for an example in the first section. For instance, Optional<String> can be thought of as a box which can either contain a String or be empty.

The Kleisli functions, are those that may not return a value. For example suppose the objects we work with are Sheep. Those sheep have families and we can write functions returning a given sheep’s parent. But if the sheep is cloned, it does not have parents, the function would return null.

If we ever want to write functions returning grandparents, we’d have to check first whether the father is defined and only in this case one could search for its parents. For great-grandparents it’s even worse; the deeper we go, the more ugly it becomes.

But this seems wrong, doesn’t it? If we have a function returning the father, returning the grandfather should be just a composition of two such functions. That’s the most clear and elegant solution, no null-checking should be involved. Indeed, in this case Java has us do the following:

where Optional.ofNullable is the $\mathrm{return}$ function and Optional.flatMap is the $\mathrm{bind}$. Looks like a monad and Kleisli composition!

However, we mentioned before that a monad has to also satisfy some laws in order to really be a monad. It turns out that Java’s Optional does not always abide these.

One of those laws is the left identity which says that $\mathrm{bind}$-ing a function to a $\mathrm{return}$-ed value has to be the same as just applying the function to the value. In Java terms this reads:

Optional.ofNullable(x).flatMap(f).equals(f.apply(x))

for any value x and any Function f. In other words, $\mathrm{return}$ is like an identity function. Whenever x is not null this works fine, but consider the following example:

This code breaks down; left side is an empty Optional, and the right one - minus one.

The problem here is that Optional.ofNullable(null) is translated to Optional.empty. So even if f is defined to work on nullable values of x, on the left-hand-side it never gets executed.

Thus Java’s Optional is not a full monad. What are the implications of this? For the answer as well as for some additional account of the topic consult [4].

Random numbers in Haskell

Consider generating random numbers in a purely functional language like Haskell. Purely functional means that any function always returns the same result for the same input. In this setting functions like random() returning a random number from $(0, 1)$ are not allowed to exist; the input is always the same (empty) so the output has to be as well. The best we could do would be to imitate xkcd:

Jokes aside, how functions like random() work in “traditional”, non-functional languages? They sample some data like mouse movements or time in nanoseconds from the system and produce a value based on these. Such implicit arguments are not allowed in functional languages, so what one has to do here is to pass this data explicitly as a parameter, in the form of some object.

So, in languages like Haskell functions returning random values take a random number generator object, rng, and return a value together with another rng (in case we want to generate more numbers in the future. We can’t reuse the rng that we passed initially – as explained above it would return the same value!). The signature of such functions is

$r\longrightarrow(v, r)$

A function of this form is called a stateful computation, sc, because what we are doing is passing some state $r$ to the function based on which it returns a value together with a new state. In the case of our example the state is the rng. The type of an sc is defined to be the type of the return value - $\mathrm{type}(v)$ in the above notation.

Using these, to write a function returning three random numbers we’d need to write code like the following:

where we have to manually pass the subsequent rngs to subsequent function calls. If the reader thinks this is ludicrous we totally agree and if she expects we are going to use monads to get around it, she is not mistaken.

So, for a given type $a$, what could monadic $ma$ be? We really have only one choice - it’s going to be stateful computations with values in $a$. Thus the values of Kleisli functions are stateful computations. Their signature is:

$f: a \longrightarrow (r\longrightarrow(b,r))$

This means that $ma$ is a functional type - it’s values are functions, the scs. Don’t get lost on this.

We need to find bind and return functions for a complete monad. Starting with the latter, what would be a trivial sc for a value $x$? It’s a function - one that yields $x$ for all states $r$.

$\mathrm{return}\ x = r \longmapsto (x,r)$

To come up with a bind suppose we have an sc $c: r \longrightarrow (a, r)$ of some type and a function $f: a \longrightarrow (r \longrightarrow (b, r))$ from that type to scs of possibly some other type. E.g. $f(a)$ could be an sc returning two values, the first of which is always $a$.

$c\ {\scriptstyle >==}\ f$ is an sc, name it $c’$. Intuitively, what we want it to do on a state $r$ is to do whatever $c$ did and then do whatever sc returned by $f$ does. To be precise the definition of $c’$ is:

Now the function generating three random numbers with a bit of Haskell’s syntactic sugar can look as follows:

Thus, what monads did to us is carrying out the pesky state passing implicitly, out of our sight, making the code look much more like what we would expect in a “traditional” language. For a more careful account of the topic you could refer to [3].

Leibniz’s monads

A taste of the meaning of monad in philosophy. Unrelated to computer science.

The term monad, from Greek monas, “unit”, was first used by Giordano Bruno. He claimed the world is homogeneous, one; nevertheless, it consists of a multitude of self-contained units. Like a point, mathematical minimum, is a basic component of space, and an atom, physical minimum, of that of matter - monad is a metaphysical minimum. Their shapes and forms are unique (which followed naturally from God being the ultimate perfect creator of infinite inventiveness).

It was then picked up by Gottfried Wilhelm Leibniz, one of the most interesting philosophers of the 17th century. While he’s mostly known for the fundamental contributions to differential and integral calculus, his metaphysical system was equally groundbreaking in its cohesiveness and generality.

Three hundred years ago the most enlightened minds of Europe were debating the problem of substance. It may seem like a silly problem now (or, to be polite, a poorly defined one) but surprisingly, these old ideas influence our language and way of thinking even today.

Rene Descartes rejected the then dominant school of thought, platonism, arguing that the corporeal substance consists of only matter itself, that there are no platonic forms. However, he went only so far in reducing the metaphysical overhead of previous generations - inspired by then state-of-the-art automatons on the streets of Paris, he introduced the distinction between body and mind. And that dualism lives on, three centuries later.

Descartes struggled with explaining how exactly these two substances influence each other. His successors suggested various solutions, most of which seem odd nowadays:

They don’t influence each other and they don’t have to because God does that for them
They may seem like two substances but in reality they are just two sides of the same coin, so to speak, namely God
There is no dualism, and only matter exists (no God here)

But perhaps the most original solution was Leibniz’s. He claimed there is an infinite number of substances but they are of the same kind. In this respect his views were similar to materialists; but looking closer we discover significant differences.

Leibniz believed that every phenomenon is unique, that everything differs from everything else, that there are no two identical leaves or drops of water. There are no two objects that differ only by their positions in space. However, things are similar. Phenomena are continuous series (notice connection to differentials). Straight line is just a special case of a curve, equation - of inequality. Consciousness is a spectrum, a lie is the smallest truth, evil the smallest good. This was Leibniz’s continuity law (lex continui). It allowed him to connect seemingly disjoint phenomena using a coherent framework. An interesting application of this theory was in psychology, where he introduced unconscious states of mind long before Freud (claiming that these were only weak forms of consciousness).

This law, along with his metaphysical pluralism was perhaps the most revolutionary thesis of Leibniz’s philosophy. And here, finally, we come to monads - this is how he called these simple individual units of being. Unfortunately for computer science-oriented readers, they have little in common with Haskell. Perhaps the most significant is that they don’t have windows, as Leibniz said, each of them being a closed universe that acts upon its environment but cannot be influenced from outside.