James Robert Lloyd

The world is dead and so, so,
The friends are long ago
And the stars are blown away.

PoetryNet, 2015

Introduction

A little while after Andrej Karpathy released char-rnn into the world, Alex Davies , Kate Crowcroft and I explored the possibility of using it to create computer generated poetry. We took the char-rnn code, a fair chunk of Wikipedia and 10,000 poems and created PoetryNet (or Clarissa as it later became known). The results were "unreasonably effective" and really quite amusing. And in an unusual turn of events, Alex and I adapted the system to provide lyrics for a musical staged in London's west end.

In this article we offer the brief thoughts of two computer scientists (Alex and I) on creativity and probe the poetic powers of a char-rnn. But mostly this is an excuse to copy paste amusing passages of computer generated ramblings.

How can a computer learn to write poetry?

Consider the following sentence fragment

The cat sat on the...

What comes next? How about this sentence (and word) fragment

The cat sat on the m...

or

The cat sat on the ma...

There are many possible ways to extend these fragments, but some are more likely than others. A computer can learn which extensions are more likely than others by seeing lots of examples. It can learn the common patterns of language and poetry.

Computers have been able to learn patterns from data for a very long time (almost any statistical analysis is a form of pattern learning), but recently the machine learning community has had a lot of success with neural networks. One of the factors for their success is their computational efficiency, which allows them to process vast quantities of data and potentially discover very complicated patterns. They may not be very efficient with respect to the amount of data they need to ingest to learn anything, but when data is plentiful this is not much of a problem. Fortunately for us, there are many poems in the world to learn from.

Learning in action - from gibberish to semi-grammatical nonsense

After collecting 10,000 poems, we used char-rnn to read in poems character by character and then try to predict what comes next. By getting the computer to make several predictions, one after the other, we can produce an entire poem, character by character. This is quite different to how a human learns to write poetry, but we'll discuss that later.

Let's see the learning process in action.

After a few seconds - character frequency and not much else

/Wteh lea e a sti es s e inne re l se l lhre ,so e e sir a f e riay r mn rdh rewsr e iie r eto e ctsse e i o en e tnea e s ryc e ot hii ntr o e dert aedste el h en i dhe e o, eol a ilr ea w a o a w r asi o oe t d t e ean r r h h s
i t e os hi i e hee i eth e eu e i
k

After about a minute - word shaped things

ursoe haoth sicge tim bonr
ghoiconiiroch is a)o
PuTTY dhr doooc nins voaed ofitot tions anewt

Ioun ao
oe wha
n ap thri shes anorhithurioidheet ofemh

After about 5 minutes - pronounceable gibberish and some small words

Er as the rears at the sear nreep gor I her the spreatsed
The tore in the he dorr wall
and stilling surs there and ware
The its sale and sam and the lay le dear boom sast me the ce coom strwarbe as so une in ussing some has the seem non coild be and serool sill momess.

Stand the fanes and chen the posser.
Srone the she was insoneed the crour faning of mas,

After about 10 minutes - very bad spelling, but starting to look like language (and it leets!)

5witch you lears who heart,
In the creature and too said to seet marged to Nature.

It is no look of the farrarge where the mills
The wassed mast in the prisering into the way of the ground,
I am all I seering how swall to the smoke of a had in the pure
All where the souls sing lakes the roage I great all desperble For you take my singer in the unimage
The sead are while the strong will smoully
they heard not light in his amstrable he would gourdeats and ond wheeter down to seeper

After about half an hour - new favourite word

It called one room,
hands in the world and earth.
In the break trees.

THRAR

SOme will dissun and make it is forth
And the hands of was my ever one sings whose sin.
The beef all of the meanal camps
the moon of my fire--
where your lays the wind,
sanding blind mind speet
and sing of indeen in the world

A few hours later

Are you not pleasant?
And as I am leaving you my life like the earthworms?
Who cannot remember to be a room of mine
I love behind three states, or differently, and nothing
but a lovely enviet of one arrival, a cutting switchback
and wide writing in the knoll, and in my brother's warmth,
like the white black chin that prizes and forms within the sky,
and the peach with the ancient broad washed-up beard of the sky
with sudden groans of mouths of cattle and so tighter red
in the water which has been curling in its cages and comes
when she sucks me back to the face of this door
and she presses her gaze and her left hand just as he falls
a little house from heaven and takes the sleep of the morning
and the forest is afraid of the sun is the sun of the land
and by the kitchen wall and the sun of the land.

They are
bombing in
the Earth.

The world can fall and get back to be erosion, or welcome the moment
That they are hung up and the pigs live with it.

What has char-rnn learnt about language and poetry?

The excerpt above may not be great poetry, but there has been an impressive amount of learning from its beginnings of a random jumble of unrelated characters. Let's try to get some understanding of what the char-rnn knows of language and poetry.

Completing words

We can get a sense of words that the char-rnn has learned by getting it to complete half formed words and phrases. For example, if we get the char-rnn to read in the characters "ap", it often completes it as "apple", and "tru" can become "truck" or "trust" amongst other possibilities.

We can verify that it recognises the difference between lower and upper case words, for example

Pe → Persian

and

Pet → Peter

There are some words that the system has not learned, but it will make a decent guess of something that looks like a word

Recog → Recography

"Recography" may not be a word, but it certainly could be!

Completing phrases

One might imagine that perhaps char-rnn has just rememebered long stretches of poetry and has just learned to recite them whilst occassionally switching poem so that no-one recognises the plagiarism. For technical readers, we made sure that we did not overfit our model, for less technical folks, here are some examples of the char-rnn failing to recreate some Shakespeare

Shall I compare thee to skill,
The white heart of the release moon still come and say,
And all the one rain is the our heads, in the green day.
The precision of countrys of the corner so little garden,
darkening blessed of sea while one of the city for the voice in the window--
The rusty ban, who deadest an old and the starpy called people,
And the draw of the doors and the time for proval, so many instant
Than are a river on the warm of the blood of his willing and play
And gave me here some move; and what any

or

Shall I compare thee to my takeaway,
O son of the bark, the boy not mother!
Says the boy he's coming from his luck.

The creativity factor

Above we mentioned that the char-rnn might complete "tru" either as "truck" or "trust". This is because instead of predicting which character comes next, the char-rnn is predicting the probability of what character comes next. Consequently, we can change how "creative" (more about that later) the system is, by altering its tendency to pick the most likely character or try something less likely. Below are the results of changing the "creativity factor".

Very low

And the soul with the sea where the wind will be
And the black sky is still and the sun is all there
And the sun is dry and something that is still and the sun
That is the sun and the sun and the wind and the street
And the wind and the sun and the light of the world
And the wind stands the wind and the sun and the sun
And the sun and the street and the sun and the sun

Low

Often the morning is but a light
And the flame of the sun is shaken,
And the air is the slow sound of the sea,
And the blood of the mountain shore
And the blood of the sunset streams
Of the roof of the sun and the hall

Normal

Of this I think of the conquerors:
The crowd, the natural camera,
And stands a seed of ripeness and burns.


I always say the book starts to see it
To say a certain slim neck below.


A child belongs to a friend
By the time the rest of us have been called
And made the wrong thing is all


He says we are so close to the heart
Of concern and a story and the product
Of our sweetness is the god of the time.


In a strange world a dull beauty
Is a faithful one in the source
Of this time that such a stranger


They were forgotten and they were there.

High

As the car starts to see some other twist of sound,
I think he is, on the paper to our sharp surface,
And for perceiving whether a spirit
To the dark boking life will learn to help
Nothing but for him. All that desk receives
And then is so young and fair, and that it is.

Very high

Before the country
light begins with snowflowers,
I suppose he'll be seen
by Similia you let them
respond, the grottes of perfect
big silk feet. Toy mass,
sprayed at a stone
riding thirty years ago

Super high

I could cease to put what you knows where I don't know my notebook,
and I'm not sharp and you are wearing the German parts for the reader, and you sometimes heir
in silence and stars. After you released it to your eggplant with the wings.

Super higher

no small acute orifice on
her sundown tranquillizer
is just my viger: He smalls
a brush within me?
Dogs jow -- this one;
though Warning-tirer cat waits.

A creativity scorecard

Now is probably a good time at which to point out the elephant in the room. PoetryNet forgets what it is talking about after a few words and each line only occasionally connects with the lines around it. This gives the poems an abstract and wandering style, but this is unlikely to be a stylistic choice, rather an inability to stay on topic. It may not be great poetry in a classical sense, but is it creative? And for that matter, what is creativity?

“Creativity” is one of those grab-bag terms, like “happiness” and “love”, that can mean so many things it runs the risk of meaning nothing at all

Maria Popova

To many, creativity falls squarely into a sacred set of fundamentally human qualities that could never be replicated by a machine, but others define it in a much more machine-accessible way. A good collection of opinions can be found here, but let's see how a char-rnn stands up to a representative set of definitions we have collected from around the Internet.

1. 1. Finding something new and interesting - Yes
2. 2. Communicate something unique about the human condition - No
3. 3. Franken, Human motivation - Creativity is defined as the tendency to generate or recognize ideas, alternatives, or possibilities that may be useful in solving problems, communicating with others, and entertaining ourselves and others - Maybe
4. 4. Dictionary.com - the ability to transcend traditional ideas, rules, patterns, relationships, or the like, and to create meaningful new ideas, forms, methods, interpretations, etc.; originality, progressiveness, or imagination - Probably not
5. 5. Wikipedia - “Creativity is a phenomenon whereby something new and somehow valuable is formed” - Yes
6. 6. Urban dictionary - “When you forget who you stole it from.” - Yes
7. 7. Steve Jobs - “Creativity is just connecting things. When you ask creative people how they did something, they feel a little guilty because they didn’t really do it, they just saw something. It seemed obvious to them after a while. That’s because they were able to connect experiences they’ve had and synthesize new things. And the reason they were able to do that was that they’ve had more experiences or they have thought more about their experiences than other people. Unfortunately, that’s too rare a commodity. A lot of people in our industry haven’t had very diverse experiences. So they don’t have enough dots to connect, and they end up with very linear solutions without a broad perspective on the problem. The broader one’s understanding of the human experience, the better design we will have.” - No

This doesn't seem too bad - there are some definitions that a char-rnn definitely fulfills, but those about understanding the human condition are out of its grasp and this is likely to be the case for some time.

How is this different from a human writing poetry?

There are many differences, but two that stand out to us are

• • Humans are writing to express feelings to other humans
• • Simple machine learning systems are trying to mimic what they have seen humans writing

and

• • Humans are connecting the words with the concepts they represent
• • Simple machine learning systems cannot currently do this. They may learn that a cat and dog are related, but have no concept of what these really are since they have not experienced the world.

Both of these differences stem from the fact that machines still have a very limited experience of the world. Some machines have seen millions of images, read thousands of novels or watched countless hours of video, but none have watched a sunset with an old friend, or been stood up on a date or eaten too many twinkies. Their life experiences have been incredibly limited and thus we shouldn't expect them to understand the world, much less what it is like to be human.

For this reason, purely generative systems, such as this one, are entertaining curiosities for computer scientists but unlikely to be valuable in the near future. Future work that seems more immediately valuable is likely to focus on computer systems that allow for interaction between human and machine .

A final favourite short poem

The Mountains

So far away the self is fast asleep
Continuously disturbed by the world.

PoetryNet, 2015