Hello, and
welcome back to the second C2HO workshop focusing on an introduction
to machine learning for creative computing with ML5.
The ML5 library allows us to utilize machine learning directly in our web
browser using P5.js.
The ML5 library also gives us access to a bunch of pre-trained models,
and today we will be playing around with the sketch RNN model.
Now the sketch again is a machine learning
model developed by Google's Magenta team.
The model is trained on millions of doodles
collected from the quick Draw game, and it's
designed to create sketches of objects and different animals.
And it uses a type of model called a recurrent neural network,
which is particularly good at understanding sequences of data,
which is exactly what a drawing is, a sequence of strokes over time.
To start, we will
use our P5 editor, which loads the ML5 library
so we can see that here
and we will add some extra things
to our index.html HTML file.
For this one,
we're going to give it a header and we're also going to give it a variable
which will show the status of the sketch art in.
So inside our body here
we will add a header
each one
and just write a sketch RNN
and close it.
So that's our header.
And we also want a variable
which we will call status
to show the status of our model.
And for now, sort of just to begin with,
we're going to give it a sort of default
text which is just loading,
loading multiple,
and then we close this one as well.
So now we can run this script and see if our headers
and status message show up and that should look something like this.
So in our sketch.js
file, we will first off, we will set up or canvas.
And since the size of the canvas,
which is this gray square, we'll sort of
vary depending on what kind of screen
you're working on, what size of screen you're working on.
I like to just set them up as variables.
So canvas height
and canvas width.
And then in our set top function,
we're going to give them their values.
So canvas
height,
we can set it for now off to these default
values.
And let's
set up a canvas variable for this one,
because we are
going to be, ah these should be the other way around.
It doesn't really matter for now, since they're the
same value, but if you guys change it,
should be the right way around like this.
And we all want the canvas to be hidden.
So let's use
canvas.hide
We want to hide the canvas until the model is ready to start drawing.
So we'll do canvas.hide.
And now we can set up our sketch RNN model.
So we'll do that.
We'll give it a name. Just model.
We use ML5.sketchRNN
Let's make this model
variable globally accessible.
Like this.
So when loading the Pre-trained model, we can choose between several
different models trained on various kinds of drawings.
So from cats to dogs and to more obscure
objects such as rabbit turtles and yoga bicycles.
And for now, we're going to use my favorite,
which is 80
Crab Rabbit Face Pig,
which is basically just an amalgamation of drawings
of crabs, rabbits, faces and pigs.
The second argument
that we are going to define
now is a callback function, which the model uses
to give us information of what happens when the loading is done.
So basically just going to call that model ready.
And we can define that one immediately.
So a function
model ready.
And what we want
this function to do is basically just show our cameras.
So canvas.show
and we want it to change this text
from loading model to something like model ready.
So to do that, we are going to use the P5 select function.
We are going to be referring
to our variable that we defined in our indexed on each HTML file.
So we gave it the ID status.
So in this select function
we write hash tag status
and we are going to give it what we want it to change to.
This is our HTML file.
We want to change it to model ready?
Like this.
So when we now run this code, we should see that this model status
changes to model ready when the sketch is loaded.
As long as everything is working as expected,
no model found.
Let's check our spelling. Yep.
I need to have the right spelling for this model.
There's an entire list of the different models, and we'll have a look at that
a little bit later.
Right now, we're just using Crab Rabbit face pig.
So let's try again.
Let's see if we can find it this time. Yeah.
So we see that this status has changed to model.
Ready? And our canvas shows up.
So since we are loading a pre-trained model,
we don't need to add any data or train the model ourselves.
Instead, we will start by creating a generate button,
which will prompt the model to generate pen strokes for us to draw on our canvas.
So we'll do that by in our set up function,
we will define a button
using the create
button
function from P5,
and we'll just give it a name what we want it to do.
Good to have that inside our brackets.
Like this.
And then we will place it on our canvas.
So we give it a position
and it doesn't really matter
that much where it is, place it slightly, slightly
down on our canvas.
Let's have a look at what that looks like.
Yeah.
So now we have a generate button, but it doesn't really
do anything at the moment, so we have to define the logic
for how this button should work, what it should do.
So we do that by
using the mouse pressed function.
So when the mouse is pressed on this button,
we want it to start generating our drawing.
So let's define a function
generate drawing.
So let's just do that right
below our set up function.
We'll call it generate
drawing.
Let's fill this one in.
So when the Generate button is pressed, we want it to.
First off, we want to clear whatever was there before.
So we can do that by calling background, which will basically just
re draw our background
and give it the same value as is default.
And then we want to reset our model.
So if it's drawing and we press generate, we want it to sort of just reset
and start over.
And we also want to define the location of the first pin stroke.
We want it to start
in the middle of our canvas.
So let's define a variable X and a variable Y.
So the middle of our canvas is our canvas,
which is divided by two and our y location
is the canvas height
divided by two.
This all centered, it's in the middle or canvas.
And then finally, we want to use the sketch
on end's own generate function
to get our first sequence of pen strokes.
So this function requires
us to posit a function that we define,
which should describe how we should handle the results.
So let's call this function.
Got stroke.
And we can define that one just below.
It's a function.
Got stroke.
And what we want this function to do is that if
there is an error cause got stroke will return
or our model generates
function will return either an error or a result.
So we should give it these arguments.
Let's just call our results for stroke for this one.
And then the first thing we want to do
is say that if there is an error,
we would like to log that error
below on our console.
So we just call console dot error.
That simply makes it so that when this message is logged,
it will turn off in red.
So it will catch our attention
a little, a little easier.
So just console and then log this error.
And if not, then we have our results.
So what we're going to do now is
we are going to define a stroke
path variable,
which we will be using to draw our drawings.
Now, since we will be using this,
these X and Y initial locations and our stroke path,
different places throughout our program, we want to define them globally.
So let's do X and Y
and our stroke path.
So let's test this out.
So there's a small spelling error that got stroke is our function.
Let's save it and run it and let's see what happens.
So everything is running smoothly for now.
Model is ready when we press generates still, nothing really happens
because we're not doing anything with this stroke path variable yet.
So before we fill in our draw function,
we should have a look at
what is actually being returned by the model.
See what we can do with it.
So let's use console.log
and we want to have a look
at this value as the returned results.
So if we now press play,
we will have a look at the results we press generate.
And now we can see that down here in our console window
this is being printed.
And what we can see here is that the sketch on end
generates the change in position and the state of this pen.
Now the pen can be in one of three states,
it can be up, down or end.
And so what we want to do
is to place our most recent results in this variable stroke path,
which is what we are doing in our results function.
So now that we'll be populating our draw function,
we should make this this pen variable.
Just remove our background call here
should make this pen that variable available globally.
So let's call it previous
pen and let's give it
just a default value of down to begin with.
So when the pen is down, we are going to start drawing.
So inside our draw function, we are first going to check
if there is something in our stroke pass variable so we can just do if
stroke pass.
And so if stroke path is true,
then sketchRNN has something new for us to draw.
So then we want to do another check to see if our pen is down.
So if previous pen
is down,
then we want to draw something.
If our previous pen is up, then there might be a change in position.
So the pen might still be moving, but we don't want to draw that line.
We only want to draw while the pen is down
to this.
So if the previous
pen is down, then we want to actually draw to our canvas.
So the way that we do this is we will set some drawing variables from P5.
So our stroke, which can contain the color that we want to draw with.
So let's give that as an RGB.
So let's say we want it to be like a greeny blue,
so let's give it no red and
equal amount of green and blue
and then we want to
set the stroke weight, which is basically just the thickness
of the line that we're drawing and just set that to
something like four.
And then we want to actually draw the line.
So we use the P5 function line,
which takes in four arguments, the
X and Y positions of the beginning
of the line and the X and Y positions for the end of the line.
So as we defined
in our initial generate drawing
function, here we have an X and a Y position,
which should be the first position of of the pen.
So let's start that X and Y
and then we should end at X
and then we're going to add the
position
given to us in this stroke path from the model.
So as we can see down here, we have the difference in X and the
difference in Y.
So strokePath.dx is
it is added to our x variable and
we add to our y variable the same,
but then we do dy.
So this should be able to draw a line
when we generate the drawing.
So let's save the model and have a look.
So let's reset things here.
And we will also
comment out this console.log.
And the next thing that we need to do
is to update our
X and Y position to be this new end of this line.
So we set that X is equal to a
strokePath.dx.
Actually we should
summarize it so we can do plus equals.
So we increase the amount, we increase X by strokPath.dx
and let's do the same for y valuable.
and we also need
to update the position or the state of our pen. So
that's two
previous pen was our variable
and now we should get its state
from our stroke.
Let's see.
Strokepath.pen
So we can also use this
the state of the pen to figure out if the model is still generating something.
So let's add another if statement.
So if strokePath.pen
is not done yet, so is not equal to end,
then we want to keep going.
So we want to empty out a stroke path.
So we set that to null
and we want to call the generate function again.
So model generate.
And we want to,
as usual, add this got stroke
function that we defined to handle these results.
And otherwise,
if the model has reached its end,
the end of its drawing, then we simply want to reset the model.
So model.reset.
So let's have a
look at what this crab rabbit face
pig model will generate by pressing play.
Our model is loaded and we can press the generate button
and they would go a crab rabbit face.
We can press, generate again
and get a new drawing
and keep going.
I'm more of a crab this time.
Instead of generating a single image, let's change some things
so that we can keep generating new drawings to fill our canvas
So by adding this call to background
inside our generate drawing function, we are clearing the previously
generated image and we draw a new one every time we press the generate button.
So if we instead want to fill our canvas with different drawings,
we can remove this line of code and make a small change to the location
of the initial stroke of the model.
So first we are going to
comment out this background function call
and then instead of setting the drawing so that it always starts
at the center of our canvas, we are going to use
the random function
from P5.
And what random does is that
it gives us a number in between two values.
So we are just going to set these two values.
So from the top
most corner of our canvas
and for the x, that will be some number between
zero and the canvas
width.
And for Y we'll do the same.
A random number between zero
and our canvas height.
And to make it
a little more interesting, let's make some of the variables
that we use for the style of the pen stroke random as well.
So we can also set that for each drawing by changing it
inside this generate drawing function.
So let's define some random colors.
We'll do that by using our IGP. So
let's set order to a random value
between zero and to 55.
For example,
and let's do the same for
green and blue, so we can just copy this.
Oops,
I go to r, g,
and b
I think for me personally, I'm a big fan of the greens and blues,
so I'm not going to allow too much red just between zero and ten.
Now the other value that we can change, so these little affect or
strokes, which is the color and our stroke weight.
So the other variable is this stroke weight
or sort of the the thickness of the lines.
So let's call that just the weight
W for short and give that
also a random value between
zero is probably too small.
We don't want to allow it to be too small, but for as a standard
and then let's make it go all the way up to 15 in a weight.
And now we update that
inside our draw function.
So stroke
should now be RGB instead
RGB and our stroke weight
should be w.
And let's just reset everything so that we can
start fresh and see what happens.
TSo we press play. Loading the model
and we get an error.
G is not defined.
Let's see.
I thought I did that.
RBB.
So RGB, save and run again,
There we go.
So now we are
adding to our image
every time we press generate.
I think we should make our canvas slightly bigger.
At least I will.
Let's make it 600 by 600
pixels instead.
And also I see that
our background is not being drawn out.
We don't want to overwrite it
every time we hit the generate button.
So let's
set it here instead in our setup function,
it's a background 220,
which is just a grayscale value for a background. Yes.
So now we can see when the model is ready, the canvas starts its better size.
I think for
for this screen.
So this is our
crab rabbit face pig model.
And you can see we're definitely getting some faces.
That's maybe a rabbit,
definitely a crab.
Yeah.
Now we can fill our screen with all these strange images.
Now, so far, we have just been using this crab rabbit face pig model,
but there are a bunch more versions of the sketch on and model available.
So there are actually over 100 available through ML5.
So let's create a list of all the different models
and then choose a new model.
Every time we press the Generate button.
Now again, we are going to use the P5 random function
so that we don't know which model is being loaded.
And this way you can kind of play a game of Pictionary against the sketchRNN
by trying to guess what it is attempting to draw.
So first off, let's load this list of models.
So on the ML5 reference page for a sketch RNN
we can find a link to all these different models.
And so let's just copy this entire list,
and we're going to put it into our sketch.
Let's call it
simply a list of models
and paste pasted in
so you can maybe got the double up here
that would go.
So now let's
change our generate drawing function
to load a new random model.
Every time we press, generate.
So let's see.
First off, we can add back this call to background
so that we are always showing a new
fresh background for a new model.
Actually, let's change this random position back.
Also back to our
canvas size
or half our canvas size so that it's always in the middle.
And then we are going to load our new model here.
So a good policy,
I think, is to use this
model status to update that when we are loading in a new model.
So let's use that one again, which is hashtag.
Um, let's see, hashtag not select
hashtag status,
dot HTML.
And this time we want it to say
new new model loaded,
something like that.
And then we are going to redefine model.
So model now equals ML5.sketchRNN,
and usually we would
have our the name of the specific model we want to use.
But now since we want it to be random,
we are going to pass our list of models
and we are going to inside these square
brackets, we're going to put our random function.
Now we want this to be an integer
so we can make sure it is by wrapping
this random function in an int function
and then it goes random.
That should be some value between zero
and the the length of this list of models.
So then we can use
list of models.length
like so.
And we also want
to pass our model ready function.
Other than this, we shouldn't have to change anything else
in our draw function that should function just as it has been.
So let's clear everything and
try to run it.
Okay, so let's
press play and see how this goes
and press generate.
We see that a new model is loaded and ready
and it has drawn something
about maybe one of the items on this list.
Let's try a new one.
New model is loaded and ready.
This time we can't really see it.
So what I think we should do about that
is that in our draw function, we should
when the model has successfully predicted something,
we should reset not just the model,
but the positions of X and Y.
So let's set them back to canvas
width divided by two
income as height, divided by two.
Let's reset again.
So we're trying again.
New model is loaded and ready
and it draws something.
Let's try again.
Now we're consistently
getting the first stroke starts at the center.
New model loaded can take
a little bit of time That's a good pineapple
for it to load can take some time
but now it's consistently drawing this.
I have no idea what it is
or so.
Finally, let's let the sketchRNN receive input
from our pen strokes so that we can draw collaboratively with the model.
And to achieve this, we can create a separate stroke path,
which is user input, and we'll do that inside our draw function.
So first off, let's make
a global variable underneath our list
for our strokes,
and we're going to use them to sort of seed the models.
So let's call them seed
seed strokes
and it can be empty to start
And now inside our draw function,
we are going to check
if the mouse is pressed, because if the mouse is pressed,
we want to capture our strokes, the use of strokes.
So if I'm going to use the P5 variable
mouse is pressed, which is either true or false.
So if the mouse is pressed,
we want to create a stroke, which is ours, so let's make it look different
from the sketchRNN strokes,
so let's make it black, for example.
And our stroke weight
can be
random
somewhere between four
and nine.
You can play around with this.
It's really up to you what you want to do.
And now we can create a line
which again takes four variables.
And now we're going to use our mouse position as the positions to draw a line.
So we're going to use some system variables
which are called mouse X and Y and P, mouse X and Y.
So we'll start from P, mouse x and Y,
So p mouse x,
which basically it's a system variable
that has at any time
contains the position on the canvas where our mouse was at the previous frame.
So its previous mouse X and previous mouse Y
p mouse y
and we want the end to be mouse x
and y
not moved
like so.
Now since these are going
to be input to our model,
we need to create a stroke path
with a d, x and a d, Y and a Penn State.
So we're going to call that use a stroke.
So let's say that user
stroke
and we're going to create an object
with some attributes.
So we want d X
in the user strokes to be equal to the difference
between the current position and the previous position.
And then we want to add a d y,
which I'm sure you can guess what that value should be.
So that's previous or current y
minus previous Y,
and then we need a pen state.
So we'll define pen and we'll set that
to down
because we know that the mouse is pressed if we're doing any of this.
So that's our user stroke.
And so the last thing we are going to do is to add
this user stroke that we just made
to seed strokes.
We can do that using push,
which just adds this object to our list.
So let's add user
stroke
and let's see
another
spelling error there.
Mouse pressed.
Let's try again.
There we go.
Now we want the model to start drawing where we left off.
So let's remove our generate button and instead create a function
which we will call start sketchRNN, which will then be called
when a release of the mouse is detected inside our canvas.
So let's remove our generate drawing function
and will replace it
with our start
sketchRNN function instead.
And so we want to start where
where the user left off.
So we're going to change our X and Y values that x should be current
position of the mouse and Y
should be also current position.
The mouse in the Y dimension.
And then we can call again
model.generate.
But this time we not only want to send our function,
our callback function got results,
we also want to send it some input to base.
It's generation on, so we'll send it
our seed strokes
and now we can call this function
from inside our model ready function when the mouse is released.
So we still want to show the chemist when the model is ready.
But then we also would like
to use canvas dot mouse released
and we send it what we want it to do
and the mouse is released, which is start function.
Call our function start sketchRNN
So before
we try out this new implementation, we should remove this generate button
since it's calling a function which no longer exists.
So let's just delete these three lines
and we should also change the stroke values
so they are no longer random.
This was all set in generate drawing.
Let's change it.
Let's give it a stroke weight
And just give it a color.
This is sort of my default
color that I always use.
I don't really know why.
Again, it's up to you guys to play around with this.
Okay, so now let's
try and draw out
we have an error that got results is not defined.
And that's true.
It's not called got results.
It's called got stroke.
Sorry about that.
Let's reset.
So let's see if we can now draw something
and the RNN continues our drawing.
Something like this.
Now, the sketchRNN
model will use our previous pen strokes as a starting point.
And it will it will attempt to continue
the drawing in a sensible way.
So if we are loading the face model, for example, sketch on end
should try to create a face based on whatever we draw.
But sometimes, as you can see, the input we supply can be, I don't know,
confusing to the or an end to the model, and we get sort of nonsense back.
So I think a useful thing for us
to implement here is a clear button
so that we can clear the canvas and reset the model.
So first let's just clear screen
and in setup we are going to add a new button
using create
button
and let's give it
the name clear.
We're going to place it
around the same spot
as we had our generate button
and we need to define its logic.
So what should happen when button
has a mouse pressed event.
So let's define a function clear
drawing,
so clear drawing.
And what we want this function to do
is to again call background.
So we know that that
resets
our background and races.
Everything that was on canvas.
And we also want to clear our seed strokes variable
so that the model can get
a fresh
stroke from us.
Let's see.
There we go.
And we also want to reset the model for good measure.
So model dot reset.
So let's see what that does.
So we now have our clear button
we can draw and the model will continue
our drawing
and draw
some small faces, it seems, and then we can clear it.
And then the model is
the canvas is cleared and the model is ready for some new input.
So the last thing that we are going to do is to add a dropdown menu
so that we can select different models based on what we are drawing.
So first let's add our global variable to indicate our dropdown menu.
Let's just call it dropdown
and then we can initialize it
inside or set up function
to contain all the different models from our list.
So dropdown
to initialize it, we use create
not button, but create select.
We need to place it somewhere.
So let's just
dropdown dot position.
Let's just place it
slightly lower than our clear button
and then we're going to use
a for loop to populate this dropdown
menu with all of these different models from our list of models.
So the way we do that is we say for
and we need a temporary constant,
call it just the value of our
list of models.
And inside this
for loop we are then just going to call dropdown
dot option
and give it our value.
So for each value in our list of models, we're going to make it an option
in our dropdown menu.
So we also want to add a function to handle
what'll happen when we select a new value from our dropdown menu
so we can do that using dropdown dot changed,
which is a P5
solution for making these sort of compact
function calls.
So what we're going to do is do dropdown
dot changed and then
we put these double parentheses
and a little arrow to point to our function
and we're just going to bake this entire function into this
small sort of line of code.
So what we want to do is we want to reset the model
when the dropdown menu changes
because we are going to be changing to a different one.
And so let's make a variable model selected
and then we need to access the current value of the dropdown menu.
So dropdown dot
value.
This always refers to the current
value of the dropdown menu
and then we will initialize a new model.
So we will say that model is now equal to
ml5.sketchRNN
this and
we will be using the model selected
and of course
we want to send in our callback function model
ready for good measure.
There we go.
And then outside this changed
dropdown dot changed function description.
If you remember to close this off
one last thing we want to do
is at the end of our
setup function,
let's actually move this
initial line which sets up our model for the first time
and it sets up this crab rabbit face pig model.
And instead we want the initial choice of the default
when we start running our program to
be set to
whatever the dropdown menu has as its current value.
Instead.
So the first element should be alarm clock.
So we want alarm clock to be sort of the default model
that loads when we start our program.
So there's our alarm clock
kind of let's try out another one.
For example, book,
we can clear it
and I'll do this
not a not a great addition, but
let's try another one.
Angel, for example.
Yeah, pretty good.
And so that's it for now.
You can take this further, if you like, by checking out the links on our website
under the header, going further and thanks so much for joining.