Manuel Fuß

Hey.

So I'd like to present a new project of mine, which is an assistant.

I built an assistant which had the actually the purpose to get my commands via voice and then make any change on my website.

So I have a sample website here, which is obviously no live website.

Well, so this is the page I like to work with.

This is my code structure.

And just to show you actually what this is all about, I'm about to start my assistant by a shell command and right now it's running and it's asking me or waiting for vocal input from my site.

So just let me see.

First of all, my website files are all in this folder, so there are also some not needed files just to have a sample stack actually.

So this is what you see here.

This is the index HTML.

You can see it here, it's just a basic one.

I will now remove those as we hopefully will not need it.

I will just show you an example and later on we talk about why it worked, why it maybe doesn't work, and what's it doing in the background.

And I will tell you about openi threads, which is a pretty cool thing, which yeah, honestly, nobody really talks about.

Maybe it's just interesting for developers.

And I missed this talk, but I also had a look in a discord in the open AI discord, and there was not much talking about it anyway.

So let me show you.

First of all, I am going to try to get this style Css implemented here in the HTML.

Please implement the style CsS file in the head of the index HTML file.

So all the styling comes out of this included file in the future.

All right, so it got my command in the end.

It's now checking.

Okay, what happened here?

And eventually building shell command written to file and executed a shell script which.

style Css.

So what happens here?

And this happens actually, to be honest, all the time, and I'll tell you later why this is the fact it implements.

So this works, this worked fine, but it doesn't implement it as it should have been in the hat, but it just puts it at the end of the index HTML.

So what happened here is it built a shell script which is nothing else than this.

So it just adds this line of code to the existing file and then executes this shell script.

So it works more or less as this is definitely not right.

Well anyway, we work with this from now on.

Another example to show you how you can see what is really happening here.

Make the background of the index HTML file red within the CSS file, you just include it.

All right, so he did anything.

Let's see.

command written to file and execute it.

Perfect.

And now you will see what the threat technology does.

So the threat technology is like a collection of all the messages you already said and all the replies you already got.

So let's say actually you remember I said the system.

Okay, now make the background red.

So my next command would be a little bit shorter.

Now make it blue.

So what happens now is it checks all what was already said before.

written to.

executed and takes this as a reference for what?

Yeah, for what the new command refers to.

Make it wide again.

So this actually doesn't work with the background.

with shell command written to file and executed.

done within a shell script.

So just let me give you another example.

Add a file called Manuel HTML build a link in the index HTML which links to the new file.

I don't know.

So to be honest, it works approximately 50% of time.

All right, shell command written to file and executed.

So it works.

The page is written.

Now we can also style a new page or fill a new page, implement the style CsS as well in the Manwell HTML.

So future styling will be read from this file.

And then in addition add a box, a black box in the new file with the size 200 times 200 pixels.

So let's see if it works or not.

I'm just waiting for the moment when it doesn't anymore, but right now it works pretty fine.

Which, to be honest, shell command written.

so it makes me a little bit, whoa.

Okay.

I'm surprised.

Didn't expect that.

Now take this black box and make it horizontally as well as vertically centered in the page.

Yes.

I'm happy as this works because to.

shell command written to file and executed.

Now also implement the script js in the manual HTML file.

Um, yeah, as I said, always makes me so happy when it works because it, it had times when it didn't.

All right, so let's come to what's happening.

Actually.

First of all, let's have a look at threads.

Threads is a technology from OpenAI which enables you to build assistants.

So assistants that have like a memory where they remember as you saw what was said, what was done, actually what happened in your complete conversation.

Those threads are built.

I built a thread, no, sorry, I built an assistant.

Within the assistant, I put a thread, you could have several threads in an assistant.

So several people could have shell command.

executed, whatever.

What did you do?

Well, anyway, could have the same once more, many people could have one assistant, but different communication.

And each communication is saved in one thread, and in one thread there is a bunch of messages.

So everything, as I said, everything was said, everything was replied, everything's in there .

And everything can be considered on future messages, which is pretty powerful.

So as I showed you, you can, for example, start building websites, or you can just tell your assistant something in the morning and tell them more and more over the day, tell them what happened, and in the end of the day you could say, all right, just summarize what happened today, or what was it, what I told you then and then.

So, pretty cool.

Okay, so I use the thread, when I start, my assistant to first of all put a lot of information in it.

So I put the whole folder, the whole website folder in it on code base.

So I say, grab all files, grab all folders, grab all code, put it in your save actually in your thread, and then let's talk.

So what happens is I give him like a memory that makes him act like someone who knows exactly about my website.

So I could ask him questions like where in which file is this happening?

Or what is this file doing?

Or things like that.

And then it answers, of course, and if it answers with code, so like what we did when I asked him, could you please add this box?

Could you please add code in a script, add a new page, link the page, whatsoever, then it builds it in a shell format, shell code format like this.

And after the file is built it gets executed and put in the executed folder.

So this is actually what happens.

And with that I could, yeah, not perfect yet, but I could, as you saw, manipulate files, manipulate text whatsoever.

I give another example, write a possibility in the Javascript file, which makes the black box change its color every 2 seconds smoothly without an end.

Yeah.

So as you see, I could just extend files, I could let it write.

shell command written to file and executed, and if it's working.

Cool.

So it's not limited to any language.

The thing is why it sometimes doesn't work, and it works pretty fine right now.

The thing is why it sometimes doesn't work is threads, even though it is written here for example, and on other places that it can use GPT four as a model, it can't.

So I had to do some research in what was written in and on blogs and forums and also in discord channel from open air itself.

There's just this single model that can cope with threats, and it's a GPD three five model, which is, yeah, it's okay, but it's not that good, it's not that precise.

So it took me, I think, one third of my complete development time to build a prompt, which makes it possible to do this in a more or less reliable way.

And this is a good run right now.

The next one might be that it completely fails and nothing works.

So this is unfortunately also because of the model.

But to be said, I enjoyed this project pretty much because I loved to work with the threads, which is such a powerful tool.

And actually it's like kind of a magical thing when you tell something to the assistant and ask something 10 minutes later and it still knows what you're talking about.

It's pretty cool, really.

And also, when you have a run where 20% work and 80% of the responses are rubbish, it's always such a good feeling when it works, it's sometimes a little bit like an addiction.

So I had some runs, some test runs where it was just failing and failing and failing, and then you have the one correct response and it feels so good.

As I said, like an addiction, because nine times of ten, I used to get so angry and I used to hate this tool and I didn't want to do anything with it anymore, but the one time felt wonderful.

All right, so threads is fun.

Threads is still in beta, which is also written here, but it's working pretty fine most of the time, except for the model, which is not that cool.

I put this project on my GitHub.

So this is a repository, it's public, you can just download it, use it, try it, and have some fun with it, as it is really fun.

Just to, yeah, also with the voice commands and voice response and things like that.

Thanks to elevenlabs at this moment.

And yeah, what else API did I use?

Of course, OpenAI GPT-3 five within the threads.

I used to also use GPT four for better answers, which made it actually too slow because the answers were good, but it took too long.

I used a whisper API from OpenAI to descript what I recorded by voice.

This is mainly it.

It's a pretty cool project, as you can have a lot of fun.

You can.

All right, just one more thing.

Extend the Javascript code so the box also keeps turning like a globe around itself pretty smoothly.

I don't know if it works.

Last time I did it within CSS, like with a transition, and this was pretty cool.

It worked.

Shell command written to file and executed.

You don't know how happy I am at the moment.

Yeah, this is the cool stuff, actually.

You can just tell them to do whatever.

And yeah, you could also go again to the index and say, now add a short text about Elon Musk right after my contact information on the index HTML file.

This is pretty open.

It could do a lot.

It's definitely not ready for shell Command.

executed, as you see.

is fine, and blah, blah, blah.

But why?

So you see it just added it here and to be honest, it's also, and this also work.

It's also able to just change text.

I can show you one more time now change sample website as a title to a super duper mega website.

I don't know if that happens.

Maybe.

Shell Command written to file and executed.

can't open file in the index HTML.

Change the text sample website to superduper website.

This is what I mean sometimes.

Of course it could also be up.

shell command written to file and executed.

Yeah, so we can check what he tried to do and what didn't work in place, blah, blah blah.

Yeah, I don't know.

Okay, anyway, um, and now do this via Javascript.

Maybe it works this way.

Can't open file.

command written to file and execute it.

as you see, sometimes it's just stuck.

So I can just restart it.

Restart it and let's see what happens.

Now change the text sample website in the index HTML.

I correct in the index HTML to the text superduper website.

Well, let's see anyway.

Shell command written to file and executed.

Do it again and use Awk instead of sed.

Yeah, so this might work now.

All right.

to file and execute it.

Yeah, what else to say?

It was fun.

Threads is pretty powerful.

Building voice assistants is pretty powerful and a lot of fun.

And I right now have an assistant to build a website, which is almost done, so want to see this again?

Yeah, I thought so.

Okay, so, all right, have fun with it.

Sorry.

I will put the link of my repository in the description.

Feel free to download it. And I would be pretty happy to get some feedback from you and some experiences with threads with this tool with assistant itself and yeah, let me know.

All right, thank you very much.

Bye.