Ilan (00:00)
It certainly was. This was not the tortoise and the hare.

David Vuong (00:03)
Yeah.

Ilan (00:08)
Don't know if that made any sense.

David Vuong (00:11)
I was thinking, I was processing like sure, sure, whatever.

Hey everybody, welcome to prompt and circumstance. I'm David. And today we are going to be rounding up four vibe coding tools.

Ilan (00:20)
and I'm Ilan

David Vuong (00:35)
All right, so today we are going to be looking at Lovable, Replit, V0, and Bolt, all in competition with one another at a track meet.

Ilan (00:46)
This track meet is going to compare these tools across dimensions that are important for product teams.

The three events we've created for you are the 100 meter dash, the long jump, and the pole vault.

we will be sharing the prompts that we used for these events, and we're going to be deploying the winner of each event so that you can test it out yourself.

Ilan (01:08)
But first, let's hear from one of our sponsors. Do you have a side hustle? Why not? If you're worried about validating your market or building a tool that people can use, our friends at Co.lab have just the product for you.

With Co.lab Validate and Build, they'll go out into the market, understand your users, build a first version, and find you your first paying customers. And they'll hand you a working product so that you can take it and run your side hustle from there.

It's so easy you'll be kicking yourself that you didn't do it sooner.

Let them know David and Ilan sent you, and you'll get $250 off.

Ilan (01:44)
David, can you tell us a little bit about each of them?

David Vuong (01:47)
That's right, so the 100 meter dash is going to be testing raw speed.

And with the long jump, we're going to be looking at how much depth and precision these tools are able to give the software that they build. And then lastly, with the pole vault, we're going to be giving it a giant PRD where we're going to be looking at how much complexity these things are able to handle and how well they do.

Ilan (02:10)
Well that sounds super fun, let's get right into it.

Ilan (02:13)
But before we get to that, let's hear from one of our sponsors.

We're excited to be sponsored by our friends at Querio. Do you have too many data sources and not enough time or know-how to link them all together so that you can get data informed answers to your product questions? Querio has just the product for you. Their AI agent sits on top of your data stack and allows you to ask natural language questions to get data responses from your various sources. I personally use it.

and it saves me hours per week And just for our listeners, they have a special deal. If you click on the link below and let them know that David and Ilan sent you, they'll give you two months free of Querio Try it now.

David Vuong (03:02)
All right. So our first event is the hundred meter dash.

So this is going to be an event which tests the speed that these tools are able to generate something workable. Now, what we are going to be doing is splitting this into two heats. So first, we're going to be looking at Lovable and Replit, and then we'll be comparing Bolt and V0, and at the very end, we will look at their times and see who is the winner. So the problem that we are going to be solving here with this scenario is...

The prompt and circumstance website isn't as great as we would like it to be. So we put together a prompt, a lengthy prompt that is going to describe what it is that we are looking for out of our website. We're going to give it context that it's going to be for a podcast and we're going to give it a logo for which it can determine the style to theme that website.

Ilan (03:58)
So David, basically this is a test of impatience for product teams.

David Vuong (04:03)
It absolutely

All right, so I've got the prompts queued in to both Lovable and Replit. I've also given them the logo for which to determine the style. And now what we do know about Replit is that it will make a plan first. So we are going to let it do that first before we start the timer.

Ilan (04:22)
That's awfully nice of us to Replit. Imagine if one Olympic sprinter wanted to tie their shoes at the start of each 100 meter dash and so they fired the gun but then let that one sprinter tie their shoes before everyone ran off.

David Vuong (04:25)
Hehehe.

That is quite nice of us.

All right, let's start the timer and we'll get these going. Let's go.

Ilan (04:53)
Replit took a second before it told us that it's designing the visual preview, but wow, we already have the beginnings of a visual preview from Replit.

David Vuong (05:06)
I think that's the difference with the user experience for Replit is that they will show you the app in construction.

Ilan (05:14)
Yeah, did notice that along the way that it's kind of cool in Replit you can see the whole thing being built as it's thinking.

David Vuong (05:23)
I think that that's a great UX takeaway for people building any kind of apps where, you know, it's ⁓ okay to show something transitioning into its final state because that does reduce the amount of impatience from the end user.

Ilan (05:39)
Yeah, absolutely. That's actually a difficulty that a lot of these tools are facing, especially as I get into reasoning models that take longer to think about the problem. In the modern world, we're used to page loads in milliseconds and getting information in seconds. And so when you're working with LLMs, that will take a few minutes to think through a problem.

or with these vibe coding tools that will take a few minutes to process your problem, it's important to give the user cues so that they don't lose their patience with your product.

David Vuong (06:20)
Absolutely. They are also creating a tremendous amount of code in the back. So they are working away at the problem.

Ilan (06:29)
That's right, I think it's about showing the user that issue. It's kind of like, you if I asked you a complicated question, David, you probably wouldn't just sit there for two minutes silent before answering me. You know, you'd give me some cues along the way of what you're thinking, like, well, what if we approach the problem this way? No, that won't work, but maybe I need to like think about this other thing. ⁓ okay, that's a good idea. So what I'm gonna do is, and then you'd kind of launch into it.

David Vuong (06:58)
Yeah, now these tools, what they're doing is they are telling us what they're up to in the little status bar here, which is quite nice. Likewise over here with Replit. ⁓ and I heard a little ⁓ ding and ⁓ it looks like Lovable is done.

Ilan (07:16)
One thing I just noticed in the Replit UI is the get notified when your app is ready button where you can scan a QR code and I guess maybe it'll send you a text or something to let you know that, hey, or a push notification to let you know, hey, you can check your app out now.

David Vuong (07:36)
Yeah. So this is another difference with Replit where they have a native mobile app that you can use to not only vibe code, but I suppose getting notifications that your app is ready.

Ilan (07:49)
It's interesting that Replit is still taking this time to build the site.

I think at this point the stopwatch is broken.

David Vuong (07:58)
Yeah

we wait for Replit, we can explore what Lovable ended up creating.

Ilan (08:05)
Yeah, so what did we get?

David Vuong (08:07)
It looks like it created a pretty decent website where the colors are really taken from the logo. We can see down here, it did hallucinate some example episodes, but that's okay. We haven't hooked that up to any real data yet. Does look nice though. ⁓

And down here it gives us some ⁓ examples of ways that we can position the website. And look, there's some calls to action over here.

Ilan (08:35)
Beautiful. Love it. What about the menu?

David Vuong (08:38)
The menu has the essentials. The episodes. That's an episodes page. That's quite nice with some hallucinated episodes. That's all right. And it has an about which is talking about us and who we serve and.

Ilan (08:55)
What did it say about us?

David Vuong (08:57)
Well, it just put placeholders there for us to fill in. And it looks like Replit is done. And ⁓ now we're only testing speed. Otherwise, I'd have a bit of an issue with this choice of font color. But here we are, we're done. looks like ⁓ I like the gradients choice here. That's quite nice.

Ilan (09:06)
while we were talking about it, yeah, Replit finished.

David Vuong (09:26)
I gotta say that the footer of the website looks a little bit more professional. That's good. It ⁓ ended up with the three sections of the site just as with ⁓ Lovable And ⁓ surprisingly similar content here, but it did take a little bit longer.

So why don't we switch over to the second heat.

Ilan (09:49)
Okay, so we're here ready for the second heat. Same as before, we have the prompts queued up with our logo set in V0 and in Bolt. So we're gonna get the stopwatch going and let's see what happens.

David Vuong (10:08)
Let's go.

And right away we notice a difference in user experience. know, Bolt compares to Lovable and Replit where it immediately has the split right away. Whereas it looks like V0 thought about it for a little bit and then began with the split. However, all four tools ended up with this left pane, right pane, code and preview setup.

Ilan (10:38)
That's right. And I'll say that...

V0 and Bolt clearly choose to have a loading page on the right side. They don't show you the preview as it's built. But V0 splits the difference a little bit because it actually shows you the code that it's writing as it goes through.

David Vuong (10:57)
I do like that split of the difference because, know, you don't need to render something necessarily that's half done. need to show that something is being done and showing the code pass by gives me confidence that, wow, you're doing a good job. You're doing lots of work. So I'm happy to wait.

Ilan (11:09)
That's right.

That's All right, with that Bolt is telling us that it is done, waiting for it to load. I'm not going to call it, I'm not going to hit the stopwatch until it's done and here we are. Bolt is finished. Let's see when V0 will come through.

David Vuong (11:31)
Wow.

Ilan (11:38)
One thing to... Ha ha, that's right.

David Vuong (11:38)
I suppose that's true to its name. With Bolt being

ultra fast, that might be the winner.

While we wait for V0, maybe we could explore what Bolt has made.

Ilan (11:51)
Definitely. Well, as you said that V0 also finished, but let's take a look at them in the order that they finished. So right from the top, we have a mobile responsive page from both and looking at Bolt, we got a little bit of a darker theme than what we saw from the other tools, but still pulling cues from our logo.

David Vuong (12:13)
Mm-hmm.

Ilan (12:19)
It was just pulling from the darker elements of the logo. And we have a hallucinated episode.

And there's some information about the podcast as we scroll down the page. And a professional-looking but compact footer.

David Vuong (12:35)
So if you click on, if you click on view episodes, does that take you to another page? It does not. Okay. Well, that's right. It cut a corner. All right. Let's look at what v0 made. I appreciate the color of the scroll bar that, that it changed. I think that was, that's unique to v0.

Ilan (12:39)
No, it does not.

So it was the fastest but it cheated. ⁓

That's true.

That's right.

It's come up with an appropriate color scheme based on what our logo provides and puts the logo nice and big front and center as the user comes in. It also gives a call to action and an ability to listen to episodes right up at the top, along with some hallucinated episodes.

And it also gives some values for the podcast as well. Actually very similar in styling to Lovable. Drawing inspiration from the same sources I'm sure.

David Vuong (13:37)
Yeah, it's almost as if they might share a foundation model.

Ilan (13:40)
Hahaha

And as well, it has the professional compact footer. And it did create all of the pages that we've asked for. So an episodes page where we can see a few different hallucinated episodes

David Vuong (13:56)
I like the highlight of the latest episode. That's, that's different on here.

Ilan (14:00)
Yeah, that's a great point. I also really like the UX cues that it's built in along the way.

David Vuong (14:07)
That's quite nice. Yep.

I like it.

Ilan (14:12)
So with that, let's go back to the studio and let's announce the winner we'll move on to the next event.

Ilan AI (14:23)
Alright, congratulations to Bolt on winning the first event.

Ilan (14:26)
Well with that, that was super exciting. We saw that Bolt was our medalist in the 100 meter dash. David, do you have any final thoughts after seeing that?

David Vuong (14:37)
It was fascinating to see the split between first and last place. That was a significant difference in time of split as we look at these other events.

Ilan (14:52)
That's true, David. How do you think about speed in terms of the overall value that these tools bring to product teams?

David Vuong (15:07)
I think that even though the slowest tool took a few minutes, it is still tremendously faster than a human doing it. I mean, if we were to go to an agency and pay thousands of dollars, maybe, ⁓ and have them draft up a website for us, that would have been for sure last after all these tools.

Ilan (15:33)
100%. With that, we'll get to test these across some other dimensions that'll be important for product teams. And so let's get into the next event.

David Vuong (15:44)
Let's do it.

Ilan (15:45)
For the next event, we have the long jump. And for the long jump, what we are testing is depth and precision from these tools. So what we had them do is we had them create a replica of the game Snake, as well as an algorithm to autoplay and find the best path to the food. And we gave them each 10 tries to autoplay

the game and see which one came up with the highest average score.

David Vuong (16:21)
Snake makes me think about the Nokia days, back when it was the main game on the phone.

Ilan (16:24)
Hahaha

Maybe

I should have given that as a styling cue in the prompt where, you know, you have to show it in a Nokia, whatever, 8300 phone image.

David Vuong (16:36)
Yeah.

Ilan (16:39)
well here you have it David. This is the first autonomous snake game made by V0 and the prompt, it adhered to it pretty well. I'll say that none of these tools was able to one-shot this prompt. All of them required between four and five prompts to fix bugs or ⁓ actually listen to what I said.

David Vuong (17:03)
here this one is V4.

Ilan (17:05)
That's right. One funny thing is that both V0 and Replit tried to cheat and give themselves a larger grid than what I had specified and helped the snake along by giving it more space to move around.

David Vuong (17:20)
That's funny.

Ilan (17:22)
But it turned out pretty well. We've got this nice modern gradient here. I don't know if I would choose these colors myself, but it's not too bad. And it gives you the nice little ⁓ pathfinder showing how it's decided to get to the next piece of food.

David Vuong (17:39)
I like that. That is a delighter.

Ilan (17:42)
Definitely. And one of the things that I did for each of these is I gave them the initial prompt and then afterwards, after it started working, I asked them to add a turbo mode so that we could test. And that works pretty well, except for that our snake committed seppuku as I turned on turbo mode.

David Vuong (18:02)
Have a sneak. ⁓

This is great.

Ilan (18:09)
on.

David Vuong (18:09)
What else

was part of the scope?

Ilan (18:12)
The full scope was to create the Snake game in a 20 by 20 grid to add the scoring

as well as add tracking for the current session. So you can see what was the high score, what was your average score, and the duration of the current game. They were told where to put each of these elements on the page, ⁓ as well as some of the coloring cues, for example, a green snake, red food, et cetera.

David Vuong (18:45)
I see that you also had mentioned the algorithms that they could choose from in order to find the right path.

Ilan (18:52)
Each of them was given two suggestions as well as the option to choose a different algorithm. So those algorithms were A* and Dijkstra, and these are pathfinding, known pathfinding algorithms. And as I mentioned, also the option to choose its own. And I'll mention that all of them chose A*, which happened to be the first in the list.

David Vuong (19:14)
I wonder whether that's ⁓ something, the results of some kind of optimization from, from those LLMs.

All right, so what we did was we ran the game for 10 tries where we then calculated the average score across the 10 games. And we would do this for each of the tools to see who comes out ahead.

Ilan (19:38)
That's right David. So V0's final result after 10 games was an average score of 462 points.

David Vuong (19:48)
Excellent. It seems to be doing a lot better right now, now that it's ⁓ on video.

Ilan (19:52)
Haha! Each of these tools, I'll say, had a pretty wide spread between their high and low scores. So the scores were not clustered around a single value. They were really spread out. Kind of like when you played Snake on your Nokia.

David Vuong (20:08)
That makes sense. All right, let's see the next one.

Ilan (20:11)
Alright.

Up next we have Bolt. and you can see that Bolt chose a dark theme for the game, very much like the dark theme that Bolt itself defaults to.

David Vuong (20:27)
And what Bolt made for our website. That's two for two.

Ilan (20:31)
That's true, that's true.

How hackery. And Bolt is also doing pretty well. They don't have the little pathfinder so we can't see where it's thinking of going. But it did follow the instructions and coloring in terms of the snake. And it did follow the guidelines of how big to make the board from the first shot.

David Vuong (20:54)
That's good adherence.

Ilan (20:56)
On the first attempt, it would not complete the game.

no matter how many prompts I gave it. And after five prompts, I started over. And on the second try, it almost one-shot. I think it took two attempts to build the game. So this just goes to show that these LLM tools, they're probabilistic. Sometimes when you don't get the result that you're looking for the first time, just start over fresh.

David Vuong (21:23)
Yeah, I think that really speaks to the nature of the tech that we are using now, right? It's not as deterministic as with, you know, just an if then statement.

Ilan (21:35)
That

is true. And one thing that comes up when you talk to folks about the state of AI is some people get really worried about AI and look at all the things it can do. But it is also true that when you use these tools frequently, you begin to notice limitations with the current state of the technology.

They're definitely very impressive, but they have a lot of improvement to go still.

David Vuong (22:07)
Yeah, absolutely. Hey, all the more reason for people to join and subscribe to our podcast. They stay up to date.

Ilan (22:14)
That's right. So a couple more things about the Bolt game. We also got the Turbo mode here. On the other It did put the scoring to the bottom, which was against what it was prompted to do.

And another thing I'll note is that the average score is not computing correctly. It's just adding the scores up over time.

David Vuong (22:39)
⁓ boy, okay. So how did it do overall?

Ilan (22:44)
So overall, Bolt had an average of 671 points. Not bad.

David Vuong (22:50)
And

that was, that was calculated by you as opposed to its own little algorithm here.

Ilan (22:54)
Yeah,

I think that Bolt would have said that its average score was 6,710 points.

David Vuong (23:00)
Yeah.

All right, let's move on to the next one.

Ilan (23:04)
So here we have Replit. Replit did obey the command to put the scores up at the top.

And when prompted to give a turbo mode, it also gave us a second option not requested for ultra speed.

David Vuong (23:20)
Woah! what's that do? Look at that.

Ilan (23:21)
And when

you hit that, the snake goes pretty fast.

A couple things to note about Replit, it also decided to add the path that it's going to take to the food.

but the ⁓ counts of games played is not accurate. So kind of a funny issue that happened. It seems to be somewhat random how it resets the game afterwards and how it counts how many games were played.

David Vuong (23:55)
That's interesting. That's a neat little artifact.

Ilan (23:58)
Definitely. This required us to really closely review what were the actual scores that it came up with while it was running.

David Vuong (24:09)
So what did it average at?

Ilan (24:12)
So in the end, 10 runs, Replit came out at 768 points on average. Very impressive.

David Vuong (24:19)
That's impressive.

I noticed that it put the score at the top, which is better adherence to the prompt.

Ilan (24:27)
Yeah, you're right, David. It did put the score up at the top. And so it actually did listen to us. Now, ⁓

David Vuong (24:34)
Hey, it's cheating! okay.

That was a cheat over there. Okay.

Ilan (24:39)
while it did have a high score, it did have some consistent bugs in the UI as well.

David Vuong (24:45)
So you still got to retain those senior developers to double check the code.

Ilan (24:50)
That's right.

David Vuong (24:51)
All right, let's move on to last but not least.

Ilan (24:55)
Lovable, I gotta say, it didn't adhere to the rules to put the scores up at the top, but in my opinion, this came up with the most delightful screen. It made me the happiest when I saw it after it completed the prompt.

And-

David Vuong (25:12)
I like

that it added a different color for the head of the snake, which is something that we did not see in some of the other tools.

Ilan (25:20)
That's true. The nice contrast is very appreciated. And we did get the turbo mode. correctly calculate the number of games played and the average score ⁓ right off the bat. So that was good. ⁓ Lovable, like the other tools though, did require a few...

attempts to get a playable game. So the first version just randomly created green and red dots around the screen while it ran. ⁓

David Vuong (25:58)
That's certainly not what you wanted, I'd imagine.

Ilan (26:01)
No, it was not ⁓ the intention of the prompt. Another thing I'll note about Lovable is that it does have some additional information in its scoring area, ⁓ which is part of what makes it delightful. You understand as a user, what am I looking at? What is this? Why is it doing what it's doing? And what are some of the decisions that were made here?

David Vuong (26:25)
Yeah, to mention that's ⁓ an artifact of the system prompt that Lovable would be using.

Ilan (26:34)
that Lovable skews towards good UX. That's their differentiator in their system prompts from some of the other tools we've looked at.

David Vuong (26:44)
That's right. Now, ⁓ that snake does not seem to be performing too well in the few games that we've just seen. How did Lovable do over its 10 game trial?

Ilan (26:56)
Over the 10 game trial, Lovable's average score was 269 points. Womp womp.

David Vuong (27:04)
Aww.

And it uses the same algorithm, does it?

Ilan (27:10)
It claims that it uses the same algorithm, yes.

David Vuong (27:14)
Yeah, it does

seem to be different. Yeah. I've seen it make some poor choices as a, as a novice snake player. think I can even identify some of the bad choices that this snake has made like just now.

Ilan (27:30)
As we get back to the studio, I think that brings up an interesting point, David, which is that each of these tools optimized for this algorithm in different ways. I think that Lovable put in a pure pathfinding algorithm while ignoring the fact that it also had to survive in order to reach the food, where Replit and Bolt were

had a balance between how do we find the path to the food while also avoiding collision with the wall or with the snake itself. And so you can see that they did a little bit more with the prompt that we gave it so that they would optimize for the highest score.

David Vuong (28:17)
Yeah, we can really see the differences between these tools now that they are competing side by side with something that is a bit more complex ⁓ and especially after two events. So I'm very much looking forward to the third event where we are going to be testing maximum complexity.

All right, so with the scores side by side, how did we do?

Ilan (28:44)
With the score side by side, we have a clear winner with Replit, with an average of 768 points across 10 attempts, followed by Bolt with 671 points, V0 with 462 points,

and poor Lovable coming up last with 269 points on average.

David Vuong (29:06)
Wow, and once again we see an enormous split between first and last.

Ilan (29:11)
Absolutely, definitely a testament to how they optimize as they go deeper into a prompt.

David Vuong (29:18)
All right, after two really fun events at the track meet for these four tools, we have a variety of different results and I'm really looking forward to seeing the third event, which is going to be the most complex that we've ever put these tools through and we're going to be evaluating them in great detail as we walk through that. And that's going to be for our next episode.

Ilan (29:45)
So look out for that in your feed. And in the meantime, like and subscribe, leave us a comment, a five star review, and you can follow us at pandcpodcast on all of the socials.

David Vuong (29:56)
Alright, see you at the next one.

Ilan (29:58)
Bye.