Episode Summary

A developer attempts to fix WordPress SEO and indexing issues using Claude AI, discovering that AI agents are treated as bots by robots.txt and encountering limitations when automating bulk changes. The experiment reveals both the potential and pitfalls of AI-assisted site maintenance, particularly the importance of human checkpoints.

Pull Quotes

On discovering that AI agents trigger robots.txt restrictions

“You're not a user, you're a bot. So something to think about when you're crafting your website, things are going to, you know, you have to pay attention to what do you look like to these bots because it may be good or bad.”

“I just kind of went, 'Nah, just go do it.' I got lazy and then it got off the rails. So I think you got to pay attention to these checkpoints. The human loop is good, right?”

“If you screw up one little episode, it's not the end of the world. But if you screw them all up or you halfway screw some of them up and then it runs out of memory context, that's a bad day.”

On AI system limitations

“There's only so much you can do in a chat window, you know, so maybe think about asking it for advice or asking it to write steps or asking it to do certain things.”

“I think everybody likes WordPress and hates WordPress. If you've ever dealt with it, it's just kind of weird and it's time for something better. But a lot of the internet isn't running on WordPress, so I don't know what are you going to do.”

Transcript

Okay, so here's one that was a little not so great. Everybody's favorite WordPress. I think everybody likes WordPress and hates WordPress. If you've ever dealt with it, it's just kind of weird and it's time for something better. But a lot of the internet isn't running on WordPress, so I don't know what are you going to do. So I said there's some indexability problems on my WordPress. It's been rotting. Time to fix it. It's one of these problems. It's hard enough. I just keep putting it off. And if I just had somebody I could call to just fix it, that'd be great. Hey, let's throw AI on it. Help me understand why Google won't index the site. There's some technical thing, right? So is there a layout or indexing or something wrong? For example, this one episode can't be indexed. And it goes and it researches and it goes through and then it says, "Oh, okay, failed to fetch robots.txt, which is weird. Oh, that's another one. You're a robot when you're coming from a GPT. So anything that is denied from Google will be denied from you. And I ran into that with logos for some reason on a lot of these corporate websites trying to put logos in, you know, into my proposals, statement work, that kind of thing. But it was just kind of weird as an observation that happened more often than not. So I thought that was interesting what the context is. You're not a user, you're a bot. So something to think about when you're crafting your website, things are going to, you know, you have to pay attention to what do you look like to these bots because it may be good or bad. But something you got to manage through.

Oh my god, major indexing problems detected. Severe. Zero results. Okay. That's awesome. Likely SEO missing robots. Okay, let's go check it. Do this manually. Missing plugin. Okay, no site map. I don't think that was a problem. Content structure. Okay, let's hit on that one. And then immediate action. Go here. Do this. Set up Google. Okay, did that proper plugin. Okay, focus on step number four. So it told me how to do that. And then how to configure your schema markup and then some other stuff and then it'll be here and then optimize your homepage and then so I think it's just probably like how do we tell Google appropriately that this is a podcast and some interesting things here and then critical schema markup and then red flags.

So I thought that was pretty cool and then I think I said here's and I ran it to a limit. So I said, "Hey, go here's the plan. Begin implementing these changes. My YouTube is here." And they said, "Okay, awesome. I'll do that." And then it started going through and I got, you know, why did it pick these out of 35 episodes or something? Oh god, it's going to be a mess. And then it just started getting in there and it had some good ideas, right? And I don't know about iframe. I don't feel like iframe is ever the right answer to anything. But it started getting weird. And then I did install the WordPress tool set the MCP and then you can upload or delete or create pages which is kind of I don't know. I guess that's useful. But it's you know it's not going to do admin stuff for you but it can manage posts. Oh god. And then it kind of got in a bunch of weird things and then didn't quite get them all and then it had trouble running out of context I guess and then it's just you know this is a weird one. You know, we're down to ticky-tack, you know, implement this plan for episode 25, complete SEO template, and now I got to implement all these templates by hand evidently, and I just got tired. And then I don't know, this one was not a great day for Claude. Started doing strange things and I reverted early and often. So I took some of the changes, but it was odd, you know, and it was like, sometimes when you just let it rip, it's not great. So I guess that's a learning. It told me, you know, before you make these changes, you know, back everything up and then let's go for it. And then Claude tries to step you through, hey, let's do one and then I'll fix one episode or one page and then you tell me if you like it or not. And then I just kind of went, "Nah, just go do it." I got lazy and then it got off the rails. So I think you got to pay attention to these checkpoints. The human loop is good, right? It's a little extra work. Don't get lazy. Do review. Do run the code that it gives you. Go through and do spend the time to methodically check the results before it goes forward because if you screw up one little episode, it's not the end of the world. But if you screw them all up or you halfway screw some of them up and then it runs out of memory context, that's a bad day. Oh, what are you gonna do? And then things that are big, I'm coming to the conclusion of like there's only so much you can do in a chat window, you know, so maybe think about asking it for advice or asking it to write steps or asking it to do certain things. You have to start getting a little creative like externalizing the memory, you know, writing it out to a disk or a database or a notion task by task and then have it come back through and iterate on the next task. I don't know. I have to think about what that looks like. But anyway, WordPress was a little bit of a disaster, but it always is. So anybody that's ever upgraded plugins knows it's a crapshoot when you go in there. But you know, kind of cool. You know, you have to squint a little harder on WordPress to make it make sense.

Want to go deeper?

Work With Me

Related Episodes

SpeedRun Ep 06 Durable Agent Writing -Tales from the Sky Lounge

SpeedRun Ep 06 Durable Agent Writing -Tales from the Sky Lounge

A CTO demonstrates how to convert Claude Desktop prototypes into production-ready AI agents using Python, Pydantic, and LangGraph. The episode walks through extracting MCP tool configurations and system prompts from chat sessions, then generating deployable code that can run on schedules or respond to events—illustrating how agent development costs are approaching zero.