In this episode of Consulting Spotlight, host Michael Bernzweig interviews Matthew Edgar, co-founder and technical SEO consultant at Elementive. They discuss the critical role of technical SEO in ensuring that websites are properly crawled and indexed by search engines. Matthew shares insights on the importance of foundational SEO practices, the need for collaboration between marketing and IT, and the challenges organizations face in the technical SEO landscape. The conversation also touches on the impact of AI on SEO strategies and the importance of internal and external links. Matthew provides examples of common technical issues and shares a success story of a client who improved their website's performance through technical SEO efforts.
In this episode of Consulting Spotlight, host Michael Bernzweig interviews Matthew Edgar, co-founder and technical SEO consultant at Elementive. They discuss the critical role of technical SEO in ensuring that websites are properly crawled and indexed by search engines. Matthew shares insights on the importance of foundational SEO practices, the need for collaboration between marketing and IT, and the challenges organizations face in the technical SEO landscape. The conversation also touches on the impact of AI on SEO strategies and the importance of internal and external links. Matthew provides examples of common technical issues and shares a success story of a client who improved their website's performance through technical SEO efforts.
Takeaways
Sound Bites
Chapters
Michael Bernzweig (00:02.242)
Are you ready to unlock the secrets of consulting success? Tired of sifting through endless noise to find actionable insights that actually move the needle?
Welcome to Consulting Spotlight, your weekly deep dive into the transformative world of professional consulting. Your host is Michael Bernzweig, who in 1998 launched Software Oasis as one of the first platforms enabling businesses to download, license, and deploy software instantly across their networks with a single click. Today, Software Oasis has evolved into one of the leading communities where businesses find top consultants across the USA and Canada.
Each week, Michael sits down with industry titans, innovative leaders, and game-changing executives to bring you exclusive insights you won't find anywhere else. From emerging methodologies to market trends, Consulting Spotlight delivers the strategic intelligence you need to accelerate your consulting career and stay ahead of the curve.
Join our growing community of consulting professionals and decision-makers. Subscribe now on your favorite podcast platform and visit softwareoasis.com/subscribe to get our bi-weekly newsletter delivered straight to your inbox.
Get ready for weekly data, trends, analysis, interviews, and insights that will propel your consulting career forward.
Subscribe to our bi-weekly newsletter https://softwareoasis.com/subscribe/ to stay updated with more insights from technology leaders and transformation experts.
Connect with an exclusive audience of consulting organizations, B2B SaaS founders, and tech leaders through Software Oasis. For over two decades, we've been the trusted platform for professionals seeking smart solutions. Now, leverage our highly engaged community with targeted advertising options designed to maximize your brand's impact and drive real results. View advertising options https://experts.softwareoasis.com/ads
I'd like to welcome everyone to this week's edition of the Consulting Spotlight. I'm your host, Michael Bernzweig, and joining us live this week, have Matthew Edgar. He's the co-founder and technical SEO consultant over at Elementive. So with that, Matthew, welcome to the podcast.
Matthew Edgar (00:24.285)
Hello, thanks for having me on.
Michael Bernzweig (00:26.466)
Yeah, no, very exciting to have you here. And I can see from a few of the questions that came in from the audience that there is quite a bit of interest in learning more about technical SEO. So with that, I was hoping, you know, just as a little background for anyone that may not be familiar with either yourself or Elementive, if you could give us a little bit of your...
journey getting to where you are and what's going on over there at Elementive. What does your team do?
Matthew Edgar (00:57.617)
Yeah, absolutely. So at Elementive, we help clients out and specialize in the technical side of SEO. So what that means is we're helping clients figure out how does Google crawl your site? How does Google index your site? And what are the various technical factors that play into that that might be preventing Google from doing things correctly? Because if you don't have those foundational aspects down right, then nothing else really matters. You can have the best content in the world, the best brand in the world. But if
Google can't get to your site, none of that is going to help you. So that's what we do. We help clients out with that. I've been doing this work now for a couple of decades. I started in the early 2000s helping clients optimize websites. And when I started, it was more just broad, you know, technical updates to websites, technical optimization. But increasingly over the years, I started to see how important search was as Google
grew and started to dominate the web through the 2000s, the 2010s, it just became more more important to get those foundations down right for clients. And so in 2015, my business partner and I founded Elementive together to really specifically help with those areas and make sure that we do help clients really figure out, how do you get those foundational aspects right to make sure that you're not
you know, blocking yourself from being able to do well in organic search.
Michael Bernzweig (02:31.118)
I love it. And you know, it's interesting, not too many organizations in your space can say that they've been around for a decade. that is kudos to you and everything you're doing over there. It obviously speaks volumes to the fact that clients are actually happy with the end result of the work that they're having done.
Matthew Edgar (02:55.239)
Yeah, I mean it is great to have been doing this for so many years and to see so many changes in this industry. And I'm very grateful that some of the clients I'm working with I've worked with now for over a decade and continuing to help them out, continuing to help their business grow and expand. And it's great to have those ongoing relationships with clients. It's always fun to work with new clients as well, but it's also really fun to have.
some years of history because you can just get in there and get to know the client better, get to understand their problems, get to understand their website. And it really is just really great to see the results that come from that.
Michael Bernzweig (03:35.98)
Yeah, and you know, a lot of, you know, any kind of work that's rewarding, you know, is seeing the journey and the progress and how organizations that you're working with, you know, what the net results are of what you're doing. And, you know, as you see organizations grow or.
you know, things like that. It's nice to know that you were a part of everything that's happening there. So what are the types of organizations that you're working with that you find you're able to, you know, move the needle for the most? Like what types of organizations see the most benefit from a deep technical SEO, you know, mentor, know, such as Elementive?
Matthew Edgar (04:22.153)
Yeah, you know, it varies depending on the client. We work with all different types of industries, all different types of businesses from smaller companies all the way up to large Fortune 500 companies, government organizations, nonprofits. So it really runs the spectrum. I'd say the sort of central link between all of our clients is that they all have somewhat bigger website that has some more complexity on it.
content or maybe it's an e-commerce site selling lots of products or they have a directory or, you know, right, there's some more complexity to the website. The other kind of universal factor across all of our clients is the website is really central to their business. And it really is, if not the majority of their business is on the website, a good portion of their business is on the website. So it's really important to make sure that the website performs correctly, works correctly for Google.
show up there. So as a result, we work with startups that might be a small business. It's not an overly large company, but their website has hundreds or thousands of pages because they're listing lots of different things or they have lots of different products they're selling or lots of different profiles in a directory and so on. the bigger and more complex the website, the more challenges that you run into technically that need to be watched out for and need to
Make sure you're taking steps to find and fix those issues.
Michael Bernzweig (05:54.742)
Yeah, so it sounds like a lot of the work is obviously digging into the research to understand what's happening, but also providing the roadmap to get the correct foundation in place for these organizations.
Matthew Edgar (06:08.425)
Exactly. And a lot of what we end up doing, mean, we're unfortunately, we start working with clients is when they're in the middle of an emergency, something has gone wrong. And so the first step is, okay, how do we figure out what went wrong and how do we resolve that? Yeah, it's a lot of investigation and digging in and trying to work backward. And that's a lot of digging into the code and all of that, but also digging into just release notes and
Michael Bernzweig (06:17.997)
Yeah.
Michael Bernzweig (06:23.544)
forensic analysis.
Matthew Edgar (06:36.573)
what's happened and how things are going at the organization. And then after that, yeah, it does get more into let's get the right systems in place, get the right structures in place to avoid things. My focus is always telling clients, okay, an emergency can happen one time, but let's never have this happen again. Let's try and not have the same emergency, the same crisis over and over again. And that's where...
Michael Bernzweig (06:56.6)
Yes.
Matthew Edgar (07:04.509)
Yeah, let's try and repair the foundational level and actually get those things set up correctly so that way we don't run into the same problems over and over.
Michael Bernzweig (07:13.474)
Yeah, it makes a lot of sense. So when you're getting started with new clients, do you have a kind of a playbook in terms of process that you like to go through to make sure you're getting a client, you know, onboarded correctly and that you're doing a similar type of audit on each, you know, client that you're working with. So you're delivering the results they're hoping for or how do you go about it?
Matthew Edgar (07:39.047)
Yeah, it's a good question. And this is one where I feel like we really differentiate ourselves, is that we don't have a standard checklist that we're just working against for every single client. So every audit we've done is different. You end up looking at different things. Where we kind of approach an audit is a central list of questions that we need to start with and start.
Michael Bernzweig (07:51.362)
Yeah.
Matthew Edgar (08:05.257)
investigating from those questions. And depending on where those questions take you, start to find, oh, OK, I really need to look at this area, and I don't need to look at this other area at all. And those questions start with some of the basics. How does Google crawl the website currently? Increasingly, it's not just Google. It's also generative AI bots. How do they crawl the website? So looking at that most foundational level of, OK, all these search engines are sending out bots to investigate a website and find content on the website.
Can they get to that website? If they get to the website, what do they see when they get there? Can they actually load the page correctly? Are there problems with how the website is built? We see this on a lot of websites that are built with heavy reliance on JavaScript that it's just not built in a way that bots can crawl it. So you don't even see the content. So you start by asking some of these questions. yeah, and as you start asking those questions, it kind of guides you through, OK, what's actually going to matter on this?
Michael Bernzweig (08:56.398)
Any other questions?
Matthew Edgar (09:04.883)
The audit for every single client just ends up taking a different course. It just looks different because every site is unique. Every business is unique. it's just, you don't want to necessarily have the same audit for everybody or the same checklist for everybody because it needs to be adapted for whatever is unique to them.
Michael Bernzweig (09:18.037)
Right.
Michael Bernzweig (09:24.342)
And I would think working with clients and in the SEO space, it's really important to make sure you have clear expectations because, you know, it's not like, like paid advertising where you just flip a switch and rush a traffic and, you optimize your funnel and, it either works or it doesn't with SEO. It's obviously, a, a longer, longer marathon, not a, not a quick sprint, to,
the finish line.
Matthew Edgar (09:55.849)
That is exactly the case. mean, that is the point that I think everybody needs to just remember about SEO is that it is a long-term game. And especially when you're dealing with something technical. mean, occasionally you come across technical problems where, OK, we can fix this and you see a big jump around something. But those are far and few between. it's things that you have to go through and you have to rebuild different parts of your website or rework.
Michael Bernzweig (10:15.896)
Right.
Matthew Edgar (10:24.681)
you know, some things on the website and that takes time in itself to do that. Google, for their part, isn't going to pick up on the changes instantly, right? It's going to take Google a few weeks at a minimum to come back to the website. Once they go back and they crawl through the website, they see the changes that have been made. It's going to take them time to process things. So right away between the time it takes to implement things, the time it takes Google to see those implementations, you're at a couple of months easily. And that's, know, before
any of the changes really start to influence rankings and start to change things. So you do need to give it time. The other thing I'd add with that, thinking about the time that's involved is generative AI is starting to change the timelines as well because a lot of these systems have longer timelines before you start to see updates than even what we see with Google. it is, like you said, mean, it is definitely more that.
This is marathon that you're preparing for. It's not a quick sprint. We're just going to have results tomorrow. It takes time.
Michael Bernzweig (11:31.278)
So how do you see SEO fitting into the whole marketing mix with the different organizations you're working with? Is it a component or is it a foundational piece of the pie?
Matthew Edgar (11:46.641)
It depends on which aspect of SEO we're talking about, because there's different areas of SEO. Something like content-focused SEO, where you're looking at, how do we write content? How do we do that? That often fits very well within marketing, and it's really a key part of that. Local SEO usually fits really well within marketing as well. Something like technical SEO, it kind of fits somewhere in between.
marketing and IT, because there are marketing things you need to think about, you know, what content needs to be promoted, how do we do that? What do need to emphasize? Those kinds of questions are marketing oriented. But then there's a lot of technical questions that come up. And so when we work with clients, typically end up spending, you know, about half our time with the people in marketing and our other half the time with the developers and the server engineers. Yeah. Right. Yeah.
Michael Bernzweig (12:39.95)
That was my question, who were the stakeholders who you're usually working with?
Matthew Edgar (12:44.681)
Because you have to think about both. that's where, to me, when I think about especially the technical side of SEO, it shouldn't be thought of as, this is just marketing anymore. And it should be thought of, this is just IT. You need a good collaboration between the two. And that's honestly one of the biggest struggles I see with companies that I'm working with, particularly somewhat bigger organizations, is that IT and marketing don't talk to each other.
Michael Bernzweig (13:06.072)
Yeah.
Michael Bernzweig (13:11.704)
Yeah.
Matthew Edgar (13:13.031)
You've got the SEO people typically in the marketing side of things and they aren't talking to the people in the IT side of things. And if they aren't talking, they aren't able to actually do everything they need to do and they're not going to make progress.
Michael Bernzweig (13:26.294)
Yeah, and I think that that's one of the interesting things. And earlier in my career, did, you know, traditional SEO, and a lot of larger organizations, you know, you may have the best plan in the world, but, you know, sometimes just getting to step one where the foundational things that need to be put into place from a technical aspect are done, then you can begin. It's, you know, sometimes it can take take
weeks or months if everyone's not on the same page.
Matthew Edgar (13:57.897)
Oh yeah. mean, yeah, you were asking about timeline before. mean, the timeline just expands exponentially. If you have to start with, you know, actually just getting somebody to a meeting to actually talk about something like that, that in and of itself can be a week's long process and just slows things down further.
Michael Bernzweig (14:17.368)
So do organizations typically, when they're reaching out to you, already have something in place or are they just kind of responding to an emergency or starting from scratch that with something that they were not able to zero in on internally? What do you see as the typical starting point for an organization?
Matthew Edgar (14:39.697)
Yeah, it's a mixture. For larger organizations, what I think is really great to see about the SEO industry now is that there are a lot of companies that will have an SEO in-house. And usually that person is wearing multiple hats. They're doing all sides of SEO. So they're kind of that jack of all trades. They know enough to kind of do things. then those people usually are reaching out saying,
Michael Bernzweig (14:59.902)
A little bit of yellow.
Matthew Edgar (15:10.046)
I kind of know this thing, but I don't know everything.
Michael Bernzweig (15:12.854)
I know what my problem is, but I need a specialist, right?
Matthew Edgar (15:15.461)
Exactly. you know, that's fun to come in and work with other SEOs because, you then you can help them just in that one specific area or that one specific problem that they have. In other cases, you know, especially at smaller companies, it's more of VP of marketing or director of marketing kind of person who, you know, they're just, you know, trying to do everything with marketing, including SEO. And, you they definitely need a specialist, but they also need somebody just to help with.
Michael Bernzweig (15:30.53)
Mm-hmm.
Michael Bernzweig (15:36.194)
Yeah.
Matthew Edgar (15:41.942)
Can you guide this through the process? Can you help me with implementation? And they need maybe a little more support in those ways. So it varies a bit.
Michael Bernzweig (15:49.942)
So for any of our listeners who may not be 100 % familiar with technical SEO, can you give our audience an example of some of the things a technical SEO expert might do versus what types of things they would not be doing?
Matthew Edgar (16:05.641)
Sure. So when it comes to technical SEO, would say probably one of the biggest categories that somebody who's working on the technical side is going to focus on is crawling. So how do bots get to the website? Where are they going on the website? Where are their problems with that? So for example, you might look and start digging into it and realize that Google's bots are unable to crawl all of your blog.
or unable to crawl all of your images. That's a problem that a technical SEO would be able to help figure out and address. The other place that you might see a technical SEO really digging into and working on is with indexing problems. So right, Google has a free tool you can use called Google Search Console that lets you track how they're crawling the website, how they're indexing the website, SEO performance, those types of things.
And you can look at the page indexing report. And under the page indexing report, they will tell you why pages are not indexed. And there's loads of reasons why pages may not be indexed. Some of those are more to do with content quality. But a lot of them are down to technical issues. So under page indexing, you might see there's a lot of not found errors or a lot of redirect errors or a lot of problems with duplicate content. Those are going to be things that
Michael Bernzweig (17:09.134)
.
Matthew Edgar (17:29.481)
You're going to want to get somebody more technically minded involved on to figure out what's going on here. Why do we see these? Which of these are a problem? What is that underlying problem behind this? We're really doing that investigation into that to figure out where is that stemming from. So those are going to be some of the biggest categories. I would say that somebody more on the technical side is going to focus on and assist with.
Michael Bernzweig (17:53.944)
Sure, now that makes a lot of sense. And I realize we're already a few minutes into the episode and I want to let anybody that's listening that may just be listening to the audio. If you would like to see the video of this episode, just go to softwareoasis.com in the community section, you'll see the forum and you can see the video feed of any of our podcasts.
So anyways, I think what's really interesting is that, you know, over the last few years, and even more so, you know, today, 2025, we're seeing a massive shift with AI search. And at the end of the day, you know, that absolutely impacts SEO. You know, we're seeing...
continued declines in traditional search, know, Google, Bing, Yahoo, and searches that are beginning with AI, you know, everybody's using AI nowadays, even my grandmother, everyone is using AI for all kinds of various things, but specifically search is an area that a lot of individuals are starting their day there. So what types of considerations, like if you were to bullet point a few of the things that
Matthew Edgar (18:55.753)
Great.
Michael Bernzweig (19:12.94)
know, organizations should be keeping top of mind, you know, in this new landscape. What's important?
Matthew Edgar (19:19.881)
Great. So I would say one of the biggest things to keep in mind is whether or not you want to show up in generative AI or not, whether or not it's helpful for your business show up in those search results. For a lot of companies, the answer is going to be absolutely yes. But there's some organizations, particularly organizations who your business is about getting traffic to your website that may not be advantageous for you.
And it's not necessarily always a universal yes or no across your entire website. There may be certain pages on your website that absolutely make sense to be included in generative AI. You want those pages to be included in chat GPT's responses to people's questions. There might be pages on your website where you think, no, this is proprietary content or this is information I want to keep out of AI. don't want them necessarily using that.
in response. And so you want to go through and figure that out. And then from a technical standpoint, you can block off these AI bots from different parts of the website. The other part to that nuance, though, it adds nuance to that, is that it's not just one bot type for AI search. There's really kind of three main bot types that we're starting to see.
And chat GPT open AI is really the one who's doing the best to differentiate all three types though, you the others are kind of falling into different buckets with this. One type is the one that's training the underlying model, how it responds to questions. Another type of bot is one that's feeding the search results for these AI tools. And then the other type of bot that's totally different than anything we've seen in the past are agents. So in chat GPT's case,
They have an agent that will go out and it will fetch pages and bring in live information from the web in response to people's questions. So this is the ChatGPT user bot. And it's a different kind of action, right? Because now it's the bot hitting your website on a user's behalf. And this is starting to grow with different types of agents as well that can actually go out and crawl the web. So not only is it a question of, what content do I want people or bots to see,
Matthew Edgar (21:39.727)
And what content do I want people to see in AI responses? And how do I make sure they're able to access those pages or not access those pages? But also which bot type do I want to have access certain types of pages or not access other types of pages on my site? Because you may very well want some pages to show up for an agent accessing your website, but not show up in the underlying model. And so that to me is one of the most fundamental questions of
just thinking about AI search that's different than traditional search is, how do we work with these bots and how do we want these bots working with our site? And that starts to present a lot of new questions, a lot of new opportunities, some pretty unique challenges too as we start to think through that.
Michael Bernzweig (22:27.0)
Yeah, no, that makes a lot of sense. And I guess the other question, when someone is working with a technical SEO, they might be working with obviously other developers and marketing departments and maybe even just a traditional generalist SEO. From a technical SEO, is it really the job of the technical SEO to do the research, the planning and the strategy?
or is implementation typically part of technical SEO as well?
Matthew Edgar (23:01.595)
Implementation, for me at least, and everybody who works in technical SEO is a little bit different. For me at least, I don't get too involved in the implementation. And that tends to be what I see from probably the majority of people working in technical SEO. You're not directly involved with it. Your involvement is, I'm gonna write out detailed specifications.
Michael Bernzweig (23:19.992)
Sure.
Matthew Edgar (23:27.203)
and make sure the developer understands here's exactly what needs to be done and how it needs to be done. Like in my case, I regularly do working sessions where a developer and I are just on Zoom together and they have their screen shared. And I'm just kind of advising as they're going through making changes, make sure it's done correctly. But I'm not the one actually doing the implementation, writing the code. The other thing that I do a lot of and just kind of generally
Michael Bernzweig (23:32.365)
Yeah.
Michael Bernzweig (23:49.87)
Yeah.
Matthew Edgar (23:55.101)
you know, would fall to somebody on the technical SEO side is doing all the testing, the QA, make sure that everything did get implemented correctly and check in for all the weird corner cases that can come up around different types of pages and different ways that bots might be interacting with the page or Google might be processing that page. Make sure those are all accounted for. So a lot of that QA work falls more on the tech SEO side as opposed to maybe the
the developer side of things.
Michael Bernzweig (24:27.382)
Makes sense. So it's a matter of having your whole team in place to get everything accomplished. I love it. And are there different industries that you're seeing focus on SEO that you didn't see in the past? Because it seems like you've been doing this for a while.
Matthew Edgar (24:33.969)
Absolutely, yeah.
Matthew Edgar (24:49.017)
The AI search does bring up some different industries and give them some different opportunities that maybe we didn't see in the past. So for example, with traditional SEO, you see a lot of informational sites show up in response to things. Here's the best whatever articles. Those types of sites used to do really well.
And in some cases, smaller companies who were selling products or services more directly, they wouldn't necessarily show up because they were crowded out by some of those bigger sites providing that informational content. So they wouldn't show up except on like pages two or three. But what's interesting is, know, chat GBT in its responses, perplexity in its responses, even Google's AI overviews to a certain extent will surface some of those other types of sites, some of those smaller business sites.
And now all of a sudden, SEO starts to matter to them in a way that maybe it just couldn't realistically matter to them before because it just wasn't a way to show up against those competitors. And I think that kind of presents an interesting opportunity for some of those companies that, okay, if you're not maybe playing in that informational space, well, okay, maybe you can actually start ranking now for some of those informational type prompts, even if you couldn't.
necessarily rank for those informational type search queries, just given the differences and how those searches work. So to me, that's one of the trends I've seen the most recently. And I don't know if it's tied to a specific industry. I think it's tied more to a specific size organization.
Michael Bernzweig (26:32.406)
So, you know, over the years you've obviously worked with all kinds of clients and all kinds of industries and all sized businesses. Are there some common themes that you see across organizations in terms of the, maybe even to bullet point the types of technical challenges that you see show up on your doorstep day in and day out that organizations are making overall?
Matthew Edgar (26:57.833)
yeah, mean, there are some common challenges that come up. One of the top challenges that comes up is rendering. So how the pages are actually built and constructed. It's very nice that you can do all these really cool features on websites where maybe you have different types of animation or you have things fade in as you scroll down the page. Maybe you have some type of infinite scroll, something like that, right?
really cool interactive features that might look nice, be kind of fun to work with, but they're terrible for bots. And Google, for its part, can't necessarily work with those. So when you start testing out those pages, in Google's view, those pages are blank. They don't see any content. And then people wonder, why don't I have traffic to these pages? Well, it's because of how it's built. So I see that on tons of different websites where things just are not built in a way that
bots can actually interact with. Google, for its part, can get through some of that JavaScript code, but AI bots, ChatGPT currently, it cannot execute any JavaScript. So in its mind, any page that's heavily relying on JavaScript, it's all blank. And I've seen that on tons and tons of websites. That'd be one big one I've seen. Another issue I've seen come up on a number of sites over the years is just structural problems.
with the site, so the way that the site's actually structured and the pages are all kind of interconnected to each other, really kind of the underlying navigation of the site. There's a lot of pages that you really can't access, you as you start clicking through the site or maybe there's only a couple links pointing to it, but those pages are basically just cut off from the majority of the site. And so you need to have clear pathways for
those pages to surface, to be found. And that operates in two ways. That helps bots actually go in and find those pages in the first place, because there's more links pointing to them. So it's easier for them to discover those pages. But then the other way that helps is it sends a signal to Google that, hey, these pages matter, because we're linking to it a lot. And so I see that one come up quite a bit, where the page itself is fine. It could work. But you're not giving any indication to Google that this page
Matthew Edgar (29:22.129)
is important or that this page ought to work. So that's another big one I see come up. And then the third area I would say that I kind of point to as a very common issue I see come up on lots of sites is speed and just slower speeds that exist on websites. not necessarily due to a singular cause. It's not just, there's one bad thing. It's just lots of little things that all kind of add up and just slow down the website.
And that's going to affect SEO in some pretty direct ways with Google's core vitals being one of many ranking factors, but it also has implications for how bots are able to process and look at the site's code. If it's slow, they're just not going to be able to do anything. So those will probably be some of the top issues I call out that I've seen on a good number of sites.
Michael Bernzweig (30:16.43)
So the whole concept of crawl budget, is that a real thing? Does Google really allocate a certain amount of crawl budget to each website, or is that more of a myth?
Matthew Edgar (30:28.801)
It's more of a myth and it's not necessarily something that you need to worry about up until your site is probably millions of pages. Once your site hits that level, then it starts to become something that you can really start to notice. If your site's a few hundred pages, even up to a thousand pages, Google is probably gonna crawl that just fine. You wanna...
you know, kind of watch for crawl traps or things that might be preventing them. But it's not necessarily a budget that's going to restrict that, you know, or prevent Google from being able to go through that. Once your site gets into hundreds of thousands of pages, millions of pages, there's so many pages that, you know, at some point, Google does have to make some determinations about we're going back to this page or this other page. But even clients I've worked with that have paid sites that are millions of pages or more.
most pages still get crawled and still actually get paid attention to by bots. So generally speaking, I don't think it's right to think of it as a crawl budget necessarily that you're working in. It's more so making sure that Google can actually crawl things, that you're sending them the right signals about what pages matter on the site with internal links or with external links.
That's where you need to kind of put your focus with crawling. Not so much on, Google only allocates this much. It's more about making sure that they actually can crawl and then checking to make sure that they actually are crawling all the pages on your website and then seeing what problems might be preventing.
Michael Bernzweig (32:10.894)
And then one question that came in in many different ways from the audience, I'd be remiss if I didn't ask, but overall, I think a lot of listeners are wondering if you could share a little bit of the importance of internal versus external links to a website.
Matthew Edgar (32:32.841)
Sure. So when we think about that, I guess the way I would kind of start is by saying links help in a few different categories. one of the big things that links do is they help with discoverability of content. So just letting Google know, hey, this page exists. This is a page that's available somewhere on the web. The other way that links can help is it sends a signal to Google about priority and importance.
And it tells Google, OK, if a lot of links point to this page, that's probably a sign that maybe this page matters a bit more. The third way that links matter is it starts to send some type of quality signal to Google and gives a sense of, this page matters a great deal more or is more important from just a standpoint of these are pages that people
think matter, think, you know, care about. And Google's doing some evaluation on its end to figure out, these links are more authoritative than others. These links matter more than others. So it starts to communicate a level of site quality. So if other high quality sites are linking then to your site, then your site starts to look like it's higher quality, right? And so that's another part of links that matter. The other area that I would say matters for links is
topics and what topic does your website fit into. So if you have a website that's selling whatever products, if your backlinks, your external links don't necessarily fit into that and don't explain that topic or your internal links don't explain that topic, nothing else matters. Because Google doesn't understand what that topic is. That's how they're learning that. And so as you think about internal versus external,
Internal links are way more important for discoverability. Internal links are also way more important for communicating topical alignment of things, so explaining how different pages on your website relate to each other. External links can help with discoverability, but external links are more about helping Google understand those quality signals of, how important is this site, both in terms of quantity and quality, but
Matthew Edgar (34:57.715)
somewhat more so quality in terms of external links, how important is that in order to kind of understand the nature of that site? Now, you also asked about earlier AI search, links are kind of interesting with AI search. It definitely still matters from a discoverability standpoint, but for AI search specifically, external links matter a lot more to help communicate what topics this website is about.
and how different websites relate to each other within different topic areas. So you have a website about X, Y, and Z. If you have other links and other mentions on other websites about X, and Z, that's going to reinforce to AI models and to Google that your website really is about, you know, topics X, and Z.
Michael Bernzweig (35:45.794)
Got it, got it. And do you have a favorite, you know, obviously you've been doing this for a while, but do you have a favorite client story? You don't have to mention any specific clients, but just a specific measurable result or win over the years that makes you really proud.
Matthew Edgar (36:04.603)
Yeah, sure. One client that comes to mind as you ask that question, it's a probably a small to midsize company and worked with them for a number of years at this point. And they started off with kind of just a website that really wasn't performing all that well. There were lots of errors on the website, lots of issues present. They had been through a few failed redesigns that just had kind of continued to tank.
websites traffic. They'd also gotten really bad advice about what content to write. They'd ended up writing a lot of content that kind of had nothing to do with their core business model. So the topics were all out of whack about what Google understood this website was about. So there's a lot of repair work to be done. They brought in somebody on the content side, brought us in on the technical side and all together worked on
repairing things on this site. Let's actually clean up signals. And as we went through and started cleaning up stuff, just started to find that there were more more problems that existed. Some of the redesigns had resulted in lots of duplicate sites that were out there. So just old dev sites that contained various copies of the website's content. Yeah. So those all had to be shut down. And get those resolved. And then there were lots of broken images and broken
pages on the site, those all had to be resolved, lots of issues with the server crashing and leading to lots of server errors. So just on and on and on with lots and lots of problems. As content got cleaned up, lots of pages got removed. So having to update lots of internal links for that. so after all the years of doing repair work, they actually started to see major gains in their traffic. They actually started to see performance start to happen on
Michael Bernzweig (37:40.579)
Yeah.
Matthew Edgar (38:00.137)
things that were more topically aligned, they started to get traffic to those areas of the site, which to me was really encouraging that, now everything from a content standpoint, everything technically is actually pointing in the right direction, and helping the site actually start to show up for more relevant terms. Google was starting to crawl more effectively through the site. You didn't see it crawling errors as much as it used to. It was not focusing on the problems because we actually fixed them and resolved them.
Now the site is actually doing well, both in traditional search results, but also doing pretty well from AI search as well. And they're starting to show up in AI search results. They're starting to get leads from AI search results, which is exactly what we want to see.
Michael Bernzweig (38:46.382)
That's exciting. obviously over this long in the technical SEO industry, you've seen a lot. But looking forward as to what's coming at us, is there anything that's particularly exciting that you think is going to be either a game changer for a lot of individuals that are looking into technical SEO for their organization or anything that really feels amazing to you?
Matthew Edgar (39:15.145)
know, one thing that I would say I get excited about and I'm curious to see where it goes is how AI is changing development. Because so much of how you have to change things to support the technical side of SEO, it is having to work with a developer and having to get in there and change some level of the code. Sometimes in big ways, sometimes in small ways.
Michael Bernzweig (39:32.942)
Thanks.
Matthew Edgar (39:39.261)
And there's a lot of bottlenecks that come up with that because SEOs aren't the only ones who want to change things in the code. There's lots of other people at an organization who wants to change things. So you end up with these long dev lists. But the way AI is changing that and allowing developers to work much faster making changes is, to me, exciting because that means that you can actually implement more changes and you can get through things more quickly.
AI also makes it easier to do QA because you can automate that process. So there again, you have some pretty good speed gains because it can help check things and test for things. And so to me, that's one aspect that I'm pretty curious to see. the results I've seen so far are pretty promising of just being able to speed up that work and hopefully shorten some of that timeline as well.
Maybe things can get done a little faster, things can get picked up, then that much sooner if you can implement them that much faster.
Michael Bernzweig (40:42.018)
Neat. And for anyone listening to this podcast, I'm sure that there are some telltale warning signs that you may see on the horizon that would indicate we need a technical SEO. What are some of those, you if you were to bullet point some of those signs that organizations may be seeing that they should be scratching their head and saying, I know what I need.
Matthew Edgar (41:04.745)
Yeah. Well, one big indicator that you probably need to talk to somebody about the technical side of SEO is if you are planning any major site changes, redesign, redevelopment, those always can cause lots of challenges for SEO. You want to make sure you get that handled correctly from a technical standpoint. So right away, if you start hearing conversations about that at an organization and you want to make those kinds of changes, good to
check in, see if those are being handled correctly. The other warning signs I point to is if you see in Google Search Console and that page indexing report I was talking about earlier, if you see major spikes with a bunch of pages suddenly not indexed, and you suddenly see, OK, we have a lot of pages that are crawled not indexed or discovered not indexed or a lot of pages that are in duplicate content or whatever category it is.
Those major increases need to be investigated to figure out what's going on. Sometimes it's a non-issue, which case, OK, fine, nothing to do. But you want to look at it because sometimes it can be a pretty big problem. The other thing that you can keep an eye on in Google Search Console is the crawl stats report. This is a little buried. You have to go under Settings to find it in Google Search Console. But if you start to watch your crawl stats report, you can see how often Google is crawling the site. If you see major drops,
where you see major surges under there, there's something that changed. There's something going on and we need to figure out what that is. So that's a really good time to have a look at the technical side of things, make sure things are still behaving correctly there. Again, sometimes it can be a non-issue, but a lot of times it indicates there's problems. The other big warning sign is if you do see major traffic drops that you don't really understand why they happened, all of a sudden,
your traffic just goes away. It's down by a third or down by a half or just completely gone. And you don't have organic traffic anymore. Why is that? If it is more of a gradual decline, that's a different kind of problem than if it's a sudden drop off in traffic. But depending on what kind of traffic drop off it is, there's different kinds of problems that are behind that. A lot of times there's technical issues that can explain a lot of those traffic drops.
Matthew Edgar (43:28.187)
anybody's seeing that kind of stuff, that's a good time to reach out and see what's going on.
Michael Bernzweig (43:35.278)
Neat. Well, Matthew, I really appreciate the deep dive on technical SEO. I think this has been great. I know we had a lot of interest in this episode. So once this is live, we will in the show notes include details if anyone wants to reach out. So we'll do that. And for anybody that's joining us for the first time on this episode.
software always just goes back to the early days of the internet, 1998, but, we have three different podcasts all within the family. There's consulting spotlight, which we're listening to now software spotlight and career spotlight. So all three are in the same family. we have, one thing I'll share with everyone. The, software spotlight is in the B2B space and we've just.
broken over 31,000 downloads as of last month. So the podcast has really found a chord in the B2B community. We have one advertising spot that will be opening up in June for anybody that's in the B2B SaaS space. And from this end, for anybody that wants to keep up to date on what's going on here at Software Oasis or any of the podcasts, you can subscribe on your
favorite podcast player, you can also go to softwareoasis.com backslash subscribe and get on our weekly email newsletter list to keep up on everything going on here in terms of research reports, new interviews and webinars and all that's going on. So once again, we've had Matthew Edgar over at Elementive on the consulting spotlight this week. Matthew, thank you so much for your time.
Matthew Edgar (45:26.665)
Thank you.