There are numerous reasons you could possibly will need to uncover all the URLs on a web site, but your specific target will figure out Whatever you’re seeking. As an illustration, you might want to:

Recognize each individual indexed URL to investigate difficulties like cannibalization or index bloat
Accumulate recent and historic URLs Google has witnessed, especially for website migrations
Discover all 404 URLs to recover from write-up-migration faults
In Just about every circumstance, just one Instrument received’t Provide you anything you require. However, Google Lookup Console isn’t exhaustive, plus a “internet site:instance.com” search is proscribed and tough to extract information from.

In this write-up, I’ll stroll you through some equipment to construct your URL record and in advance of deduplicating the information using a spreadsheet or Jupyter Notebook, based upon your web site’s dimension.

Aged sitemaps and crawl exports
When you’re seeking URLs that disappeared in the Are living internet site lately, there’s an opportunity an individual in your workforce could have saved a sitemap file or perhaps a crawl export ahead of the adjustments were manufactured. In the event you haven’t now, check for these data files; they could frequently deliver what you will need. But, if you’re looking at this, you probably didn't get so lucky.

Archive.org
Archive.org
Archive.org is an invaluable Software for Web optimization tasks, funded by donations. When you hunt for a site and select the “URLs” selection, you are able to obtain around ten,000 outlined URLs.

On the other hand, There are some constraints:

URL Restrict: You'll be able to only retrieve nearly web designer kuala lumpur 10,000 URLs, that's inadequate for greater web pages.
High quality: Quite a few URLs may be malformed or reference source documents (e.g., images or scripts).
No export alternative: There isn’t a designed-in method to export the checklist.
To bypass The dearth of the export button, use a browser scraping plugin like Dataminer.io. Even so, these limits mean Archive.org may not give an entire solution for greater web-sites. Also, Archive.org doesn’t suggest regardless of whether Google indexed a URL—but when Archive.org uncovered it, there’s a good likelihood Google did, way too.

Moz Professional
Though you may normally make use of a backlink index to uncover exterior websites linking to you, these resources also find out URLs on your web site in the process.


How you can use it:
Export your inbound backlinks in Moz Pro to obtain a swift and simple listing of focus on URLs from the web-site. For those who’re dealing with a massive website, consider using the Moz API to export information past what’s manageable in Excel or Google Sheets.

It’s vital that you Be aware that Moz Pro doesn’t verify if URLs are indexed or identified by Google. Nonetheless, considering that most internet sites implement precisely the same robots.txt policies to Moz’s bots because they do to Google’s, this process typically functions nicely as being a proxy for Googlebot’s discoverability.

Google Search Console
Google Search Console features quite a few important resources for constructing your list of URLs.

Inbound links studies:


Similar to Moz Pro, the One-way links area offers exportable lists of focus on URLs. Sadly, these exports are capped at one,000 URLs Every. You could use filters for precise pages, but due to the fact filters don’t apply for the export, you might need to trust in browser scraping tools—limited to 500 filtered URLs at a time. Not excellent.

Overall performance → Search Results:


This export provides you with a listing of internet pages acquiring search impressions. Even though the export is limited, you can use Google Look for Console API for much larger datasets. In addition there are cost-free Google Sheets plugins that simplify pulling much more extensive facts.

Indexing → Webpages report:


This portion supplies exports filtered by problem form, even though they are also confined in scope.

Google Analytics
Google Analytics
The Engagement → Webpages and Screens default report in GA4 is a superb resource for gathering URLs, with a generous Restrict of one hundred,000 URLs.


Even better, you can implement filters to generate distinct URL lists, efficiently surpassing the 100k Restrict. One example is, if you need to export only website URLs, adhere to these steps:

Action 1: Insert a segment into the report

Phase two: Click on “Develop a new phase.”


Move three: Determine the phase having a narrower URL pattern, like URLs that contains /site/


Observe: URLs located in Google Analytics may not be discoverable by Googlebot or indexed by Google, but they provide precious insights.

Server log data files
Server or CDN log data files are Potentially the final word Software at your disposal. These logs seize an exhaustive listing of every URL path queried by people, Googlebot, or other bots in the recorded interval.

Criteria:

Data sizing: Log data files is often large, a great number of web-sites only retain the last two months of data.
Complexity: Examining log documents might be demanding, but different tools are offered to simplify the method.
Mix, and good luck
When you’ve collected URLs from these resources, it’s time to combine them. If your web site is sufficiently small, use Excel or, for more substantial datasets, resources like Google Sheets or Jupyter Notebook. Make sure all URLs are consistently formatted, then deduplicate the record.

And voilà—you now have a comprehensive listing of latest, previous, and archived URLs. Great luck!