Today I learned about Invoke-WebRequest and Get-Unique. Combine them together and you can quickly get all of the unique URLs linked on a website. Here’s a quick example using my jasonpearce.com website.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 | PS D:\> $site = Invoke-WebRequest -Uri https://jasonpearce.com/ PS D:\> $site.Links.Href | Sort-Object | Get-Unique #content / /please-do-not-solicit/ https://jasonpearce.com https://jasonpearce.com/2016/03/04/using-powershell-to-change-the-upn-suffix/ https://jasonpearce.com/2016/04/05/upcoming-pure-storage-user-group-meeting-april-20-at-the-district-tap/ https://jasonpearce.com/2016/04/06/load-powercli-into-the-powershell-session-as-a-module/ https://jasonpearce.com/2016/04/06/load-powercli-into-the-powershell-session-as-a-module/#respond https://jasonpearce.com/2016/04/06/running-powershell-as-different-user/ https://jasonpearce.com/2016/04/18/use-powershell-to-require-users-with-the-oldest-passwords-to-change-their-passwords/ https://jasonpearce.com/2016/04/18/use-powershell-to-require-users-with-the-oldest-passwords-to-change-their-passwords/#respond https://jasonpearce.com/2016/05/11/scamhoax-computer-system-alert-system-has-been-infected-due-to-an-unexpected-error/ https://jasonpearce.com/2016/05/11/scamhoax-computer-system-alert-system-has-been-infected-due-to-an-unexpected-error/#respond https://jasonpearce.com/2016/05/15/plummer-pearce-book-1811-1856/ https://jasonpearce.com/2016/05/15/plummer-pearce-book-1811-1856/#respond https://jasonpearce.com/2016/06/04/irv-pearce-inducted-into-faia/ https://jasonpearce.com/2016/06/04/irv-pearce-inducted-into-faia/#respond https://jasonpearce.com/about/ https://jasonpearce.com/author/jasonpearce/ https://jasonpearce.com/contact/ https://jasonpearce.com/genealogy/ https://jasonpearce.com/page/178/ https://jasonpearce.com/page/2/ https://jasonpearce.com/travel/ PS D:\> |