User PowerShell to get all links on a website

Today I learned about Invoke-WebRequest and Get-Unique. Combine them together and you can quickly get all of the unique URLs linked on a website. Here’s a quick example using my jasonpearce.com website.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
PS D:\> $site = Invoke-WebRequest -Uri https://www.jasonpearce.com/
PS D:\> $site.Links.Href | Sort-Object | Get-Unique
#content
/
/please-do-not-solicit/
https://www.jasonpearce.com
https://www.jasonpearce.com/2016/03/04/using-powershell-to-change-the-upn-suffix/
https://www.jasonpearce.com/2016/04/05/upcoming-pure-storage-user-group-meeting-april-20-at-the-district-tap/
https://www.jasonpearce.com/2016/04/06/load-powercli-into-the-powershell-session-as-a-module/
https://www.jasonpearce.com/2016/04/06/load-powercli-into-the-powershell-session-as-a-module/#respond
https://www.jasonpearce.com/2016/04/06/running-powershell-as-different-user/
https://www.jasonpearce.com/2016/04/18/use-powershell-to-require-users-with-the-oldest-passwords-to-change-their-passwords/
https://www.jasonpearce.com/2016/04/18/use-powershell-to-require-users-with-the-oldest-passwords-to-change-their-passwords/#respond
https://www.jasonpearce.com/2016/05/11/scamhoax-computer-system-alert-system-has-been-infected-due-to-an-unexpected-error/
https://www.jasonpearce.com/2016/05/11/scamhoax-computer-system-alert-system-has-been-infected-due-to-an-unexpected-error/#respond
https://www.jasonpearce.com/2016/05/15/plummer-pearce-book-1811-1856/
https://www.jasonpearce.com/2016/05/15/plummer-pearce-book-1811-1856/#respond
https://www.jasonpearce.com/2016/06/04/irv-pearce-inducted-into-faia/
https://www.jasonpearce.com/2016/06/04/irv-pearce-inducted-into-faia/#respond
https://www.jasonpearce.com/about/
https://www.jasonpearce.com/author/jasonpearce/
https://www.jasonpearce.com/contact/
https://www.jasonpearce.com/genealogy/
https://www.jasonpearce.com/page/178/
https://www.jasonpearce.com/page/2/
https://www.jasonpearce.com/travel/
PS D:\>