Investigating URLs webpages
| Investigating URLs webpages | |
|---|---|
| Name | Investigating URLs webpages |
| Location | H&D Studio |
| Date | 2026/07/17 |
| Time | 14:30-15:30 |
| PeopleOrganisations | Rebekah Lyndon |
| Type | HDSC2026 |
| Web | Yes |
| No | |
This event was only for participants of the Summer Camp 2026 and not open to the public. For public events you could check: HDSC2026 Clouds to Commons Public Program
This skills-sharing explored what we can learn from websites beyond the webpages that are presented to us. We looked at URLs, page sources, and various workarounds to see what information is being taken about us and what semi-hidden information is available to us.
This included a no-coding introduction to inspecting the html of a webpage to identify key information (like time stamps, sources, as well as redirections and connections from chatbots, trackers, and big tech companies.). Together we can open a discussion about how this relates to physical infrastructure/impacts and data privacy.
Some notes and tips gathered during that session!
In a url, keep your eyes open for Percent Encoding and UTM Parameters. Wikipedia is good for more info on these!
For ‘advanced googling’/dorking/search operators, start with https://search.brave.com/help/operators or https://kit.exposingtheinvisible.org/en/google-dorking.html (a little outdated). You can work around a lot of websites’ internal search limitations even just with site: and filetype:
To get to page source:
- Right click in your browser and select ‘View Page Source’
- or Cmd+Option+U / Ctrl+U
- or add ‘view-source:’ at the start of the URL
- some starting points to search for in the page source:
- edit, modified, created, author, key, google, tag, api, (the start of comments)
To get to inspect/elements/network:
- Right click in your browser and select ‘Inspect’
- Navigate to the ‘network’ heading. Then refresh the page. Snoop around the links coming up!
- Right click on the headers bar (the one with name, status etc) and select ‘Remote address’ if you want it to display the IP address of the site.
Website structure:
- add ‘/robots.txt’ to the end of the domain
- add ‘/sitemap.xml’ to the end of the domain
- doesn’t always work but can show you what bots/scraping the website allows and what the layout is, respectively.
To check site registration and ownership: [1]
In terminal:
- ping <IPaddress>
- curl -L ip.thc.org/<IPaddress>
- (replace the ‘<IPaddress> ‘ with the sequence of numbers for the domain you are interested in)
There are loads of tools out there that can facilitate the steps above and more, but be conscious that many are unclear about their methodology, motivation, and ownership – in using them you may be giving over your own data. Poking around yourself you may be able to find the information without the middleman. Also, as the internet evolves many tools often quickly become outdated, so it can be helpful to get some sense of how they work and what information they draw on. I do recommend [2] for checking for trackers etc. And any of the investigations on the Markup may be of interest. Also cool reporting on the TBIJ, GIJN, ProPublica, Craig Silverman, Lighthouse Reports, and [3] has some good resources.
Participant could bring the following:
- your own laptop if you could :)
- a couple of websites that you were curious about, but there were also be examples and time to explore during the session
Workshop timeline:
- 5 mins group discussion – how have we noticed web navigation changing over time, in our own experiences using the internet?
- 5 min – intro/why bother?
- 10 mins – what is a ‘clean’ link? What isn’t, and what does that tell us?
- 5 mins – playing around with ‘decoding’ URLs in a online tool
- 10 mins – how do we find the less obvious pages? dorking and robots (interactive follow-along)
- 5 mins – intro to practicalities of accessing page source/inspecting page
- 10 mins – case studies of what this can reveal (companies relied on, specificities of timings, physical locations). Maybe also looking into internet archive.
- 10 mins – flexible for people to add any examples of things I’ve missed + discuss what this means for a ‘commons’ approach to using the internet.
Access note:
- This activity was involving screentime. You could let me know before if it would help to have print-outs of slides etc.
- I avoided bright colours and high-contrast colours in slides due to my own needs; I endeavoured to check for colourblindness compatibility and to provide a link to slides so people can view and adjust on their own devices if preferable.
- You could let me know if there were any specific steps I could take to support you :)
Bio
I used to work in digital investigations (/OSINT, but I don’t love that term) for organisations including Human Rights Watch, the UN, and Amnesty International. I’ve taught media literacy workshops to a range of audiences and led research on disinformation.
It’s been a few years since then, and more recently I’ve turned to learning python, webscraping, and data journalism essentials, for personal projects for now and hopefully more collaborations down the line. My academic training was originally in history, so I bring an interest in close source analysis to most things I do! When I’m overwhelmed by the internet I like to get offline with community gardening, DIY, and textile projects.