How to Parse URLs

Problem

You need to extract specific parts of a URL, such as the hostname or query parameters.

Prerequisites

• Understanding of URL structure
• Familiarity with URL components

Steps

1. 1. Understand URL components

   A URL consists of a scheme (protocol), authority (userinfo@host:port), path, query string, and fragment. Each part is optional in certain contexts. The standard URL format is defined in RFC 3986.

   Input

   https://[email protected]:8080/path?q=1#frag

   Output

   scheme: https, host: example.com, port: 8080, path: /path, query: q=1, fragment: frag

2. 2. Parse the URL

   Use a URL parser rather than splitting on characters. A parser correctly handles edge cases like IPv6 addresses in brackets, encoded characters, and missing components. The WHATWG URL API is the modern standard.

   Input

   https://example.com/path?q=1

   Output

   parsed components

3. 3. Extract and use components

   Access the specific components you need: hostname for DNS lookups, pathname for routing, searchParams for query values. Parse query parameters into a key-value map. Handle missing components gracefully.

   Input

   searchParams: q=1&page=2

   Output

   {q: '1', page: '2'}

Expected result

The URL decomposed into its individual components, ready for inspection or use.

Common errors

Splitting on ':' or '/' instead of using a parser

Use a proper URL parser. Naive splitting fails on IPv6 addresses, encoded slashes, and missing components.

Query parameters not decoded

URL-decode query parameter values after parsing. %20 should become a space, %26 should become &.

Troubleshooting

• If parsing fails, the URL may be malformed or relative. Use a base URL to resolve relative URLs.
• IPv6 addresses must be in brackets: http://[::1]:8080/.

Security limits

• Parsed URLs may point to malicious hosts. Validate the scheme and host before making requests. Be aware of open redirect vulnerabilities.