テキストから URL を抽出する方法
パターンマッチングを使って、非構造化テキストからすべての URL を検索して抽出します。
问题
URL を含むテキストブロックがあり、プログラムでそれらを抽出する必要があります。
前提条件
- URL 構造の理解
- 正規表現またはパターンマッチングに関する知識
ステップ
-
1. Define what constitutes a URL
URLs start with a scheme (http, https, ftp) followed by :// and a host. They may include port, path, query, and fragment. Decide whether to match only http(s) or all schemes. Also decide whether to include trailing punctuation that is not part of the URL.
入力text with URLs输出matching rules defined -
2. Apply the extraction pattern
Use a regex or URL extraction tool to find all matches. A common pattern is https?://[\w-]+(\.[\w-]+)+[/\w.?=&%-]*. Test against various URL formats: with ports, query strings, fragments, and international domains.
入力Visit https://example.com/page?q=1 today.输出https://example.com/page?q=1 -
3. Clean and validate the results
Strip trailing punctuation (periods, commas, parentheses) that is not part of the URL. Validate each extracted URL by parsing it. Handle edge cases like URLs in parentheses or URLs followed by punctuation.
入力raw matches输出clean, valid URLs
期待される結果
テキストから抽出された有効な URL のリスト。末尾の句読点や無効なマッチは含まれません。
よくあるエラー
- Trailing punctuation included in the URL
- Strip trailing periods, commas, and closing parentheses that are not part of the URL. These are common in sentence-final URLs.
- URLs in parentheses not handled
- URLs inside parentheses (like Wikipedia style) need special handling. Use a parser that balances parentheses.
トラブルシューティング指引
- If valid URLs are missed, relax the pattern or add support for more schemes.
- If invalid strings are matched, tighten the pattern to require a valid host structure.
セキュリティ上の制限
- Extracted URLs may be malicious. Do not automatically visit them. Validate and sandbox any URL processing.