Week1 Day 5 Lab 爬取网站链接分析

 

 

业务目标

输入公司名称 + 官网地址，自动抓取网站链接，用大模型筛选业务相关链接（关于我们、招聘、公司介绍等，排除隐私 / 服务条款），抓取对应页面内容，最后生成面向潜在客户、投资者、求职者的企业宣传册（Markdown 格式），支持流式打字机输出效果。

核心流程

1. 抓取网站全部链接
2. LLM 筛选业务相关链接，输出 JSON 结构化结果
3. 抓取首页 + 筛选出的相关页面文本内容
4. 组装页面内容到 Prompt，调用 LLM 生成 Markdown 宣传册
5. 优化：流式输出，实现打字机动态渲染

依赖说明：代码中scraper.py是外部爬虫模块，提供 2 个函数

• fetch_website_links(url)：获取网页上所有链接
• fetch_website_contents(url)：获取网页正文文本

---

完整可运行代码

注意：

1. 同目录需要有scraper.py爬虫文件；
2. 创建.env文件写入OPENAI_API_KEY=sk-xxx；
3. Jupyter Notebook 环境运行（用到 IPython.display）；
4. 模型名称按你的环境修改。

python

 

运行

 

 

# %% [markdown]
# # A full business solution 完整的业务解决方案
# 业务挑战：根据公司名称与官网，自动生成企业宣传册，面向潜在客户、投资者、求职者

# %%
# imports
import os
import json
from dotenv import load_dotenv
from IPython.display import Markdown, display, update_display
# 外部爬虫模块 scraper.py
from scraper import fetch_website_links, fetch_website_contents
from openai import OpenAI

# Initialize and constants
load_dotenv(override=True)
api_key = os.getenv('OPENAI_API_KEY')

if api_key and api_key.startswith('sk-proj-') and len(api_key)>10:
    print("API key looks good so far")
else:
    print("There might be a problem with your API key? Please visit the troubleshooting notebook!")

MODEL = 'gpt-5-nano'
openai = OpenAI()

# %% [markdown]
# ## 第一步：大模型筛选网站相关链接，输出JSON格式
# %%
link_system_prompt = """
You are provided with a list of links found on a webpage.
You are able to decide which of the links would be most relevant to include in a brochure about the company,
such as links to an About page, or a Company page, or Careers/Jobs pages.
You should respond in JSON as in this example:

{
    "links": [
        {"type": "about page", "url": "https://full.url/goes/here/about"},
        {"type": "careers page", "url": "https://another.full.url/careers"}
    ]
}
"""

def get_links_user_prompt(url):
    user_prompt = f"""
Here is the list of links on the website {url} -
Please decide which of these are relevant web links for a brochure about the company,
respond with the full https URL in JSON format.
Do not include Terms of Service, Privacy, email links.

Links (some might be relative links):

"""
    links = fetch_website_links(url)
    user_prompt += "\n".join(links)
    return user_prompt


def select_relevant_links(url):
    print(f"Selecting relevant links for {url} by calling {MODEL}")
    response = openai.chat.completions.create(
        model=MODEL,
        messages=[
            {"role": "system", "content": link_system_prompt},
            {"role": "user", "content": get_links_user_prompt(url)}
        ],
        response_format={"type": "json_object"}
    )
    result = response.choices[0].message.content
    links = json.loads(result)
    print(f"Found {len(links['links'])} relevant links")
    return links

# 测试链接筛选
# select_relevant_links("https://edwarddonner.com")
# select_relevant_links("https://huggingface.co")

# %% [markdown]
# ## 第二步：抓取首页 + 全部相关链接页面内容，组装上下文
# %%
def fetch_page_and_all_relevant_links(url):
    contents = fetch_website_contents(url)
    relevant_links = select_relevant_links(url)
    result = f"## Landing Page:\n\n{contents}\n## Relevant Links:\n"
    for link in relevant_links['links']:
        result += f"\n\n### Link: {link['type']}\n"
        result += fetch_website_contents(link["url"])
    return result

# %% [markdown]
# ## 生成宣传册Prompt配置
# %%
# 标准版宣传册system prompt
brochure_system_prompt = """
You are an assistant that analyzes the contents of several relevant pages from a company website
and creates a short brochure about the company for prospective customers, investors and recruits.
Respond in markdown without code blocks.
Include details of company culture, customers and careers/jobs if you have the information.
"""

# 幽默趣味版本，需要可以切换打开
# brochure_system_prompt = """
# You are an assistant that analyzes the contents of several relevant pages from a company website
# and creates a short, humorous, entertaining, witty brochure about the company for prospective customers, investors and recruits.
# Respond in markdown without code blocks.
# Include details of company culture, customers and careers/jobs if you have the information.
# """

def get_brochure_user_prompt(company_name, url):
    user_prompt = f"""
You are looking at a company called: {company_name}
Here are the contents of its landing page and other relevant pages;
use this information to build a short brochure of the company in markdown without code blocks.

"""
    user_prompt += fetch_page_and_all_relevant_links(url)
    user_prompt = user_prompt[:5_000]  # Truncate if more than 5,000 characters
    return user_prompt

# %% [markdown]
# ### 普通一次性输出宣传册
# %%
def create_brochure(company_name, url):
    response = openai.chat.completions.create(
        model="gpt-4.1-mini",
        messages=[
            {"role": "system", "content": brochure_system_prompt},
            {"role": "user", "content": get_brochure_user_prompt(company_name, url)}
        ],
    )
    result = response.choices[0].message.content
    display(Markdown(result))

# %% [markdown]
# ### 流式输出：打字机动画效果（推荐）
# %%
def stream_brochure(company_name, url):
    stream = openai.chat.completions.create(
        model="gpt-4.1-mini",
        messages=[
            {"role": "system", "content": brochure_system_prompt},
            {"role": "user", "content": get_brochure_user_prompt(company_name, url)}
          ],
        stream=True
    )
    response = ""
    display_handle = display(Markdown(""), display_id=True)
    for chunk in stream:
        response += chunk.choices[0].delta.content or ''
        update_display(Markdown(response), display_id=display_handle.display_id)


# ====================== 运行示例 ======================
# create_brochure("HuggingFace", "https://huggingface.co")
stream_brochure("HuggingFace", "https://huggingface.co")
 

 

关键知识点梳理

1. One‑shot Prompting（一次性提示）：在 system prompt 中直接给出 JSON 输出样例，指导 LLM 输出指定格式；
2. response_format={"type":"json_object"}：强制大模型输出合法 JSON；
3. 上下文截断：user_prompt[:5_000]，限制传入 LLM 的文本长度，防止超长报错；
4. 流式输出 stream=True：逐块获取返回结果，配合IPython.display.update_display实现打字机动态渲染；
5. Prompt 调音调优：修改 system prompt 即可轻松改变输出风格（正式 / 幽默），不用改动业务逻辑。