# 昨天在做爬虫 (September 19, 2024)

基于 playwright 的爬虫，让 LLM 来做标注

<img src="/nlark/yuque/0/2024/png/26070246/1726736636718-e7943c6b-bc7e-4d1c-959c-489d21ab12a2.png" width="1876.5714285714287" title="" crop="0,0,1,1" id="u50d02280" class="ne-image">

结果包含了 LLM 生成的选择器（[经过一个 reflection 步骤](https://github.com/CNSeniorious000/my-auto-annotation/blob/main/app/core/prompt.py)确保了一定能解析）、选择到的元素、这些元素的 computedStyle 交集

核心代码：

```python
@main_loop.callback
class _(Callback):  # noqa: N801
    max_retries = 4

    async def end_process(self, context: ChainContext):
        dom: Selector = context["dom"]

        res: dict[str, str | None] = loads(context.result)
        problems = context["problems"] = []

        for key, selector in res.items():
            if selector:
                print_label(f"{key} - {selector}")
                try:
                    if results := dom.css(selector):
                        show(results)
                    elif self.max_retries:
                        problems.append((key, None))
                    else:
                        res[key] = None

                except SelectorSyntaxError as e:
                    print_exception_only(e)
                    problems.append((key, format_exception_only(e)))

        if not problems or self.max_retries <= 0:
            raise Jump(out_of=main_loop)

        self.max_retries -= 1
```