feat: add OpenInsider Top-of-day crawler and multi-CRAWLER_TYPE support

New crawler: app/crawlers/openinsider_top.py\n- Scrapes three pages (sales/purchases/officer purchases)\n- Filters rows with Value/Amount >= ,000,000 (configurable via INSIDER_MIN_AMOUNT)\n- Builds concise notifications; saves to data/openinsider_top.json

Runner: support comma-separated CRAWLER_TYPE and new openinsider_top type\n- Accepts e.g., CRAWLER_TYPE=openinsider_top,openinsider,barrons\n- Preserves order, removes duplicates; warns on unknown types\n- Uses shared schedule: RUN_DAILY_AT or CHECK_INTERVAL; initial run per crawler

Entrypoint: rename enhanced_crawler.py -> main.py\n- Update Dockerfile CMD and README references

Config & docs:\n- Reorganize .env.template into clear sections with examples\n- Update .env with multi-crawler example and INSIDER_MIN_AMOUNT\n- README: document new crawler, usage, and multi-type CRAWLER_TYPE
This commit is contained in:
2025-09-09 21:17:50 +08:00
parent b2c58c0560
commit f708f3bf1d
6 changed files with 308 additions and 27 deletions

View File

@@ -1,9 +1,10 @@
# 股票爬蟲服務(模組化架構)
可擴充的股票爬蟲服務,內建 HTTP API 與多種通知Email/Webhook/Discord
目前提供類爬蟲:
目前提供類爬蟲:
- Barron's 股票推薦
- OpenInsider 內部人交易(支援多標的)
- OpenInsider 當日大額內部人交易(跨三頁合併、金額過濾)
## 功能
- 定時抓取(支援每 N 秒或每日固定時間)
@@ -24,7 +25,7 @@ app/
crawlers/template.py # 新站點範本(複製後改名擴充)
services/storage.py # JSON 儲存
services/notifications.py # Email/Webhook/Discord
enhanced_crawler.py # 入口,委派到 app.runner
main.py # 入口,委派到 app.runner
Dockerfile
docker-compose.yml
requirements.txt
@@ -58,7 +59,7 @@ curl "http://localhost:8080/notify_test?channel=email" # 或 webhook/discord
## 本機執行(非 Docker
```bash
pip install -r requirements.txt
python enhanced_crawler.py
python main.py
```
## 環境變數說明
@@ -80,11 +81,17 @@ python enhanced_crawler.py
- `LOG_DIR`: 日誌輸出路徑Docker 預設 `/app/logs`;本機預設 `./logs`
- 爬蟲選擇與參數
- `CRAWLER_TYPE`: `barrons` | `openinsider`
- `CRAWLER_TYPE`: `barrons` | `openinsider` | `openinsider_top`(可用逗號同時啟多種,例如:`openinsider_top,openinsider`
- Barron's無額外參數
- OpenInsider
- OpenInsider(依個別股票查詢)
- 單一標的:`SYMBOL=PLTR`
- 多個標的:`SYMBOLS=PLTR,NVDA,TSLA`
- OpenInsider 當日大額(跨頁合併):
- 來源頁面:
- `http://openinsider.com/top-insider-sales-of-the-day`
- `http://openinsider.com/top-insider-purchases-of-the-day`
- `http://openinsider.com/top-officer-purchases-of-the-day`
- 金額門檻(含千分位、自動去 `$``INSIDER_MIN_AMOUNT`,預設 `1000000`
Email 使用建議:
- Gmail 請使用「應用程式密碼」並開啟兩步驟驗證
@@ -141,3 +148,4 @@ docker-compose down
- 2025-09
- 重構為模組化架構API 與爬蟲邏輯分離
- 新增 OpenInsider 內部人交易爬蟲與多標的支援
- 新增 OpenInsider 當日大額內部人交易(≥$1,000,000爬蟲