feat: add OpenInsider Top-of-day crawler and multi-CRAWLER_TYPE support
New crawler: app/crawlers/openinsider_top.py\n- Scrapes three pages (sales/purchases/officer purchases)\n- Filters rows with Value/Amount >= ,000,000 (configurable via INSIDER_MIN_AMOUNT)\n- Builds concise notifications; saves to data/openinsider_top.json Runner: support comma-separated CRAWLER_TYPE and new openinsider_top type\n- Accepts e.g., CRAWLER_TYPE=openinsider_top,openinsider,barrons\n- Preserves order, removes duplicates; warns on unknown types\n- Uses shared schedule: RUN_DAILY_AT or CHECK_INTERVAL; initial run per crawler Entrypoint: rename enhanced_crawler.py -> main.py\n- Update Dockerfile CMD and README references Config & docs:\n- Reorganize .env.template into clear sections with examples\n- Update .env with multi-crawler example and INSIDER_MIN_AMOUNT\n- README: document new crawler, usage, and multi-type CRAWLER_TYPE
This commit is contained in:
18
README.md
18
README.md
@@ -1,9 +1,10 @@
|
||||
# 股票爬蟲服務(模組化架構)
|
||||
|
||||
可擴充的股票爬蟲服務,內建 HTTP API 與多種通知(Email/Webhook/Discord)。
|
||||
目前提供兩類爬蟲:
|
||||
目前提供三類爬蟲:
|
||||
- Barron's 股票推薦
|
||||
- OpenInsider 內部人交易(支援多標的)
|
||||
- OpenInsider 當日大額內部人交易(跨三頁合併、金額過濾)
|
||||
|
||||
## 功能
|
||||
- 定時抓取(支援每 N 秒或每日固定時間)
|
||||
@@ -24,7 +25,7 @@ app/
|
||||
crawlers/template.py # 新站點範本(複製後改名擴充)
|
||||
services/storage.py # JSON 儲存
|
||||
services/notifications.py # Email/Webhook/Discord
|
||||
enhanced_crawler.py # 舊入口,現委派到 app.runner
|
||||
main.py # 入口點,委派到 app.runner
|
||||
Dockerfile
|
||||
docker-compose.yml
|
||||
requirements.txt
|
||||
@@ -58,7 +59,7 @@ curl "http://localhost:8080/notify_test?channel=email" # 或 webhook/discord
|
||||
## 本機執行(非 Docker)
|
||||
```bash
|
||||
pip install -r requirements.txt
|
||||
python enhanced_crawler.py
|
||||
python main.py
|
||||
```
|
||||
|
||||
## 環境變數說明
|
||||
@@ -80,11 +81,17 @@ python enhanced_crawler.py
|
||||
- `LOG_DIR`: 日誌輸出路徑(Docker 預設 `/app/logs`;本機預設 `./logs`)
|
||||
|
||||
- 爬蟲選擇與參數
|
||||
- `CRAWLER_TYPE`: `barrons` | `openinsider`
|
||||
- `CRAWLER_TYPE`: `barrons` | `openinsider` | `openinsider_top`(可用逗號同時啟多種,例如:`openinsider_top,openinsider`)
|
||||
- Barron's:無額外參數
|
||||
- OpenInsider:
|
||||
- OpenInsider(依個別股票查詢):
|
||||
- 單一標的:`SYMBOL=PLTR`
|
||||
- 多個標的:`SYMBOLS=PLTR,NVDA,TSLA`
|
||||
- OpenInsider 當日大額(跨頁合併):
|
||||
- 來源頁面:
|
||||
- `http://openinsider.com/top-insider-sales-of-the-day`
|
||||
- `http://openinsider.com/top-insider-purchases-of-the-day`
|
||||
- `http://openinsider.com/top-officer-purchases-of-the-day`
|
||||
- 金額門檻(含千分位、自動去 `$`):`INSIDER_MIN_AMOUNT`,預設 `1000000`
|
||||
|
||||
Email 使用建議:
|
||||
- Gmail 請使用「應用程式密碼」並開啟兩步驟驗證
|
||||
@@ -141,3 +148,4 @@ docker-compose down
|
||||
- 2025-09:
|
||||
- 重構為模組化架構,API 與爬蟲邏輯分離
|
||||
- 新增 OpenInsider 內部人交易爬蟲與多標的支援
|
||||
- 新增 OpenInsider 當日大額內部人交易(≥$1,000,000)爬蟲
|
||||
|
Reference in New Issue
Block a user