mhhung/stock-info-crawler
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
8 Commits 1 Branch 0 Tags
master
Commit Graph

8 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
MH Hung
f676f44645 feat(openinsider_top): add trade type tag in notifications and ensure absolute amount display 
- Prefix titles and email lines with normalized trade tag (BUY/SALE/...)\n- Include amount as absolute value in lines; keep threshold logic unchanged
 2025-09-09 21:32:07 +08:00
MH Hung
e015eef61e fix(openinsider_top): treat negative Value on sales as absolute for threshold parsing 
Handle strings like '-,234,567' by normalizing and parsing magnitude; ensures sales rows are included when exceeding INSIDER_MIN_AMOUNT.
 2025-09-09 21:26:52 +08:00
MH Hung
f708f3bf1d feat: add OpenInsider Top-of-day crawler and multi-CRAWLER_TYPE support 
New crawler: app/crawlers/openinsider_top.py\n- Scrapes three pages (sales/purchases/officer purchases)\n- Filters rows with Value/Amount >= ,000,000 (configurable via INSIDER_MIN_AMOUNT)\n- Builds concise notifications; saves to data/openinsider_top.json

Runner: support comma-separated CRAWLER_TYPE and new openinsider_top type\n- Accepts e.g., CRAWLER_TYPE=openinsider_top,openinsider,barrons\n- Preserves order, removes duplicates; warns on unknown types\n- Uses shared schedule: RUN_DAILY_AT or CHECK_INTERVAL; initial run per crawler

Entrypoint: rename enhanced_crawler.py -> main.py\n- Update Dockerfile CMD and README references

Config & docs:\n- Reorganize .env.template into clear sections with examples\n- Update .env with multi-crawler example and INSIDER_MIN_AMOUNT\n- README: document new crawler, usage, and multi-type CRAWLER_TYPE
 2025-09-09 21:17:50 +08:00
MH Hung
b2c58c0560 refactor(notifications): unify email sending via _send_email; standardize crawler notifications\n\n- Extract _send_email and have send_email/send_custom_email share it\n- BaseCrawler: centralize _send_notifications and add _build_email hook\n- BarronsCrawler: override _build_email to keep original subject/body\n- OpenInsiderCrawler: remove custom _send_notifications, add _build_email\n- /notify_test: use crawler _build_email + send_custom_email for emails 2025-09-04 22:59:51 +08:00
MH Hung
e89567643b feat(openinsider): 新增 OpenInsider 內部人交易爬蟲，支援多標的與每日排程 
- 新增 app/crawlers/openinsider.py，來源 http://openinsider.com/search?q={symbol}

- 支援多標的：以 SYMBOLS=PLTR,NVDA,... 同時追多檔（或使用 SYMBOL 單一）

- runner: 多實例排程與啟動；/check 會依序觸發全部爬蟲

- API: /info、/stats、/check、/notify_test 支援多爬蟲回應

- config/base: 新增 RUN_DAILY_AT 每日固定時間；未設定則用 CHECK_INTERVAL

- notifications: 新增 send_custom_email、send_text_webhook、send_text_discord

- README 與 .env.template 更新；.env 改為 CRAWLER_TYPE=openinsider

- 移除 quiver_insiders 爬蟲與相關設定

BREAKING CHANGE: 不再支援 CRAWLER_TYPE=quiver_insiders；請改用 openinsider。
 2025-09-04 22:32:29 +08:00
MH Hung
58cc979b5b refactor: modularize project structure and separate API from crawlers 
- Introduce app/ package with config, services (storage, notifications), API server, and crawler modules
- Add BaseCrawler and BarronsCrawler; extract notifications and storage
- Keep enhanced_crawler.py as back-compat entry delegating to app.runner
- Add template crawler for future sites
- Update README with new structure and usage
- Extend .env.template with DATA_DIR/LOG_DIR options
 2025-09-04 21:39:24 +08:00
MH Hung
099f156e6f feat(email): add SMTP security modes (starttls/ssl/none) with sensible default ports; add /notify_test endpoint; support ALWAYS_NOTIFY_ON_STARTUP to force first-run notification 
chore(docker): run enhanced_crawler.py as entrypoint

ops(compose): load env via env_file and remove hardcoded secrets

docs: update README and .env.template for SMTP and startup notification
 2025-09-03 21:32:50 +08:00
MH Hung
852f206d2e feat: init project 2025-09-03 16:47:02 +08:00