o
ohk$ @ s d dl mZ d dlZd dlZd dlmZmZ d dlmZmZm Z m
Z
d dl Z d dl m
Z
d dlmZmZ d dlmZ e ZdZd Zd
d iZddd d
defddZdedefddZdddedededededB f
dd Zd ede e
eeef fd d Z d ede
e e ef fd!d"Z d#e e
de e fd$d%Z d&ede e fd'd(Z d)ed*ed+ede e fd,d-Z d.ede!fd/d0Z"d1ede!fd2d3Z#deeeeB f fd4d5Z$e%d6kre&e$ dS dS )7 )escapeN)datetimetimezone)DictListOptionalTuple)
BeautifulSoup)
get_clientlogger)broadcast_htmlz5https://www.moef.go.kr/nw/notice/hr.do?menuNo=4050300z,https://www.moef.go.kr/nw/notice/hrDetail.doz
User-AgentzCMozilla/5.0 (compatible; govbot/1.0; +https://work.jjickjjicks.com) paramsheaderstimeouturlc C s d}t dD ]=}ztj| |||d}| |W S tyC } z |dkr' td|d | t| |d9 }W Y d }~qd }~ww t d) N ? r zGET retry (%s): %s unreachable)
rangerequestsgetraise_for_status Exceptionr warningtimesleep RuntimeError)r r r r delayire r& )/var/www/html/bot/app/crawler/moef_n8n.py_get_with_retry s
r( targetreturnc C s dd l }t| }ztd|| dddd ttj
d W |S t y@ } zt
d| W Y d }~|S d }~ww )Nr crawler_runrunning)idr) statuspagesrows fail_reason
started_atz[crawler_run insert] )uuidstruuid4supabasetableinsertr nowr utc isoformatexecuter r r )r) r3 run_idr% r& r& r'
_run_start) s&
r> )r1 r= r. r/ r0 r1 c
C s |t tj ||d}|d ur||d< ztd|d| W d S t
yA } zt d| W Y d }~d S d }~ww )N)r. finished_atr/ r0 r1 r+ r- z[crawler_run finish] )
r r9 r r: r; r6 r7 updateeqr< r r r )r= r. r/ r0 r1 payloadr% r& r& r' _run_finish: s " rC href_jsc C s4 t d| pd}|s dS |d|d|dfS )za
javascript:fn_egov_select('4050300','MOSFBBS_...','POST_ID') -> (menuNo, bbsId, postId)
z=fn_egov_select\('(\d+)','([A-Za-z0-9_]+)','([A-Za-z0-9_]+)'\) Nr r r )researchgroup)rD mr& r& r' _extract_js_paramsK s rJ
title_textc C sD | sdS t d| }|r |d |d fS d| fS )uL
'[태그] 제목' -> ('태그','제목'), 아니면 (None, 원제)
)NrE z^\[(.+?)\]\s*(.+)$r r N)rF matchstriprH )rK rI r& r& r' _split_tag_from_titleT s rN containerc C sB | sdS | d}|r |jdddpd }|d}|p dS dS )u~
목록 DOM에서 'state*' 클래스를 갖는 태그(예: 인사발령)를 우선 추출
Nz@span[class*="state"], em[class*="state"], strong[class*="state"] TrM rE z[]())
select_oneget_textrM )rO eltr& r& r' _extract_tag_from_state_ s
rV htmlc C s t d| }|st d| }|sdS |ddd}z t|d}|dW S tyY |d}dd t |D }ztd
|d}|dW Y S tyX Y Y dS w w )
uj
상세 페이지에서 '등록일' 또는 날짜 패턴(YYYY.MM.DD / YYYY-MM-DD) -> 'YYYY-MM-DD'
u3 등록일[^0-9]*(\d{4}[.\-](\d{1,2})[.\-](\d{1,2}))z#(\d{4}[.\-](\d{1,2})[.\-](\d{1,2}))Nr .-z%Y-%m-%dc S s& g | ]\}}|d kr| dn|qS )r r )zfill).0r# pr& r& r'
{ s & z0_parse_date_from_detail_html..) rF rG rH replacer strptimestrftimer split enumeratejoin)rW rI rawdtpartsr& r& r' _parse_date_from_detail_htmll s&
rg menuNobbsIdpostIdc C s& ||| d}t t|tdd}t|jS )N) searchBbsId1 searchNttId1rh r
r )r(
DETAIL_URLHEADERSrg text)rh ri rj r r$ r& r& r' _detail_posted_at s
rp row_idc C s, t ddd| d }t|jS )Nmoef_idr- r )r6 r7 selectrA limitr< booldata)rq resr& r& r' _exists_in_supabase s "
rx rowc s g dg dg dg dg}|D ]6} fdd|D }zt d| W dS tyF } ztd | d
| W Y d }~qd }~ww d S )
u<
tag/posted_at 유무와 무관하게 유연 삽입
)r- ri rj titletag posted_at)r- ri rj rz r| )r- ri rj rz r{ r- ri rj rz c s i | ] }| v r| | qS r& r& )r[ kry r& r'
s z _safe_insert..rr Tzinsert retry with fields z failed: NF)r6 r7 r8 r< r r r )ry fields_priorityfieldsrv r% r& r r' _safe_insert s "r c C s< d} t | }d}d}d}zbtttdd}|d7 }t|jd}|d}|D ]-}|dd } t| }
|
s6q&|
\} } }
| d
|
}t |rGq&d }|
d }|r^| d }|r^|j d
dd}|si|j d
ddphd}|
dp|
dp|
dp|
dp|j}t|}t|\}}|s|}n|}zt| | |
}W n ty } ztd| d| d }W Y d }~nd }~ww || |
|p|d}|r||d< |r||d< t|}|std| q&|d7 }t d| d |
d | }|rd t| d t|d d d!n d"t|d d d!}d#| d$| d%}zt|} |t| tr't| n| r,dnd7 }W n tyN } ztd&| d| W Y d }~nd }~ww t d' q&t |d(||d) t d*| d+| |||d(d,W S ty } z!t |d-||t |d. t d/| t ||d-d0W Y d }~S d }~ww )1Nrr r r
)r r r z html.parserz$a[href*="javascript:fn_egov_select"]hrefrE rY h3arP TrQ u 기재부 인사발령litrarticledivzfetch posted_at failed for z: r} r{ r| zinsert failed for z?searchBbsId1=z&searchNttId1=z&menuNo=z[(z) rz ][u! 기재부 인사발령입니다.
z
[자세히 보기]ztelegram send failed for r passed)r. r/ r0 zMOEF run: inserted=z, sent=)insertedsentr= r. aborted)r. r/ r0 r1 zMOEF run aborted: )errorr= r. )!r> r( LIST_URLrn r ro rs r rJ rx
find_previousfindrS find_parentparentrV rN rp r r r r rm r r
isinstanceintr r rC infor4 exception) r) r= r/ r r r$ soupanchorsr rD r rh ri rj rq rK r linkrO r{ tag2 clean_titler| r% ry okheadmessagesent_resr& r& r' run_once s
@*" r __main__)'rW r rF r r r typingr r r r r bs4r app.services.supabase_servicer
r app.services.telegram_senderr r6 r rm rn r4 r( r> r rC rJ rN rV rg rp ru rx r r __name__printr& r& r& r' s8 (
h