IT Talks/BigData 7

๋ฐ์ดํ„ฐ ์ง๋ฌด๋ณ„ ์Šคํ‚ฌ - Types of Data Professionals

์ „์— ๋ฐ์ดํ„ฐ ์ง๋ฌด๋ณ„ ํ•˜๋Š”์ผ์— ๋Œ€ํ•œ ํฌ์ŠคํŒ…์„ ์˜ฌ๋ฆฐ ์ ์ด ์žˆ์—ˆ๋Š”๋ฐ (์•„๋ž˜ ๊ธ€ ์ฐธ๊ณ ) ์ด๋ฒˆ์—๋Š” ๊ฐ ๋ฐ์ดํ„ฐ ์ง๋ฌด๋ณ„ ์ฃผ์š” ์—…๋ฌด์— ๋Œ€ํ•ด ์ฐจํŠธ ํ˜•ํƒœ๋กœ ๋œ ๊ทธ๋ฆผ์ด ์žˆ์–ด์„œ ๊ณต์œ ํ•ด ๋ณธ๋‹ค. Data Scientist vs Engineer vs Analyst ์˜ˆ์ „์— ์ธํ„ฐ๋„ท์—์„œ ๋ดค๋˜ ์ด๋ฏธ์ง€ ์ธ๋ฐ, ๋ฐ์ดํ„ฐ ๊ด€๋ จ ์ง๋ฌด์— ๋Œ€ํ•ด ์ •๋ฆฌ๊ฐ€ ์ž˜ ์•ˆ๋˜๋Š” ๊ฒฝ์šฐ๊ฐ€ ๋งŽ์•„์„œ ์ฐธ๊ณ ํ•˜๋ฉด ์ข‹์„ ๊ฒƒ ๊ฐ™๋‹ค. (ํ˜„์‹ค์€ ๊ทธ๋ ‡์ง€ ์•Š์ง€๋งŒ) Data Scientist ํ†ต๊ณ„๋‚˜ ๋จธ์‹  ๋Ÿฌ๋‹์„ ์ด์šฉํ•ด ์ฃผ์š” ๋น„ blog.ojj.kr

IT Talks/BigData 2023.08.02

Pandas Data Frame vs Spark DataFrame

DataFrame ์€ ํ–‰๊ณผ ์—ด์ด ์žˆ๋Š” ๋ฐ์ดํ„ฐ ํ…Œ์ด๋ธ”์„ ๋‚˜ํƒ€๋‚ด๋ฉฐ, DataFrame ๊ฐœ๋…์€ ์–ด๋–ค ํ”„๋กœ๊ทธ๋ž˜๋ฐ ์–ธ์–ด์—์„œ๋„ ๋ณ€ํ•˜์ง€ ์•Š์ง€๋งŒ Spark ์™€ Pandas ์˜ DataFrame ์€ ์ƒ๋‹นํžˆ ๋‹ค๋ฅด๋‹ค. ์ด ๊ธ€์—์„œ๋Š” Spark DataFrame๊ณผ Pandas DataFra,e์˜ ์ฐจ์ด์ ์„ ์•Œ์•„๋ณด๋ ค๊ณ  ํ•ฉ๋‹ˆ๋‹ค. Pandas DataFrame Panda๋Š” NumPy ๋ผ์ด๋ธŒ๋Ÿฌ๋ฆฌ๋ฅผ ๊ธฐ๋ฐ˜์œผ๋กœ ํ•˜๋Š” ์˜คํ”ˆ ์†Œ์Šค Python ๋ผ์ด๋ธŒ๋Ÿฌ๋ฆฌ์ž…๋‹ˆ๋‹ค. ๋‹ค์–‘ํ•œ ๋ฐ์ดํ„ฐ ๊ตฌ์กฐ์™€ ์—ฐ์‚ฐ์„ ์‚ฌ์šฉํ•˜์—ฌ ์ˆ˜์น˜ ๋ฐ์ดํ„ฐ์™€ ์‹œ๊ณ„์—ด์„ ์กฐ์ž‘ํ•  ์ˆ˜ ์žˆ๋Š” Python ํŒจํ‚ค์ง€์ž…๋‹ˆ๋‹ค. ์ฃผ๋กœ ๋ฐ์ดํ„ฐ ๊ฐ€์ ธ์˜ค๊ธฐ ๋ฐ ๋ถ„์„์„ ์ƒ๋‹นํžˆ ์‰ฝ๊ฒŒ ํ•˜๊ธฐ ์œ„ํ•ด ์‚ฌ์šฉ๋ฉ๋‹ˆ๋‹ค. Panda DataFrame์€ ๋ ˆ์ด๋ธ”์ด ์ง€์ •๋œ ์ถ•(ํ–‰ ๋ฐ ์—ด)์„ ๊ฐ€์ง„ ์ž ์žฌ์ ์œผ๋กœ ์ด์งˆ์ ์ธ 2์ฐจ์› ํฌ๊ธฐ ๊ฐ€๋ณ€ ํ‘œ ํ˜•์‹ ..

IT Talks/BigData 2023.03.16

Data Scientist vs Engineer vs Analyst

์˜ˆ์ „์— ์ธํ„ฐ๋„ท์—์„œ ๋ดค๋˜ ์ด๋ฏธ์ง€ ์ธ๋ฐ, ๋ฐ์ดํ„ฐ ๊ด€๋ จ ์ง๋ฌด์— ๋Œ€ํ•ด ์ •๋ฆฌ๊ฐ€ ์ž˜ ์•ˆ๋˜๋Š” ๊ฒฝ์šฐ๊ฐ€ ๋งŽ์•„์„œ ์ฐธ๊ณ ํ•˜๋ฉด ์ข‹์„ ๊ฒƒ ๊ฐ™๋‹ค. (ํ˜„์‹ค์€ ๊ทธ๋ ‡์ง€ ์•Š์ง€๋งŒ) Data Scientist ํ†ต๊ณ„๋‚˜ ๋จธ์‹  ๋Ÿฌ๋‹์„ ์ด์šฉํ•ด ์ฃผ์š” ๋น„์ฆˆ๋‹ˆ์Šค ์งˆ๋ฌธ์— ๋Œ€ํ•œ ์˜ˆ์ธก๊ณผ ๋‹ต๋ณ€์„ ๋งŒ๋“ฆ. Data Engineer ๋ฐ์ดํ„ฐ ๊ณผํ•™์ž์™€ ๋ถ„์„๊ฐ€๊ฐ€ ์ž‘์—…์„ ์ˆ˜ํ–‰ํ•  ์ˆ˜ ์žˆ๋Š” ์‹œ์Šคํ…œ์„ ๊ตฌ์ถ•ํ•˜๊ณ  ์ตœ์ ํ™”. Data Analyst ๋ฐ์ดํ„ฐ๋ฅผ ํ™œ์šฉํ•˜์—ฌ ๋น„์ฆˆ๋‹ˆ์Šค ์˜์‚ฌ ๊ฒฐ์ •์— ๋„์›€์ด ๋˜๋Š” ๊ฒฐ๊ณผ๋ฅผ ์ „๋‹ฌํ•จ์œผ๋กœ์จ ๊ฐ€์น˜๋ฅผ ์ œ๊ณต. (์ฐธ๊ณ ) ๋ฐ์ดํ„ฐ ์ง๋ฌด๋ณ„ ์Šคํ‚ฌ ๋ฐ์ดํ„ฐ ์ง๋ฌด๋ณ„ ์Šคํ‚ฌ - Types of Data Professionals ์ „์— ๋ฐ์ดํ„ฐ ์ง๋ฌด๋ณ„ ํ•˜๋Š”์ผ์— ๋Œ€ํ•œ ํฌ์ŠคํŒ…์„ ์˜ฌ๋ฆฐ ์ ์ด ์žˆ์—ˆ๋Š”๋ฐ (์•„๋ž˜ ๊ธ€ ์ฐธ๊ณ ) ์ด๋ฒˆ์—๋Š” ๊ฐ ๋ฐ์ดํ„ฐ ์ง๋ฌด๋ณ„ ์ฃผ์š” ์—…๋ฌด์— ๋Œ€ํ•ด ์ฐจํŠธ ํ˜•ํƒœ๋กœ ๋œ ๊ทธ๋ฆผ์ด ์žˆ์–ด์„œ..

IT Talks/BigData 2022.10.06

AI Platform์˜ ๊ณ ๊ฐํ‰์ƒ๊ฐ€์น˜ ์˜ˆ์ธก: ์†Œ๊ฐœ

์ด ๋ฌธ์„œ๋Š” Google Cloud์—์„œ AI Platform์„ ์‚ฌ์šฉํ•˜์—ฌ ๊ณ ๊ฐํ‰์ƒ๊ฐ€์น˜(CLV)๋ฅผ ์˜ˆ์ธกํ•˜๋Š” ๋ฐฉ๋ฒ•์„ ์„ค๋ช…ํ•˜๋Š” 4๋ถ€๋กœ ๊ตฌ์„ฑ๋œ ์‹œ๋ฆฌ์ฆˆ ์ค‘ ์ฒซ ๋ฒˆ์งธ ๋ฌธ์„œ์ž…๋‹ˆ๋‹ค. ์ด ์‹œ๋ฆฌ์ฆˆ์˜ ๋ฌธ์„œ๋Š” ๋‹ค์Œ๊ณผ ๊ฐ™์Šต๋‹ˆ๋‹ค. 1๋ถ€: ์†Œ๊ฐœ(๋ณธ ๋ฌธ์„œ). ๊ณ ๊ฐํ‰์ƒ๊ฐ€์น˜(CLV)์™€ ์ด๋ฅผ ์˜ˆ์ธกํ•˜๋Š” ๋‘ ๊ฐ€์ง€ ๋ชจ๋ธ๋ง ๊ธฐ๋ฒ•์„ ์†Œ๊ฐœํ•ฉ๋‹ˆ๋‹ค. 2๋ถ€: ๋ชจ๋ธ ํ•™์Šต. ๋ฐ์ดํ„ฐ๋ฅผ ์ค€๋น„ํ•˜๊ณ  ๋ชจ๋ธ์„ ํ•™์Šต์‹œํ‚ค๋Š” ๋ฐฉ๋ฒ•์„ ์„ค๋ช…ํ•ฉ๋‹ˆ๋‹ค. 3๋ถ€: ํ”„๋กœ๋•์…˜์— ๋ฐฐํฌ. 2๋ถ€์—์„œ ์„ค๋ช…๋œ ๋ชจ๋ธ์„ ํ”„๋กœ๋•์…˜ ์‹œ์Šคํ…œ์— ๋ฐฐํฌํ•˜๋Š” ๋ฐฉ๋ฒ•์„ ์„ค๋ช…ํ•ฉ๋‹ˆ๋‹ค. 4๋ถ€: AutoML Tables ์‚ฌ์šฉ. AutoML Tables๋ฅผ ์‚ฌ์šฉํ•˜์—ฌ ๋ชจ๋ธ์„ ๋นŒ๋“œ ๋ฐ ๋ฐฐํฌํ•˜๋Š” ๋ฐฉ๋ฒ•์„ ์„ค๋ช…ํ•ฉ๋‹ˆ๋‹ค. * ์›๋ฌธ : https://cloud.google.com/solutions/machine-learning/clv-predi..

IT Talks/BigData 2020.11.20

IP ์ฃผ์†Œ๋Š” ๊ฐœ์ธ์‹๋ณ„์ •๋ณด์ธ๊ฐ€?

* ์ถœ์ฒ˜ : http://www.boannews.com/media/view.asp?idx=35078 [์ •๋ณด๋ณดํ˜ธ๋ฒ•๋ฐ”๋กœ์•Œ๊ธฐ 28] IP ์ฃผ์†Œ๋Š” ๊ฐœ์ธ์‹๋ณ„์ •๋ณด์ธ๊ฐ€? ๊ฐœ์ธ์ •๋ณด๋ณดํ˜ธ์— ๊ด€ํ•œ ์ธ์‹์€ ๋…์žฌ๊ตญ๊ฐ€์˜ ๋ฌด๋ถ„๋ณ„ํ•œ ๊ฐœ์ธ์ •๋ณด ์ˆ˜์ง‘์œผ๋กœ๋ถ€ํ„ฐ ๋ถ€๊ฐ๋˜์—ˆ์ง€๋งŒ, ์ „ ์„ธ๊ณ„์ ์œผ๋กœ ์ฒด๊ณ„์ ์ธ ๊ฐœ์ธ์ •๋ณด๋ณดํ˜ธ๋ฅผ ์ œ๋„ํ™”ํ•˜๋ ค๋Š” ๋…ธ๋ ฅ์ด ์‹œ์ž‘๋œ ๊ฒƒ์€ 1980๋…„ OECD๊ฐ€ ํ”„๋ผ์ด๋ฒ„์‹œ 8 www.boannews.com ํŠน์ • ์ •๋ณด์— ๋Œ€ํ•œ โ€˜๋ณดํ˜ธ์˜ ํ•„์š”์„ฑโ€™์ด ์žˆ๋Š”์ง€๊ฐ€ ์ค‘์š” ๋•Œ์™€ ์žฅ์†Œยท์ƒํ™ฉ์„ ๊ณ ๋ คํ•œ ์ƒ๋Œ€์ ์ธ ๊ฐœ๋… ๊ณ ๋ ค๋ผ์•ผ [๋ณด์•ˆ๋‰ด์Šค=๋ฒ•๋ฅ ์‚ฌ๋ฌด์†Œ ๋ฏผํ›„ ๊น€๊ฒฝํ™˜ ๋Œ€ํ‘œ๋ณ€ํ˜ธ์‚ฌ] ๊ฐœ์ธ์ •๋ณด๋ณดํ˜ธ์— ๊ด€ํ•œ ์ธ์‹์€ ๋…์žฌ๊ตญ๊ฐ€์˜ ๋ฌด๋ถ„๋ณ„ํ•œ ๊ฐœ์ธ์ •๋ณด ์ˆ˜์ง‘์œผ๋กœ๋ถ€ํ„ฐ ๋ถ€๊ฐ๋˜์—ˆ์ง€๋งŒ, ์ „ ์„ธ๊ณ„์ ์œผ๋กœ ์ฒด๊ณ„์ ์ธ ๊ฐœ์ธ์ •๋ณด๋ณดํ˜ธ๋ฅผ ์ œ๋„ํ™”ํ•˜๋ ค๋Š” ๋…ธ๋ ฅ์ด ์‹œ์ž‘๋œ ๊ฒƒ์€ 1980๋…„ OECD๊ฐ€ ํ”„๋ผ์ด๋ฒ„์‹œ 8์›์น™..

IT Talks/BigData 2015.02.24

ํ•˜๋‘ก์„ ์“ฐ์ง€๋งˆ์„ธ์š”- ๋‹น์‹ ์˜ ๋ฐ์ดํ„ฐ๋Š” ๊ทธ๋ฆฌ ํฌ์ง€ ์•Š์Šต๋‹ˆ๋‹ค

* ์›๊ธ€ : http://www.chrisstucchio.com/blog/2013/hadoop_hatred.html * ๋ฒˆ์—ญ : http://codeflow.co.kr/question/1033/%ED%95%98%EB%91%A1%EC%9D%84-%EC%93%B0%EC%A7%80%EB%A7%88%EC%84%B8%EC%9A%94-%EB%8B%B9%EC%8B%A0%EC%9D%98-%EB%8D%B0%EC%9D%B4%ED%84%B0%EB%8A%94-%EA%B7%B8%EB%A6%AC-%ED%81%AC%EC%A7%80-%EC%95%8A%EC%8A%B5%EB%8B%88%EB%8B%A4/

IT Talks/BigData 2014.01.29
728x90