Python 3

Pandas Data Frame vs Spark DataFrame

DataFrame ์€ ํ–‰๊ณผ ์—ด์ด ์žˆ๋Š” ๋ฐ์ดํ„ฐ ํ…Œ์ด๋ธ”์„ ๋‚˜ํƒ€๋‚ด๋ฉฐ, DataFrame ๊ฐœ๋…์€ ์–ด๋–ค ํ”„๋กœ๊ทธ๋ž˜๋ฐ ์–ธ์–ด์—์„œ๋„ ๋ณ€ํ•˜์ง€ ์•Š์ง€๋งŒ Spark ์™€ Pandas ์˜ DataFrame ์€ ์ƒ๋‹นํžˆ ๋‹ค๋ฅด๋‹ค. ์ด ๊ธ€์—์„œ๋Š” Spark DataFrame๊ณผ Pandas DataFra,e์˜ ์ฐจ์ด์ ์„ ์•Œ์•„๋ณด๋ ค๊ณ  ํ•ฉ๋‹ˆ๋‹ค. Pandas DataFrame Panda๋Š” NumPy ๋ผ์ด๋ธŒ๋Ÿฌ๋ฆฌ๋ฅผ ๊ธฐ๋ฐ˜์œผ๋กœ ํ•˜๋Š” ์˜คํ”ˆ ์†Œ์Šค Python ๋ผ์ด๋ธŒ๋Ÿฌ๋ฆฌ์ž…๋‹ˆ๋‹ค. ๋‹ค์–‘ํ•œ ๋ฐ์ดํ„ฐ ๊ตฌ์กฐ์™€ ์—ฐ์‚ฐ์„ ์‚ฌ์šฉํ•˜์—ฌ ์ˆ˜์น˜ ๋ฐ์ดํ„ฐ์™€ ์‹œ๊ณ„์—ด์„ ์กฐ์ž‘ํ•  ์ˆ˜ ์žˆ๋Š” Python ํŒจํ‚ค์ง€์ž…๋‹ˆ๋‹ค. ์ฃผ๋กœ ๋ฐ์ดํ„ฐ ๊ฐ€์ ธ์˜ค๊ธฐ ๋ฐ ๋ถ„์„์„ ์ƒ๋‹นํžˆ ์‰ฝ๊ฒŒ ํ•˜๊ธฐ ์œ„ํ•ด ์‚ฌ์šฉ๋ฉ๋‹ˆ๋‹ค. Panda DataFrame์€ ๋ ˆ์ด๋ธ”์ด ์ง€์ •๋œ ์ถ•(ํ–‰ ๋ฐ ์—ด)์„ ๊ฐ€์ง„ ์ž ์žฌ์ ์œผ๋กœ ์ด์งˆ์ ์ธ 2์ฐจ์› ํฌ๊ธฐ ๊ฐ€๋ณ€ ํ‘œ ํ˜•์‹ ..

IT Talks/BigData 2023.03.16

Python3 ๊ฐ€์ƒ ํ™˜๊ฒฝ ์ƒ์„ฑ

venv ๋ชจ๋“ˆ์€ ์ž์ฒด ์‚ฌ์ดํŠธ ๋””๋ ‰ํ„ฐ๋ฆฌ๋ฅผ ๊ฐ–๋Š” ๊ฒฝ๋Ÿ‰ โ€œ๊ฐ€์ƒ ํ™˜๊ฒฝโ€์„ ๋งŒ๋“ค๊ณ , ์„ ํƒ์ ์œผ๋กœ ์‹œ์Šคํ…œ ์‚ฌ์ดํŠธ ๋””๋ ‰ํ„ฐ๋ฆฌ์—์„œ ๊ฒฉ๋ฆฌํ•  ์ˆ˜ ์žˆ๋„๋ก ์ง€์›ํ•ฉ๋‹ˆ๋‹ค. ๊ฐ ๊ฐ€์ƒ ํ™˜๊ฒฝ์€ ๊ณ ์œ ํ•œ ํŒŒ์ด์ฌ ๋ฐ”์ด๋„ˆ๋ฆฌ(์ด ํ™˜๊ฒฝ์„ ๋งŒ๋“œ๋Š” ๋ฐ ์‚ฌ์šฉ๋œ ๋ฐ”์ด๋„ˆ๋ฆฌ ๋ฒ„์ „๊ณผ ์ผ์น˜ํ•จ)๋ฅผ ๊ฐ€์ง€๋ฉฐ ์ž์‹ ์˜ ์‚ฌ์ดํŠธ ๋””๋ ‰ํ„ฐ๋ฆฌ์— ๋…๋ฆฝ์ ์œผ๋กœ ์„ค์น˜๋œ ํŒŒ์ด์ฌ ํŒจํ‚ค์ง€ ์ง‘ํ•ฉ์„ ๊ฐ€์งˆ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ๊ฐ€์ƒํ™˜๊ฒฝ ์ƒ์„ฑ $ python3 -m venv /home/myhome/venv * ์ง€์ •ํ•œ ๋””๋ ‰ํ† ๋ฆฌ์— ๊ฐ€์ƒ ํ™˜๊ฒฝ ์ƒ์„ฑ (๋ฆฌ๋ˆ…์Šค ๊ธฐ์ค€) * ์‹คํ–‰ ํ›„ ์ด๋ž˜์™€ ๊ฐ™์€ ์˜ค๋ฅ˜๊ฐ€ ๋ฐœ์ƒํ•œ๋‹ค๋ฉด --without-pip ์˜ต์…˜์œผ๋กœ pip ๋Š” ๋ณ„๋„๋กœ ์„ค์น˜ํ•ด ์ค€๋‹ค. Error: Command '['/home/ojj/venv/bin/python3', '-Im', 'ensurepip', '--upgrade',..

IT Talks 2022.09.22

python ์˜ sys.stderr.write(f"ERROR: {exc}") ์˜ค๋ฅ˜

๊ฐ‘์ž๊ธฐ ์ด๋Ÿฐ ๋ฉ”์‹œ์ง€๊ฐ€ ๋‚˜์˜จ๋‹ค๋ฉด ? $ sudo python3 -m pip Traceback (most recent call last): File "/usr/lib/python3.5/runpy.py", line 184, in _run_module_as_main "__main__", mod_spec) File "/usr/lib/python3.5/runpy.py", line 85, in _run_code exec(code, run_globals) File "/usr/local/lib/python3.5/dist-packages/pip/__main__.py", line 29, in from pip._internal.cli.main import main as _main File "/usr/local/lib/pyth..

IT Talks/Linux 2021.05.15
728x90