Docker

Docker Environment - Image, Container

목차
- Install
- Image
  - 원격 image 다운로드
  - image 목록 확인
  - image 삭제
  - Dockerfile로 새로운 image 만들기
  - image를 원격 registry에 업로드
  - image를 압축파일로 저장
  - image 압축파일 다운로드
  - Dockerfile 작성
- Container
  - image로부터 container 새로 띄우기
  - 이미 띄운 container에 -v 폴더 연결
  - container 목록 확인
  - container 이름 변경
  - 종료했던 container 띄우기
  - 띄운 container에 접속
  - container 종료하지 않고 탈출
  - container 종료
  - container 삭제
  - 외부 파일/폴더를 container로 복사
  - container를 image로 변환
  - container를 압축파일로 저장
  - container 압축파일 다운로드하여 image 만들기

Install

vscode에 docker 설치: vscode 익스텐션 중 Docker, Dev Containers 설치
에러 해결:
- docker: Error response from daemon: could not select device driver “” with capabilities: [[gpu]].
  Sol.: nvidia-cuda-toolkit을 설치 또는 재설치한다.
- docker: Error response from daemon: failed to create shim task: nvidia-container-cli: requirement error: unsatisfied condition: cuda>=xx.x, please update your driver to a newer version
  Sol.: 설치된 cuda 버전이 docker image가 요구하는 cuda version보다 낮아서 생기는 에러로, cuda 또는 nvidia-driver를 버전 업그레이드

sudo 매번 안 치게:

sudo usermod -aG docker $USER
sudo usermod -aG docker semyu
sudo reboot

Image

# 원격 image 다운로드
docker pull pytorch/pytorch:latest
# image name은 <Repository>:<Tag> 꼴

# image 목록 확인
docker images

# image 삭제
docker rmi 0x3f3fsc # <Image ID>

# Dockerfile로 새로운 image 만들기
docker build --progress=plain -t myownimg/myownimg:latest -f ./Dockerfile .

# image를 원격 registry에 업로드
docker push myownimg/myownimg:latest

# image를 압축파일로 저장
docker save -o ddrf.tar pytorch/pytorch:latest

# image 압축파일 다운로드
# docker save -> docker load
docker load -i ddrf.tar

Dockerfile 작성:
https://hub.docker.com/r/nvidia/cuda/tags?name=12.8 들어가서 원하는 cuda 버전의 tag 확인
(base, runtime 는 compile 하기 위한 extension이 없지만 가벼움, devel은 compile까지 가능하지만 무거움)
https://pytorch.org/get-started/previous-versions/ 들어가서 pip으로 원하는 버전의 pytorch 설치하는 명령어 확인
다 확인하면 아래와 같이 Dockerfile 작성

# 아까 확인한 cuda tag
FROM nvidia/cuda:<CUDA_VERSION>-<base/runtime/devel>-ubuntu<UBUNTU_VERSION> 

ENV DEBIAN_FRONTEND=noninteractive

# 기본 패키지 설치 (옵션)
RUN apt-get update && apt-get install -y \
    build-essential \
    wget \
    git \
    curl \
    ca-certificates \
    python3.11 python3.11-dev python3.11-distutils pip\
    && rm -rf /var/lib/apt/lists/*

# 필요하면 추가로 환경 설정 (옵션)
ENV PATH=/usr/local/cuda/bin:${PATH}
ENV LD_LIBRARY_PATH=/usr/local/cuda/lib64:${LD_LIBRARY_PATH}


# 아까 확인한 pytorch 설치 명령어
RUN pip3 install torch torchvision --index-url https://download.pytorch.org/whl/cu128 

CMD ["/bin/bash"]

Dockerfile 예시:
Blender with GPU로 synthetic multi-object multi-view videos 만드는 Kubric code에 있는 Dockerfile
https://github.com/ethz-vlg/kubric/blob/multiview-point-tracking/challenges/point_tracking_3d/Dockerfile

# Creating and testing the created image:
#   docker build --progress=plain -t franorajic/nvidia-pytorch-blender-kubric:0.2 -f challenges/point_tracking_3d/Dockerfile . 2>&1 | tee build_log_7.txt
#   docker images
#   docker run --gpus all -it --rm -v /home/frrajic/xode/01-code/kubric:/workspace franorajic/nvidia-pytorch-blender-kubric:0.2 /bin/bash
#     cd /workspace && python challenges/point_tracking_3d/worker.py --job-dir output/multiview_v3/debug/72 --seed 72 --n_cameras 10 --save_state --scratch_dir output/tmp
#   docker push franorajic/nvidia-pytorch-blender-kubric:0.2

FROM nvcr.io/nvidia/pytorch:24.04-py3

LABEL Title="Kubric with Blender 3.6 and GPU support"
LABEL Author="m43 <frano.rajic@inf.ethz.ch>"

ENV DEBIAN_FRONTEND=noninteractive
ENV LC_ALL=C.UTF-8
ENV LANG=C.UTF-8

WORKDIR /blenderpy

# Install basic system dependencies
RUN apt-get update --yes --fix-missing && apt-get install --yes --quiet --no-install-recommends git git-lfs ffmpeg

# Build Blender from source
RUN git clone https://github.com/blender/blender.git --branch v3.6.16 --depth 1
RUN cd blender && \
    sed -i 's/"sudo",//g' ./build_files/build_environment/install_linux_packages.py && \
    sed -i 's/libshaderc-dev/wget/g' ./build_files/build_environment/install_linux_packages.py && \
    ./build_files/build_environment/install_linux_packages.py --all
RUN cd blender && ./build_files/utils/make_update.py --use-linux-libraries

# Build Blender with CUDA support
RUN mkdir blender/build_blender && \
    cd blender/build_blender && \
    cmake -DWITH_CYCLES_CUDA_BINARIES=ON ..
RUN cd blender/build_blender && make -j24
RUN cd blender/build_blender && make install

# Build the Python module (bpy)
RUN mkdir blender/build_bpy && \
    cd blender/build_bpy && \
    cmake -C ../build_files/cmake/config/bpy_module.cmake -DWITH_CYCLES_CUDA_BINARIES=ON ..
RUN cd blender/build_bpy && make -j24
RUN cd blender/build_bpy && make install
RUN cd blender && python ./build_files/utils/make_bpy_wheel.py ./build_bpy/bin
RUN cd blender && pip install ./build_bpy/bin/bpy*.whl

# Add shared libraries to the path
ENV LD_LIBRARY_PATH="/blenderpy/blender/lib/linux_x64/alembic/lib:$LD_LIBRARY_PATH"
ENV LD_LIBRARY_PATH="/blenderpy/blender/lib/linux_x64/boost/lib:$LD_LIBRARY_PATH"
ENV LD_LIBRARY_PATH="/blenderpy/blender/lib/linux_x64/brotli/lib:$LD_LIBRARY_PATH"
ENV LD_LIBRARY_PATH="/blenderpy/blender/lib/linux_x64/dpcpp/lib:$LD_LIBRARY_PATH"
ENV LD_LIBRARY_PATH="/blenderpy/blender/lib/linux_x64/embree/lib:$LD_LIBRARY_PATH"
ENV LD_LIBRARY_PATH="/blenderpy/blender/lib/linux_x64/epoxy/lib:$LD_LIBRARY_PATH"
ENV LD_LIBRARY_PATH="/blenderpy/blender/lib/linux_x64/ffmpeg/lib:$LD_LIBRARY_PATH"
ENV LD_LIBRARY_PATH="/blenderpy/blender/lib/linux_x64/fftw3/lib:$LD_LIBRARY_PATH"
ENV LD_LIBRARY_PATH="/blenderpy/blender/lib/linux_x64/freetype/lib:$LD_LIBRARY_PATH"
ENV LD_LIBRARY_PATH="/blenderpy/blender/lib/linux_x64/fribidi/lib:$LD_LIBRARY_PATH"
ENV LD_LIBRARY_PATH="/blenderpy/blender/lib/linux_x64/gmp/lib:$LD_LIBRARY_PATH"
ENV LD_LIBRARY_PATH="/blenderpy/blender/lib/linux_x64/harfbuzz/lib:$LD_LIBRARY_PATH"
ENV LD_LIBRARY_PATH="/blenderpy/blender/lib/linux_x64/haru/lib:$LD_LIBRARY_PATH"
ENV LD_LIBRARY_PATH="/blenderpy/blender/lib/linux_x64/imath/lib:$LD_LIBRARY_PATH"
ENV LD_LIBRARY_PATH="/blenderpy/blender/lib/linux_x64/jemalloc/lib:$LD_LIBRARY_PATH"
ENV LD_LIBRARY_PATH="/blenderpy/blender/lib/linux_x64/jpeg/lib:$LD_LIBRARY_PATH"
ENV LD_LIBRARY_PATH="/blenderpy/blender/lib/linux_x64/level-zero/lib:$LD_LIBRARY_PATH"
ENV LD_LIBRARY_PATH="/blenderpy/blender/lib/linux_x64/llvm/lib:$LD_LIBRARY_PATH"
ENV LD_LIBRARY_PATH="/blenderpy/blender/lib/linux_x64/materialx/lib:$LD_LIBRARY_PATH"
ENV LD_LIBRARY_PATH="/blenderpy/blender/lib/linux_x64/mesa/lib:$LD_LIBRARY_PATH"
ENV LD_LIBRARY_PATH="/blenderpy/blender/lib/linux_x64/openal/lib:$LD_LIBRARY_PATH"
ENV LD_LIBRARY_PATH="/blenderpy/blender/lib/linux_x64/opencollada/lib:$LD_LIBRARY_PATH"
ENV LD_LIBRARY_PATH="/blenderpy/blender/lib/linux_x64/opencolorio/lib:$LD_LIBRARY_PATH"
ENV LD_LIBRARY_PATH="/blenderpy/blender/lib/linux_x64/openexr/lib:$LD_LIBRARY_PATH"
ENV LD_LIBRARY_PATH="/blenderpy/blender/lib/linux_x64/openimagedenoise/lib:$LD_LIBRARY_PATH"
ENV LD_LIBRARY_PATH="/blenderpy/blender/lib/linux_x64/openimageio/lib:$LD_LIBRARY_PATH"
ENV LD_LIBRARY_PATH="/blenderpy/blender/lib/linux_x64/openjpeg/lib:$LD_LIBRARY_PATH"
ENV LD_LIBRARY_PATH="/blenderpy/blender/lib/linux_x64/openpgl/lib:$LD_LIBRARY_PATH"
ENV LD_LIBRARY_PATH="/blenderpy/blender/lib/linux_x64/opensubdiv/lib:$LD_LIBRARY_PATH"
ENV LD_LIBRARY_PATH="/blenderpy/blender/lib/linux_x64/openvdb/lib:$LD_LIBRARY_PATH"
ENV LD_LIBRARY_PATH="/blenderpy/blender/lib/linux_x64/osl/lib:$LD_LIBRARY_PATH"
ENV LD_LIBRARY_PATH="/blenderpy/blender/lib/linux_x64/png/lib:$LD_LIBRARY_PATH"
ENV LD_LIBRARY_PATH="/blenderpy/blender/lib/linux_x64/potrace/lib:$LD_LIBRARY_PATH"
ENV LD_LIBRARY_PATH="/blenderpy/blender/lib/linux_x64/pugixml/lib:$LD_LIBRARY_PATH"
ENV LD_LIBRARY_PATH="/blenderpy/blender/lib/linux_x64/python/lib:$LD_LIBRARY_PATH"
ENV LD_LIBRARY_PATH="/blenderpy/blender/lib/linux_x64/sdl/lib:$LD_LIBRARY_PATH"
ENV LD_LIBRARY_PATH="/blenderpy/blender/lib/linux_x64/shaderc/lib:$LD_LIBRARY_PATH"
ENV LD_LIBRARY_PATH="/blenderpy/blender/lib/linux_x64/sndfile/lib:$LD_LIBRARY_PATH"
ENV LD_LIBRARY_PATH="/blenderpy/blender/lib/linux_x64/spnav/lib:$LD_LIBRARY_PATH"
ENV LD_LIBRARY_PATH="/blenderpy/blender/lib/linux_x64/tbb/lib:$LD_LIBRARY_PATH"
ENV LD_LIBRARY_PATH="/blenderpy/blender/lib/linux_x64/tiff/lib:$LD_LIBRARY_PATH"
ENV LD_LIBRARY_PATH="/blenderpy/blender/lib/linux_x64/usd/lib:$LD_LIBRARY_PATH"
ENV LD_LIBRARY_PATH="/blenderpy/blender/lib/linux_x64/vulkan/lib:$LD_LIBRARY_PATH"
ENV LD_LIBRARY_PATH="/blenderpy/blender/lib/linux_x64/webp/lib:$LD_LIBRARY_PATH"
ENV LD_LIBRARY_PATH="/blenderpy/blender/lib/linux_x64/xml2/lib:$LD_LIBRARY_PATH"
ENV LD_LIBRARY_PATH="/blenderpy/blender/lib/linux_x64/xr_openxr_sdk/lib:$LD_LIBRARY_PATH"
ENV LD_LIBRARY_PATH="/blenderpy/blender/lib/linux_x64/zlib/lib:$LD_LIBRARY_PATH"
ENV LD_LIBRARY_PATH="/blenderpy/blender/lib/linux_x64/zstd/lib:$LD_LIBRARY_PATH"
RUN ldconfig

# Test bpy
RUN python -c "import bpy, os; bpy.ops.wm.save_as_mainfile(filepath=os.path.abspath('my.blend'))" && rm my.blend
RUN python -c "import bpy; bpy.context.preferences.addons['cycles'].preferences.get_devices(); print(bpy.context.preferences.addons['cycles'].preferences.devices.keys())"

# Install Kubric python dependencies
COPY requirements.txt .
COPY requirements_full.txt .
RUN pip uninstall -y opencv && rm -rf /usr/local/lib/python3.10/dist-packages/cv2
RUN pip install --upgrade pip wheel
RUN pip install --upgrade --force-reinstall -r requirements.txt -r requirements_full.txt
RUN pip install imageio[ffmpeg]
RUN python -c "import cv2; print(cv2.__path__); print(cv2.imread)"
RUN rm -f requirements.txt
RUN rm -f requirements_full.txt

# Silences tensorflow
ENV TF_CPP_MIN_LOG_LEVEL="3"

# Install Kubric from a wheel (created by running `python setup.py bdist_wheel` in the Kubric root directory)
COPY dist/kubric*.whl .
RUN pip3 install `ls kubric*.whl`
RUN rm -f kubric*.whl

# Install additional tools
RUN apt-get install -y tree nvtop ncdu htop tmux

Container

image로부터 container 새로 띄우기

docker run -it --gpus all -n colmap -v /home/semyeong/data:/workdir/data pytorch/pytorch:latest /bin/bash
#############################################################################################
# -it: 종료 없이 container 탈출 가능(iterative terminal)
# --gpus all: container 내부에서 GPU 쓸 수 있도록 (nvidia toolkit이 설치돼있어야 함)
# -n: container name
# -v: host directory를 container 내부에 mount하여 폴더 공유 (<host_path>:<container_path> 꼴)
# /bin/bash: container 생성할 때 시작할 process
#############################################################################################

이미 띄운 container에 -v 폴더 연결 : by container -> image -> container

docker commit e3df03c colmapv2 # <현재 container ID> <new image name>
docker run -it -v /home/semyeong/data:/workdir/data colmapv2 /bin/bash

container 목록 확인

docker ps  # 현재 띄워져 있는 container만
docker ps -a  # 종료된 container까지
docker ps -a --filter ancestor=ddrf:latest  # 특정 container 검색

container 이름 변경
```
docker rename old_name new_name
```
종료했던 container 띄우기
```
docker start colmap
```
띄운 container에 접속 vscode에서 container 우클릭 후 Attach Visual Studio Code > Open Folder
또는
```
docker attach colmap
```
container 종료하지 않고 탈출 :
Ctrl+P > Ctrl+Q
container 종료 :
Ctrl+D 또는 exit
container 삭제
```
docker rm colmap
```

외부 파일/폴더를 container로 복사

docker cp /home/semyeong/data/a.png colmap:/workdir/data/a.png
# docker cp <source_file> <container_name>:<container_path>

container를 image로 변환 :
8GB 이상의 큰 파일이 있으면 commit 안 되므로
큰 파일은 굳이 container에서 image로 변환하지 말고,
container 생성할 때 host 폴더 공유 옵션(-v)으로 연결해서 사용하자
```
docker commit -p e3df03c colmapv2 # <현재 container ID> <new image name>
```

container를 압축파일로 저장

docker export colmap > ddrf.tar  
docker export e3df03c > ddrf.tar

container 압축파일 다운로드하여 image 만들기

# docker export -> docker import
docker import ddrf.tar ddrf:v1 # <Repository>:<Tag>

Docker

Contents

Install

Image

Container