I already test almost everything but I'm not quite sure if this is a bug or something else. Basic Airflow cluster configuration for CeleryExecutor with Redis and PostgreSQL. If I remove the PythonOperator code and imports the code works fine. This works fine, however I would like to move the python scripts, which are my tasks out of the mounted plugins folder and into their own docker containers. Apache Airflow, Apache, Airflow, the Airflow logo, and. My njtransit_scrapper.py file is just a file that collects all tweets for a tweeter account and saves the result in a Postgres database. I have a docker container running on my windows machine, which was build with an adapted version of the docker-compose file provided in the official docs. The Apache Airflow community, releases Docker Images which are reference images for Apache Airflow. This is my PythonOperator: tweets_load = PythonOperator( I already tried airflow initdb and restarting the web server but it keeps showing the error ModuleNotFoundError: No module named 'plugins'įor the import statement I'm using: from plugins import njtransit_scrapper py fileĬmd to run docker airflow image: docker run -p 8080:8080 -v /My/Path/To/Dags:/usr/local/airflow/dags puckel/docker-airflow webserver I'm using docker image to run Airflow but it doesn't recognize a module where I have my. You can read more about using, customising, and extending the images in the Latest docs, and learn details on the internals in the IMAGES.rst document.
#AIRFLOW DOCKER FILE INSTALL#
I created a very simple DAG to execute a Python file using PythonOperator. PyPI releases to install Airflow using standard pip tool Docker Images to install airflow via docker tool, use them in Kubernetes, Helm Charts, docker-compose, docker swarm, etc.
0 kommentar(er)
