logo
Tags down

shadow

How to set request_cpu globally for airflow worker pods using the kubernetes executor?


By : Trent G
Date : August 01 2020, 09:00 PM
like below fixes the issue You can set this executor_config at task level for KubernetesExecutor as follows.
code :
    exmaple_task = PythonOperator(
        task_id="exmaple_task",
        python_callable=print_stuff,
        executor_config={
            "KubernetesExecutor": {"request_cpu": "1",
                                   "request_memory": "128Mi",
                                   "limit_memory": "128Mi"}}
    )


Share : facebook icon twitter icon

How to kill airflow worker properly (with celery executor)?


By : iuhhuh
Date : March 29 2020, 07:55 AM
will be helpful for those in need I normally set up the Airflow processes as systemd processes and just run systemctl stop airflow-* however I think you can simply kill the main airflow-worker process with a standard linux kill command? Probably what the systemd daemon does anyway.

Add custom volumeMount into Airflow worker pods (with k8s Executor)


By : Al-Abuasi
Date : March 29 2020, 07:55 AM
I wish did fix the issue. Sorry i'd to answer my own question. Maybe will help someone. Inside the dag file, where I define the task, I simply had to add the executor_config as follows;
code :
        IngestionStatusSensor(
            task_id=...,
            executor_config={"KubernetesExecutor": {
                                  "image": "airflow:runner2",
                                  "volume_mounts": [
                                      {
                                          "name": "airflow-dags",
                                          "mountPath": "/usr/local/airflow/libs",
                                          "subPath": "airflow/development/libs"
                                      },
                                      {
                                          "name": "airflow-dags",
                                          "mountPath": "/usr/local/airflow/plugins",
                                          "subPath": "airflow/development/plugins"
                                      }],
                                  }
                             },
            dag=dag,
            ingestion_feed=table,
            poke_interval=60 * 30,
            offset=0,
            start_date=start_date
        )

How to mount volume of airflow worker to airflow kubernetes pod operator?


By : user3149642
Date : March 29 2020, 07:55 AM
fixed the issue. Will look into that further The example in the docs seems pretty similar to your code, only the parameters are plurals volume_mounts and volumes. For your code it would look like this:
code :
self.operator = KubernetesPodOperator(
            namespace=self.namespace, image=self.image, name=self.name,
            cmds=self.cmds,
            arguments=self.arguments,
            startup_timeout_seconds=600,
            is_delete_operator_pod=True,
            # the operator could run successfully but the directory /tmp is not mounted to kubernetes operator
            volumes=[self.volume],
            volume_mounts=[self.volume_mount],
            **self.context)

Airflow kubernetes executor: Run 2 jobs on the same pod


By : Gary
Date : March 29 2020, 07:55 AM
With these it helps Sorry this isn't possible - one job per pod.
You are best to use task 1 to put the data in a well known location (e.g in a cloud bucket) and get it from the second task. Or just combine the two tasks.

Airflow scheduler fails to start with kubernetes executor


By : FloxaY
Date : March 29 2020, 07:55 AM
around this issue For the helm value, the template uses a loop that places the airflow.config map into double quotes ". This means any " in a value needs to be escaped for the output templated YAML to be valid.
code :
airflow:
  config:
    AIRFLOW__KUBERNETES__KUBE_CLIENT_REQUEST_ARGS: '{\"_request_timeout\":60}'
Related Posts Related Posts :
  • Using disabledDate in Antd Datepicker in table
  • iterator .end() from std::list returns "0xcdcdcdcdcdcdcdcd" but .begin() as expected
  • how to convert HAC flexible query to DAO query
  • Cannot refresh UI if update in ItemView
  • How to make a function to use dict keys as variables to a class?
  • Best approach to remove cassandra-topology.properties file in running cluster nodes
  • plsql store procedure loop compare value
  • Replace values in XML file with values of a vector
  • Convert old SQL Database in compatibility mode
  • Sum same property object by group
  • What do you do about the JLabel classes? It says, "JLabel not a statement" for the error
  • Is std::sqrt the same as sqrt in C++
  • Iterate through std::initializer_list
  • Why does the overidden run method in java.lang.Thread produce a bizarre output?
  • Typescript: type one parameter based on the other
  • How to add a CSS to this JavaScript or HTML on click buttons?
  • Is it OK to inherit an empty Interface?
  • Functional Interface call for a new Instance
  • Microsoft Bot Framework: Smilies in MS Teams
  • changing background image of div using javascript
  • How to convert two arrays of strings to the array of objects like key and value with particular keys in javascript?
  • What is the fastest way to find if a column has at least one NULL value in ORACLE database?
  • Rename headers - 'list' object is not callable
  • Codeblocks c++ code doesn't run in VS 19 (vector subscript out of range)
  • Passing res.send value from node.js backend to react.js
  • Vim shortcuts to select and copy the current line without the next line
  • Is it possible to pass data from an angular7 component or service to index.html file?
  • When I tried to add ArrayList into ArrayList second ArrayList is repeating
  • If I implement IEquatable<T>, will I lose the option to compare by reference?
  • Authorize with both ASP.NET core MVC/Razor site AND a WebAPI
  • Compare two version of zip file and find which file has been modified within that zip
  • Dynamically generated href won't show properly
  • Best way to saving completed progress in table?
  • Does UIWindow function not work in Xcode11.3?
  • TypeError: __init__() takes 2 positional arguments but 6 were given
  • Converting string (with timezone) to datetime in python
  • How to overwrite the theme in shopify
  • Get the no of consecutive days a Field value is Stale
  • How to keep track of previous value of variable in swift?
  • Can't get result when running the query from Spring Data Jpa
  • If Condition Simplification
  • Python list generation from two strings
  • How to find distinct records in vespa.ai?
  • Why erase on std::vector promote iterator
  • How to use data to set other data in Vue.js
  • Azure AD does not return groups on claims
  • ASP Net Core Web API: Client side GroupBy is not supported
  • How to correct TypeError: Unicode-objects must be encoded before hashing with ReportLab
  • how to destroy an object in C++
  • How to do pagination using groupby in vespa.ai?
  • How can I print the longest word from a user defined list?
  • C# I have a DLL file and I need to make a class that inherits from a class that's in the DLL file?
  • Can someone explain to me why my factorial recursion code can't be compiled
  • Pass a PHP variable to a JS variable
  • Showing messages based on scroll position
  • How to copy cells via vba macro without getting subscript out of range
  • Replace substring in shell script
  • enabling authentication in ignite
  • Swipe to delete rows with multi section in tableview?
  • [BootstrapVue warn]: popover - Unable to find target element in document
  • shadow
    Privacy Policy - Terms - Contact Us © voile276.org