Repository navigation
Feature/IDP-Run-workspace - Run notebooks in a workspace - #1023
D-GopalKrishna wants to merge 19 commits into
Conversation
|
Only for the jupyterlab-minimal image - otherwise checked jupyterlab works perfectly fine. |
filippomc
left a comment
There was a problem hiding this comment.
In order to make the feature sustainable, have to respect quotas, and have to think a bit how to configure the runtime
|
|
||
| RUN_NOTEBOOKS_BASENAME = "osb-run-notebooks-job" | ||
| # CustomTask sets no limits by default; the notebooks are third-party code. | ||
| RUN_NOTEBOOKS_RESOURCES = {"requests": {"cpu": "100m", "memory": "256Mi"}, "limits": {"cpu": "1", "memory": "2Gi"}} |
There was a problem hiding this comment.
resources must be taken from the user quotas
There was a problem hiding this comment.
using get_run_resources to get the requests an limits now.
| ADD run.sh /run.sh | ||
|
|
||
| RUN chmod +x /run.sh | ||
| CMD ["/run.sh"] |
There was a problem hiding this comment.
In order to run notebooks, have to make sure that all the libraries needed are in the env. Unless we make sure that notebooks all have the requirements installed first inside the notebook itself. One alternative can be to use the jupyterlab image, so everything can run there, can run here
There was a problem hiding this comment.
Perhaps an even more flexible and scalable option, the request specifies the image where to run
There was a problem hiding this comment.
We do the installation as part of run.sh - by expecting a standard requirements.txt and install path for setup.py|pyproject.toml. This will allow us to run in the argo pod, but agreed for them to run this - notebooks have to declare the requirements installation as part of it.
Using the JupyterLab image now, and passing which script to run from the request param - This is now in the Configmap - and picked based on where the request asks to run.
…uting Jupyter notebooks in workspace
filippomc
left a comment
There was a problem hiding this comment.
I'd like to improve the code clarity a little, and the parametrization strategy
Task description:
RunNotebooksTaskleaves the Keycloak secret andallvaluesmounts out of the run pod, andrun.shruns the code without theCH_*variables (env -i).and
run.sh): copies the repository to scratch (repo.discardremoves the original), installssetup, then runs each notebook in order with papermill, passingINPUT_DIR/OUTPUT_DIR; the first failure stops the run.<results.notebooks>.running, renamed on exit to<results.notebooks>if all passed or<results.notebooks>.failedif not; the workflow's exit-handler scan then lists them, so a client tells success from failure by which folderGET /workspace/{id}resources shows.