Questions tagged [trains]
Questions regrading TRAINS pypi package & server (auto-magical experiment manager & version control for AI)
23
questions
4
votes
1answer
260 views
Can ClearML (formerly Trains) work a local server?
I am trying to start my way with ClearML (formerly known as Trains).
I see on the documentation that I need to have server running, either on the ClearML platform itself, or on a remote machine using ...
2
votes
1answer
77 views
How should Trains be used with hyper-param optimization tools like RayTune?
What could be a reasonable setup for this? Can I call Task.init() multiple times in the same execution?
2
votes
2answers
78 views
Tracking separate train/test processes with Trains
In my setup, I run a script that trains a model and starts generating checkpoints. Another script watches for new checkpoints and evaluates them. The scripts run in parallel, so evaluation is just a ...
2
votes
1answer
47 views
Parallel Coordinates Plot in TRAINS
Is there a way to create a parallel coordinates plot in TRAINS (https://github.com/allegroai/trains) package to compare several hyper-parameters in respect to a specific metric?
2
votes
1answer
103 views
ClearML server IP address not used with localhost and SSH port forwarding
Trying to use clearml-server on own Ubuntu 18.04.5.
I use env variables to set the IP Address of my clearml-server.
export CLEARML_HOST_IP=127.0.0.1
export TRAINS_HOST_IP=127.0.0.1
But it still is ...
2
votes
1answer
42 views
How to fix trainserver empty server? [closed]
Im trying to install an allegroai trains-server on a k8s cluster.
I tried the following 3 methods
bare linux installtion
k8s manifest installation
helm installation
I followed the linux installation ...
1
vote
2answers
208 views
pip install trains fails
upon running pip install trains in my virtual env
I am getting
ERROR: Command errored out with exit status 1:
command: /home/epdadmin/noam/code/venv_linux/bin/python -u -c 'import sys, ...
1
vote
1answer
146 views
How to manage datasets in ClearML Web UI?
Using a self-deployed ClearML server with the clearml-data CLI, I would like to manage (or view) my datasets in the WebUI as shown on the ClearML webpage (https://clear.ml/mlops/clearml-feature-store/)...
1
vote
1answer
25 views
ClearML get max value from logged values
I use ClearML to track my tensorboard logs (from PyTorch Lightning) during training.
At a point later I start another script which connects to existing task and do some testing.
But unfortenautly I do ...
1
vote
1answer
62 views
ClearML multiple tasks in single script changes logged value names
I trained multiple models with different configuration for a custom hyperparameter search. I use pytorch_lightning and its logging (TensorboardLogger).
When running my training script after Task.init()...
1
vote
1answer
42 views
Trains: reusing previous task id
I am using reuse_last_task_id=True to overwrite an existing task (with same project and task name). But the experiment contains the torch model and therefore does not overwrite the existing task but ...
1
vote
1answer
82 views
Will Trains automagically log Tensorboard HParams?
I know that it's possible to send hyper-params as a dictionary to Trains.
But can it also automagically log hyper-params that are logged using the TF2 HParams module?
Edit: This is done in the HParams ...
1
vote
1answer
101 views
Trains: Can I reset the status of a task? (from 'Aborted' back to 'Running')
I had to stop training in the middle, which set the Trains status to Aborted.
Later I continued it from the last checkpoint, but the status remained Aborted.
Furthermore, automatic training metrics ...
1
vote
1answer
50 views
Can Trains config file be specified dynamically or relative to the running script path?
Suppose I have a server where many users run different experiments, possibly with different Trains Servers.
I know about the TRAINS_CONFIG_FILE environment variable, but I wonder if this can be made ...
1
vote
2answers
101 views
trains with grid search
I would like to test trains usage during grid search and it not clear how to do so.
from trains import Task
Task.init(project_name="project name", task_name='name')
creates an experiment in the ...
1
vote
1answer
3k views
How to Backup/Restore TRAINS-server when moving from AMI to local machine
I recently started using TRAINS, with the server in AWS AMI. We are currently using v0.9.0.
I would like to move the TRAINS-server to run on our on-premises kubernetes cluster. However, I don't want ...
1
vote
1answer
143 views
How to manually register a sci-kit model with TRAINS python auto-magical experiment manager?
I'm working mostly with scikit-learn, as far as I understand, the TRAINS auto-magic doesn't catch scikit-learn model store/load automatically.
How do I manually register the model after I have '...
1
vote
1answer
114 views
Is there a way to create a graph comparing hyper-parameters vs model accuracy with TRAINS python package?
I would like to run multiple experiments, then report model accuracy per experiment.
I'm training a toy MNIST example with pytorch (v1.1.0), but the goal is, once I can compare performance for the ...
1
vote
1answer
30 views
ClearML Web UI custom column not persistent
I'm using the experiments page of a project in ClearML Web UI to visualize some custom metrics. Therefore I've customized my table vie (https://allegro.ai/clearml/docs/docs/webapp/webapp_exp_table....
1
vote
1answer
49 views
ClearML SSH port forwarding fileserver not available in WEB Ui
Trying to use clearml-server on own Ubuntu 18.04.5 with SSH Port Forwarding and not beeing able to see my debug samples.
My setup:
ClearML server on hostA
SSH Tunnel connections to access Web App ...
0
votes
1answer
19 views
ClearML How to get configurable hyperparameters?
How do I get args like epochs to show up in the UI configuration panel under hyperparameters? I want to be able to change number of epochs and learning rate from within the UI.
0
votes
1answer
24 views
Does clone experiment work on sklearn functions?
I'm trying to run a script and I'm constantly getting this while cloning experiment in allegro.ai
AttributeError: 'Namespace' object has no attribute 'get'
Can anybody help?
-1
votes
1answer
40 views
How resilient is reporting to Trains server?
How would Trains go about sending any missing data to the server in the following scenarios?
Internet connection breaks temporarily while running an experiment
Internet connection breaks and doesn't ...