Common Directories and File Locations#
Jupyter stores different files (i.e. configuration, data, runtime) in a number of different locations. Environment variables may be set to customize for the location of each file type.
Jupyter separates data files (nbextensions, kernelspecs) from runtime files (logs, pid files, connection files) from configuration (config files, custom.js).
Configuration files#
Config files are stored by default in the
Set this environment variable to use a particular directory, other than the default, for Jupyter config files.
Besides the JUPYTER_CONFIG_DIR , additional directories to search can be specified through JUPYTER_CONFIG_PATH .
Set this environment variable to provide extra directories for the config search path. JUPYTER_CONFIG_PATH should contain a series of directories, separated by « os.pathsep« ( ; on Windows, : on Unix).
An example of where the JUPYTER_CONFIG_PATH can be set is if notebook or server extensions are installed in a custom prefix. Since notebook and server extensions are automatically enabled through configuration files, automatic enabling will only work if the custom prefix’s etc/jupyter directory is added to the Jupyter config search path.
Besides the user config directory mentioned above, Jupyter has a search path of additional locations from which a config file will be loaded. Here’s a table of the locations to be searched, in order of preference:
To list the config directories currently being used you can run the below command from the command line :
The following command shows the config directory specifically:
Data files#
Jupyter uses a search path to find installable data files, such as kernelspecs and notebook extensions. When searching for a resource, the code will search the search path starting at the first directory until it finds where the resource is contained.
Each category of file is in a subdirectory of each directory of the search path. For example, kernel specs are in kernels subdirectories.
Set this environment variable to provide extra directories for the data search path. JUPYTER_PATH should contain a series of directories, separated by os.pathsep ( ; on Windows, : on Unix). Directories given in JUPYTER_PATH are searched before other locations. This is used in addition to other entries, rather than replacing any.
Linux (& other free desktops)
/.local/share/jupyter/ (respects $XDG_DATA_HOME )
JUPYTER_DATA_DIR or (if not set) %APPDATA%\jupyter
The config directory for Jupyter data files, which contain non-transient, non-configuration files. Examples include kernelspecs, nbextensions, or voila templates.
Set this environment variable to use a particular directory, other than the default, as the user data directory.
As mentioned above, to list the config directories currently being used you can run the below command from the command line :
The following command shows the data directory specifically:
Runtime files#
Things like connection files, which are only useful for the lifetime of a particular process, have a runtime directory.
On Linux and other free desktop platforms, these runtime files are stored in $XDG_RUNTIME_DIR/jupyter by default. On other platforms, it’s a runtime/ subdirectory of the user’s data directory (second row of the table above).
An environment variable may also be used to set the runtime directory.
Set this to override where Jupyter stores runtime files.
As mentioned above, to list the config directories currently being used you can run the below command from the command line :
Jupiter notebook где хранятся файлы
By default, it saves to the directory of anaconda3/ with an extension of «ipynb» when I click «File» => «Save».
How to save it to a directory under anaconda3/, instead of the default location? There is not a «save as» command in notebook?
3 Answers 3
You can save a notebook to a location of your choice by using the «File» -> «Download as» -> «Notebook (.ipynb)» option from the menu.
Alternatively you can start your notebook server from a different directory and it will save all notebooks to that directory.
A third option is to navigate to the directory you want the notebook to be saved to in the tree view «http://127.0.0.1:8888/tree» prior to creating the notebook.
There are two methods:
1.You can use the magic command %notebook to save as ipynb
2.You can use the magic command %%writefile to save as py file
In the second method, you should put this command at the top of the cell, otherwise it will throw an error.
It is easiest to select a destination before you create a program using Jupyter Notebook; as then you do not run into this issue.
However, since you have already made a program, one possible solution is to make a copy of the file, move it to your desired location, and then delete the old file. Before doing this, ensure that you have saved it first, otherwise data might be lost.
Summary
Almost every notebook contains a pd.read_csv(file_path) or a similar command to load data. Dealing with file paths in notebooks, however, is kinda troublesome: moving notebooks around becomes a problem, and the notebook now has to know project locations. Here, we discuss a couple of approaches to handle this problem.
Introduction
Starting a notebook is always easy, you just start a couple of cells which often just contain a df.head() . However, as the project grows (and in industry they always do), you will need to organize your folders. You will need a folder for the data, and another folder for notebooks. As the EDA progresses, you will need more folders representing different subsections of the main analysis. On top of that, your project should be reproducible, so that your peers can download the code, run the script, and it will work as intended, hopefully yielding the same results you had 🙂
So, if you have a read_csv(relative_path_to_data) on your notebook, moving it from one folder to another will require a change in the code. This is undesirable, we would like it to work regardless of its location. You could solve this by using read_csv(absolute_path_to_data) , but this is even worse: you will deal with paths lengthier than they need to be, and your code will probably break if you try to run it on another machine.
Let’s say you have your working directory on /system_name/project , from which you run jupyter lab or jupyter notebook . The data directory is located at /system_name/project/data , and your notebooks are in system_name/project/notebooks
We propose two ways to solve this problem:
- Using a environment variable inside a notebook
- Using a data module
Environment Variable
With this approach, we inform the system the location of the data directory through the usage of an environment variable. Before starting a jupyter server, we will set the variable by doing
If you are on the /system_name/project folder, you can do:
to achieve the same effect. Now, this variable is accessible to all child processes you start from your bash terminal. In your notebooks, you can now do:
Now the only thing your notebook needs to know is the file_name of the dataset. Sounds fair, right?
Another thing you can try to do is changing the working directory of the notebook itself by doing this:
This works, but I prefer the former, as the latter makes the notebook work in a directory that it is not, feels slightly shady :).
Finally, it might be a bit boring to set the environment variable every time you start a jupyter server. You can automate this process using python-dotenv. It will search for a .env file, first in the local directory, and then in all it's parents. Once it does, it will load the variables defined there. Check the project documentation if you like the idea!
Data Module
We used an environment variable to hold information about the project configuration, and exposed this to the notebook. But what about moving this responsibility somewhere else? We can create a module whose responsibility is to know the data directory, and where the datasets are. I prefer this approach, as it will make datasets explicit symbols inside the code.
We will need a project_package folder to represent, well, the project's package. Inside it, we will create a project_data.py module:
We use the __file__ dunder method which returns the current file path, and the built-in Path class to navigate through the directory. We make this package installable by creating the setup.py inside the project folder:
We are almost there! Now, we install the package we just created in development mode, so that changes to the package won’t require a reinstall:
python -m pip install -e .
This should install the project_package package, which can be accessed from the notebook:
How to save a file into a directory in Jupyter notebook?
By default, it saves to the directory of anaconda3/ with an extension of «ipynb» when I click «File» => «Save».
How to save it to a directory under anaconda3/, instead of the default location? There is not a «save as» command in notebook?
3 Answers 3
You can save a notebook to a location of your choice by using the «File» -> «Download as» -> «Notebook (.ipynb)» option from the menu.
Alternatively you can start your notebook server from a different directory and it will save all notebooks to that directory.
A third option is to navigate to the directory you want the notebook to be saved to in the tree view «http://127.0.0.1:8888/tree» prior to creating the notebook.
There are two methods:
1.You can use the magic command %notebook to save as ipynb
2.You can use the magic command %%writefile to save as py file
In the second method, you should put this command at the top of the cell, otherwise it will throw an error.
It is easiest to select a destination before you create a program using Jupyter Notebook; as then you do not run into this issue.
However, since you have already made a program, one possible solution is to make a copy of the file, move it to your desired location, and then delete the old file. Before doing this, ensure that you have saved it first, otherwise data might be lost.
-
The Overflow Blog
Linked
Related
Hot Network Questions
Subscribe to RSS
To subscribe to this RSS feed, copy and paste this URL into your RSS reader.
Site design / logo © 2023 Stack Exchange Inc; user contributions licensed under CC BY-SA . rev 2023.3.11.43300
By clicking “Accept all cookies”, you agree Stack Exchange can store cookies on your device and disclose information in accordance with our Cookie Policy.